Cisco is honored to be a partner of the Black Hat NOC (Network Operations Center), and this was our seventh year supporting Black Hat Asia. Cisco is the Official Mobile Device Management, Malware Analysis and DNS (Domain Name Service) Provider.
We work with other official providers to bring the hardware, software and engineers to build and secure the network, for our joint customer: Black Hat.
- Arista: Network Equipment
- Corelight: Network Analytics and Detection
- MyRepublic: Broadband
- NetWitness: Threat Detection & Response, Identity
- Palo Alto Networks: Network Security Platform
The primary mission in the NOC is network resilience. The partners also provide integrated security, visibility and automation, a SOC (Security Operations Center) inside the NOC.
On screens outside the NOC were displayed partner dashboards for the attendees to view the volume and security of the network traffic.
It All Started with Malware
Cisco joined the Black Hat NOC in 2016, when asked to provide automated malware analysis with Thread Grid. The Cisco contributions to the network and security operations evolved, with the needs of the customer, to include more components of the Cisco Security Cloud.
- Breach Protection Suite
- Cisco Secure Malware Analytics (Formerly Threat Grid): sandboxing and integrated threat intelligence
- User Protection Suite
- Cisco Umbrella: DNS visibility for the conference network and protection for iOS devices
- Cisco Security Connector: iOS device security and visibility, managed with Meraki Systems Manager
- Cloud Protection Suite
- ThousandEyes: Network observability / availability
The NOC leaders allowed Cisco (and the other NOC partners) to bring in additional software to make our internal work more efficient and have greater visibility; however, Cisco is not the official provider for Extended Detection & Response, Network Detection & Response or Collaboration.
- Breach Protection Suite
- Cisco XDR: Threat Hunting / Threat Intelligence Enrichment / Executive dashboards / Automation with Webex
- Cisco XDR Analytics (Formerly Secure Cloud Analytics / Stealthwatch Cloud): network traffic visibility and threat detection
- Cisco Webex: Incident notification and team collaboration
The Cisco XDR Command Center dashboard tiles made it easy to see the status of each of the connected Cisco Security technologies, and the status of ThousandEyes agents.
When the partners deploy to each conference, we set up a world class network and security operations center in three days. Our goal remains network up time and creating better integrated visibility and automation. Black Hat has the pick of the security industry tools and no company can sponsor/buy their way into the NOC. It is invitation only, with the intention of diversity in partners, and an expectation of full collaboration.
As a NOC team comprised of many technologies and companies, we are continuously innovating and integrating, to provide an overall SOC cybersecurity architecture solution. We look forward to continuing the work with partner Palo Alto Networks, for further automation at Black Hat USA 2024.
Below are the Cisco XDR integrations for Black Hat Asia, empowering analysts to investigate Indicators of Compromise (IOC) very quickly, with one search.
We appreciate alphaMountain.ai, Pulsedive and Recorded Future donating full licenses to Cisco, for use in the Black Hat Asia 2024 NOC.
| Cisco Networking and Security | Third Party | |
| 1 | Meraki System Manager | alphaMountain.ai |
| 2 | Secure Endpoint for iOS | APIVoid |
| 3 | Secure Malware Analytics | AlienVault OTX |
| 4 | ThousandEyes | CyberCrime Tracker |
| 5 | Umbrella DNS | Google Safe Browsing |
| 6 | Webex | NetWitness / Custom relay |
| 7 | XDR Analytics | Pulsedive |
| 8 | Cisco Telemetry Broker | Recorded Future |
| 9 | Shodan | |
| 10 | Threatscore | Cyberprotect | |
| 11 | VirusTotal | |
| 12 | Slack | |
| 13 | urlscan |
An example of this is an investigation of a potentially malicious activity on the 2nd day of Training. An IP address was identified by NetWitness for possible geolocation leakage.
Investigation of the IP correlated the syslog sightings from the partner technologies in the NetWitness logs, with threat intelligence from Pulsedive, Recorded Future, alphaMountain and others.
Reviewing the DNS logs and the details of the packet capture in both Corelight and NetWitness, it was confirmed no geolocation data was leaked and it was part of a Training course. The activity would have been blocked in a production environment.
A core integrated workflow in the Black Hat NOC is NetWitness and Corelight sending suspicious files to Secure Malware Analytics. Over 4,900 samples were submitted.
The NOC analysts also used Malware Analytics to investigate suspicious domains, without the risk of infection. Rather than going to the website on a corporate or Black Hat assets, we were able to interact with the website in the glovebox, including downloading and installing the website payload.
Detonating files or browsing websites in Secure Malware Analytics protects the analysts from accidental infection.
We saw a series of similar (but different hash values) exploit kits downloaded on the first day in the Business Hall. The downloads were on the conference Wi-Fi and not in a Training course, so the event had to be investigated to confirm there was not an attack on the attendees. Working with the Corelight team, the NOC responders parsed the traffic and confirmed it was a Capture the Flag event, which continued into the last day of the conference.
Threat Hunters’ Story, by Aditya Raghavan and Shaun Coulter
In the Black Hat Asia 2024 NOC, Shaun staffed the morning shifts, and Aditya the afternoon shifts, as threat hunters focused on the Cisco XDR and Secure Malware Analytics consoles. Mornings were usually pretty chill. However, and for some heretofore unknown (coffee related?) reason, the activity ramped up in the afternoon on most days, leading Aditya to a place of “involved joy”, and Shaun to a place of tormented jealousy :D. With dogged determination both hunters spent their time reviewing alerts, activities, and conducted IOC scans using XDR Investigate. They reviewed submitted samples and network logs for signs of intrusion or suspicious activity.
Using Secure Malware analytics, they dissected malware samples, analyzed phishing campaigns, and scrutinized network traffic patterns for anomalies. Numerous alerts flagged as spikes in traffic from unexpected sources, strange destinations and odd variants of malicious code popped up multiple times, initiating thorough investigations. In most cases, they traced the anomaly to an authorized Black Hat Training or Briefing source and closed such cases as “Black Hat Positive”; meaning you would not allow this on your production network, but for Black Hat, it is business as usual. Since Black Hat is a conference designed for learning about offensive security, these malware samples are expected, and marked as such.
Thankfully or unthankfully, as the system tuning was completed, most alerts raised were as above and expected or actually ‘near misses’ – items that warrant investigation but did not extend to impactful behaviours, as we were able to stop them in time.
On the first day of Briefings, as Shaun is dutifully poring through the console of Secure Malware Analytics, in walks Aditya to relieve the shift. Greetings aside, Shaun quickly pivots over excitedly “Brother, I want to show you a couple of interesting things.” Aditya’s interest is piqued, and Shaun opens a new dashboard showing one of the recently released features of Cisco XDR – MITRE ATT&CK ® Coverage Map.
This new capability quickly displays all the tactics and techniques in the MITRE ATT&CK® matrix for which Cisco XDR has detections/coverage. In addition to the XDR Native, detections from Secure Endpoint and Secure Malware Analytics are also used to derive the coverage map making it a holistic view. This view allows the user to visualize the detections of XDR natively, as well as the integrated solutions and identify the scope of coverage and importantly, map out the gaps for future consideration. Thanks to the Cisco Talos team, all solutions within the Cisco Breach Protection Suite are mapped today and this would be rolled out to include other suites and solutions, including 3rd party integrations, soon.
As our threat hunters geek out on the behind-the-scenes stuff on XDR, Jessica politely calls out “Adi. Shaun. Guys, there is some new activity on Umbrella. Can you look into it?” Nudged back to reality, our threat hunters get to work – finding needles in the stack of needles at Black Hat as it was rightly put by Grifter! Talking about that, the new activity appears to be a query for a domain classified as a Command & Control (C&C) domain. Let’s dig into it.
A quick look into Umbrella Activity Search shows the latest traffic activity matching the C&C category that was allowed. Expanding the details pane, we can see the domain being queried and the identity of the endpoint issuing the query which appears to be from the ‘Hacking Enterprises 2024 Red Team’. That is a legitimate Training class at Black Hat Asia 2024. We pivot over to Umbrella Investigate and see the reason for this domain being categorized as C&C and its indicators.
Let’s head over to XDR and query this observable against all the integrated solutions for more intel. We quickly get a visual connected graph and tabulated events on all the relevant intel. The integration with NetWitness Logs provides us with events related to that domain, as well as populating the graph with those relationships, along with the Umbrella event which was the source for this hunt.
Looking at the evidence, this turned out to be another needle! Nothing untoward here, we classified this as a ‘Black Hat Positive’ and moved on. As the afternoon shift winds down, the team is discussing potential destinations for dinner and there is always dessert to look forward to at the end. Aditya and Ryan were pining for rich ice cream and Home Best Dessert turns out to be the right solution for the ask. In the NOC, the right solution is almost always teamwork with all our partners.
One such instance was when a Corelight hunter picked up a spike of traffic to unusual destinations. These appear to be DNS queries to a bunch of C&C domains. We quickly delve into Umbrella showing us all the domains being queried in a short window and most of them being Malware and/or C&C categorized. This appears to be a system either being compromised or someone intentionally doing a test / recon for those domains.
Let’s investigate some of these domains in XDR. We can see a lot of red icons on this visualization! In fact, every queried domain is classified as Malicious and known to host other malicious content. This doesn’t look expected for sure and that puts the intentional test / recon theory to rest quickly. Ben Reardon, the hunter from Corelight, puts it succinctly “This box is pwned six ways to Sunday!” What else can we find about this system then?
Looking at the DHCP logs for the IP address, the Corelight hunter was able to pinpoint the device MAC address and hostname, which resembled a name. A short Google search later, we have a potential device owner and the fact that he was delivering a session at Black Hat in one of the rooms next door! A short conversation with the person after his session ensued, where the NOC leads advised the NOC’s findings on his compromised system. He was grateful for the finding and reached out for additional context. This one turned out to be a ‘True Positive.’
The following day, the team has zeroed in on Turkish food for the evening. Ryan halts Shaun as he departs at the end of his shift and demands his hotel name and room number. “I’m gonna come knock at your door and wake you up tonight, man. I mean it. No day is too long. I used to do my shifts on three hours of sleep. Now, let’s go!” Ryan is deadpan serious. That is what we thought while investigating our next potential malware finding.
Another event on the Umbrella console comes to our attention and this time it is a query for a domain categorized as Malware. The source endpoint is quickly identified from the Identity and Umbrella investigate tells us this domain is part of the Malware block list. In a normal production network, this would ideally be blocked.
Black Hat isn’t your normal production network, and it attracts all kinds of security folk. And that is exactly what it turned out to be this time. The National University of Singapore has a group organizing regular capture the flag (CTF) events and is running a similar get-together at Black Hat. Go NUS Greyhats!
Activities involving malware what would be blocked on a corporate network must be allowed, within the confines of Black Hat Code of Conduct.
Network Observability with ThousandEyes, by Adam Kilgore and Patrick Yong
Deploying ThousandEyes at Black Hat is a rigorous process involving a lot of hardware (some shown below), configuration, testing, troubleshooting, and running around the conference center.
In addition to our typical deployment tasks, we implemented multiple improvements to the service. These improvements included an overhaul of the dashboards to show granular data for each conference room, alongside aggregate data for the entire conference; and better labeling and organization of deployed agents.
The ThousandEyes dashboard was projected on the large screen in the NOC, for alerting on any network issues, prior to reports from users.
On the troubleshooting side, we improved our log analysis and collection techniques and set up centralized monitoring of wireless data. These efforts contributed to improvements in visibility and agent uptime throughout the conference.
During the initial two days of Training sessions at Black Hat, ThousandEyes agents showed only minor deviations from baseline as the Training sessions came online. As the Training sessions continued, performance was stable, with only rare alerts for minor degraded throughput or moderate latency spikes. On Thursday, all the two-day Training sessions were closed, and the conference shifted towards Briefings, alongside two four-day Training sessions that ran for the conference’s length. With start of Briefings and opening the Business Hall, headcounts drastically increased. ThousandEyes saw degraded performance on the network, primarily in the large conference rooms hosting the Briefings. The below image shows a test result from the Hibiscus 3610 ballroom:
The network path above shows heavy latency on the first link to the default gateway, compounded by another high latency link outside the conference network. A breakdown of connectivity for the above path is shown below:
The throughput number above is key to this investigation. The Access Points (APs) for the Hibiscus 3610 ballroom had an average throughput of around 174 Mbps. Reviewing AP logs, we found that 92 users were connected to the same AP from which the test was run. Dividing the 174 Mbps by 92 gives an average throughput in line with the 1.7 Mbps shown above, so the poor connectivity was driven by oversaturation of user connections in this area.
The Hibiscus 3610 room and other agents in a nearby hallway consistently had the worst connection among the conference rooms, as shown by our agent polling results.
While there were limitations in the amount of bandwidth available for the conference in general, the data above suggests more of the available AP and bandwidth resources should be allocated to the Hibiscus 3610 ballroom and adjacent hallways for future conference topologies, which was shared with our Network Equipment partner.
Meraki Systems Manager, by Paul Fidler and Connor Loughlin
Our eighth deployment of Meraki Systems Manager as the official Mobile Devices Management platform went very smoothly, and we introduced a new caching operation to update iOS devices on the local network, for speed and efficiency. Going into the event, we planned for the following types of devices and purposes:
- iPhone Lead Scanning Devices
- iPads for Registration
- iPads for Session Scanning
We registered the devices in advance of the conference. Upon arrival, we turned each device on.
Then we ensured Location Services enabled, always on.
Instead of using a mass deployment technology, like Apple’s Automated Device Enrollment, the iOS devices are “prepared” using Apple Configurator. This includes uploading a Wi-Fi profile to the devices as part of that process. In Las Vegas, this Wi-Fi profile wasn’t set to auto join the Wi-Fi, resulting in the need to manually change this on 1,000 devices. Furthermore, 200 devices weren’t reset or prepared, so we had those to reimage as well.
Black Hat Asia was different. We took the lessons from Black Hat USA 2023 and coordinated with the contractor to prepare the devices. Now, if you’ve ever used Apple Configurator, there’s several steps needed to prepare a device. However, these can be combined into a Blueprint.
For Black Hat Asia this included:
- Wi-Fi profile
- Enrollment, including supervision
- Whether to allow USB pairing
- Setup Assistant pane skipping
In Meraki Systems Manager, we controlled the applications by the assigned use, designated by Tags. When we came in on the first morning of the Briefings, three iPhones needed to be changed from lead scanning in the Business Hall, to Session Scanning for the Keynote, so the attendees could fill the hall faster. Reconfiguring was as simple as updating the Tags on each device. Moments later, they were ready for the new mission…which was important as the Keynote room filled and had to go to an overflow room.
We also were able to confirm the physical location of each device if wiping was required due to loss or theft.
When it was time for the attendees to register, they just displayed their QR code from their personal phone, as received in email from Black Hat. Their badge was instantly printed, with all personal details secured.
This goes without saying, but the iOS devices (Registration, Lead Capture and Session Scanning) do have access to personal information. To ensure the security of the data, devices are wiped at the end of the conference, which can be completed remotely through Meraki Systems Manager.
Content Caching
One of the biggest problems affecting the iOS devices in Black Hat USA 2023 was the immediate need to both update the iOS device’s OS due to a patch to fix a zero-day vulnerability and to update the Black Hat iOS app on the devices. There were hundreds of devices, so this was a challenge for each to download and install. So, I took the initiative into looking into Apple’s Content Caching service built into macOS.
Now, just to be clear, this wasn’t caching EVERYTHING… Just Apple App store updates and OS updates.
This is turned on withing System Setting and starts working immediately.
I’m not going to get into the weeds of setting this up, because there’s so much to plan for. But, I’d suggest that you start here. The setting I did change was:
Location and Jailbreak detection
One thing that we haven’t spoken about in some time is Jailbreak detection and Location. There are many elements that we get back from a device, but two of them, Location and Jailbreak must be retrieved from a device using a supplemental application: In this case, the Meraki Systems Manager agent.
HOWEVER, these can only be retrieved from the device if the application is running in the background. If the device has been rebooted, or the application terminated, then we don’t get anything.
One of the other painful, but understandable, aspects of MDM is that you can’t launch an application remote on a mobile device…. But you can!
On both Android and iOS, there’s a capability called Kiosk or Single App mode: Use cases for this are normally unattended devices, like in restaurants, or scanning devices like delivery drivers, etc. And when sending the command to the device to go into kiosk mode will launch the application. You can also send a command to remove kiosk mode from the device too. The great thing about this last point is that the application remains in focus and open!
So, the other capability that using Meraki Systems Manager gives us is the ability to schedule settings. Therefore, we can turn on kiosk mode in the middle of the night and remove it an hour later.
To ensure that this doesn’t impact the registration staff, we can go one step further: after we’ve launched Meraki Systems Manager, an hour later we can relaunch the registration application, Swapcard Go.
Systematic ThousandEyes Agent Deployment
ThousandEyes has been a hit at Black Hat. At an event where understanding immediately where issues lie in the network and beyond to ensure a great conference is paramount, the visibility ThousandEyes gives is incredible. Given that, and the complexity of the network here, and given that we have a Mac Mini deployed for caching software updates, as we are using Meraki Systems Manager (SM) for other purposes, I thought I’d take the opportunity to deploy the ThousandEyes Agent using SM.
The other reason is that, whilst we have a considerable amount of cloud and enterprise agents, we had no endpoint agents deployed. However, things are never that easy with software deployment, primarily because you need to provision / configure software once deployed. On mobile devices, this is straightforward, either using settings payloads, or by using Managed Appe Config to configure an app.
On desktop, using MDM, we can normally use things like Managed Plists to do the same thing, but the TE agent does NOT support this. Once installed, we must call the agent with a string.
So, to achieve all this, we can package the agent and command into a package using a command line utility on the Mac called PKGBUILD (more details here).
I also used a guide I’d written for the Meraki Community, available here.
Facts of note:
The Postflight:
#!/bin/bash
# this name will change with each version of the agent
installer -pkg /tmp/Endpoint\ Agent-x64-1.193.1.pkg -target /
/Applications/ThousandEyes\ Endpoint\ Agent.app/Contents/MacOS/te-agent –register “YOURUNIQUESTRING”
exit 0
The command to build the package using PKGBUILD
More details here or watch the video.
Repurposing of Devices for the next show
We were asked if there was anything we could do to leave the devices as they were for the next show. After careful consideration, we decided that we could leave the devices in a state that was amenable to everyone. The major requirement was leaving the Swapcard Go app on the device. But, as the app is provisioned for each show, it’s quite the process to remove configuration and then re-add it….
So, the other thing to note is the options that we have when installing (and removing) an application on a managed iOS device:
Remove with MDM is the interesting one, as it allows us to, rather than WIPING the device at the end of the show, to remove management, including any apps and settings, and their corresponding data.
The problem with this is that this was never a requirement at the start of the show. So, we now need a process in a particular order to facilitate this…. As this is for only a handful of devices:
- Deprovision the app from devices by unscoping the application in Meraki Systems Manager
- Wait to see this command has completed across all devices
- Reprovision the app using MDM again, but with this being a new app install, it will allow the OS to keep the app in situ after an unenrollment
- Wait until completed
- Unenroll the device
Domain Name Service Statistics, by Christian Clasen
Since 2018, we have been tracking DNS stats at the Black Hat Asia conferences.
The historical DNS requests are in the chart below.
With over 18.2M DNS requests made, we had the most to date at an Asia show. We made visibility advancements at the previous year’s Asia conference. Prior to Asia 2023, we were allowing attendees to use their chosen DNS resolvers over our assigned internal Umbrella Virtual Appliances. In coordination with Palo Alto Networks (the conference Firewall provider), we began intercepting and redirecting DNS queries for other resolvers, to force resolution through the Umbrella gear. While this is only effective for plain-text DNS queries and not encrypted protocols like DNS over HTTPS, it never-the-less dramatically increased visibility as evidenced by the numbers in the accompanying charts.
The Activity volume view from Umbrella gives a top-level level glance of activities by category, which we can drill into for deeper threat hunting. On trend with the previous Black Hat Asia events, the top Security categories were Malware and Newly Seen Domains.
In a real-world environment, of the 18.2M requests that Umbrella saw, over 2,000 of them would have been blocked by our default security policies. However, since this is a place for learning, we typically let everything fly.
We also track the Apps using DNS, using App Discovery.
- 2024: 4,327 apps
- 2023: 1,162 apps
- 2022: 2,286 apps
App Discovery in Umbrella gives us a quick snapshot of the cloud apps in use at the show. Not surprisingly, Generative AI (Artificial Intelligence) has exploded over the previous year as a top application.
Umbrella also identifies risky cloud applications. Should the need arise, we can block any application via DNS, such as Generative AI apps, Wi-Fi Analyzers, or anything else that has suspicious undertones.
Again, this is not something we would normally do on our General Wi-Fi network, but there are exceptions. For example, every so often, an attendee will learn a cool hack in one of the Black Hat courses or in the Arsenal lounge AND try to use said hack at the conference itself. That is obviously a ‘no-no’ and, in many cases, very illegal. If things go too far, we will take the appropriate action.
During the conference NOC Report, the NOC leaders also report of the Top Categories seen at Black Hat.
Overall, we are immensely proud of the collaborative efforts made here at Black Hat Asia, by both the Cisco team and all the partners in the NOC.
Black Hat USA will be in August 2024, in Las Vegas. Christian Clasen will lead the Cisco team in the NOC, so follow his blog to see if what happens in Vegas, stays in Vegas.
Acknowledgments
Thank you to the Cisco NOC team:
- Cisco Security: Christian Clasen, Shaun Coulter, Aditya Raghavan, Adam Kilgore, Patrick Yong and Ryan Maclennan
- Meraki Systems Manager: Paul Fidler and Connor Loughlin
- Additional Support and Expertise: Adi Sankar, Robert Harris, Jordan Chapian, Junsong Zhao, Vadim Ivlev and Ajit Thyagarajan
Also, to our NOC partners NetWitness (especially Iain Davidson and Alessandro Zatti), Palo Alto Networks (especially James Holland and Jason Reverri), Corelight (especially Mark Overholser and Eldon Koyle), Arista Networks (especially Jonathan Smith), MyRepublic and the entire Black Hat / Informa Tech staff (especially Grifter ‘Neil Wyler’, Bart Stump, Steve Fink, James Pope, Michael Spicer, Jess Jung and Steve Oldenbourg).
About Black Hat
Black Hat is the cybersecurity industry’s most established and in-depth security event series. Founded in 1997, these annual, multi-day events provide attendees with the latest in cybersecurity research, development, and trends. Driven by the needs of the community, Black Hat events showcase content directly from the community through Briefings presentations, Trainings courses, Summits, and more. As the event series where all career levels and academic disciplines convene to collaborate, network, and discuss the cybersecurity topics that matter most to them, attendees can find Black Hat events in the United States, Canada, Europe, Middle East and Africa, and Asia. For more information, please visit www.blackhat.com. See the press release for Black Hat Asia 2024.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Security on social!
Cisco Security Social Channels
Instagram
Facebook
Twitter
LinkedIn
