Recently Fernando Montenegro of 451 Research, part of S&P Global Market Intelligence, released a new thought leadership report, “Cloud Security is a Team Sport,”* (*Login required) that breaks down the need for collaboration and teamwork when tackling cloud security in two key areas:

• Within the vendor community, highlighting the need for collaboration between cloud service providers (CSPs), third-party security vendors, and services firms, as most customers are looking to secure their cloud deployments with a combination of these three key players.
• Within the organization itself, as internal teams such as DevOps and Security need to come together when building and operating cloud security in order for it to be effective.

The report explores how each and every cloud provider is unique in its own approach to cloud infrastructure security, which can raise additional challenges as very few organizations are run on one cloud alone. With each individual cloud provider differing in how they define identities, assign privileges, and so on, the security challenges across clouds, and across teams, grow.

Fernando also takes a look at how the third-party vendor landscape breaks down to provide additional security and bridge a lot of these gaps. (Full disclosure – Capsule8 is listed as a newer vendor with offerings around cloud workload security, primarily containers and Kubernetes).

Now when it comes to Security and Operations teams, it may seem like they are on different sides of the table during discussions about who can do what and why in a production environment, but to quote an entirely overused movie quote, “We’re not so different, you and I.” Both departments are working towards the same goal, a successful and secure production environment.

While we can only briefly summarize Fernando’s most recent report above, we’ve compiled a number of resources that may be helpful in overcoming some of the challenges raised by his report, notably moving to secure the cloud and helping Security and Operations teams work harmoniously toward their goals.

The Journey Toward Smoother and More Secure Workload Modernization Efforts

451 Research, part of S&P Global Market Intelligence,  recently investigated the considerations that organizations made in deciding how to proceed with application modernization efforts and released their findings in the Business Impact Brief, “The Journey Toward Smoother and More Secure Workload Modernization Efforts.” In it, they discuss how organizations can successfully and securely use cloud environments and technologies to drive their business forward.

Webinar Replay: How Security Teams Can Better Engage With DevOps

In a recent live webcast, 451 Research’s Fernando Montenegro and Capsule8’s Kelly Shortridge discussed the relationship between Security and DevOps. For DevOps, performance degradation issues or production crashes are not a risk they are willing to take by adding security to their systems, regardless of how important it is for business. So how can CISOs help both teams work together? Fernando and Kelly give tips to help CISOs improve engagement among the teams.

30 Techniques to Align Security with What DevOps Already Loves

David Spark of CISO Series jokes that the difficult relationship between security teams and developers needs couples counseling. They both know that they need the other, but cannot compromise on how to work together. So, when Sparks asks security professionals what they think security is already doing or could do that would be embraced by DevOps, there are a multitude of responses. Read about what 30 techniques security professionals suggest to help align security and DevOps.

Security Is Suffering From DevOps FOMO

On this episode of the CISO/Security Vendor Relationship Podcast, hosts David Spark and Mike Johnson welcome as a guest, Dayo Adetoye, senior manager of security architecture and engineering at Mimecast. Together, they discuss how to include security in DevOps without disrupting operations.

^{Photo credit: @montaelkins – Kelly Shortridge Keynote at S4x20}

Last week, I keynoted S4x20, the biggest industrial control systems (ICS) security conference in the world, and was able to catch quite a few talks, too. While it took place in sunny Miami Beach, my highlights from the conference suggest a far cloudier outlook. Specifically, there seems to be a growing rumble about the adoption of cloud-based infrastructure, including the DevOps mindset it entails, and what it means for ICS security.

Three talks I saw at S4x20 really stuck out to me, covering critical infrastructure as code, applying chaos engineering to field-based critical systems, and rethinking how we measure security. In this post, I’ll highlight what I found most interesting from each of them to whet your appetite for when the videos of the talks are posted.

Critical Infrastructure as Code

Configuration as code is blooming with popularity in the DevOps world, but is less popular in the security world — let alone the ICS security world, which tends to be even more conservative given the importance of critical infrastructure. However, Matthew Backes from Lincoln Labs outlined how to bring the decidedly modern practice of config as code to ICS, in what he calls “Infrastructure as Code” (infrastructure in the ICS sense), to help reassert control over ICS systems.

One thing I learned from his talk is that configuration of ICS devices is a hot mess. It tends to be highly manual, ad-hoc, and vendor-specific, with a high probability of disrupting systems or bricking devices when it goes wrong. Even pulling the current configuration of devices is a nasty affair — it usually resides in the sysadmin’s head or in scattered, undocumented locations. Otherwise, the device must be directly accessed (or “touched” as is apparently common ICS parlance) to gain the ground truth of the config. Backups are a fantasy. As you can tell, the pain is real.

As Matthew described in his talk, the goal of infrastructure as code for ICS is to cleanly define configurations and push them down to the device — no touching or mind reading required. There must also be version control (something like git) and accessibility (something like an API). Lincoln Labs’ approach was to use Ansible with a JSON config file and then a standardized API written in Python. They experimented this on protection relays, remote terminal units, genset controllers, and serial-to-ethernet converters (of which I knew nothing about and recommend you look out for the recording of the talk to learn more).

The workflow will look familiar to those of you who know how config as code works — ICS is now getting in on the YAML party. There are standardized templates to define device / service functionality, a host inventory (list of devices in the environment), and variables at the global, vendor, and device level (like IP addresses, protection settings, or specific files). These components generate a config file, which can be used to push settings across devices using a tool like Ansible, or even as a compliance report for auditors.

The lessons learned were probably the most interesting to me, given they’re based on real experiments Matthew’s team conducted among operators in field environments. First, there’s a tradeoff between manageability and security, depending on the device type. For instance, protection relays expose their config files through telnet FTP, not SSH, and exposed management interfaces often lack password protection. Second, there isn’t a lot of interoperability, which can lead to bricked devices or other issues when attempting to parse config files. Thus, quite a bit of modification is required to get config as code to work for your ICS.

It’s clear that simplicity and manageability are sorely lacking in ICS, making it difficult to control those systems. There’s appetite for central config storage and standardized workflows, but the tooling hasn’t caught up. While Matthew made a call to ICS vendors to improve their config functionality for the benefit of the community (which is absolutely needed!), it struck me that the Ops community could provide a valuable helping hand here, too. 

The ICS security and DevOps communities are far from close, but my hypothesis is that Ops engineers from Silly Valley and elsewhere can pattern match their way through some of the roadblocks ICS defenders, being new to the concept, will face in attempting to implement config as code. It’s a way to harness your hours of using Terraform, Puppet, Ansible, Chef, Cloud Formation, and more for the benefit of us all, as even helping set up the basics will help ICS security seriously level up their abilities!

Chaos Engineering to Avoid National Chaos

Virginia Wright, a Program Manager in Idaho National Laboratory’s Cybercore division, discussed how to apply DARPA’s RADICS program, which stands for “Rapid Attack Detection, Isolation, and Characterization Systems”, to chaos security engineering. She acknowledged it involves a mindset shift from contemplating how to keep attackers out of our systems towards figuring out how to cope once they’re inside. This closely aligns with my own advice of moving away from the ideal of perfect prevention and instead striving towards resilience — ensuring you can recover gracefully in the face of inevitable failure.

Applying chaos engineering to critical systems in the field is a daunting task. But operators need to feel confident in their ability to recover from attacks, and cannot do so without practicing within real, or at least realistic, environments. Virginia walked through an exercise her team conducted consisting of a “black start recovery of a crank path amidst a cyber attack on the power infrastructure to enable grid restart operations.” In non-ICS speak, I believe it means “restoring operations of an isolated part of the energy grid to seed power back into the overall grid.”

Importantly, this exercise ran on real equipment, not simulated systems, and played with failure modes, like an extended regional power outage. Thus, while the scenarios were a bit scripted to create some structure to the exercise, it still felt pretty random to the operators themselves. 

The campaigns themselves use “consequence-based engineering,” beginning with defining the highest-consequence outcomes and figuring out how the attacker would achieve that outcome. This is essentially the same as the advice in my keynote to start with the most important assets to your organization (like customer data in an S3 bucket, uptime of a particular service, etc.) and work back to how the attacker would most easily compromise that asset. Either way, repeatedly running campaigns with these scenarios helps you to determine the potential blast radius of a real incident and to continuously refine your processes and tools.

I was especially delighted that Virginia espoused the philosophy of confronting your worst fears, bringing to mind one of my favorite lines by Chuck Palahniuk, from his novel Invisible Monsters: “Find out what you’re afraid of and go live there.” We need to develop the “muscle tone,” as Virginia put it, to deal with that fear and move forward from it — otherwise we will feel lost, uncertain, and afraid when our fears (like a real attack on a power plant) materialize. This confrontation via experimentation can (and should!) start small, letting us build confidence and iterate over time.

My own talk recommended starting with less critical systems that are still a part of ICS, like predictive maintenance systems, Office 365, cloud storage, or grid edge systems. Relatedly, I recommend Bryan Owen’s insightful talk on the ICS shift to cloud from last year’s S4 to learn more about the current transition underway. I still think that beginning with those less critical systems is a fantastic way to become more comfortable with the practice of chaos security engineering, but Virginia’s advice can help you safely navigate practicing it on in-the-field critical systems.

Rethinking Security Measurement 

It turns out that figuring out how to rank or score the security of companies is hard. Derek Vadala, Global Head of Cyber Risk at Moody’s, discussed their approach to rating the security risk of companies. While this arena is incredibly complex and full of slippery variables, I liked how Derek offered a straightforward division of companies into two buckets: those who can defend against commodity attacks, and those who can additionally defend against “sophisticated” attacks. Digestible mental models are underrated in security, in my opinion.

I found his recognition that technology shifts — like the shift to cloud-based systems — impact measurements to be particularly insightful. One example that Derek highlighted is that uptime is no longer a desirable metric. The longer systems are online, the higher the chance attackers will compromise and persist in them. This notion neatly fits within my own talk, in which I advised a goal of ephemerality and using a metric of reverse uptime to keep it in check (minimizing the amount of time a host is online to reduce persistence opportunity).

The type and quality of data provided for security ratings is also highly influential. As Derek noted, there’s a tradeoff between data fidelity (how valuable it is as a signal) and effort involved to retrieve it. Validated data, such as that collected from a live exercise, is invaluable in cultivating confidence in the security of deployed systems. This is another overlapping point on the benefits of chaos security engineering from my keynote. Repeated experimentation via the injection of security failure generates feedback about the resilience of your systems, giving you greater fidelity with minimal ongoing effort (albeit with non-trivial upfront effort).

The general inaccessibility of valuable signals will assuredly present a stumbling block for cyber insurance companies as well, who want the highest quality signal possible for the potential risk a company poses, but are likely to receive self-provided data at best and external data only at worst. Nevertheless, these sorts of security ratings pass the muster of a “good enough” health check to give insurers a high-level sense of a company’s security health. Given quote speed is a key driver of insurance purchases, fast but “good enough” will win over slow but “highly accurate” within the industry.

Final Thoughts

In the quintessential hallway track, much of my personal discussions involved the stickiness of the existing ICS security mindset. Specifically, one of the barriers in ICS to adopting modern infrastructure and security practices isn’t technical in nature at all — it’s the emotional cocktail of skepticism and inertia. Part of the goal of my keynote was to demonstrate benefits to security of all this cloudiness and containerity, but it will take many members of the community continuing the drumbeat before the collective perspective shifts.

Luckily, S4x20 had a palpably strong community vibe, which is essential for changing hearts and minds, not just technical approaches (and free lunch from a taco truck doesn’t hurt in fostering a positive spirit). The range of backgrounds, from practitioners in ICS, national laboratories, risk quantifiers, to vendor-resident thinkers like me, helped provide a healthy variety of topics and perspectives. 

As always in infosec, the conference was predominantly white and male. I noticed that the sponsor stage was especially homogenous, which presents an opportunity for S4’s organizers to offer incentives to vendors who put forward speakers from underrepresented backgrounds (such as a small discount on the speaking slot price). S4 is far from alone in this (looking at you, RSAC), but I was impressed by how professional and welcoming it felt, giving me greater confidence in its ability to lead the charge on this front.

Overall, I can attest the conference was impeccably organized, so would highly recommend speaking there if you have the opportunity. I learned a lot about the ICS niche and can empathize better with the challenges ICS security practitioners face, and hopefully those who attended my talk feel more comfortable with DevOps, modern infrastructure, and chaos security engineering. We all know ICS security is vital for our economy and society (even if just to avoid “cyberpocalypse” headlines in the event of a major compromise of critical infrastructure), so it’s heartening to see an openness to fresh ideas and an eagerness to work together to level up the community.

What do you get when you take a security conference and pare back its typical formula of swag-laden vendor tables, high-concept lighting that promises to be “an experience”, bougie parties with LED-lit stemware and a surplus of decibels — not to mention all of the offsec-focused talks? You find a group of dedicated defenders who, freed from the flash and fanfare, come together not to exchange pitches but to exchange ideas. And donuts.

That’s Art into Science.

Also known as ACoD — a shortening of its tagline “A Conference for Defense” — the Austin-based Art into Science has a fairly rare value-prop in the security conference landscape: “glorify the defense”. It’s common for security cons to over-index on stunt hacks, 0day drops, and tales of ultimate pwnage; ACoD aims to create space for the valuable and often-overlooked defender perspective. 

Having spent the better part of four years working in security operations, and now supporting a product that secops teams use, ACoD had been on my radar for a while, but this was my first year in attendance. (Our own Kelly Shortridge was a speaker there the previous three years, so this year in Kelly’s stead they got a slightly taller Capsulator with the same initials.) Most of the talks I went to were in the operations track. A few themes I observed:

Building/customizing rather than buying/outsourcing

Several of the talks in the operations track followed the general narrative of: 

1. We kept encountering this problem
2. Here’s how we approached trying to solve the problem
3. Here’s the open-source tool we built to solve it

Perhaps the most striking of these was Philip Gardner and Derek Chamorro detailing building their own serverless SIEM in order to handle the high volume of log data they ingest — and measurably decreasing their cloud services bill. I noticed a good bit of variety in this and other alerting pipelines that were discussed, reflecting the complex infrastructure of all of our respective environments. For detection tooling, rather than buying one pane of glass to rule them all, there’s a drive toward making the existing panes of glass play nicely with one another. Words like “webhooks”, “API”, and “integration”featured prominently.

Often in build vs buy debates, the question arises, “will it scale?”. In “Democratizing Chrome Extension Security”, Jacob Rickerd explored the threat landscape of Chrome extensions, and building CRXcavator to surface extensions’ unique risks. No viable tooling previously existed to comprehensively assess extensions’ attack surface such as permissions changes over time and potentially risky API calls. (Full disclosure, Jacob is a former colleague and my old team was customer zero for this tool.) 

What stood out to me was that CRXcavator was intentionally built for use beyond its original internal customer and use case — distinguishing it from many internal tools that struggle to retrofit themselves for external use and a broadening scope. A full API was released along with the tool, as well as API documentation and discussion of its use at enterprise scale.

A welcome shift in language

For all of the FUD-speak of 0days, nation-state attacks, and APTs, the more realistic threat model for many blue teamers involves well-intentioned mistakes, misconfigurations, and undetected changes. It was refreshing to see this reality reflected — from a standpoint of blamelessness and enablement — in ACoD talks. 

Jordan Wright, Nick Mooney (also both former colleagues), and Matt McNiece released Secret-Bridge for monitoring Github for secrets leakage — a near-universal occurrence among developers, rarely with malicious intent. Rather than the counterproductive angle of trying to shame users into behavioral change, tools like this reflect a judgement-free acceptance of the fact that humans are messy and complex, and when mistakes inevitably happen we should have a way to detect and remediate them.

Reframing resilience

The one philosophy track talk I did catch was Guillaume Ross’s “Reliability as a Liability”. Conventional wisdom often frames complete availability at all possible costs as the measure of success. Uptime, however, can mean uptime of attack surface and pivot points; a server that stays online for 200 consecutive days gives ops pristine availability metrics, but also gives an attacker a comfy and stable place to chill. 

Drawing upon examples of attacks that were inadvertently mitigated by interrupted availability (such as the DoS of a DHCP server that left more sensitive endpoints unroutable), Guillaume reimagined building for “resilience” not as unilaterally preserving uptime, but as designing strategically-placed breaks in uptime to reduce blast radius. He drew parallels to cars engineered with “crumple zones” to bear the full impact of a collision, leaving more critical zones (read: driver and passenger areas) intact. Ephemeral infrastructure is a more familiar concept in the DevOps world; it’s heartening to see these ideas begin to gain traction in security spaces.

And the winner for buzziest word is…

The MITRE ATT&CK framework is clearly the current hotness — it came up in several talks. (The pivotal question remains unanswered as to whether “ATT&CK” is pronounced “attandck” or “attampersandck”.) The two matrices I saw mentioned most were Windows and cloud. Tim Frazier used the Windows matrix as the foundation for his attack simulation tool to assess detection capabilities. While I’d caution against taking the ATT&CK framework as 100 percent comprehensive, many teams are leveraging it as a tool to more easily identify coverage gaps based on their infrastructure, rather than starting from a completely blank threat model.

Read it now: Forrester Q&A on the MITRE ATT&CK Framework and Modern Cloud Infrastructure

What’s missing: Linux and containers 

Detection and response at the public cloud level (mainly AWS) came up in a few talks. The container and orchestration layers, however, were greenfield. A few of the folks I spoke with whom I’d consider security experts noted their lack of familiarity with container security. (Incidentally, we’re working on writing a blog series on container security — more to come!)

I also noticed that many of the malware-related talks were heavily Windows-focused. It’s difficult to work in security without ever interacting with Windows at some point, but Linux is everywhere from production workloads to embedded systems, and there’s a ton of Linux security content that’s ripe for exploration. 2021 may not be the year of Linux on the desktop, but maybe it’ll be the year of Linux on the ACoD?

A note on accessibility and diversity

A win for accessibility: every talk I attended had a microphone for audience questions. However, the venue still presented some challenges for those who are hard-of-hearing or have auditory processing disorders: the two tracks could hear each other, and no sound barrier existed between the operations track and the area where people mingled. Having that separation also would have aided in facilitating more of the organic conversations that pop up during cons — the foundation of a successful “hallway track” is something akin to an actual hallway.

Though the atmosphere was friendly and open, the diversity of the ACoD crowd both onstage and off left room for improvement. I hope that going forward, the organizers will, particularly in building a speaker lineup, take an intentional, intersectional approach to leveling up the con’s diversity. The goal of conference program committees should not be to have speakers reflect a demographic cross-section of the current industry (which just perpetuates the status quo), but to give a platform to a wide range of practitioners whose diverse lived experiences inform their threat models and approaches to solving a problem. “The status is not quo.”

One small change that can facilitate a broader speaker base: move the CFP window earlier. While on the surface it seems inconsequential, ACoD closing its CFP two weeks before the con privileges those who are either local to high cost-of-living Austin or in a position to make last-minute travel plans (they’re probably not a primary caregiver, and they either have an employer who’ll foot the bill or they can afford more expensive flights).

I mention these not to throw shade, but to give a loving nudge to make a good conference even better. Even at a con that’s informal by design, there’s always some degree of care and feeding needed to foster an inclusive environment.

Conclusion

I left ACoD tired yet energized, my head buzzing with new ideas and perspectives — the mark of a successful conference. If you’re looking for a con with a chaotic-good character alignment that elevates the art of blue teaming, this is one to watch.

Understanding DevOps trends is essential for infosec professionals. Before you angrily close the tab because you are tired of lectures about the need for infosec to work with DevOps, consider whether the idea of a job focused on strategic, innovative work rather than firefighting and gatekeeping is appealing. If so, then these trends matter for you.

In this post, I will highlight the insights from this year’s Accelerate: State of DevOps Report that are especially pertinent to infosec, divided into the following five themes:

1. Speed & stability (including security) are bffs
2. Elite DevOps = faster IR
3. Feedback loops work
4. Stricter reviews don’t work
5. Security’s role is shifting, not dissolving

Note: Based on survey results and rigorous statistical methods, the report defines benchmarks for elite, high, medium, and low performers (see page 18 of the report). While elite performers are determined by these benchmarks, there are other characteristics and drivers of elite performance that I will be highlighting in this post, as well.

Speed & Stability are BFFs

In the modern enterprise, information security and stability have become inexorably linked. When we see public instances of instability, we often assume a security incident is its cause. Is that service down because of an attack or due to a developer’s error? Systems sometimes must be taken offline in order to patch vulnerabilities or otherwise remediate security issues. Compromised databases containing customer information need to be investigated internally and by regulatory bodies, impeding performance. 

Despite this, infosec professionals commonly believe that speed and stability are like Paris Hilton and Lindsay Lohan — more frenemies than friends. This year’s Accelerate State of DevOps report, and all the ones before it, demonstrate that speed and stability are more like Leslie Knope and Anne Perkins — working together to enable and elevate each other.

As the report outlines, stability is measured through time to restore service and change failure rate, while speed is measured by deployment frequency and lead time for changes. Time to recover is a fitting metric for security programs, as I will discuss more in the next section. In fact, this metric is codified in article 32 of GDPR, which states that an organization must be able to restore access to personal data in a timely manner in the event of an incident. 

Change failure rate has an additional application to security by measuring the percentage of changes to user-facing resources that result in an insecure service and subsequently require remediation. These metrics improve as our ability to deliver software quickly improves, because the faster we can deliver a software change in normal conditions, the faster we can deliver security-related software changes in emergency conditions. Therefore, your security program must be designed to work with speed, not against it.

Likewise, being secure one day does not mean you stay secure.

Some organizations that were high performers in last year’s report slipped back into being medium performers. Many seemed to struggle with increased complexity, some of which may be a result of complacency — the assumption that their high performance would persist even as their systems burgeoned. Likewise, being secure one day does not mean you stay secure. The world changes, and you are not owed a participation trophy. Your context will change as your technological capabilities advance and frequently become more complex. Security must be measured by progress, not acute health checks. You cannot become complacent, or you will be left behind, relegated to work that feels tedious and pointless.

Elite DevOps = Faster IR

This statistic perhaps says it all: elite DevOps teams recover 2,604x faster from incidents — taking less than one hour to restore services rather than one week to one month like low performers. Security practitioners face enormous pressure to respond quickly to incidents, so responding over two thousand times faster might feel like being blessed with the winged sandals of Hermes.

According to IBM X-Force, the average time to identify a breach is 206 days, and 73 days to remediate a breach. However, companies who detected and contained breaches in fewer than 200 days spent $1.23 million less on cleanup costs. Imagine being able to reduce recovery time to a day rather than 73 — a fantasy made far more feasible through distributed, immutable, and ephemeral infrastructure (aka the D.I.E. model coined by Sounil Yu).

While far from the sole driver of elite software delivery performance, the use of cloud resources does propel better performance. Elite performers are 24x more likely to meet NIST’s five essential cloud characteristics than low performers (and they are savvier at estimating cloud costs). In shadowy corners of infosec meetups, you can still find professionals raging against cloud infrastructure, though when you start asking probing questions, it becomes immediately apparent that their cloud implementation is sorely deficient. Only 29% of respondents using cloud infrastructure agreed that they meet all five characteristics, so statistically speaking, you are probably doing the cloud incorrectly.

Cloud infrastructure facilitates the use of distributed, immutable, and ephemeral infrastructure, all of which can significantly improve response times (as Dr. Forsgren and I outlined in our Black Hat talk). Shuffling IP blocks makes lateral movement trickier for attackers, even if they gained an initial foothold. If a container is compromised, you can kill it and replace it with a fresh image. And resources with lifespans in the seconds to minutes range provide inherent containment, too.

Improved containment is not the only way being an elite DevOps performer engenders better incident response — detection and remediation get a boost as well. One of the essential characteristics for cloud computing is “measured service,” which requires monitoring capabilities. The research shows that monitoring and observability fuels performance, because it helps teams understand their systems and improve them. The same telemetry used for performance monitoring can absolutely be leveraged for security detection. Additional capabilities needed for elite DevOps, such as continuous integration, deployment automation, and even code maintainability benefit remediation efforts by making it much easier to patch systems, too.

Think of the benefits unlocked through DevOps for security as the “Three Rs of Modern Security — Reduce, Reuse, Recycle” (inspired by the three Rs of the environment). Reduce your persistent attack surface through immutable and ephemeral infrastructure. Reuse monitoring telemetry. Recycle scripts to automate and enable remediation. By collaborating with DevOps efforts and embracing the practices that help elevate them to “leetness,” you can elevate the security program as well.

Reducing response time also helps reduce burnout, which, empathy aside, is noted in the report as being vampiric to productivity. Infosec can be an insanely negative sphere in which to work, which is only magnified if you feel that none of your work meaningfully improves your organization’s security long-term — that you are scrambling to rectify ad-hoc events that are purported to be as devastating as the burning of the Library of Alexandria. Burnout does not have to be a fact of life of infosec, and we should embrace a combination of work recovery and strong software delivery processes to reduce burnout. As the research shows, elite performers are half as likely to report feeling burned out. How lovely would that be for infosec?

Feedback Loops Work

Information security is decent at collecting data (though generally with more noise than signal), but poor at using it to continuously enhance their situational awareness, thus leading to a somewhat stagnant strategy. Teams may collect data on the number of vulnerabilities in a particular component at a single point in time, but not consider its evolving relationships with other components and complexity at the system-level. Further, few organizations employ strong red team (offense) programs to create realistic exercises requiring the blue team (defense) to practice their ability to detect and recover from incidents.

The data from this year’s Accelerate State of DevOps report shows this neglect is a mistake. Organizations are 1.4x more likely to be elite if they create and implement action items based on the insights gleaned from disaster recovery exercises. Only 40% of respondents even perform disaster recovery tests at least once annually, but those who do have higher levels of service availability. Preparing for events that could impact your organization’s operations means your organization is less disrupted when they happen. This sounds obvious, but security is much more bark than bite in these practices.

Amassing a heap of processes offers dubious efficacy, so security teams should instead attempt to correct errors sooner. As the report explains, “techniques such as continuous testing, continuous integration, and comprehensive monitoring and observability provide early and automated detection, visibility, and fast feedback.” This is pertinent for infosec. Infosec teams vary in their level of detection and visibility capability, but usually maintain some basic level of capability. However, “fast feedback,” continuous testing, and continuous integration are rarely seen in practice — and are still somewhat obscure concepts for many infosec professionals.

Investment in performance improvements is difficult when you are drowning in a data swamp befouled by false positive alerts, tickets always marked as the highest priority, and last-minute requests for security reviews. However, to reach the goal of continuous and automated security — and thus a less frenetic workload for the security team — automated test suites must be built, just as they are required for continuous integration. From my observations, security teams sometimes implement automated vulnerability scanning, but not much else. 

However, testing goes beyond vulnerability scanning and into testing how systems respond to failure, as Dr. Forsgren and I discussed in our Black Hat talk. Just as disaster recovery exercises test how systems and teams respond to failure, security teams should implement their own failure testing, known as chaos testing, to see how systems and teams respond to security failures. For instance, inject expired API tokens to test your authentication mechanisms, or modify encrypted data to test your detection of unauthorized modification. Such tests do not simply evaluate the health of a system, but evaluate the system’s ability to respond to unwanted events — and whether progress is being made in improving the system’s resilience over time.

Stricter Reviews Don’t Work

Organizations who experience a breach or other scary security incidents are wont to panic and begin implementing strict security processes in the hope of preventing future events. In this regard, the security team ends up operating much like a change advisory board, an external authority whose approval is required, through a set of formal processes, before software is delivered to users. It is not uncommon to see security teams that serve as figurative gatekeepers to releases, performing stringent security reviews before imparting their blessing and allowing the release to proceed.

This year’s report digs into why this strategy is folly, and not just because it erodes performance. Spoiler alert: gatekeeping is negatively correlated with resilience because it corrodes adaptability. The data bears this out: organizations were 2.6x more likely to be low performers if they maintain a formal approval process — an unsurprising statistic. There was also no evidence that formal approval processes were associated with lower change failure rates. 

In fact, gatekeeping contributed to more delays and more instability. It is difficult to promptly mitigate risk upon discovery if each change involves a protracted process. Additionally, waiting for approvals leads to an unwieldy backlog of code, which increases the blast radius of changes once they hit production. This increases the likelihood that these changes will engender failure, and once they do, debugging such a massive change inflicted on your codebase becomes difficult. From a security standpoint, the end result is exposing your organization to higher failure rates and a nice, juicy window of opportunity for attackers.

Instead, the security team must become, collectively, a strong and transparent communicator. If you speak to engineers, they often feel that infosec scries a crystalline orb containing occult shimmers that reveal the divine requirements for determining whether a new feature or service is sufficiently secure. This uncertainty over what steps are precisely required to gain the stamp of approval from infosec and how long the approval process will take deeply shackles performance.

Teams are 1.8x more likely to be elite performers if they possess a clear understanding of approval processes and what steps are required to go from “submitted” to “accepted” each time. These elite teams also express confidence that they can work through the approval process in a timely fashion. Fundamentally, when the organization has confidence in their processes, the organization is effective. Infosec teams, therefore, need to maintain clear, documented, and standardized processes for their security reviews — your obfuscatory vibe isn’t cute. 

Security’s Role is Shifting, not Dissolving

The above trends do not mean security’s role is dissipating before our very eyes. However, infosec’s mandate is decisively shifting. Despite propaganda that DevOps does not care about security, it does, and very much so, but it lacks the understanding of how to efficiently implement security into software delivery processes. The unwillingness to add excessive friction to software delivery is perceived as apathy, despite the fact that software delivery performance is the engine of the organization. If anything, security is too often apathetic towards ongoing business concerns. 

Security must adopt the mantle of infused SMEs, rather than attempting to recreate the wheel in a vacuum. Knowledge must be shared and simplified. As much as infosec takes an all-or-nothing approach to knowledge, there exists a level of “good enough” that will help engineering teams more securely design and architect their products and services. My belief is that security professionals of the future will be embedded on engineering teams, offering their knowledge and guidance throughout the software delivery lifecycle and helping their team level-up their understanding and implementation of security.

For instance, helping engineering teams understand the concept of “attacker math,” the fact that attackers will take the easiest and most inexpensive path towards their goals, can help improve threat modeling during the design phase. They do not need details around the latest side channel zero day, just an abstract understanding of why an attacker will try to phish for admin credentials if they can instead of exploiting a zero day vulnerability. Understanding that the easiest paths must be eliminated first to raise the cost of attack is a simple enough concept, but possesses an outsized impact on how engineers think about improving security.

One of the report’s findings is that training centers are not the most effective way for organizations to improve performance, because they simply introduce more silos and isolated expertise. Infosec is regrettably enthralled by training modules and awareness programs (or use them for checkbox compliance), which backfire by becoming bottlenecks of expertise and fostering exclusivity. Further, infosec experts are removed from the actual work involved in taking these generic lessons and implementing them into real workflows. Far too often, infosec is not learning and growing with the rest of the organization, but pushing its propaganda and hoping for indoctrination to take root.

Instead, “Community Builders” is the chosen strategy for most elite performers, in which teams learn and grow together. Teams evangelize their tooling and methods with other teams to share their knowledge and expertise. Or, a team may figure out how to transform themselves into elite performers and spread this wisdom with other teams. This strategy is resilient to product and organization changes, something essential for teams like infosec that are already spread too thin. For infosec to adopt this strategy, they must embrace the idea of cross-pollination. Infosec must learn about DevOps workflows to bridge security best practice to pragmatic implementation, or else best practice is just a set of untested hypotheses.

Relatedly, this year’s report notes that engineers on the highest performing teams are 1.5x more likely to report having useful, user-friendly tools that support their work. This is a problem for infosec, as even experts find many security tools cumbersome. If security tooling is not intuitive and easy to use, it will not become a core part of the software delivery lifecycle. Less painful tools will be adopted, even if they are considered less rigorous. 

It is natural that a team which selects and requires complicated tools is not one with a promising future. Other teams will figure out how to get their work done without those tools, and likely without that team, too. Worse yet, they can find workarounds that do not even accomplish the task — something every infosec professional has seen one too many times.

As a result, I would argue that experts in security UX are more relevant for the future than the legions of analysts who tune firewalls, SIEMs, and other blinky boxes. We need people who understand how to integrate the right security tools into software delivery workflows without adding friction. As the past two decades have demonstrated, security cannot be solved by throwing more tools and data at security analysts. Lamentably, the skills required to achieve this paradigm of usable security, a blend of threat modelling and UX knowledge, are sorely missing on typical enterprise security teams.

Conclusion

Security scoffed at DevOps and missed its opportunity to integrate security into DevOps practices at the dawn of the movement. Now, infosec vainly attempts to insert itself into the conversation through the term “DevSecOps,” creating its own sphere of siloed influence and practices that ignore the lessons DevOps learned on its path from low performance to elite performance. Automating security scans is not enough without investment into changing our priorities, processes, and culture. Luckily, by being late to the party, we have a guidebook at our fingertips for how to ascend to eliteness.

Everyone in security wants to be leet. The Accelerate State of DevOps report outlines the path to elite performance for us, if only we open ourselves up to the idea that there is much to learn outside of our infosec bubble. If we embrace the characteristics and practices of these elites, we can reduce the amount of painful gatekeeping, firefighting, and burnout we experience. We can support secure software delivery while also preserving its performance, the only path for security teams to follow if they want to stay relevant in the modern organization.

Another year at Black Hat has come and gone, with attendees  from around the world coming together to share and discuss their ideas, research, and discoveries. Did you attend Black Hat this year? If not, don’t worry. We’ve put together the highlights of this year’s conference.

Here are our top takeaways from Black Hat 2018:

1. The Days of Reacting to an Attack are Past

The occurrence of data breaches is growing, as is their size and complexity. We can no longer just react to breaches after they happen, but must be more proactive as to how we detect and defend against attacks..

During Black Hat 2018, there were a variety of hands-on training sessions focused on how security teams can become more proactive. A sold-out session titled A Guide to Threat Hunting Utilizing the ELK Stack and Machine Learning allowed attendees to create their very own enterprise-wide hunting platform and develop a method for retrieving the data from various endpoints and sources. Attendees learned to use normalization and visualization techniques to organize data and find outliers within each dataset as well as how to collect data efficiently from all endpoints in the network. The goal of all of these training sessions was to help bridge the gap between academic knowledge of threat modeling and the real world.

Raffael Marty discussed the dangers of algorithms. He emphasized that many businesses are blindly relying on algorithms to track and detect anomalies without having a deeper understanding of what the algorithms are doing. We shouldn’t rely solely on algorithms to detect all anomalies in our data, but instead, rely on a combination of machine learning and detection strategies driven by human experts.

2. The Influence of Social Engineering

Social engineering continues to be a prevalent attack technique,  manipulating individuals into giving away information to be used for fraudulent acts. The Advanced Practical Social Engineering discussion helped attendees to develop a strategy to build the skills, mindset, and tools needed to become a professional social engineer. Attendees learned about human vulnerability, the skills of influence and persuasion, and other psychological and technical tactics to master elicitation. In another discussion, Achieving Security Awareness Through Social Engineering Attacks, the presenter taught attendees about evaluating human vulnerability and using modern techniques to demonstrate this vulnerability through penetration testing.

3. Learn to Think Like a Hacker

Black Hat also held several training sessions on various hacking techniques. In order to stop hackers and disrupt attacks that can happen, one must think like a hacker. Hackers are becoming more sophisticated in the way they attack, so understanding the latest technology and techniques of hacking will assist in the recognition of attacks and allow disruption of the attack to happen faster. There were training sessions about car hacking, basic and advanced infrastructure hacking, and more. To highlight a couple, the Basic Infrastructure Hacking session was used to familiarize attendees with the fundamentals of network hacking and the Advanced Infrastructure Hacking session discussed a variety of penetration techniques to exploit platforms such as Docker, Kubernetes, and Windows and Linux modern operating systems using the latest, cutting-edge technology.

4. The Future is a Growing Concern

The future of security is always evolving, and there will become more concerns as technology continues to advance. One session at Black Hat covering this topic was by Charlie Miller and Chris Valasek on Applied Self-Driving Car Security. They gave insight into potential security concerns of self-driving cars. Should we be worried about cyber attacks in these new vehicles as they become more popular? As we look forward to the future, we must consider all of the additional devices in our lives that require additional security. This presentation focused on security for cars, but as smart products continue to develop (speakers, household appliances, and so on), new security measures may need to be developed.

 

Did we miss anything? If you attended the conference, let us know what you enjoyed most!

And if you didn’t have a chance to meet with us at the show and you’re interested in how Capsule8 can instantly detect and disrupt zero-day attacks in your production environment, request a demo!