This is post 1 of 2 on the subject of security in a DevOps environment. For step-by-step security recommendations, see my next post.
Recently, a roommate and I got into a discussion about experiences with various CI/CD tools. When I asked about security practices, his enlightening comment was, “I do the least amount of security possible.” Another roommate said he stays as far away from security as project leadership will allow. These are common, understandable refrains among developers under pressure to meet deadlines and pump out code quickly - and unfortunately sentiments that work against an organization’s best interest.
Expounding upon the discussions with my roommates, I wrote this blog post to provide security recommendations to developers and teams working in a DevOps environment. I am excited to share insights about a variety of new products and updated techniques that facilitate application security testing in CI/CD pipelines. My goal is to make it easier and faster for developers to design, build, test, and deploy secure software.
Introduction
DevOps is the practice of software developers and IT operations engineers collaborating throughout the entire application lifecycle, and it is characterized by integrated teams, frequent deployments, and automation. It results in faster delivery of new features, improved customer satisfaction, and more reliable service quality.
Continuous Integration (CI) is a closely related development technique that involves developers merging changes into a central repository, often several times a day. Each code check-in triggers automated builds and tests, which may or may not be automatically deployed as part of the Continuous Delivery (CD) model. CI allows teams to capture the benefits of DevOps and streamlines the process of validating software updates.
Security was a relative latecomer to the DevOps landscape, as it lagged in its ability to be software-defined (“infrastructure as code”) thus did not easily integrate into automated DevOps workflows. On the organizational side, the DevOps movement was successful in breaking down the barriers between development and operations, but security still resides in a separate silo for many companies. Additionally, the driving motivations for DevOps adoption are speed and innovation, but IT leaders often view security as inhibiting to DevOps agility because it can reside in slow-moving business units with outdated policies.
However, many organizations recognize that the rapid, incremental nature of DevOps methodologies offers advantages when applied to security practices. DevOps is more than just adopting Agile to the development process: it also involves a toolchain focused on automating repetitive, error-prone tasks to improve overall development effectiveness. Let’s explore how DevOps can offer advantages in security at scale.
In the traditional model, a development team hands off a near-complete feature to a security team, which tries to break it and find flaws. When flaws are inevitably found, there is a mad dash to make fixes in time to meet release deadlines. If the flaw is related to architecture or large-scale design choices (e.g. processing sensitive data in a cloud-based tool not approved for such use), the project faces delays and cost overruns or must implement band-aid solutions that leave the organization prone to issues in the future.
Contrast that with a future model where a development team consumes automated security services, fixing code and making decisions based on immediate feedback. Developers don’t want to wait days or a week for a security team to generate a report showing flaws - they want to know right away if code is bad. The quicker the feedback, the more impactful it is: developers won’t make the same mistakes again.
In this normative scenario, the security team monitors development progress and sees the results of automated testing. They consult with the development team on features being designed and can provide training as needed. Although automated security testing does not identify all bugs and a dedicated security team is still required, more flaws are found and fixed earlier in the code pipeline. Thus the security team has more time to build and evolve the automated security tools. Management is satisfied because there are reliable and repeatable security controls in place to document how risk is being mitigated.
Summarizing the differences:
| Traditional Model (reactive) | DevOps + Security (proactive) |
|---|---|
| Pre-production security audit on a near-complete system (if at all) leading to rushed, costly, incomplete fixes | Bugs and flaws found earlier in SDLC and security professionals able to be cooperative, iterative, and proactive in fixes |
| No involvement of security practitioners in design and architecture decisions | Security practitioners able to influence toolset, design, and architecture decisions |
| Manual, tedious compliance verification | Automated, real-time compliance status; programmatic policy enforcement; strong audit trail |
| Manual, incomplete, time-intensive code review when vulnerabilities discovered in open source software components (e.g. Heartbleed, Shellshock) | Automated detection of at-risk packages and libraries within minutes |
| Few ways to gauge the security quality of delivered code aside from complaints from the security team or actual security compromises in production systems | Quantitative way to measure security progress (e.g. defect density/programming faults per Lines of Code) |
| Developers and operations engineers view security as the responsibility of a compliance/risk department | DevOps teams are empowered to take a personal responsibility for security |
| Security policies and configurations must be manually set up for new environments and servers | Highly-scalable, versioned security infrastructure |
A quick note on terminology: various organizations such as Gartner and SANS Institute call the merging of development, operations, and security silos, “DevOpsSec” or “DevSecOps.” Both describe the practices outlined in this post. Since neither has wide recognition, I am hesitant to use them for fear that organizations will treat “DevOpsSec”/“DevSecOps” as an optional addition to DevOps rather than recognizing that security should be an integral part of it.
Continue on to my second post in the series, which offers security recommendations at each step in the CI/CD pipeline.