Integrating Security into the CI/CD Pipeline: Step-by-Step Recommendations

This is post 2 of 2 on the subject of security in a DevOps environment. For an introduction to the concept, see my previous post.

Starting Out

The first step towards the delivery of secure software is to conduct risk assessment and threat modeling. The exercise allows you to formulate security objectives and outline an appropriate security posture. For most of my audience, this task is handled elsewhere and your team faces directives such as, “We have HIPAA data so we need an automated way to ensure XYZ.” Risk assessment and threat modeling are beyond the scope of this post, but I encourage you to visit OWASP (Open Web Application Security Project) to learn more.

What Do We Actually Need to “Secure”?

Security can mean hundreds of things, so let’s break it down into manageable components and not get overwhelmed by the scope. A team moving to integrate an expansively-defined “security” into DevOps faces considerations such as:

• Security of the CI/CD Pipeline: authentication required to push changes, login tracking, key management, secure storage of build artifacts, etc.
• Security in the CI/CD Pipeline: automated security testing, static code analysis, etc.
• Security Automation: configuration management, automated incident response and forensics, secure backups, log monitoring, etc.

Thankfully, Security of the CI/CD Pipeline is relatively straightforward assuming you and your team followed best practices during creation of your project’s Git repository, AWS Security Groups, etc. At this point, I want to focus on Security in the CI/CD Pipeline and the components of Security Automation that can be leveraged in a DevOps workflow. The sections below provide a general overview of the areas with an opportunity to integrate security and can be used as a rough measure of organizational maturity towards security best practices. Steps are listed in roughly the order they appear in a typical CI/CD pipeline. For each, I include a basic description along with recommendations and example security-related tools.

IDE Plugins and Linters

Description: Lightweight static analysis tools that detect defects such as coding errors, poor coding practices, and security vulnerabilities as a developer writes code.

Recommendations: IDE plugins are commonplace, but many developers don’t realize how easy it is to download and utilize security-focused plugins. These save time by providing informative fixes during construction of a program and minimize code review effort in later development stages. One problem is that it is hard to enforce their usage on a team basis: some developers may avoid using the tools if security remains “someone else’s problem.”

Example Tools:

• Veracode plugins for Eclipse, Intelli-J, Visual Studio
• OWASP ASIDE for Eclipse (open source)
• Synopsys SecureAssist for Eclipse, Visual Studio
• FindSecBugs for Eclipse, Intelli-J, Netbeans (open source)

Peer Code Reviews

Description: Improve code quality with pair programming, informal walkthroughs, or formal inspections.

Recommendations: Since not every line of code can be reviewed, consider curating a “security checklist” among your team or organization that can provide simple test cases and validations. These checks can be performed manually with even a very basic knowledge of application security and ensure your team is thinking about common vulnerabilities during the development process. As a starting point for your checklist, OWASP provides an excellent “Cheat Sheet Series” that provides concise recommendations for a variety of design and testing categories.

Example Tools:

• Gerrit (open source)
• Phabricator (open source)
• Atlassian Crucible
• SmartBear Collaborator
• Codacy

Static and Dynamic Code Analysis

Description: Code quality tools integrated into CI applications such as Jenkins, Travis CI, or CircleCI. Static and dynamic code analysis is commonplace in a modern release pipeline and saves time by automating code review in areas such as styling, best practices, compatibility, and security.

Recommendations:
Keep in mind that most modern applications include components written by third parties, so reviewing your source code alone is not enough. The tools below will help monitor whether any open-source components that your application depends upon have known vulnerabilities in addition to finding flaws in your native code. Discovering and mitigating defects earlier in the development process will save time and money while supporting a more secure final application.

Testing conducted during the build process needs to run quickly. For any longer-running security services, consider scheduling nightly executions in the same manner that integration testing is scheduled at some organizations (asynchronous testing). In a later section, I will discuss additional factors that need to be addressed in security testing. All of the tools below integrate with Jenkins and/or other build servers.

Example Tools:

• Snyk
• SonarQube (with security-related rules enabled) (open source)
• OWASP ZAP (open source)
• Minion (open source)
• Rogue Wave Software Klocwork
• Codacy
• Synopsys Coverity
• IBM AppScan
• Veracode Scanner
• Analysis Core Plugin (open source) - allows you to generate reports and track issues over time
• RetireJS

Unit Testing

Description: Tests if classes, functions, and methods behave as expected. Unit tests don’t typically interact with external resources (e.g. databases or the network). Allows developers to refactor code and add features with confidence.

Recommendations:
Unit testing as traditionally conducted doesn’t offer much security protection. Developers focus on the smallest testable components of an application. To improve, ensure that scripts test misuse/abuse cases and check for common vulnerabilities such as buffer overflows and unvalidated input/output. For additional information, check out the OWASP Top Ten risks cheat sheet.

As with my advice for static code analyzers, unit testing needs to run quickly while longer-running security services should be scheduled for regular executions. The tools listed below either run from Jenkins or utilize a plug-in (microservices) architecture that integrates with Jenkins and other CI servers.

Example Tools:

• Common language-specific unit testing frameworks such as JUnit (Java), Mockito (Java), Jasmine (JavaScript), QUnit (JavaScript), pytest (Python)
• GauntIt (open source)
• BDD-Security (open source)
• Mittn (open source)
• Big List of Naughty Strings (open source)

Functional Security Testing

Description: Ensures software fulfills requirements in areas such as authentication and session management.

Recommendations: Traditional testing emphasizes what a program should do and functional testing cases are usually written in a similar format (positive requirements). However, there is a far greater importance to negative requirements in security testing, describing what a system should not allow. The agile concept of user stories, describing what users can do, has a converse concept in the form of evil user stories that is useful for formulating functional security test cases.

Example Tools:
Popular functional testing tools such as Selenium can be modified to automate actions that a user could take from a web browser, such as an unauthenticated user navigating to a restricted page or attempting cross-site scripting attacks. Most of the tools listed for unit testing can also be utilized for functional testing.

Configuration Automation

Description: Stores information about versions and builds of the software to provide traceability between software and test results. Automates deployments and can provision environments and servers.

Recommendations:
Humans are great at coming up with creative solutions but make mistakes when it comes to repetitive tasks. Configuration management tools make it easy to provision secure infrastructure repeatedly and at scale. Most DevOps teams utilize Ansible, Puppet, Chef, or Salt, so integrating basic security can be as simple as adding audit, access control, and server hardening policies to configuration templates and deployment scripts. Ansible is probably the most advanced in terms of security customizations and ease of integration into your CI/CD pipeline.

For deployments, take advantage of the concept of immutable infrastructure. If the instance needs to be updated, simply deploy a new server and retire the existing one. This method reduces scaling issues associated with patch management and improves security via hardened servers and regular replacement of server instances.
The tools below contain configuration management services, although their capabilities may extend into other areas.

Example Tools:

• Ansible (open source)
• Puppet (open source)
• Chef
• Salt (open source)
• AWS Config
• AWS CloudFormation
• AWS OpsWorks
• AWS Elastic Beanstalk
• HashiCorp Terraform
• OpenSCAP and STIGMA (open source)

Performance Optimization, Prioritization, and Reporting

Automated security scans can be complex and require an unacceptable amount of time to execute if steps are not taken to tune testing to your CI/CD needs. An appropriate testing configuration must run relatively fast, minimize false positives, and give indications of flaw severity.

For synchronous testing, or testing completed during the build process, a few approaches ensure the development team isn’t slowed down. First, conduct incremental/differential scans rather than full scans if using a tool that allows it. Second, trim the testing scope and rulesets to focus on critical vulnerabilities and high-confidence tests rather than an exhaustive set. The goal is quick communication of important vulnerabilities and risks to the development team. Tightening the feedback loop addresses issues immediately, fixes the build, and avoids repeats of the same issue in the future.

For cumbersome testing tasks, it’s advisable to relegate them to an asynchronous testing pipeline. Some teams run more exhaustive static analysis on a dedicated Git branch that is merged into on a regular basis (e.g. nightly) to trigger analysis. This two-part strategy allows you to capture the benefits of automated testing without slowing down your team’s workflow.

In the traditional CI/CD model, teams rightly avoid passing a build that contains quality errors. The reality of security testing (and security flaw remediation) is that following the same approach would be an undue burden on the development workflow. The time required to completely eliminate security flaws is infeasible. A pragmatic approach that weighs defect risks versus remediation time is needed. For example, application scanners categorize vulnerabilities as high, medium, or low severity. An appropriate go/no-go criteria for when the build fails might read: 95% of high severity checks pass, 60% of medium severity checks pass, and 40% of low severity checks pass. An additional rule could examine the category of vulnerabilities identified and specify that unvalidated input automatically fails a build regardless of the scanner’s severity assessment. For incremental/differential scans, the success criteria can reference the number of new vulnerabilities introduced rather than aggregate figures. Your team’s ultimate decision criteria should be determined by a thorough risk and threat assessment (described earlier).

Finally, the way that vulnerabilities are communicated to the development team must be understandable. The results should be delivered using the same bug tracking processes used in the normal CI/CD workflow. Grouping defects by location within the codebase or type can help developers fix errors quickly. Grouping defects by severity is useful for prioritizing the order of remediation. The testing configuration should minimize false positives because an excessive amount will require manual (and time-intensive) filtering or cause developers to ignore the results of security testing. Historical vulnerabilities should be tracked but reports should direct attention to the vulnerabilities introduced in the most recent build.

Conclusion

The integration of security into automated DevOps workflows makes it easier and faster to develop and deliver secure software. Fortunately, the DevOps ecosystem is rapidly evolving and new products and tools arrive on almost a weekly basis. I hope that I have sparked some ideas about how to leverage automated security tools in your own code pipelines. Let me know if there are any tools or plugins that you find especially useful!

Additional Resources:
Open Web Application Security Project
"Awesome AppSec" on GitHub
“Application Security and DevOps” report by Hewlett Packard Enterprise
"Continuous Security: Implementing the Critical Controls in a DevOps Environment" report by SANS Institute