Applying DevSecOps in Your Organization

In my last blog I wrote about the advantages and disadvantages of modern AppSec tooling and how penetration testing fits into a DevSecOps development environment. With all the tooling choices these days, development teams have a lot to consider. Static analysis (SAST), which is as far left as you can go in the development process, may be best for one organization, while a dynamic analysis tool might be better for another. Or a combination approach of static and dynamic might fit best for some organizations.

Each of these modern tools has strengths and weaknesses, but don’t forget that penetration testing should always be included as one of the testing strategies for your applications. Tooling and testing strategy is just one part of DevSecOps -- you will also need to understand the entire DevSecOps lifecycle and what aspects of tooling and testing you can build into the way your organization develops applications.

ARE YOU ON THE ROAD TO IMPLEMENTING DEVSECOPS?

More and more organizations are thankfully moving away from the traditional Waterfall development model towards DevOps, which promotes faster development. Note that DevOps is based on Agile principles so we should never think that DevOps replaces Agile. Agile is an iterative approach for software development that is typically based on customer feedback. Small, short releases called sprints are generally part of this Agile approach. DevOps, on the other hand, is a focus on closing the gap between the development and operations teams to enable faster and more efficient delivery of software.

However, the reality is that many organizations will end up with a hybrid model or try to build in a few DevOps aspects into their development process to meet their goals of satisfying customers and speeding up software delivery. And that’s totally okay. For example, some teams begin by moving away from sprints to CI/CD pipelines to release their code more frequently while others may want to automate some of their security testing by moving away from time consuming manual checks and implementing a Static Application Security Testing (SAST) and/or a Software Composition Analysis (SCA) tool early in the development process. Basically, you need to focus on quick wins and building in DevOps aspects where you can.

But many organizations are caught up in the hype and popularity of DevSecOps (aka “the new hotness”) without really putting together a plan on how to get there as an organization. This leads to one of the problems I currently see in the industry: the tendency to rush into DevSecOps without having a plan in place to get the most from this approach. But before trying to understand what DevSecOps is, we need to first understand what DevOps is.

THE THREE WAYS OF DEVOPS

If you are a fan of the Disney+ series “The Mandalorian” you’ll be familiar with the famous saying from the show, “This is the Way.” In the context of this blog, that way is the three ways of DevOps made popular by the book The Phoenix Project by Gene Kim, Kevin Behr and George Spafford. If you haven’t read The Phoenix Project, I highly recommend that you do. It reads like a novel telling the story of a fictional company called “Parts Unlimited.” One of the great things about this book is how the company feels just like an organization that many of us can relate to. In the book, you have different departments like IT, product managers, information security, and others that work together to get a product out the door under a tight timeline. Just like in real life though, unplanned work like escalations and outages derail the best of plans. This unplanned work leads to failure, which in theory, should lead to change in an organization. The book does a great job of showing how DevOps facilitates a culture of efficient workflows, fast feedback, and promoting innovation and experimentation.

The First Way: Efficiency of the Entire System

The first way of DevOps is ensuring efficiency of the entire system through speed, scalability, and automating everything from development, IT operations, and to the customer. This includes automating CI/CD pipelines, automating testing, and ensuring code is being released as fast as possible. The first way also includes removing constraints which can include people, process, or technology that gets in the way of system efficiency.

The Second Way: Fast Feedback

The second way is fast feedback, where you focus on getting testing results and feedback to the developers immediately and using tools to automate your security testing, where possible. You may remember in the old Waterfall development model; security testing was typically the last step in the process, which became the most expensive and time-consuming part of development. This model won’t fly in a DevOps world. Results of testing and problem identification need to be in real-time so that developers can make fixes and perform remediation quickly. The goal, of course, is to streamline the development process and avoid those old roadblocks from the Waterfall model.

The Third Way: Continuous Learning and Risk Taking

The third way of DevOps is constantly learning, making improvements, and taking risks by implementing new tools and testing techniques. In the third way, development teams should always be learning how to develop secure code and should be testing out new tools that will make their job and the development process more efficient. There are many types of security testing tools that are available for development teams and it’s important that developers are included in tooling conversations with the security team. Developers should be empowered to take risks and try new things to improve the security of their software. This is often a tough concept for people on security teams to understand, but we need to evolve into this way of thinking.

Here are some examples to help encourage developers to be more empowered with the security of their software:

• Encourage developers to conduct peer reviews of their code
• Set up frequent lunch and learns to collaborate and whiteboard ideas for secure development
• Start a security “champions” program to help train and mentor other developers
• Move away from old “CBT” style secure code training and implement new interactive “gamified” training options
• Let developers help with the development of secure coding policies and guidelines for the organization

Lastly, it is important to understand that DevOps is a methodology and a culture shift for the organization. DevOps is a shared responsibility, and leadership needs to embrace DevOps from the top down to be successful.

WHAT IS DEVSECOPS?

DevSecOps (Development, Security, and Operations) is where security is implemented into DevOps. In DevSecOps, security is woven into the entire development process. To work, security must shift to becoming an enabler instead of a blocker. Do you remember the days when the security team was the “gatekeeper” for the organization? That gatekeeping won’t work in DevSecOps, instead, security should be empowering developers to run security tools, improve their own processes and the security team is there to provide training, help and guidance. For example, security could help evaluate a new SAST tool and train the developers on how to use it, or they could help develop and mentor “Security Champions” to help promote secure coding practices to other developers.

There are also some creative ways to get developers to think about security. Years ago, when I was teaching developers on secure coding practices, my team developed a capture the flag event where developers had to hack and exploit real (vulnerable) applications. Not only did this exercise show them what it’s like being a penetration tester or real attacker, it also showed them how easy it was to exploit some of the same vulnerabilities found in their own applications. With one client, this event was so successful that developers pulled up code in their production environment during the training showing where some of these vulnerabilities were in their own code! This was also a good time to show them how to fix this code and make it more secure. Talk about a hands-on training experience!

Lastly, it’s important that security adapts to the way apps are being developed, not the other way around. For example, if you are using sprints, security testing is integrated into that process and doing sprints as well. We should let the developers help drive the conversation on where security fits into the way they develop applications. I should also note that moving to DevSecOps should be considered a journey, and not necessarily a destination. Along this journey will be many bumps in the road which will need to be overcome by a top-down leadership approach. What does that mean? Leaders in an organization must encourage and support their development teams through this journey. Without leadership support and partnership, moving to DevSecOps will be nearly impossible.

PLANNING CONSIDERATIONS WHEN MOVING TO DEVSECOPS

Not all organizations will be ready to jump right in and ride the DevSecOps train. For the majority, its best to start small and implement the most important aspects of a solid AppSec program first, then continue to move towards DevSecOps over time. When planning out and beginning your organization’s move to DevSecOps, you should include at least some of the activities noted in the typical DevSecOps Lifecycle noted below:

Development (Plan, Code, Build)

First of all, what security requirements are needed for the application(s) being developed? For example, if you’re processing and storing sensitive data or PII, what are the security requirements to store and process this data? How do you know what your security requirements are? You could start with the OWASP Application Security Verification Standard (ASVS) or the OWASP Software Assurance Maturity Model (SAMM).

Development within DevSecOps also includes at least some threat modeling and a process for reviewing architecture designs looking for security flaws. During coding, this is a great opportunity to deploy static analysis tools which can be integrated with the developer’s IDE. Security unit tests should also be developed, and the organization’s secure coding policies should be followed by the development team.

Testing (Test, Release, Deploy)

In DevSecOps, security testing continues throughout release and deployment. Automated tools like SAST, DAST, IAST, and SCA should be considered as well as manual testing activities like code review and of course, penetration testing. To better understand these tools and determine which can work for your needs, check out my article “Choosing the Right Modern Application Security Tools.”

Note that any manual activity needs to be more focused and you should automate it wherever possible. For example, code reviews and penetration testing can be scaled down to specific unit type tests and findings from these tests should be pushed directly into bug trackers so developers can get feedback immediately.

Operate & Monitor

Logging and monitoring tools should be used and a process for incident response should be implemented to respond to security alerts. This is also where automated testing continues such as DAST and where defense-in-depth solutions like RASP or more traditional WAF technology should be deployed.

CONCLUSION

Moving your organization fully to DecSecOps is no small undertaking. It will require a solid understanding of DevOps and leadership needs to embrace this type of development as a culture and shared responsibility for the organization. However, the benefits of having security being built into every aspect of the development lifecycle, while also deploying code as fast and efficient as possible, has tremendous benefits to an organization. It is commonly known that implementing DevOps in an organization can reduce application deployment by more than 50% and according to Veracode’s annual State of Software Security report, organizations that follow DevSecOps practices remediated flaws faster and had much less security debt to worry about. The key is to begin small, find quick wins where you can, and ensure the development, operations, and security teams are working together in unison to achieve your organization’s goals.

---