So Many Repos, So Little Time: Speeding Up Deployments Securely

As we entered 2020, digital transformation was already gathering steam. Once Covid-19 spread, workers went remote and e-commerce and business-to-business transactions went predominantly online. This propelled digital transformation into a raging inferno that led to unprecedented pressure on developer pipelines from incessant business demands for new code functionality and faster updates. Around the same time vulnerabilities related to software supply chains came to light. Significant breaches that included secrets in code, personally identifiable information (PII), and malicious or accidental code configuration errors gained national-level attention 

Development teams accelerated their adoption of DevOps and continuous integration /continuous deployment (CI/CD) processes to help improve and speed up software development and deployment. DevOps and CI/CD pipelines rely on processes that start with pull requests, then merge code, and then complete integration and unit testing. This is followed by acceptance testing to deploy code to a staging server. The final step is deploying to production servers, commonly referred to as prod. GitHub is one of the most prevalent tools in this area. 

GitHub is an online open-source version control system that is based on Git. GitHub is the virtual watering hole for developers in a communal setting where developers can host and share projects, have philosophical discussions with other developers, see what other developers are building and create a personal profile to showcase their own body of work. GitHub hosts a number of repositories that can be made private or public. 

There has been a dramatic growth in the number of repositories in the last few years. As of 2021, there are well over 200 million repositories on GitHub making it the largest source code host. GitHub was acquired by Microsoft. However, it remains true to its open-source roots. Other similar tools include GitLab (which recently went public via an initial public offering of their stock) and BitBucket, a solution now part of commercial software vendor Atlassian.

The goal of DevOps is to help an organization deliver software much faster. It is widely accepted that iterating with smaller increments helps improve quality and speed. CI/CD pipelines today are automated and help accelerate the software development process. Automated Release Orchestration platforms have provided a much easier route to creating development pipelines leveraging best-of-breed tools for each stage of the pipeline. This means that developers can use the tools they love most instead of having to adopt one vendor’s set of tools across the entire process. These tools can include automated lifecycle management tools, source control, version control, code analysis, static and dynamic testing, and many others. 

Based on a survey of 4,300 developers, GitLab released their report A Maturing DevSecOps Landscape, 2021 Survey. Based on the survey:

  • 60% of developers are releasing code 2x faster, thanks to DevOps. 
  • Leading Organizations are investing to improve the process leading to acquisition of tools such as SCM, CI/CD, a DevOps platform, and automated testing.
  • Roles are shifting – developers continue to take on jobs that used to be the sole responsibility of IT Ops.
  • Security is shifting left into the purview of developers

The rush to increase the velocity of software delivery continues to stretch application security resources. The security industry has developed application security tools that solve point problems. The growth of software development lifecycle (SDLC) security tools was first applied to on-premise applications. This has extended to cloud-based applications with both static and dynamic testing solutions (SAST and DAST) and more recently has extended to include infrastructure as code (IaC). However, these are being applied as a bolt-on to the current development processes creating friction and delays in deploying software.

Developers in many organizations are observing that application security is slowing them down. The overhead of coordinating between dev, IT ops, and security functions largely falls on developers who are viewing security as a necessary evil. The good news is that this trend is changing and enterprises are improving coordination between their dev and security teams. 

Based on a DevOps organizational survey conducted by 451 Research, at the manager level, 49% of the organizations polled have close collaboration at the management level between information security and DevOps teams. However, one can also infer that 51% of organizations do not have this level of collaboration. At the staff level, the picture is troublesome with 61% of organizations lacking a formal program for cooperation between information security and DevOps.

Survey results showing the level of collaboration between Information Security and DevOps functions in organizations

What about Application Security? So Many Repos and So Much Code to Deploy

A common problem for DevOps teams is that adding security into the mix introduces too much friction into the development process. Developers who usually don’t have admin-level access for most infrastructure find ways to bypass security, not because they are malicious, but because they want to get their code working perfectly. They have every intention of coming back to fix issues related to security once it’s all working perfectly. The reality is that rarely happens and application security ends up taking a back seat to speedy deployment.

Security and DevOps experts both agree that keeping application security from slowing down CI/CD pipelines requires security to be integrated into all steps of the DevOps process. This has given rise to the discipline of DevSecOps, where security gets built-in and ideally provides a bridge between the other two functions.

Additional studies on the cost of mitigating security threats have shown that security threats that are addressed later in the development stage are more expensive to mitigate, whereas application vulnerabilities uncovered early in the process are easier to address and lost costly to remediate.

The Shift Left movement enables organizations to focus on application security earlier in the development cycle to lower the cost and complexity of remediating code vulnerabilities

What do Security Pros Think?

The previously referenced GitLab survey on DevSecOps maturity mentions that “A full 72% of security pros rated their organizations’ security efforts as either strong or good. Wishful thinking it might be, but it’s a significant increase in optimism from a group not necessarily known to be upbeat.” The report goes on to add that security and development functions are friendlier, however, it states that, “there is still confusion over who owns the security and the finger-pointing game is strong.” 

 Enabling a Risk-Based Approach to Code Security.

A best practice approach to enabling an application security strategy that is well integrated with the DevOps program includes the following:

  • Recognize and acknowledge that the way code is written has changed vastly to include open source code and externally developed code that resides in many, many repositories, distributed all over both inside and outside the enterprise.
  • It is increasingly more difficult to monitor and track who has access to your code, where it resides and whether it has been cloned somewhere.
  • As underlying infrastructure gets virtualized, it’s configuration is represented in code as well, making infrastructure vulnerable to code related attacks
  • Generally accepted application security solutions like SAST, DAST and even open source tools provide point protection, however they do not close all the security gaps. Additionally, open source tools can generate a very large number of alerts that include false positives as well. This can make it virtually impossible to remediate.
  • Explore risk-based application security solutions that can scan large volumes of code in repositories that will include not just the latest version, but all the prior versions of code. Many of the security fixes made to code in mid-development or later stages never make it back to the original repositories 

BluBracket Delivers a Seven-Step Approach to Risk-Based Code Security

The BluBracket solution is most effective in uncovering vulnerabilities, determining the associated risks, and, best of all, delivering a path to mitigating the threats. BluBracket can deliver a risk score that is a quantifiable measure of risk over time. Unlike many point solutions, BluBracket can virtually eliminate the very high false positives that practitioners face from the point solutions and open source tools

Finally, as a solution that is built with developers in mind, BluBracket is easy to integrate into the daily development workflows. It has the flexibility to operate across all git repositories, both internal and external. Integration with existing DevOps and CI/CD tools commonly found in the enterprise allow developers to easily include BluBracket into their daily routines. 

BluBracket’s coverage of risk includes:

  • Secrets in Code: secrets in code exist as artifacts that an app uses to connect to an external service, account, or application. Secrets used by developers include API keys; encryption keys; OAuth tokens; certificates; and passwords. BluBracket for secrets in code, alerts on them and then provide a means to eliminate the risk. 
  • Personally Identifiable Information (PII): organizations have various policies governing the use, storage, and disposition of PII. Some examples are social security numbers, credit card numbers, date of birth etc. In certain industries there are strong regulatory mandates with huge penalties for failure to protect this data. BluBracket can block such data from being exposed. 
  • Infrastructure as Code (IaC): helps with the managing and provisioning of compute infrastructure and cloud services using (software) configuration files. Misconfigurations in IaC scripts can lead to serious disruptions. BluBracket can identify misconfigurations in IaC prior to deployment.
  • Access and Identity: as code is cloned and proliferated via Git repositories, it becomes difficult to identify the owners of the code, the collaborators, and the interlopers. BluBracket allows developers and AppSec teams to discover who has access at all times.
  • Code Leaks: Developers sometimes unknowingly place company IP at risk when they share code sections or commits on to public Git repositories. BluBracket regularly scans public repositories for code fingerprints that may have leaked into the extended universe. 
  • Compliance with Git Configuration Rules: Git misconfigurations can result from insecure default configs, incomplete or un-patched applications etc. BluBracket looks across hundreds of software components, libraries, and application frameworks for vulnerable misconfigurations.
  • Identifying Non-Inclusive Language: In keeping with current times and developer’s desires to further the tech industry to remove insensitive racial or gender bias in words that we use in our code, BluBracket scans to identify such terms and delivers options to remediate.where they originated from, allowing attackers the opportunity to exploit those vulnerabilities.

BluBracket provides the most effective code security solution to secure developer environments. Developers can identify and mitigate key risks across the entire CI/CD pipeline while maintaining speed and agility. For more information on BluBracket or to provide comments and feedback, please contact us.

Share this post!