Priorities in Agile Lead to Software Observability in Annual Agile Report

It has been nearly 20 years since a group of 17 software developers conceived the “Manifesto for Agile Software Development.” Frustrated by a development methodology that followed a set path—with long development cycles before a final product was ready to be released—the group wanted an approach that would bring new software to the market faster. They believed this could be accomplished by speeding up development times through rapid feedback and having the flexibility to make changes throughout the development process.

Guided by 12 principles, the manifesto moved away from a more rigid linear Waterfall methodology to a methodology built around making the customer the highest priority—the first principle of the manifesto.

Defining the Agile Development Methodology

The Agile methodology of development is an iterative approach that focuses on fulfilling the needs of customers by delivering working software in small, rapid releases—or sprints—typically of a month or less. Small, cross-functional teams are utilized to build teamwork and understanding. Each team member has a wide variety of similar and equal skill sets and can perform every job, reducing delays that can happen while waiting for other team members to complete necessary tasks.

Focusing on collaboration between developers and customers, maintaining simplicity, and welcoming changes to meet the needs of the customer—even late in development—are all principles that promote the primary measure of Agile success, which is working software.

In addition to the manifesto’s 12 principles, Agile has four core values:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

By adhering to these core values, Agile teams are able to collaborate with customers throughout product development, allowing them to receive frequent feedback and addressing quality issues after each sprint. This greatly reduces project risk and the time and costs associated with remediating errors at the end of the software development life cycle (SDLC), as would happen under the Waterfall methodology. 

Agile Report Provides Insight Through Demographics

Digital.ai recently published its 14^th annual State of Agile Report, which provides insight into the application of Agile, why organizations use it, and what has changed since the last survey. Over 1,100 survey responses covering six continents were collected and analyzed from a wide range of industries throughout the global software development community. The responses were analyzed based on how long a respondent’s organization has been practicing Agile, the size of the respondent’s company, and the respondent’s self-reported role.

During the survey analysis, three notable survey responses were found to have significant changes of at least 10% over last year’s survey:

• 26% of respondents cited reduced project cost as an important reason for adopting Agile—down from 41% the previous year—but similar to the 24% response in the 12^th annual report.
• Identifying technical risk prior to deployment was reported as very valuable by 34% of respondents, up 12% over the previous year.
• Possibly as a result of mandated compliance requirements, automated audit compliance and governance across control points was reported as very valuable by 28% of respondents, up 18% over the previous year.

Filtering results along demographic lines reveals that a greater level of Agile maturity exists in organizations that have been practicing Agile longer, resulting in improved time to market and an increased ability to manage changing priorities. The vast majority of respondents (84%) acknowledged that their organizations were below a high level or better competency with Agile practices, indicating that there is much room for improvement that can be made through more training and coaching.

Organizations utilizing Agile for five or more years were more likely to have DevOps initiatives underway and were more interested in value stream management, which includes everything in the software development life cycle (SDLC)—from idea to production required to deliver software products or services to customers. Organizations with over 20,000 people were much more likely to have practiced Agile for five or more years, while organizations with less than 1,000 people reported a higher percentage of all of their teams being Agile.

Organizations with five or more years of Agile experience revealed that:

• A greater percentage of the organization practices Agile
• They are more likely to have DevOps initiatives underway
• They are more likely to use tooling
• They have more than 20,000 employees

While larger organizations may have more years of experience with Agile, organizations with less than 1,000 employees report a higher percentage of teams utilizing Agile. They are also more likely to utilize Agile in other areas of the organization outside of development, IT, and operations.    

Additional Growth in Agile Development Is Possible

Despite 95% of respondents reporting that their organizations practice Agile methods, 82% revealed they have teams that are not using Agile practices, indicating there is room for growth. An additional area of growth, revealed in a new survey question this year, suggests that Agile principles and practices are being employed across multiple departments—marketing, human resources, sales, and other. This allows entire organizations to experience the benefits of this methodology.  

Agile Adoption Prepared for COVID-19 Distributed Teams

One of the 12 principles of Agile specifies that development teams must have face-to-face conversations in order to efficiently and effectively communicate with each other. In the new “normal” of COVID-19, this has not been possible. Most organizations have needed to adjust to telework situations involving virtual meetings and other forms of distanced communications.

But according to the study by Digital.ai, many organizations using Agile were in a better position than other organizations that were not as far along in their adoption of Agile. Specifically, the survey reveals that 81% organizations have teams with members who do not all work in the same location and were already communicating across geographical boundaries and time zones. In fact, 21% of respondents indicate that one of their reasons for adopting Agile was to better manage distributed teams. Whether this becomes a growing rationale for Agile adoption remains to be seen.

Reasons for Agile Adoption

There are numerous reasons why organizations embrace Agile development. Accelerating software delivery (71%) and enhancing the ability to manage changing priorities (63%) remain the top two reasons. Increasing productivity (51%), improving business/IT alignment (47%), and enhancing software quality (42%) finish out the top five.

Two areas of objectives saw significant change over the previous year: Reducing project risk rose to 37% from 28%, and reducing project cost dropped to 26% from 41%. These reasons correlate with the benefits organizations receive from practicing an Agile methodology, including the ability to manage changing priorities (70%), project visibility (65%), business/IT alignment (65%), delivery speed/time to market (60%), and team morale (59%).

DevOps Addresses Gaps in Agile

Increasingly, DevOps is used in conjunction with Agile to help development organizations go the “extra mile.” Of those surveyed, more than half (55%) indicated that their organizations currently have DevOps initiatives underway, with an additional 21% planning on one within the next year. Of those with initiatives underway, 90% indicated that DevOps transformation is important in their organizations, including 43% citing that it is very important.

Agile and DevOps methodologies are two distinct methodologies, though their similarities can make them appear to be the same. Both methods strive to lower cost, reduce risk, and increase quality. They both also focus on development, but DevOps goes a step further into the operations and deployment of projects.

While some consider DevOps to be an extension of Agile, there are some key differences:

• Agile focuses on responding to changes during short sprint cycles, while DevOps focuses on continuous testing and delivery.
• Agile employs an iterative approach that obtains customer feedback at the end of each sprint, while DevOps maintains continuous collaboration with the customer.
• Agile focuses solely on software development, while DevOps extends to operations.

In order to improve DevOps practices, 39% of respondents indicate that having metrics to identify disruptions in the flow of business value would be most valuable. Observability from development to deployment and being able to identify and measure technical risk prior to deployment were equally valuable at 34% each. Unfortunately, 38% of respondents still rely on static analysis tools to address risks, which is the same as the previous year, even though only 14% of respondents planned to use static tools in 2019.

Software Observability Is Critical to Meeting Agile Objectives

The still high reliance of development teams on static application security toolsets is an obstacle that will continue to impede Agile objectives such as faster development releases. A turn to software observability that facilitates automation and improves accuracy is needed. Legacy application security generates high volumes of false positives that consume valuable time triaging and diagnosing. They also lack automation, with manual scanning, detection, and remediation workflows compounding the problems.

Faced with the prospect of missing release deadlines, development teams increasingly skirt or forego security policy controls and vulnerability alerts. This is driven from the C-suite: 68% of development teams have a mandate from their CEOs to ensure that nothing slows down development cycles.

But these pose serious risk, particularly with application threats making it more difficult to detect, remediate, and block attacks. Accordingly, Verizon, in its latest Data Breach Investigations Report, found that the percentage of data breaches tied to application vulnerabilities doubled over the past year—hitting 43%.

Software instrumentation has proven to be highly beneficial in application performance monitoring. It holds the same potential for application security—the ability to embed instrumentation within software delivers comprehensive observability across the application attack surface. Just as internal culture, leadership participation, and consistent processes and practices across teams are key to the adoption of Agile, they are just as important when it comes to the adoption of software observability. For more detail on how developers can use security observability to remove security roadblocks, read our eBook.