Automated Retrieval and Visualization of DORA Metrics

Automated Retrieval and Visualization of DORA Metrics

A Practical Implementation with OutSystems, Azure DevOps, and Elastic

DevOps principles are crucial to ensuring the fast, reliable, and consistent delivery of software. One important way to measure the success of a DevOps strategy is by using DORA metrics. These metrics, developed by Google’s DevOps Research and Assessment (DORA) team, predict with high accuracy the value a team delivers to the organization. We have previously written and made a video about DORA and its metrics.

In this blog, we will discuss what DORA metrics are, the implementation of an automated system to retrieve and visualize these metrics, and how we achieved this with a combination of OutSystems, Azure DevOps, and Elastic.

What are DORA Metrics?

Years of research into what makes high-performing teams successful have resulted in four metrics that most accurately predict the value a team delivers to the organization. These are expressed as low, medium, high, or elite performers.
Let’s briefly go over each metric:

Deployment frequency: How often does an IT team deploy software to production? Is it once a month? Or multiple times a day? To be considered a high performer in this area, a high degree of automation in your development process is an absolute requirement.

Change lead time: Indicates how quickly an organization can respond to changes such as market demand, regulations, or security risks. This measures the time from the start of developing a piece of software to its deployment.

Change failure rate: Indicates the percentage of changes to the software that result in a degradation of the product, requiring fixes, rollbacks, and patches. For a low performer, this can be 46 to 60%, while for high performers, it is only 0 to 15%.

Time to restore service: Tells how quickly a team can restore a service that impacts the user experience. A low performer may take a week to a month, while a high performer can resolve it within a day. It is important to detect issues as quickly as possible, making a good monitoring solution indispensable.

Why Automate?

We see that DORA metrics are increasingly used by customers, but often still determined manually or “by feel” with the team. Automating this has many advantages:

• Increase accuracy: Automation minimizes human errors, making the data more reliable and the measurements more precise.
• Reduce manual work: By automating processes, teams do not have to spend time manually collecting and processing data.
• One standard system for multiple teams: A generic solution ensures that all teams work with the same measurement methods, making it more consistent and easier to extend to other teams.
• Visualization: The data is automatically sent to Elastic, where clear and insightful dashboards can be created.

Automating Data Retrieval

Let’s now discuss how we achieved this.

Deployment Frequency

As previously discussed, Deployment Frequency measures how often a team deploys code to production. Within OutSystems, this data is stored in the LifeTime database – LifeTime coordinates the transfer of code to an environment. By writing a direct SQL query within Logstash that accesses the LifeTime database, we can automatically retrieve data on how many times per day/week/month a particular application has been deployed to production and which team it is associated with. This data is transformed and then sent to Elastic. This approach makes it easy to monitor deployment frequency for multiple applications and teams simultaneously, without teams having to manually track data.

Change lead time

For Lead Time for Changes – how long it takes to go from committed code to code running successfully in production – we looked at Azure DevOps boards. Teams log how long it takes to work on a feature. We agreed with teams that when they start working on a feature, it is moved to active, and when it is released to the end-user, it is moved to done. With a custom PowerShell script and APIs, we retrieve this data and send it to Elastic.

Time to restore service and Change failure rate

For Time to restore service and Change failure rate, we took a similar approach. In the event of a service disruption, teams log this as a bug in Azure DevOps. This bug is moved to done when the disruption is resolved. The data on the bugs is retrieved via the PowerShell script and APIs, the time between the creation and resolution of the bug is calculated, and then it is sent to Elastic.
Change failure rate is also based on these bugs, but specifically when a disruption occurs after a new deployment. In that case, information about the deployment is included in the bug. This data is also sent to Elastic. Within Elastic, we can then compare the number of deployments to the number of deployments related to a bug.

Automatically visualise metrics in Elastic

As mentioned earlier, we bring the data from the different sources together in Elastic – our chosen datastore for data analysis and visualization. Elastic offers many possibilities for data input – in this case from Logstash and directly via APIs. Additionally, Elastic provides good options for visualizing DORA metrics with dashboards based on the different data sources discussed. This allows teams to see both their current performance and their progress over time.

Advantages of this Approach

By combining direct data retrieval from OutSystems LifeTime for Deployment Frequency and using a PowerShell script for the other metrics from Azure DevOps, we have set up a robust and flexible system. The main advantage of this approach is that it is generic and works for multiple teams, without each team having to manually create reports. Additionally, it is easy to extend to other OutSystems teams that also use Azure DevOps. However, it is important that teams maintain their workflow within Azure DevOps properly.

Conclusion

Automated retrieval of DORA metrics provides DevOps teams with valuable insights into their performance and helps them continuously improve. By using a combination of OutSystems LifeTime, PowerShell, Azure DevOps, and Elastic, we have created a scalable and flexible solution that works for every team. This process ensures that teams can focus on optimizing their processes and delivering value, without having to worry about manually tracking their progress.

By Guido Vandecasteele, Data Analist and Consultant