Client could not measure impact of DevOps/QE (Quality Engineering) transformation due to lack of KPIs and metrics.
Manual data collection from multiple sources was time-consuming and tedious.
Qualitest collaborated with the Client to identify, baseline and measure key data and KPIs.
We deployed our unified data analytics platform, Qualiview, to create dashboards with real-time data tailored for specific stakeholder groups.
Clearly visualized, targeted metrics and KPIs enable ROI measurement and rapid reaction to change across the organization.
Streamlined data collection reduced manual collection time by 85% and decreased data and reporting errors by 95%.
The Client is a large US PPO (Preferred Provider Organization), supplying cost-saving solutions to healthcare payors and increased access to provider networks for their clients. The company dominates in the commercial, property & casualty and government-sponsored healthcare markets.
A quality partner of longstanding with Qualitest, the Client had recently engaged us to help spearhead their transformation journey. Their objectives were to enhance organizational agility, expedite time to market, elevate product quality and establish robust continuous delivery practices.
Initially our focus was on empowering teams through various DevOps and QE features and shift-left principles. We created robust CI/CD pipelines with Quality Gates and implemented in-sprint automation practices and continuous testing, among other actions. We designed this proactive approach to ensure the Client not only met but exceeded industry standards in a rapidly evolving landscape.
As the program continued, it became clear that something was missing: concrete factors to measure progress through real, data-driven insights. There were three major challenges:
In collaboration with the Client, the Qualitest team crafted a comprehensive solution with two distinct phases.
Phase One: Identifying, baselining and measuring KPI metrics
We enlisted DevOps and QE evangelists to help identify KPIs and engaged in discussions to align these metrics with the Client’s transformation goals. This ensured a comprehensive understanding of the key success factors.
Next, we conducted a thorough assessment of existing processes to establish baseline metrics, using actuals from before the transformation, and implemented automated processes to collect data after baseline assessment. We measured baseline vs. actual KPIs to determine teams’ progress for the three most recent sprints.
We also held workshops with the Client to define new or modify existing governance processes. Then we defined and communicated new or changed processes to the teams, emphasizing the importance of accurate data capture and conducting demo sessions to ensure a smooth transition.
Phase Two: Creating customized dashboards with Qualiview
The teams decided that differently focused dashboards displaying data most relevant and useful for certain stakeholders would supply everyone with the most meaningful metrics and KPIs and improve company-wide adoption. Qualiview, Qualitest’s customizable data aggregation and visualization platform, would create the dashboards.
Qualiview streamlines data extraction across major platforms and tool sets to provide holistic consolidated and standardized reporting, using web-based dashboards. Flexible and effective, Qualiview enables unlimited users and can create ready-to-use dashboards in a wide variety of areas.
To enable the most meaningful measurement of the transformation’s impact for key stakeholders, we created two dashboards:
DevOps and QE Dashboard: This dashboard displayed real-time data most relevant for team members and technical product delivery managers (TPDM) within the team. A few examples:
DORA (DevOps Research and Assessment) Dashboard: Tailored for the leadership team, this dashboard offered a holistic perspective on the transformation’s impact in alignment with DORA’s four standard metrics: deployment frequency, lead time for change, mean time to recover (MTTR) and change failure rate.
After the dashboards were in place, we implemented weekly/biweekly standups so teams could promptly identify areas requiring additional training or support, contributing to ongoing enhancement.
This project continued our successful collaboration and built on the positive outcomes achieved in the earlier stages of the Client’s delivery transformation, including an acceleration of DevOps adoption across the organization, the installation of Continuous Integration/Continuous Delivery (CI/CD) pipelines and implementation of NextGen automation technologies, among many others. Some noteworthy results from this initiative:
Download the PDF
