Introducing Criterion Score Traceability for scorecards that use derived metrics

[#1]

Compass scorecards play a vital role in tracking metrics, and among these, some are known as derived metrics. These metrics are calculated based on events like builds and deployments, creating a meaningful connection between the events and the scores we evaluate.

When a new event occurs, Compass recalculates the metric value, which in turn prompts a reassessment of any scorecards that rely on that metric.

For instance, consider a scorecard that measures build time. This metric is derived from build events, so whenever a new build takes place, the metric value is updated, and the scorecard reflects this change by reassessing its score.

Related to this, we're excited to introduce a new feature: criterion score traceability. This enhancement allows you to dive deeper into the events that shape the metrics used in your scorecards.

This feature lets you to trace back through the chain of events. If your scorecard starts failing, you can now easily identify the specific event - and its corresponding value - that led to this change. This capability can significantly reduce the time spent troubleshooting and enhance your understanding of the underlying data.

With this feature, you gain a clearer understanding of the data driving your metric calculations, offering a more detailed view into how events influence metric values and thus scorecard scores.

How it works

This feature is can be used with criteria that use built-in metric definitions supported by events. When you check out a scorecard using these derived metrics, you'll notice a Show events button. Clicking this button opens a modal displaying all the events contributing to the current derived metric.

In this modal, you'll see individual values for each event. Each event is assessed against the criterion to determine if it would pass on its own or if it's considered unusual. This makes it easy to spot events that might need a closer look. By clicking on an event, you'll be taken directly to the event source, such as a build pipeline, for a more in-depth analysis.

Limitations

Currently, the following metrics are supported:

• Build time

• Deployment time

• [Incident] MTTR

• Critical severity vulnerability MTTR

• High severity vulnerability MTTR

Metrics that are purely count-based, like Deployment frequency, aren't supported by this traceability feature yet. Also, keep in mind that traceability data is only available from this point forward; there's no historical data available before this feature's release.

---

We hope this feature can offer detailed insights into the events behind derived metrics so that you can understand scorecard scores more easily. We encourage you to explore this new functionality and share your experiences with us!