Platform engineers’ guide to HEART Framework developer experience measurement
Platform engineers call that journey the developer experience (DX), which includes how developers feel and interact with tools and services during software construction, test, deployment, and maintenance.
DX must be prioritized: Developer frustration causes inefficiency, talent loss, and IT shadowing. Instead, positive DX fosters innovation, community, and productivity. You must measure your performance to deliver a favorable DX.
Google HEART Framework
“Improving your developers’ platform experience by applying Google frameworks and methods” at PlatformCon 2024, discussing Google’s HEART Framework, which delivers a holistic perspective of your organization’s developers’ experience through actionable data.
In this essay, we will discuss how to apply the HEART framework to Platform Engineering to better understand your organization’s developer experience. Before that, let me define the HEART Framework.
Introduction to the HEART Framework
In summary, HEART measures developer behaviors and attitudes from their platform experience and provides insights into what’s going on behind the numbers by creating precise metrics to track objective success. Progress through feedback is essential to platform engineering, helping platform and application product teams make data-driven, user-centered decisions.
HEART is a user-sentiment framework for choosing metrics based on product or platform goals, not a data gathering instrument. It balances quantitative data like active portal users with qualitative insights like “My users feel the portal navigation is confusing.” Consider HEART a framework for measuring user experience, not a technology. It helps you choose what to measure, not how.
Let’s examine each individually.
Happiness: Does your product make users happy?
Highlight: Developer feedback collection and analysis
Subjective measures:
- Surveys: Regularly assess satisfaction, usability, and pain points. Toil lowers developer morale and contentment. Manual, repetitive work can cause frustration exhaustion and platform dissatisfaction.
- Input mechanisms: submit easy ways for developers to submit direct input on platform features or areas like Net Promoter Score (NPS) or Customer Satisfaction surveys.
- Interview developers and hold user groups for open-ended input.
- Analyze developer sentiment in feedback channels, support tickets, and online forums.
System stats:
Track developer feature requests by number and type. This helps you prioritize happiness-boosting improvements by revealing their requirements and wants.
Platforms may increase developer productivity but not job satisfaction. This needs further analysis, especially if your research reveals disgruntled developers.
Engagement: What is developer platform experience breadth and quality?
Highlight: Intensity and quality of platform engineer-developer interaction, chat channel participation, training, dual ownership of golden paths, joint troubleshooting, architectural design discussions, and the breadth of interaction by new hires to senior developers.
Subjective measures:
- Survey interaction quality focus on depth and type of interaction, such as chat channel, trainings, dual ownership of golden routes, cooperative troubleshooting, or architectural design debates.
- High toil can lower platform developer engagement. When engineers spend too much time on tiresome chores, they’re less likely to try new features and evolve the platform.
System stats:
- Active users: Track daily, weekly, and monthly developers’ task durations.
- Examine the most popular platform features, tools, and portal resources.
- Platform engineer-developer interaction frequency.
- Breadth of user engagement: Measure new hire proficiency onboarding time and senior developer contribution to golden paths or portal features.
Avoid: Confusing engagement with contentment. In polls, developers may grade the platform highly, yet usage data may show low frequency of core feature use or a small number of teams using it. “How has the platform changed your daily workflow?” instead of “Are you satisfied with the platform?”
Adoption: Platform growth and developer feature adoption?
Highlight: Platform adoption and development workflow integration.
System stats:
- New user registrations: Track platform developer growth.
- Track time between registration and platform use, including golden pathways, tooling, and portal functionality.
- Active portal users each week, month, quarter, half-year, or year who authenticate and use golden paths, tooling, and portal capabilities.
- Feature adoption: Monitor how quickly and widely new features or updates are used.
- Developer CI/CD platform usage percentage
- You decide how many deployments per user, team, day, week, or month.
- Analysis of adoption changes following training.
Beware: Adoption “long tail” neglect. If a platform doesn’t evolve to suit developer needs, early adoption may plateau or deteriorate. Track usage across weeks, months, and years, not just early adoption.
Retention: Do developers stick with the platform?
Highlight: Long-term engagement and churn reduction.
Subjective measures:
If a user is dormant for 12 months, use an exit survey.
System stats:
- Churn rate: Track developers who cease utilizing the site.
- Dormant users: Investigate why developers stop working after 6 months.
- Track rarely utilized services.
Worry about: Misinterpreting churn. Understand why developers churn from your platform. Identifying the problem incorrectly might waste time and prevent improvement. Think outside the platform project requirements, team structures, and industry trends can drive churn.
Task success: Can developers finish tasks?
Overview: Platform efficiency and effectiveness in supporting developer activities.
Subjective measures:
Survey to analyze the continued presence of toil and its negative impact on developer productivity, resulting in slower job completion.
System stats:
- Completion rates: Measure the percentage of platform-run golden routes and tools without errors.
- Task completion time using golden pathways, portal, or tools.
- Error rates: Monitor log files or golden pathways, portal, or tooling dashboards for developer errors and failures.
- Mean Time to Resolution (MTTR): How long does it take to fix errors? Low MTTR suggests a more durable platform and speedier recovery from failures.
- Developer platform and portal uptime: Measure the proportion of time they’re operational. Higher uptime guarantees developers can continuously use the platform and perform tasks.
Beware: Don’t mix work completion with success. Tests of developers’ platform proficiency don’t necessarily indicate success. Even if they succeed, developers may use workarounds or execute jobs inefficiently. Manually monitoring developer workflows in their natural surroundings may reveal pain points and friction points.
Avoid misaligning task success with corporate goals. Business goals may be overlooked during task execution. The utility of a platform is unclear if it only helps developers execute tasks that don’t contribute to business goals.
HEART framework application for platform engineering
Not all categories must be used. You can include all categories or narrow them down depending on the assessment’s purpose and context. Some examples:
- Improve new developer onboarding: Adoption, task success, and happiness.
- Focus on adoption and happiness while launching a feature.
- Track engagement, retention, and task success to increase platform usage.
Remember that one category may give an incomplete picture.
When should you use the framework?
Ideally, you would utilize the HEART framework to conduct a baseline assessment a few months after launching your platform to gain insight into early developer experience. As your platform grows, this early data serves a benchmark for growth and trends. Early assessment lets you anticipate UX difficulties, inform design decisions, and iterate swiftly for optimal functionality and developer satisfaction. If you’re starting with an MVP, do the baseline assessment when the core functionality is in place and you have a few early users to provide input.
You can add metrics after 12 months to show a fresh or mature platform. This can help you understand how your developers use the platform, quantify the impact of platform changes, suggest areas for improvement, and prioritize future development efforts. Metrics are necessary to evaluate the effectiveness of new golden routes, tooling, and functionality as well as their influence on developer behavior.
HEART metrics assessment frequency relies on numerous parameters, including:
- The maturity of your platform: Monthly or quarterly assessments help newer platforms track development and fix early difficulties. As the platform evolves, you can reduce HEART evaluations to bi-annual or annual.
- When your platform is evolving rapidly, such as large platform updates, new portal features or golden paths, or user behavior changes, apply the HEART framework more often to ensure beneficial impacts. This lets you closely monitor critical KPIs after each update.
- Size and complexity of your platform: Complex platforms may need more frequent assessments to identify nuances and concerns.
- Capability of your team: Time and resources are needed for HEART assessments. Consider your team’s bandwidth when setting frequency.
Use the HEART framework to conduct quarterly or bi-annual deep dives to better understand your platform’s performance and find areas for improvement.
Moving toward platform engineering
This blog article shows how platform engineering can measure and improve developer experience using the HEART framework. It examined the framework’s five core areas happiness, engagement, adoption, retention, and task success and provided metrics and guidelines on when to use them. These insights can help platform engineering teams improve developer morale and productivity, boosting software development success.
