5. Product Data

Now let’s discuss the second pillar: Product Data. While the first pillar focused on qualitative data and the Voice of the Customer, this pillar is centered around quantitative product data—how we measure performance, usage, and outcomes in a structured, data-driven way.

Understanding Product Data

Product data can be categorized into several key areas:

  1. Customer Journey Data: Quantitative insights that track how customers engage with the product throughout their journey.
  2. Product Performance Data: Metrics that reflect how well the product performs. The definition of performance will vary by company and product. For instance, performance could include server uptime, response times, or even product usage.
  3. Product Usage Data: Metrics like active users, frequency of use, feature engagement, and depth of interaction with the product.
  4. Financial Data: Financial metrics, such as profits, revenue, and P&L for specific product lines. While financial data is often one of the more complex areas to unify, it’s crucial for product teams to understand the business impact of their product offerings.

In many companies, Product Data is one of the most mature pillars, especially in organizations that have embraced a data-driven culture. These companies often have teams dedicated to building robust databases that track customer and product information. However, in other cases, data may be siloed, and different departments may maintain their own versions of the data. Product Operations plays a crucial role in unifying this data and generating insights for the product team.

For example, at VTEX, we initially lacked well-organized, segmented product data. The Product Ops team played an essential role in pushing for this unification, particularly in areas like product usage data, where we had numerous clients using different features, but no clear way to track or analyze their behavior effectively.

Examples of Product Data Initiatives

Let’s look at two key initiatives related to Product Data: the Amplitude implementation and a Product Team Efficiency Scorecard.

1. Implementing Amplitude for Product Usage Analysis

Amplitude is a powerful tool for tracking user behavior and product usage. It allows you to analyze which features users are engaging with, how frequently they access them, and the paths they take through your product. The implementation of Amplitude at VTEX provided us with critical insights into product usage and allowed us to track specific behaviors at scale.

Solution Implementation

The first step was to choose the right tool. At VTEX, we evaluated several options, including Amplitude, Mixpanel, and Pendo. We ultimately decided to go with Amplitude, along with plans to incorporate Segment later for more advanced data scaling.

The second step was to start small. We implemented the tool with two carefully selected squads. These squads worked closely with the Product Ops team to define the tags and data points we would track. During this phase, we also established a data dictionary, which is crucial for maintaining consistency across teams. For example, terms like active user or event types needed to be clearly defined to ensure that everyone understood the data in the same way.

We used Notion as our knowledge management tool to document the entire process, helping us train other teams on how to use the dashboards and interpret the data.

A key part of this initiative was educating the product team. Even with a powerful tool like Amplitude, it’s vital that teams know how to use the data effectively. We provided training on dashboard setup and worked with product managers (PMs) to help them understand how data could inform their decisions.

2. Product Team Efficiency Scorecard

The second example is a scorecard that we developed to measure the efficiency of product teams. This initiative was inspired by Agile Coaches but with a slightly different focus. Many PMs and engineering managers were already aware of metrics like velocity and lead time, but there was a lack of understanding about how to interpret these metrics and how they could be used to improve team performance.

Solution Implementation

We developed a Product Team Efficiency Scorecard, which was based on a balance of six key pillars, each representing a different aspect of team performance:

  1. Sustainability of Product Development: Measures the long-term sustainability of development efforts.
  2. Business Impact: Tracks how product launches contribute to business outcomes.
  3. Launch Volume: Quantifies the number of features or improvements released.
  4. Launch Speed: Measures how quickly teams can bring features to market.
  5. Launch Quality: Tracks the stability and reliability of new releases.
  6. Team Efficiency: Combines metrics like deploys, pull requests, and lead time to assess team performance.

This scorecard is not intended to compare teams or squads against one another. Instead, it is designed to help each squad reflect on its own performance and identify areas for improvement. By providing these insights, the Product Ops team helped each squad improve predictability, understand backlog management better, and ultimately, deliver more effectively.

Key Learnings for Dashboard Success

When working with dashboards—a key tool for product data—you need to treat them as a product themselves. This means thinking carefully about the audience, context, and goals for each dashboard:

Drawing from my background in data analytics, I often use the Fogg Behavior Model when working with teams to build useful dashboards. This model, developed by BJ Fogg at Stanford University, explains that behavior is driven by three elements:

  1. Motivation: Is the user motivated to take action?
  2. Ability: Is the task easy enough for the user to complete?
  3. Triggers: Are there clear signals that prompt the user to take action?

For example, if a team has high motivation (e.g., they are eager to improve a feature) but the dashboard is complex, they may struggle to use it effectively. Likewise, if they know how to use the dashboard but don’t have regular reminders or triggers, they might forget to engage with it.

By carefully considering motivation, ability, and triggers, we can ensure that dashboards are actionable and impactful, rather than just being a static tool that no one uses.

Conclusion

The Product Data pillar is about much more than just collecting data—it’s about making that data usable and actionable for product teams. By implementing tools like Amplitude, developing efficiency scorecards, and thinking strategically about how dashboards are used, Product Operations can help product teams make data-driven decisions that improve product performance and customer satisfaction.