Concepts on Grafana Labs

What is profiling?

Fri, 03 Apr 2026 19:43:06 +0000

What is profiling?

Profiling is a technique used in software development to measure and analyze the runtime behavior of a program. By profiling a program, developers can identify which parts of the program consume the most resources, such as CPU time, memory, or I/O operations. You can use this information to optimize the program, making it run faster or use fewer resources.

Pyroscope can be used for both traditional and continuous profiling.

Traditional profiling (non-continuous)

Traditional profiling, often referred to as sample-based or instrumentation-based profiling, has its roots in the early days of computing. Back then, the primary challenge was understanding how a program utilized the limited computational resources available.

With sample-based profiling, the profiler interrupts the program at regular intervals, capturing the program’s state each time. By analyzing these snapshots, developers can deduce the frequency at which parts of the code execute.

With instrumentation-based profiling, developers insert additional code into the program that records information about its execution. This approach provides detailed insights but can alter the program’s behavior due to the added code overhead.

Benefits

Traditional profiling provides:

Precision: Offers a deep dive into specific sections of the code.
Control: Developers can initiate profiling sessions at their discretion, allowing for targeted optimization efforts.
Detailed reports: Provides granular data about program execution, making it easier to pinpoint bottlenecks.

Continuous profiling

As software systems grew in complexity and scale, the limitations of traditional profiling became evident. Issues could arise in production that weren’t apparent during limited profiling sessions in the development or staging environments.

This led to the development of continuous profiling, a method where the profiling data is continuously collected in the background with minimal overhead. By doing so, developers gain a more comprehensive view of a program’s behavior over time, helping to identify sporadic or long-term performance issues.

Benefits

Continuous profiling provides:

Consistent monitoring: Unlike traditional methods that offer snapshots, continuous profiling maintains an uninterrupted view, exposing both immediate and long-term performance issues.
Proactive bottleneck detection: By consistently capturing data, performance bottlenecks are identified and addressed before they escalate, reducing system downtime and ensuring smoother operations.
Broad performance landscape: Provides insights across various platforms, from varied technology stacks to different operating systems, ensuring comprehensive coverage.
Bridging the Dev-Prod gap: Continuous profiling excels in highlighting differences between development and production:
- Hardware discrepancies: Unearths issues stemming from differences in machine specifications.
- Software inconsistencies: Sheds light on variations in software components that might affect performance.
- Real-world workload challenges: Highlights potential pitfalls when real user interactions and loads don’t align with development simulations.
Economical advantages:
- Resource optimization: Continual monitoring ensures resources aren’t wasted, leading to cost savings.
- Rapid problem resolution: Faster troubleshooting means reduced time and monetary investment in issue rectification, letting developers channel their efforts into productive endeavors.
Non-intrusive operation: Specifically designed to work quietly in the background, continuous profiling doesn’t compromise the performance of live environments.
Real-time response: It equips teams with the ability to act instantly, addressing issues as they arise rather than post-occurrence, which is crucial for maintaining high system availability.

Choose between traditional and continuous profiling

In many modern development workflows, both methods are useful.

	Traditional profiling	Continuous profiling
When to use	During development or testing phases	In production environments or during extended performance tests.
Advantages	Offers detailed insights that can target specific parts of code.	Provides a continuous view of system performance, often with minimal overhead, making it suitable for live environments.
Disadvantages	Higher overhead provides only a snapshot in time.	It might be less detailed than traditional profiling due to the need to minimize impact on the running system.

What is continuous profiling?

Fri, 03 Apr 2026 19:43:06 +0000

What is continuous profiling?

Continuous profiling is a systematic method of collecting and analyzing performance data from production systems.

Traditionally, profiling is used to debug applications on an as-needed basis. For example, you can run a benchmark tool locally and get a pprof file in Go or connect to a misbehaving prod instance and pull a flame graph from a JFR file in Java. This method is good for debugging, but not robust enough for production.

Refer to Flame graphs to learn more.

Continuous profiling is a modern approach which is safer and more scalable for production environments. It uses low-overhead sampling to collect profiles from production systems and stores the profiles in a database for later analysis. Using continuous profiling gives you a more holistic view of your application and how it behaves in production.

Grafana offers Grafana Pyroscope and Grafana Cloud Profiles (powered by Pyroscope) to collect and store your profiling data. You can use Grafana Profiles Drilldown to inspect profile data and investigate issues.

Benefits

Why prioritize continuous profiling?

In-depth code insights: It provides granular, line-level insights into how application code utilizes resources, offering the most detailed view of application performance.
Complements other observability tools: Continuous profiling fills critical gaps left by metrics, logs, and tracing, creating a more comprehensive observability strategy.
Proactive performance optimization: Regular profiling enables teams to proactively identify and resolve performance bottlenecks, leading to more efficient and reliable applications.

Use cases

Adopting continuous profiling with tools like Grafana Pyroscope and Profiles Drilldown can lead to significant business advantages:

Reduced operational costs: Optimization of resource usage can significantly cut down cloud and infrastructure expenses
Reduced latency: Identifying and addressing performance bottlenecks leads to faster and more efficient applications
Enhanced incident management: Faster problem identification and resolution, reducing Mean Time to Resolution (MTTR) and improving end-user experience

Reduced operational costs

By providing in-depth insights into application performance, profiling empowers teams to identify and eliminate inefficiencies, leading to significant savings in areas like observability, incident management, messaging/queuing, deployment tools, and infrastructure.

By using sampling profilers, Pyroscope and Cloud Profiles can collect data with minimal overhead (~2-5% depending on a few factors). The custom storage engine compresses and stores the data efficiently. Some advantages of this are:

Low CPU overhead thanks to sampling profiler technology

Control over profiling data granularity (10s to multiple years)
Efficient compression, low disk space requirements and cost

Reduced latency

Profiles play a pivotal role in reducing application latency by identifying performance bottlenecks at the code level. This granular insight allows for targeted optimization, leading to faster application response times, improved user experience, and consequently, better business outcomes like increased customer satisfaction and revenue.

Enhanced incident management

Pyroscope and Profiles Drilldown streamline incident management by offering immediate, actionable insights into application performance issues. With continuous profiling, teams can quickly pinpoint the root cause of an incident, reducing the mean time to resolution (MTTR) and enhancing overall system reliability and user satisfaction.

Understand profiling types and their uses in Pyroscope

Fri, 03 Apr 2026 19:43:06 +0000

Understand profiling types and their uses in Pyroscope

Continuous profiling is a systematic method of collecting and analyzing performance data from production systems.

Refer to Flame graphs to learn more.

Benefits

Why prioritize continuous profiling?

In-depth code insights: It provides granular, line-level insights into how application code utilizes resources, offering the most detailed view of application performance.
Complements other observability tools: Continuous profiling fills critical gaps left by metrics, logs, and tracing, creating a more comprehensive observability strategy.
Proactive performance optimization: Regular profiling enables teams to proactively identify and resolve performance bottlenecks, leading to more efficient and reliable applications.

Use cases

Adopting continuous profiling with tools like Grafana Pyroscope and Profiles Drilldown can lead to significant business advantages:

Reduced operational costs: Optimization of resource usage can significantly cut down cloud and infrastructure expenses
Reduced latency: Identifying and addressing performance bottlenecks leads to faster and more efficient applications
Enhanced incident management: Faster problem identification and resolution, reducing Mean Time to Resolution (MTTR) and improving end-user experience

Reduced operational costs

Low CPU overhead thanks to sampling profiler technology

Control over profiling data granularity (10s to multiple years)
Efficient compression, low disk space requirements and cost

Reduced latency

Enhanced incident management

Profiling is an essential tool for understanding and optimizing application performance. In Pyroscope, various profiling types allow for an in-depth analysis of different aspects of your application. This guide explores these types and explain their impact on your program.

Profiling types

In Pyroscope, profiling types refer to different dimensions of application performance analysis, focusing on specific aspects like CPU usage, memory allocation, or thread synchronization.

Pyroscope supports these profile types:

CPU (CPU time, wall time)
Memory (allocation objects, allocation space, heap)
In use objects and in-use space
Goroutines
Mutex count and duration
Block count and duration
Lock count and duration
Exceptions

Refer to the Profile types tables for information on supported profile types based on instrumentation method.

For information on auto-instrumentation and supported language SDKs, refer to Configure the client.

Flame graphs

Fri, 03 Apr 2026 19:43:06 +0000

Flame graphs

Flame graphs provide a visual summary of your profile data. A flame graph is a complete visualization of hierarchical data, for example stack trace and, file system contents, with a metric, typically resource usage, attached to the data.

A fundamental aspect of continuous profiling is the flame graph, a convenient way to visualize performance data. These graphs provide a clear, intuitive understanding of resource allocation and bottlenecks within the application.

Brendan Gregg, the creator of flame graphs, was inspired by the inability to view, read, and understand stack traces using the regular profilers to debug performance issues.

How Pyroscope creates flame graphs

This diagram shows how code is turned into a flame graph. In this case, Pyroscope samples the stacktrace of your application to understand how many CPU cycles are spent in each function. It then aggregates this data and turns it into a flame graph.

What does a flame graph represent?

Horizontally, the flame graph represents 100% of the time that this application was running. The width of each node represents the amount of time spent in that function. The wider the node, the more time spent in that function. The narrower the node, the less time spent in that function.

Vertically, the nodes in the flame graph represent the hierarchy of functions called and time spent in each function. The top node is the root node and represents the total amount of time spent in the application. The nodes below it represent the functions called and time spent in each function. The nodes below those represent the functions called from those functions and time spent in each function. This continues until you reach the bottom of the flame graph.

This is a CPU profile, but profiles can represent many other types of resource such as memory, network, disk, etc.

Flame graph visualization panel UI

To learn more about the flame graph user interface in Grafana, Grafana Cloud, and Grafana Profiles Drilldown, refer to Flame graph visualization panel.

To learn more about the flame graph in the Pyroscope UI, refer to Pyroscope UI.