You can view an aggregate representation of your trace data, filter your trace data, and explore individual traces by using the Trace Explorer page. By default, the Trace Explorer queries the data stored in your Google Cloud project. However, you can configure the page to query trace data stored in multiple projects.
The following screenshot illustrates how trace data is shown:
To perform a custom aggregate analysis of your spans using SQL, use the Observability Analytics page or BigQuery. When you use the Observability Analytics page, you can view query results as a table or as a chart. For information about using SQL to query your trace data, see Query and analyze traces.
About the Trace Explorer page
To help you identify trends and patterns in your trace data, the latency data is aggregated and displayed in charts. The heatmap uses color to represent the number of spans in a cell. A cell with many spans has a darker color than one with few spans. Each cell's tooltip displays latency and other information. You can also view latency percentiles and span rate information.
When you are investigating an issue, you might want to view a specific trace or only spans with certain properties:
To restrict which spans are shown, apply filters. For example, you can filter by service, application, error state, and other properties.
To display a specific trace, use the toolbar's pageview Search for trace option.
To identify outliers and view details of individual spans, use the tabular section:
You can sort the rows of the Spans and Grouped tabs by the duration header.
To understand how many spans different services are generating, use the Grouped tab. Each row corresponds to a service and span name, and for each entry, latency statistics are shown.
To explore details of a span or grouping, select the entry from a row in the tabular section. A flyout opens that displays the span in the context of its trace. From this view, you can explore the span, its attributes, and view events and log data. You can also search the spans and attributes in the trace for keywords.
The trace data that the Trace Explorer page displays depends on the following:
The Scope setting, which determines which views the system searches for trace data. When the page opens, the Scope element is set to the default trace scope and the system queries the views listed in that scope for trace data.
On a new project, the default trace scope lists the
_AllSpansview for the project. That view provides read access to all trace data that the project stores.You can create trace scopes that let you query data stored in multiple projects. For more information, see View traces across projects.
Your Identity and Access Management (IAM) permissions on the searched view. If you don't have permission to view trace data for a project, then the Google Cloud console displays a warning message and the data from that project isn't displayed.
The time-range setting.
The filters you apply.
You can save your trace data on a custom dashboard. For more information, see Display trace data on a custom dashboard.
The remainder of this page provides more information about how to find and explore your trace data.
Before you begin
To get the permissions that
you need to view trace data by using the Google Cloud console and to select a trace scope,
ask your administrator to grant you the
Cloud Trace User (roles/cloudtrace.user) IAM role on your project.
For more information about granting roles, see Manage access to projects, folders, and organizations.
This predefined role contains the permissions required to view trace data by using the Google Cloud console and to select a trace scope. To see the exact permissions that are required, expand the Required permissions section:
Required permissions
The following permissions are required to view trace data by using the Google Cloud console and to select a trace scope:
-
To select a trace scope:
cloudtrace.traceScopes.[get, list] -
To read the default trace scope:
observability.scopes.get
You might also be able to get these permissions with custom roles or other predefined roles.
For more information about roles, see Control access with Identity and Access Management.
View aggregated trace data
To view the aggregated information about your trace data, do the following:
-
In the Google Cloud console, go to the
Trace explorer page:
You can also find this page by using the search bar.
It might take several minutes after the first trace data is written to a Google Cloud project before that data is available to view. If you don't see any trace data displayed after waiting a few minutes, then your project might not have any data to display or there might be a configuration issue. For information about how to resolve these issues, see Troubleshoot: No data in the Trace interface.
Configure the Trace Explorer page:
- In the toolbar of the Google Cloud console, select your Google Cloud project. For App Hub configurations, select the App Hub host project or management project.
Optional: Configure which views the system searches for trace data by using the Scope element:
To show the trace data that your project stores, set the first menu of the Scope element to
Project or to 
_Default.
These two settings are equivalent.To show the trace data that multiple projects store, expand the first menu of the Scope element, select Trace scope, and then select the trace scope that lists the
_AllSpansview for those projects. After you make your selection, the Scope menu displays a trace scope icon,, and the name of the
selected trace scope.
The data that the system displays depends on your IAM roles on the searched views. For example, if you don't have the permissions necessary to see data in a view, then the system doesn't display that data.
For more information, see Create and manage trace scopes.
Optional: Update the time range by using the time-range selector or by using your pointer to highlight a range on the x-axis.
For example, you might set this selector to Last 2 weeks when you want to see whether there are any trends in the latency data. Span data is stored for 30 days.
Explore the heatmap:
Use color to identify trends and outliers. A cell with many spans has a darker color than one with few spans.
View details of the cell by using the cell's tooltip. The tooltip lists the number of spans, date and time, and the time interval of the cell.
Explore latency percentiles and error rates:
To view latency trends, set the Chart view menu to Span duration (percentile). The duration chart displays the 50th, 90th, 95th, and 99th percentiles.
To view the response status as a function of time, set the Chart view menu to Span rate. The chart displays the rate of spans being sent to your project.
Use the Spans and Grouped tables to identify outliers or to view statistical information. The Spans table lists individual spans. The Grouped table groups spans by their name and by service or workload name.
To view outliers or find a span name, sort the table. You can sort by any column header.
For example, sorting the Spans table by duration lets you find spans with high latency.
To view statistics like average latency, use the Grouped table.
To find App Hub-registered services and workloads, search the Service/workload column for entries that include the service icon,
, or the
workload icon, 
.
The link provided with an App Hub-registered
service or workload opens the corresponding Application Monitoring dashboard.
If you want more information about a particular grouping, then select the span name for a row. A flyout opens that displays the span in the context of its trace. You can then explore the span and trace.
To restrict which spans are shown, add filters. For information about filtering your trace data, see the next section.
Filter your trace data
To display only information that is of interest to you, apply filters. Filters restrict what data is shown. For example, you might filter by service name and by status. Or, if you've deployed applications to App Hub, you might want to view the trace data only for the application, or for a specific service or workload that is part of the application.
When you add or remove a filter, the data displayed by the Trace Explorer page is refreshed and only shows those spans that match all applied filters.
Filter by trace ID
When you are troubleshooting an incident or failure, you might know the trace ID. To explore that trace, go to the toolbar, click pageview Search for trace, and enter the trace ID.
When you enter a valid ID, the Details flyout opens and displays information about the trace and its spans. You can use the options in that pane to explore the trace. For example, you might search the spans for keywords.
If you enter a trace ID that isn't found, then the trace data might be stored in a view that isn't searched by the Trace Explorer page, or you might not have an IAM role that lets you view trace data. To resolve these failures, identify the projects that store the trace data. For each project, verify that your IAM role lets you view trace data and then ensure that the Scope element lists the project's default view for trace data. For more information, see Configure which views the Trace Explorer searches.
Apply span filters
The Span filters pane lists the most common filters. You can select multiple entries from any subcategory. As you add or remove filters, the Filter bar is also updated.
The values for all menus are derived from your trace data. When a menu includes an option without any text, that option refers to spans that don't include the corresponding attribute.
The following Span filters are typically available:
- OpenTelemetry service: Filters by the
service.nameattribute. - Span name: Name of the span.
- Span status: The request status. For information about the values, see the OpenTelemetry SpanStatus documentation.
- Duration: The duration of the span.
- Span kind: Describes the relationships between spans. For information about the values, see the OpenTelemetry SpanKind documentation.
- App Hub application: Filters by the
gcp.apphub.application.idresource attribute. - App Hub service: Filters by the
gcp.apphub.service.idresource attribute. - App Hub workload: Filters by the
gcp.apphub.workload.idresource attribute.
If you want to filter by an attribute that isn't listed in the Span filters pane, then use the Filter bar.
Use the filter bar
The Filter bar lets you apply a filter with a predefined filter key and a value that you select, or you can enter both the key and value.
To add a filter, select Add filter, and then do one of the following:
- Select a defined key, like Span name, and then select a value from the secondary menu.
Select Add attribute filter, and then add your custom key and value. If you enter your own filter key, then use the same syntax as a key for an attribute on a span.
For example, to filter by the host identifier, set the key to
host.id. Similarly, to filter by status code, set the key to/http/status_code. In this scenario, you might set the value to200, which results in the filter/http/status_code: 200. To let the filter match any value, select Any value.
Filter by application
You can filter your trace data by an App Hub application, workload, or service, when that data contains the following resource attributes:
gcp.apphub.application.{container,id,location}gcp.apphub.{workload,service}.{criticality_type,environment_type,id}
These resource attributes are generated by instrumentation that you add to your applications.
In the