Kafka Ingestion and Query Enhancements in Tempo 3.1

What Changed Operationally

Operational observability pipelines are undergoing a fundamental shift in how they handle data ingestion and query performance, driven by the latest updates to Tempo 3.1. The most immediate operational impact is the expansion of Kafka-based ingestion capabilities, which now supports a wider array of security protocols and compression algorithms. Organizations relying on Tempo to aggregate distributed traces from high-throughput messaging systems can now configure TLS connections alongside four additional SASL mechanisms—SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARER, and AWS_MSK_IAM. This granular control over authentication and encryption ensures that data remains secure as it traverses network boundaries, addressing compliance requirements without sacrificing the flexibility needed to connect to diverse Kafka-compatible backends. Furthermore, the introduction of the producer_compression option allows operators to select from none, gzip, snappy, lz4, or zstd, enabling them to optimize bandwidth usage and reduce infrastructure costs by minimizing the volume of data transferred over the network.

Beyond security, the release introduces a critical optimization for multi-zone deployments through the new client_rack option. This feature implements rack-aware fetching, a mechanism inspired by KIP-392, which allows Tempo consumers to read data from a replica located within the same availability zone. By minimizing cross-zone data transfer, this capability directly reduces latency for local queries and significantly lowers data transfer costs associated with egress fees. The ability to intelligently route read requests to local replicas ensures that the observability platform remains performant and cost-effective, even as the volume of trace data grows. These Kafka enhancements collectively transform Tempo from a passive collector into a more robust, secure, and efficient component of the observability stack.

Enhanced Querying and Metrics Processing

The operational capabilities of Tempo extend beyond ingestion into the realm of query execution and data management, where the 3.1 release introduces significant improvements to TraceQL and span handling. A major advancement is the implementation of sampling-aware metrics, which addresses a common discrepancy in observability dashboards where sampled traces can lead to under-reporting of traffic volume. Tempo now supports a TraceQL hint, with(extrapolate=true), which instructs the query engine to recognize the sampling rate and scale the result accordingly. For example, if a trace is sampled at 50%, each stored span represents two actual spans; the engine applies this factor to rate, count_over_time, and sum_over_time functions to provide accurate representations of traffic. This feature applies to a wide range of aggregations, including histogram and quantile calculations, ensuring that performance baselines are not artificially deflated by sampling strategies. However, it is important to note that this extrapolation does not apply to min_over_time or max_over_time, as these functions are inherently point-in-time based.

How The Capability Fits Together

Alongside extrapolation, the release introduces arithmetic operators to TraceQL, allowing users to perform mathematical operations directly within their queries. This enables the calculation of derived metrics, such as error rates, by dividing one metric query by another using the standard +, -, *, and / operators. This flexibility allows operators to construct complex, ad-hoc queries without needing to pre-process data in external systems. To support these advanced queries, Tempo has optimized its processing model; queries that do not require the full trace structure now operate on individual spans rather than whole traces. This shift reduces latency and memory consumption, allowing the system to handle larger volumes of data more efficiently. The engine can now process these lightweight queries with greater speed, making real-time analysis of span-level data feasible even in high-throughput environments.

Data Management and Redaction

Effective observability also requires the ability to manage sensitive data, and Tempo 3.1 introduces a new method for trace redaction that integrates directly into the query layer. Previously, redaction often required manual intervention or complex data pipelines. In this release, users can now define redaction rules using a TraceQL query, allowing for precise filtering and masking of specific attributes within traces. This approach provides a centralized way to handle data privacy, ensuring that personally identifiable information or proprietary secrets are removed before they are stored or analyzed. The ability to apply redaction rules at the query stage means that sensitive data is handled early in the pipeline, minimizing the risk of exposure.

In addition to redaction, the release refines how Tempo handles trace data structures and service graph metadata. The trace-by-ID v2 endpoint has been updated to collapse groups of similar leaf spans into a single summary span. This summary carries aggregation metadata, such as aggregation.span_count and the minimum, maximum, average, and total duration of the original spans. This consolidation reduces the payload size of trace responses and simplifies the data structure for consumers. For service graphs, the addition of the unmatched_span_kind label to the expired edges counter provides greater visibility into topology issues, helping operators identify which sides of a connection are missing. While some features, such as span pruning and the extrapolation hint, remain experimental, these updates demonstrate a continued effort to refine Tempo’s architecture for better performance, security, and usability.

Operational Impact

Kafka Ingestion and Security Configuration

For administrators managing Tempo’s ingestion pipeline, the 3.1 release introduces a robust set of options for securing and optimizing connections to Kafka-based backends. The most significant security update is the expansion of authentication mechanisms beyond basic TLS. Tempo 3.1 now supports four additional SASL mechanisms: SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARER, and AWS_MSK_IAM. This allows teams to align Tempo’s authentication requirements with the specific security posture of their cloud infrastructure or on-premises clusters. When configuring these settings, engineers must ensure that the broker configuration explicitly permits the selected mechanism to prevent ingestion failures.

Beyond security, the release addresses operational costs associated with data transfer. The new client_rack option implements rack-aware fetching, a feature derived from KIP-392. This setting allows Tempo consumers to prefer reading from a replica located in the same availability zone as the consumer itself. By minimizing cross-zone data transfer, organizations can significantly reduce egress costs and latency. Administrators should evaluate their current network topology to determine if enabling this option provides a tangible benefit, particularly in multi-zone deployments where data locality is critical.

Rollout And Governance Decisions

TraceQL Enhancements and Query Performance

The query engine in Tempo 3.1 has been refined to handle complex metrics analysis and memory constraints more effectively. A major addition is the ability to perform trace redaction directly within TraceQL. Instead of relying on external masking tools, administrators can write a query that filters out sensitive fields before the data is stored or displayed. This approach reduces the risk of accidental exposure and lowers storage costs by shrinking the volume of sensitive data retained in the system.

For teams working with sampled traces, accurate metrics reporting has historically been a challenge. Tempo 3.1 resolves this with the with(extrapolate=true) hint. This experimental feature allows queries to mathematically scale results based on the sampling rate. For example, if traces are sampled at 50%, a query matching 1,000 spans will report 2,000 spans to accurately reflect the total traffic volume. This functionality applies to rate, count_over_time, sum_over_time, and other aggregation functions, but it does not apply to min or max operations. Engineers should test this hint on their dashboards to ensure it provides the correct scaling for their specific sampling policies.

Furthermore, the engine’s performance has been optimized for queries that do not require the full trace structure. By processing individual spans rather than reconstructing entire traces, latency and memory usage are reduced. This is particularly useful for simple metric lookups. The release also introduces arithmetic operators (+, -, *, /) within TraceQL, enabling the calculation of derived metrics—such as error rates—directly within the query language. Finally, the traces diff endpoint has been added to facilitate comparison, allowing administrators to send two trace IDs to the query-frontend and receive a structured report of the differences between them.

Failure Modes And Limits

Failure Modes and Operational Uncertainties

Operational stability in distributed tracing environments relies on the successful ingestion and querying of telemetry data. Tempo 3.1 introduces several experimental features that, while promising, introduce new variables into the production stack. One significant area of uncertainty is the new span pruning mechanism. This feature is currently experimental, meaning that the specific parameters governing its behavior and the exact structure of the response format are subject to change in future releases. Deploying this feature in a critical production environment requires a strategy for handling potential breaking changes or behavioral shifts that could affect downstream consumers of the trace data.

Security And Privacy Considerations

Another potential failure mode involves the sampling-aware metrics capabilities. The with(extrapolate=true) hint allows for accurate representation of traffic volume by scaling results based on the sampling rate. However, this functionality is not universally available; it is strictly limited to vParquet4 blocks or later. If a user attempts to apply this hint to legacy data blocks that do not meet this requirement, the query will fail. Furthermore, while the extrapolation logic applies to standard time-series functions like rate and count_over_time, it does not apply to min_over_time or max_over_time. This limitation means that users cannot rely on extrapolation to accurately determine the absolute minimum or maximum duration of requests across a sampled dataset, potentially obscuring performance outliers.

Security, Privacy, and Configuration Checklist

Open Questions

Implementing the new features in Tempo 3.1 requires a rigorous review of security configurations and privacy protocols. The release significantly enhances Kafka ingestion security, adding support for TLS and four additional SASL mechanisms: SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARER, and AWS_MSK_IAM. Organizations must audit their Kafka broker configurations to ensure these mechanisms are enabled and properly configured to prevent unauthorized access to trace streams. Additionally, the new client_rack option enables rack-aware fetching, which optimizes data transfer costs by directing consumers to read from replicas within their own availability zone. While this improves efficiency, it requires that the Kafka cluster is properly configured with rack metadata for consumers.

Privacy concerns are addressed through the introduction of TraceQL-based trace redaction. This allows administrators to use query syntax to filter and remove sensitive data from traces before they are stored or queried. To implement this effectively, teams should define specific TraceQL queries that target sensitive attributes and apply them during the ingestion pipeline. It is also critical to verify the handling of database attributes. Tempo 3.1 accepts both legacy and new OpenTelemetry semantic conventions for database attributes. However, if a span carries both identifiers, the db.system attribute takes precedence. Dashboards and alerts currently pointing to the legacy attribute name will continue to function, but new instrumentation should be aligned with the new conventions to avoid confusion.

Environment Checklist

Verification and Disclaimer

This article is a synthesis of the provided research notes and does not represent first-hand testing or validation of the features described. The operational capabilities, performance characteristics, and exact behavior of the new features in Tempo 3.1, particularly the experimental span pruning and sampling extrapolation, have not been verified by the author. Readers must independently verify the behavior of these features within their specific infrastructure and data schemas before deploying them to production environments.

Verification Before Production Use

This article was not lab-tested. Verify the current vendor documentation, licensing and rollout conditions, and the behavior in a non-production environment before relying on it operationally.