This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Tracing Package

API reference for rivaas.dev/tracing - Distributed tracing for Go applications

This is the API reference for the rivaas.dev/tracing package. For learning-focused documentation, see the Tracing Guide.

Package Information

Package Overview

The tracing package provides OpenTelemetry-based distributed tracing for Go applications with support for multiple exporters including Stdout, OTLP (gRPC and HTTP), and Noop.

Core Features

  • Multiple tracing providers (Stdout, OTLP, Noop)
  • Built-in HTTP middleware for request tracing
  • Manual span management with attributes and events
  • Context propagation for distributed tracing
  • Thread-safe operations
  • Span lifecycle hooks
  • Testing utilities

Architecture

The package is built on OpenTelemetry and provides a simplified interface for distributed tracing.

graph TD
    App[Application Code]
    Tracer[Tracer]
    Provider[Provider Layer]
    Noop[Noop]
    Stdout[Stdout]
    OTLP[OTLP gRPC/HTTP]
    Middleware[HTTP Middleware]
    Context[Context Propagation]
    
    App -->|Create Spans| Tracer
    Middleware -->|Auto-Trace| Tracer
    Tracer --> Provider
    Provider --> Noop
    Provider --> Stdout
    Provider --> OTLP
    Tracer --> Context
    Context -->|Extract/Inject| Middleware

Components

Main Package (rivaas.dev/tracing)

Core tracing functionality including:

  • Tracer - Main tracer for creating and managing spans
  • New() / MustNew() - Tracer initialization
  • Span management - Create, finish, add attributes/events
  • Middleware() - HTTP request tracing
  • ContextTracing - Helper for router context integration
  • Context helpers - Extract, inject, get trace IDs
  • Testing utilities

Quick API Index

Tracer Creation

tracer, err := tracing.New(options...)     // With error handling
tracer := tracing.MustNew(options...)      // Panics on error

Lifecycle Management

err := tracer.Start(ctx context.Context)   // Start OTLP providers
err := tracer.Shutdown(ctx context.Context) // Graceful shutdown

Span Management

// Create spans
ctx, span := tracer.StartSpan(ctx, "operation-name")
tracer.FinishSpan(span, statusCode)

// Add attributes
tracer.SetSpanAttribute(span, "key", value)

// Add events
tracer.AddSpanEvent(span, "event-name", attrs...)

Context Propagation

// Extract from incoming requests
ctx := tracer.ExtractTraceContext(ctx, req.Header)

// Inject into outgoing requests
tracer.InjectTraceContext(ctx, req.Header)

HTTP Middleware

handler := tracing.Middleware(tracer, options...)(httpHandler)
handler := tracing.MustMiddleware(tracer, options...)(httpHandler)

Context Helpers

traceID := tracing.TraceID(ctx)
spanID := tracing.SpanID(ctx)
tracing.SetSpanAttributeFromContext(ctx, "key", value)
tracing.AddSpanEventFromContext(ctx, "event-name", attrs...)

Testing Utilities

tracer := tracing.TestingTracer(t, options...)
tracer := tracing.TestingTracerWithStdout(t, options...)
middleware := tracing.TestingMiddleware(t, middlewareOptions...)

ContextTracing Helper

ct := tracing.NewContextTracing(ctx, tracer, span)
ct.SetSpanAttribute("key", value)
ct.AddSpanEvent("event-name", attrs...)
traceID := ct.TraceID()

Reference Pages

API Reference

Tracer type, span management, and context propagation.

View →

Options

Configuration options for providers and sampling.

View →

Middleware Options

HTTP middleware configuration and path exclusion.

View →

Troubleshooting

Common tracing issues and solutions.

View →

User Guide

Step-by-step tutorials and examples.

View →

Type Reference

Tracer

type Tracer struct {
    // contains filtered or unexported fields
}

Main tracer for distributed tracing. Thread-safe for concurrent access.

Methods: See API Reference for complete method documentation.

Option

type Option func(*Tracer)

Configuration option function type used with New() and MustNew().

Available Options: See Options for all options.

MiddlewareOption

type MiddlewareOption func(*middlewareConfig)

HTTP middleware configuration option.

Available Options: See Middleware Options for all options.

Provider

type Provider string

const (
    NoopProvider     Provider = "noop"
    StdoutProvider   Provider = "stdout"
    OTLPProvider     Provider = "otlp"
    OTLPHTTPProvider Provider = "otlp-http"
)

Available tracing providers.

EventType

type EventType int

const (
    EventError   EventType = iota // Error events
    EventWarning                   // Warning events
    EventInfo                      // Informational events
    EventDebug                     // Debug events
)

Event severity levels for internal operational events.

Event

type Event struct {
    Type    EventType
    Message string
    Args    []any // slog-style key-value pairs
}

Internal operational event from the tracing package.

EventHandler

type EventHandler func(Event)

Processes internal operational events. Used with WithEventHandler option.

SpanStartHook

type SpanStartHook func(ctx context.Context, span trace.Span, req *http.Request)

Callback invoked when a request span is started.

SpanFinishHook

type SpanFinishHook func(span trace.Span, statusCode int)

Callback invoked when a request span is finished.

Common Patterns

Basic Usage

tracer := tracing.MustNew(
    tracing.WithServiceName("my-api"),
    tracing.WithOTLP("localhost:4317"),
)
tracer.Start(context.Background())
defer tracer.Shutdown(context.Background())

ctx, span := tracer.StartSpan(ctx, "operation")
defer tracer.FinishSpan(span, http.StatusOK)

With HTTP Middleware

tracer := tracing.MustNew(
    tracing.WithServiceName("my-api"),
    tracing.WithOTLP("localhost:4317"),
)
tracer.Start(context.Background())

handler := tracing.MustMiddleware(tracer,
    tracing.WithExcludePaths("/health"),
)(httpHandler)

http.ListenAndServe(":8080", handler)

Distributed Tracing

// Service A - inject trace context
req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
tracer.InjectTraceContext(ctx, req.Header)
resp, _ := http.DefaultClient.Do(req)

// Service B - extract trace context
ctx = tracer.ExtractTraceContext(r.Context(), r.Header)
ctx, span := tracer.StartSpan(ctx, "operation")
defer tracer.FinishSpan(span, http.StatusOK)

Thread Safety

The Tracer type is thread-safe for:

  • All span management methods
  • Concurrent Start() and Shutdown() operations
  • Mixed tracing and lifecycle operations
  • Context propagation methods

Not thread-safe for:

  • Concurrent modification during initialization

Performance Notes

  • Request overhead (100% sampling): ~1.6 microseconds
  • Start/Finish span: ~160 nanoseconds
  • Set attribute: ~3 nanoseconds
  • Path exclusion (100 paths): ~9 nanoseconds

Best Practices:

  • Use sampling for high-traffic endpoints
  • Exclude health checks and metrics endpoints
  • Limit span attribute cardinality
  • Use path prefixes instead of regex when possible

Comparison with Metrics Package

The tracing package follows the same design pattern as the metrics package:

AspectMetrics PackageTracing Package
Main TypeRecorderTracer
Provider OptionsWithPrometheus(), WithOTLP()WithOTLP(), WithStdout(), WithNoop()
ConstructorNew(opts...) (*Recorder, error)New(opts...) (*Tracer, error)
Panic VersionMustNew(opts...) *RecorderMustNew(opts...) *Tracer
MiddlewareMiddleware(recorder, opts...)Middleware(tracer, opts...)
Panic MiddlewareMustMiddleware(recorder, opts...)MustMiddleware(tracer, opts...)
Path ExclusionMiddlewareOptionMiddlewareOption
Header RecordingMiddlewareOptionMiddlewareOption

Version Compatibility

The tracing package follows semantic versioning. The API is stable for the v1 series.

Minimum Go version: 1.25

OpenTelemetry compatibility: Uses OpenTelemetry SDK v1.x

Next Steps

For learning-focused guides, see the Tracing Guide.

1 - API Reference

Complete API documentation for the Tracer type and all methods

Complete API reference for the Tracer type and all tracing methods.

Tracer Type

type Tracer struct {
    // contains filtered or unexported fields
}

The main entry point for distributed tracing. Holds OpenTelemetry tracing configuration and runtime state. All operations on Tracer are thread-safe.

Important Notes

  • Immutable: Tracer is immutable after creation via New(). All configuration must be done through functional options.
  • Thread-safe: All methods are safe for concurrent use.
  • Global state: By default, does NOT set the global OpenTelemetry tracer provider. Use WithGlobalTracerProvider() option if needed.

Constructor Functions

New

func New(opts ...Option) (*Tracer, error)

Creates a new Tracer with the given options. Returns an error if the tracing provider fails to initialize.

Default configuration:

  • Service name: "rivaas-service".
  • Service version: "1.0.0".
  • Sample rate: 1.0 (100%).
  • Provider: NoopProvider.

Example:

tracer, err := tracing.New(
    tracing.WithServiceName("my-api"),
    tracing.WithOTLP("localhost:4317"),
    tracing.WithSampleRate(0.1),
)
if err != nil {
    log.Fatal(err)
}
defer tracer.Shutdown(context.Background())

MustNew

func MustNew(opts ...Option) *Tracer

Creates a new Tracer with the given options. Panics if the tracing provider fails to initialize. Use this when you want to panic on initialization errors.

Example:

tracer := tracing.MustNew(
    tracing.WithServiceName("my-api"),
    tracing.WithStdout(),
)
defer tracer.Shutdown(context.Background())

Lifecycle Methods

Start

func (t *Tracer) Start(ctx context.Context) error

Initializes OTLP providers that require network connections. The context is used for the OTLP connection establishment. This method is idempotent; calling it multiple times is safe.

Required for: OTLP (gRPC and HTTP) providers
Optional for: Noop and Stdout providers (they initialize immediately in New())

Example:

tracer := tracing.MustNew(
    tracing.WithOTLP("localhost:4317"),
)

if err := tracer.Start(context.Background()); err != nil {
    log.Fatal(err)
}

Shutdown

func (t *Tracer) Shutdown(ctx context.Context) error

Gracefully shuts down the tracing system, flushing any pending spans. This should be called before the application exits to ensure all spans are exported. This method is idempotent - calling it multiple times is safe and will only perform shutdown once.

Example:

defer func() {
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()
    if err := tracer.Shutdown(ctx); err != nil {
        log.Printf("Error shutting down tracer: %v", err)
    }
}()

Span Management Methods

StartSpan

func (t *Tracer) StartSpan(ctx context.Context, name string, opts ...trace.SpanStartOption) (context.Context, trace.Span)

Starts a new span with the given name and options. Returns a new context with the span attached and the span itself.

If tracing is disabled, returns the original context and a non-recording span. The returned span should always be ended, even if tracing is disabled.

Parameters:

  • ctx: Parent context
  • name: Span name (should be descriptive)
  • opts: Optional OpenTelemetry span start options

Returns:

  • New context with span attached
  • The created span

Example:

ctx, span := tracer.StartSpan(ctx, "database-query")
defer tracer.FinishSpan(span, http.StatusOK)

tracer.SetSpanAttribute(span, "db.query", "SELECT * FROM users")

FinishSpan

func (t *Tracer) FinishSpan(span trace.Span, statusCode int)

Completes the span with the given status code. Sets the span status based on the HTTP status code:

  • 2xx-3xx: Success (codes.Ok)
  • 4xx-5xx: Error (codes.Error)

This method is safe to call multiple times; subsequent calls are no-ops.

Parameters:

  • span: The span to finish
  • statusCode: HTTP status code (e.g., http.StatusOK)

Example:

defer tracer.FinishSpan(span, http.StatusOK)

SetSpanAttribute

func (t *Tracer) SetSpanAttribute(span trace.Span, key string, value any)

Adds an attribute to the span with type-safe handling.

Supported types:

  • string, int, int64, float64, bool: native OpenTelemetry handling
  • Other types: converted to string using fmt.Sprintf

This is a no-op if tracing is disabled, span is nil, or span is not recording.

Parameters:

  • span: The span to add the attribute to
  • key: Attribute key
  • value: Attribute value

Example:

tracer.SetSpanAttribute(span, "user.id", 12345)
tracer.SetSpanAttribute(span, "user.premium", true)
tracer.SetSpanAttribute(span, "user.name", "Alice")

AddSpanEvent

func (t *Tracer) AddSpanEvent(span trace.Span, name string, attrs ...attribute.KeyValue)

Adds an event to the span with optional attributes. Events represent important moments in a span’s lifetime.

This is a no-op if tracing is disabled, span is nil, or span is not recording.

Parameters:

  • span: The span to add the event to
  • name: Event name
  • attrs: Optional event attributes

Example:

import "go.opentelemetry.io/otel/attribute"

tracer.AddSpanEvent(span, "cache_hit",
    attribute.String("key", "user:123"),
    attribute.Int("ttl_seconds", 300),
)

Context Propagation Methods

ExtractTraceContext

func (t *Tracer) ExtractTraceContext(ctx context.Context, headers http.Header) context.Context

Extracts trace context from HTTP request headers. Returns a new context with the extracted trace information.

If no trace context is found in headers, returns the original context. Uses W3C Trace Context format by default.

Parameters:

  • ctx: Base context
  • headers: HTTP headers to extract from

Returns:

  • Context with extracted trace information

Example:

ctx := tracer.ExtractTraceContext(r.Context(), r.Header)
ctx, span := tracer.StartSpan(ctx, "operation")
defer tracer.FinishSpan(span, http.StatusOK)

InjectTraceContext

func (t *Tracer) InjectTraceContext(ctx context.Context, headers http.Header)

Injects trace context into HTTP headers. This allows trace context to propagate across service boundaries.

Uses W3C Trace Context format by default. This is a no-op if tracing is disabled.

Parameters:

  • ctx: Context containing trace information
  • headers: HTTP headers to inject into

Example:

req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
tracer.InjectTraceContext(ctx, req.Header)
resp, _ := http.DefaultClient.Do(req)

Request Span Methods

These methods are used internally by the middleware but can also be used for custom HTTP handling.

StartRequestSpan

func (t *Tracer) StartRequestSpan(ctx context.Context, req *http.Request, path string, isStatic bool) (context.Context, trace.Span)

Starts a span for an HTTP request. This is used by the middleware to create request spans with standard attributes.

Parameters:

  • ctx: Request context
  • req: HTTP request
  • path: Request path
  • isStatic: Whether this is a static route

Returns:

  • Context with span
  • The created span

FinishRequestSpan

func (t *Tracer) FinishRequestSpan(span trace.Span, statusCode int)

Completes the span for an HTTP request. Sets the HTTP status code attribute and invokes the span finish hook if configured.

Parameters:

  • span: The span to finish
  • statusCode: HTTP response status code

Accessor Methods

IsEnabled

func (t *Tracer) IsEnabled() bool

Returns true if tracing is enabled.

ServiceName

func (t *Tracer) ServiceName() string

Returns the service name.

ServiceVersion

func (t *Tracer) ServiceVersion() string

Returns the service version.

GetTracer

func (t *Tracer) GetTracer() trace.Tracer

Returns the OpenTelemetry tracer.

GetPropagator

func (t *Tracer) GetPropagator() propagation.TextMapPropagator

Returns the OpenTelemetry propagator.

GetProvider

func (t *Tracer) GetProvider() Provider

Returns the current tracing provider.

Context Helper Functions

These are package-level functions for working with spans through context.

TraceID

func TraceID(ctx context.Context) string

Returns the current trace ID from the active span in the context. Returns an empty string if no active span or span context is invalid.

Example:

traceID := tracing.TraceID(ctx)
log.Printf("Processing request [trace=%s]", traceID)

SpanID

func SpanID(ctx context.Context) string

Returns the current span ID from the active span in the context. Returns an empty string if no active span or span context is invalid.

Example:

spanID := tracing.SpanID(ctx)
log.Printf("Processing request [span=%s]", spanID)

SetSpanAttributeFromContext

func SetSpanAttributeFromContext(ctx context.Context, key string, value any)

Adds an attribute to the current span from context. This is a no-op if tracing is not active.

Example:

func handleRequest(ctx context.Context) {
    tracing.SetSpanAttributeFromContext(ctx, "user.role", "admin")
    tracing.SetSpanAttributeFromContext(ctx, "user.id", 12345)
}

AddSpanEventFromContext

func AddSpanEventFromContext(ctx context.Context, name string, attrs ...attribute.KeyValue)

Adds an event to the current span from context. This is a no-op if tracing is not active.

Example:

import "go.opentelemetry.io/otel/attribute"

tracing.AddSpanEventFromContext(ctx, "cache_miss",
    attribute.String("key", "user:123"),
)

TraceContext

func TraceContext(ctx context.Context) context.Context

Returns the context as-is (it should already contain trace information). Provided for API consistency.

Event Types

EventType

type EventType int

const (
    EventError   EventType = iota // Error events
    EventWarning                   // Warning events
    EventInfo                      // Informational events
    EventDebug                     // Debug events
)

Event severity levels for internal operational events.

Event

type Event struct {
    Type    EventType
    Message string
    Args    []any // slog-style key-value pairs
}

Internal operational event from the tracing package. Events are used to report errors, warnings, and informational messages about the tracing system’s operation.

EventHandler

type EventHandler func(Event)

Processes internal operational events from the tracing package. Implementations can log events, send them to monitoring systems, or take custom actions based on event type.

Example:

tracing.WithEventHandler(func(e tracing.Event) {
    if e.Type == tracing.EventError {
        sentry.CaptureMessage(e.Message)
    }
    slog.Default().Info(e.Message, e.Args...)
})

DefaultEventHandler

func DefaultEventHandler(logger *slog.Logger) EventHandler

Returns an EventHandler that logs events to the provided slog.Logger. This is the default implementation used by WithLogger.

If logger is nil, returns a no-op handler that discards all events.

Hook Types

SpanStartHook

type SpanStartHook func(ctx context.Context, span trace.Span, req *http.Request)

Called when a request span is started. It receives the context, span, and HTTP request. This can be used for custom attribute injection, dynamic sampling, or integration with APM tools.

Example:

hook := func(ctx context.Context, span trace.Span, req *http.Request) {
    if tenantID := req.Header.Get("X-Tenant-ID"); tenantID != "" {
        span.SetAttributes(attribute.String("tenant.id", tenantID))
    }
}
tracer := tracing.MustNew(
    tracing.WithSpanStartHook(hook),
)

SpanFinishHook

type SpanFinishHook func(span trace.Span, statusCode int)

Called when a request span is finished. It receives the span and the HTTP status code. This can be used for custom metrics, logging, or post-processing.

Example:

hook := func(span trace.Span, statusCode int) {
    if statusCode >= 500 {
        metrics.IncrementServerErrors()
    }
}
tracer := tracing.MustNew(
    tracing.WithSpanFinishHook(hook),
)

ContextTracing Type

type ContextTracing struct {
    // contains filtered or unexported fields
}

A helper type for router context integration that provides convenient access to tracing functionality within HTTP handlers.

NewContextTracing

func NewContextTracing(ctx context.Context, tracer *Tracer, span trace.Span) *ContextTracing

Creates a new context tracing helper. Panics if ctx is nil.

Parameters:

  • ctx: The request context
  • tracer: The Tracer instance
  • span: The current span

Example:

ct := tracing.NewContextTracing(ctx, tracer, span)

ContextTracing Methods

TraceID

func (ct *ContextTracing) TraceID() string

Returns the current trace ID. Returns an empty string if no valid span.

SpanID

func (ct *ContextTracing) SpanID() string

Returns the current span ID. Returns an empty string if no valid span.

SetSpanAttribute

func (ct *ContextTracing) SetSpanAttribute(key string, value any)

Adds an attribute to the current span. No-op if span is nil or not recording.

AddSpanEvent

func (ct *ContextTracing) AddSpanEvent(name string, attrs ...attribute.KeyValue)

Adds an event to the current span. No-op if span is nil or not recording.

TraceContext

func (ct *ContextTracing) TraceContext() context.Context

Returns the trace context.

GetSpan

func (ct *ContextTracing) GetSpan() trace.Span

Returns the current span.

GetTracer

func (ct *ContextTracing) GetTracer() *Tracer

Returns the underlying Tracer.

ContextTracing Example

func handleRequest(w http.ResponseWriter, r *http.Request, tracer *tracing.Tracer) {
    ctx := r.Context()
    span := trace.SpanFromContext(ctx)
    
    // Create context tracing helper
    ct := tracing.NewContextTracing(ctx, tracer, span)
    
    // Use helper methods
    ct.SetSpanAttribute("user.id", "123")
    ct.AddSpanEvent("processing_started")
    
    // Get trace info for logging
    log.Printf("Processing [trace=%s, span=%s]", ct.TraceID(), ct.SpanID())
}

Constants

Default Values

const (
    DefaultServiceName    = "rivaas-service"
    DefaultServiceVersion = "1.0.0"
    DefaultSampleRate     = 1.0
)

Default configuration values used when not explicitly set.

Provider Types

const (
    NoopProvider     Provider = "noop"
    StdoutProvider   Provider = "stdout"
    OTLPProvider     Provider = "otlp"
    OTLPHTTPProvider Provider = "otlp-http"
)

Available tracing providers.

Next Steps

2 - Tracer Options

All configuration options for Tracer initialization

Complete reference for all Option functions used to configure the Tracer.

Option Type

type Option func(*Tracer)

Configuration option function type used with New() and MustNew(). Options are applied during Tracer creation.

Service Configuration Options

WithServiceName

func WithServiceName(name string) Option

Sets the service name for tracing. This name appears in span attributes as service.name.

Parameters:

  • name: Service identifier like "user-api" or "order-service".

Default: "rivaas-service"

Example:

tracer := tracing.MustNew(
    tracing.WithServiceName("user-api"),
)

WithServiceVersion

func WithServiceVersion(version string) Option

Sets the service version for tracing. This version appears in span attributes as service.version.

Parameters:

  • version: Service version like "v1.2.3" or "dev".

Default: "1.0.0"

Example:

tracer := tracing.MustNew(
    tracing.WithServiceName("user-api"),
    tracing.WithServiceVersion("v1.2.3"),
)

Provider Options

Only one provider can be configured at a time. Configuring multiple providers results in a validation error.

WithNoop

func WithNoop() Option

Configures noop provider. This is the default. No traces are exported. Use for testing or when tracing is disabled.

Example:

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithNoop(),
)

WithStdout

func WithStdout() Option

Configures stdout provider for development/debugging. Traces are printed to standard output in pretty-printed JSON format.

Example:

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithStdout(),
)

WithOTLP

func WithOTLP(endpoint string, opts ...OTLPOption) Option

Configures OTLP gRPC provider with endpoint. Use this for production deployments with OpenTelemetry collectors.

Parameters:

  • endpoint: OTLP endpoint in format "host:port" (e.g., "localhost:4317")
  • opts: Optional OTLP-specific options (e.g., OTLPInsecure())

Requires: Call tracer.Start(ctx) before tracing

Example:

// Secure (TLS enabled by default)
tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithOTLP("collector.example.com:4317"),
)

// Insecure (local development)
tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithOTLP("localhost:4317", tracing.OTLPInsecure()),
)

WithOTLPHTTP

func WithOTLPHTTP(endpoint string) Option

Configures OTLP HTTP provider with endpoint. Use this when gRPC is not available or HTTP is preferred.

Parameters:

  • endpoint: OTLP HTTP endpoint with protocol (e.g., "http://localhost:4318", "https://collector:4318")

Requires: Call tracer.Start(ctx) before tracing

Example:

// HTTP (insecure - development)
tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithOTLPHTTP("http://localhost:4318"),
)

// HTTPS (secure - production)
tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithOTLPHTTP("https://collector.example.com:4318"),
)

OTLP Options

OTLPInsecure

func OTLPInsecure() OTLPOption

Enables insecure gRPC for OTLP. Default is false (uses TLS). Set to true for local development.

Example:

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithOTLP("localhost:4317", tracing.OTLPInsecure()),
)

Sampling Options

WithSampleRate

func WithSampleRate(rate float64) Option

Sets the sampling rate (0.0 to 1.0). Values outside this range are clamped to valid bounds.

A rate of 1.0 samples all requests, 0.5 samples 50%, and 0.0 samples none. Sampling decisions are made per-request based on the configured rate.

Parameters:

  • rate: Sampling rate between 0.0 and 1.0

Default: 1.0 (100% sampling)

Example:

// Sample 10% of requests
tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithSampleRate(0.1),
)

Hook Options

WithSpanStartHook

func WithSpanStartHook(hook SpanStartHook) Option

Sets a callback that is invoked when a request span is started. The hook receives the context, span, and HTTP request, allowing custom attribute injection, dynamic sampling decisions, or integration with APM tools.

Type:

type SpanStartHook func(ctx context.Context, span trace.Span, req *http.Request)

Example:

startHook := func(ctx context.Context, span trace.Span, req *http.Request) {
    if tenantID := req.Header.Get("X-Tenant-ID"); tenantID != "" {
        span.SetAttributes(attribute.String("tenant.id", tenantID))
    }
}

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithSpanStartHook(startHook),
)

WithSpanFinishHook

func WithSpanFinishHook(hook SpanFinishHook) Option

Sets a callback that is invoked when a request span is finished. The hook receives the span and HTTP status code, allowing custom metrics recording, logging, or post-processing.

Type:

type SpanFinishHook func(span trace.Span, statusCode int)

Example:

finishHook := func(span trace.Span, statusCode int) {
    if statusCode >= 500 {
        metrics.IncrementServerErrors()
    }
}

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithSpanFinishHook(finishHook),
)

Logging Options

WithLogger

func WithLogger(logger *slog.Logger) Option

Sets the logger for internal operational events using the default event handler. This is a convenience wrapper around WithEventHandler that logs events to the provided slog.Logger.

Parameters:

  • logger: *slog.Logger for logging internal events

Example:

import "log/slog"

logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
    Level: slog.LevelDebug,
}))

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithLogger(logger),
)

WithEventHandler

func WithEventHandler(handler EventHandler) Option

Sets a custom event handler for internal operational events. Use this for advanced use cases like sending errors to Sentry, custom alerting, or integrating with non-slog logging systems.

Type:

type EventHandler func(Event)

Example:

eventHandler := func(e tracing.Event) {
    switch e.Type {
    case tracing.EventError:
        sentry.CaptureMessage(e.Message)
        myLogger.Error(e.Message, e.Args...)
    case tracing.EventWarning:
        myLogger.Warn(e.Message, e.Args...)
    case tracing.EventInfo:
        myLogger.Info(e.Message, e.Args...)
    case tracing.EventDebug:
        myLogger.Debug(e.Message, e.Args...)
    }
}

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithEventHandler(eventHandler),
)

Advanced Options

WithTracerProvider

func WithTracerProvider(provider trace.TracerProvider) Option

Allows you to provide a custom OpenTelemetry TracerProvider. When using this option, the package will NOT set the global otel.SetTracerProvider() by default. Use WithGlobalTracerProvider() if you want global registration.

Use cases:

  • Manage tracer provider lifecycle yourself
  • Need multiple independent tracing configurations
  • Want to avoid global state in your application

Important: When using WithTracerProvider, provider options (WithOTLP, WithStdout, etc.) are ignored since you’re managing the provider yourself. You are also responsible for calling Shutdown() on your provider.

Example:

import sdktrace "go.opentelemetry.io/otel/sdk/trace"

tp := sdktrace.NewTracerProvider(
    // Your custom configuration
    sdktrace.WithSampler(sdktrace.AlwaysSample()),
)

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithTracerProvider(tp),
)

// You manage tp.Shutdown() yourself
defer tp.Shutdown(context.Background())

WithCustomTracer

func WithCustomTracer(tracer trace.Tracer) Option

Allows using a custom OpenTelemetry tracer. This is useful when you need specific tracer configuration or want to use a tracer from an existing OpenTelemetry setup.

Example:

tp := trace.NewTracerProvider(...)
customTracer := tp.Tracer("my-tracer")

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithCustomTracer(customTracer),
)

WithCustomPropagator

func WithCustomPropagator(propagator propagation.TextMapPropagator) Option

Allows using a custom OpenTelemetry propagator. This is useful for custom trace context propagation formats. By default, uses the global propagator from otel.GetTextMapPropagator() (W3C Trace Context).

Example:

import "go.opentelemetry.io/otel/propagation"

// Use W3C Trace Context explicitly
prop := propagation.TraceContext{}

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithCustomPropagator(prop),
)

Using B3 propagation:

import "go.opentelemetry.io/contrib/propagators/b3"

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithCustomPropagator(b3.New()),
)

WithGlobalTracerProvider

func WithGlobalTracerProvider() Option

Registers the tracer provider as the global OpenTelemetry tracer provider via otel.SetTracerProvider(). By default, tracer providers are not registered globally to allow multiple tracing configurations to coexist in the same process.

Use when:

  • You want otel.GetTracerProvider() to return your tracer
  • Integrating with libraries that use the global tracer
  • Single tracer for entire application

Example:

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithOTLP("localhost:4317"),
    tracing.WithGlobalTracerProvider(), // Register globally
)

Option Combinations

Development Configuration

tracer := tracing.MustNew(
    tracing.WithServiceName("my-api"),
    tracing.WithServiceVersion("dev"),
    tracing.WithStdout(),
    tracing.WithSampleRate(1.0),
    tracing.WithLogger(slog.Default()),
)

Production Configuration

tracer := tracing.MustNew(
    tracing.WithServiceName("user-api"),
    tracing.WithServiceVersion(os.Getenv("VERSION")),
    tracing.WithOTLP(os.Getenv("OTLP_ENDPOINT")),
    tracing.WithSampleRate(0.1),
    tracing.WithSpanStartHook(enrichSpan),
    tracing.WithSpanFinishHook(recordMetrics),
)

Testing Configuration

tracer := tracing.MustNew(
    tracing.WithServiceName("test-service"),
    tracing.WithServiceVersion("v1.0.0"),
    tracing.WithNoop(),
    tracing.WithSampleRate(1.0),
)

Validation Errors

Configuration is validated when calling New() or MustNew(). Common validation errors:

Multiple Providers

// ✗ Error: multiple providers configured
tracer, err := tracing.New(
    tracing.WithServiceName("my-service"),
    tracing.WithStdout(),
    tracing.WithOTLP("localhost:4317"), // Error!
)
// Returns: "validation errors: provider: multiple providers configured"

Solution: Only configure one provider.

Empty Service Name

// ✗ Error: service name cannot be empty
tracer, err := tracing.New(
    tracing.WithServiceName(""),
)
// Returns: "invalid configuration: serviceName: cannot be empty"

Solution: Always provide a service name.

Invalid Sample Rate

// Values are automatically clamped
tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithSampleRate(1.5), // Clamped to 1.0
)

Sample rates outside 0.0-1.0 are automatically clamped to valid bounds.

Complete Option Reference

OptionDescriptionDefault
WithServiceName(name)Set service name"rivaas-service"
WithServiceVersion(version)Set service version"1.0.0"
WithNoop()Noop providerYes (default)
WithStdout()Stdout provider-
WithOTLP(endpoint, opts...)OTLP gRPC provider-
WithOTLPHTTP(endpoint)OTLP HTTP provider-
WithSampleRate(rate)Sampling rate (0.0-1.0)1.0
WithSpanStartHook(hook)Span start callback-
WithSpanFinishHook(hook)Span finish callback-
WithLogger(logger)Set slog logger-
WithEventHandler(handler)Custom event handler-
WithTracerProvider(provider)Custom tracer provider-
WithCustomTracer(tracer)Custom tracer-
WithCustomPropagator(prop)Custom propagatorW3C Trace Context
WithGlobalTracerProvider()Register globallyNo

Next Steps

3 - Middleware Options

All configuration options for HTTP middleware

Complete reference for all MiddlewareOption functions used to configure the HTTP tracing middleware.

MiddlewareOption Type

type MiddlewareOption func(*middlewareConfig)

Configuration option function type used with Middleware() and MustMiddleware(). These options control HTTP request tracing behavior.

Path Exclusion Options

Exclude specific paths from tracing to reduce noise and overhead.

WithExcludePaths

func WithExcludePaths(paths ...string) MiddlewareOption

Excludes specific paths from tracing. Excluded paths will not create spans or record any tracing data. This is useful for health checks, metrics endpoints, etc.

Maximum of 1000 paths can be excluded to prevent unbounded growth.

Parameters:

  • paths: Exact paths to exclude (e.g., "/health", "/metrics")

Performance: O(1) hash map lookup

Example:

handler := tracing.Middleware(tracer,
    tracing.WithExcludePaths("/health", "/metrics", "/ready", "/live"),
)(mux)

WithExcludePrefixes

func WithExcludePrefixes(prefixes ...string) MiddlewareOption

Excludes paths with the given prefixes from tracing. This is useful for excluding entire path hierarchies like /debug/, /internal/, etc.

Parameters:

  • prefixes: Path prefixes to exclude (e.g., "/debug/", "/internal/")

Performance: O(n) where n = number of prefixes

Example:

handler := tracing.Middleware(tracer,
    tracing.WithExcludePrefixes("/debug/", "/internal/", "/.well-known/"),
)(mux)

Matches:

  • /debug/pprof
  • /debug/vars
  • /internal/health
  • /.well-known/acme-challenge

WithExcludePatterns

func WithExcludePatterns(patterns ...string) MiddlewareOption

Excludes paths matching the given regex patterns from tracing. The patterns are compiled once during configuration. Returns a validation error if any pattern fails to compile.

Parameters:

  • patterns: Regular expression patterns (e.g., "^/v[0-9]+/internal/.*")

Performance: O(p) where p = number of patterns

Validation: Invalid regex patterns cause the middleware to panic during initialization.

Example:

handler := tracing.Middleware(tracer,
    tracing.WithExcludePatterns(
        `^/v[0-9]+/internal/.*`,  // Version-prefixed internal routes
        `^/api/health.*`,          // Any health-related endpoint
        `^/debug/.*`,              // All debug routes
    ),
)(mux)

Matches:

  • /v1/internal/status
  • /v2/internal/debug
  • /api/health
  • /api/health/db
  • /debug/pprof/heap

Header Recording Options

WithHeaders

func WithHeaders(headers ...string) MiddlewareOption

Records specific request headers as span attributes. Header names are case-insensitive. Recorded as http.request.header.{name}.

Security: Sensitive headers (Authorization, Cookie, etc.) are automatically filtered out to prevent accidental exposure of credentials in traces.

Parameters:

  • headers: Header names to record (case-insensitive)

Recorded as: Lowercase header names (http.request.header.x-request-id)

Example:

handler := tracing.Middleware(tracer,
    tracing.WithHeaders("X-Request-ID", "X-Correlation-ID", "User-Agent"),
)(mux)

Span attributes:

  • http.request.header.x-request-id: "abc123"
  • http.request.header.x-correlation-id: "xyz789"
  • http.request.header.user-agent: "Mozilla/5.0..."

Sensitive Header Filtering

The following headers are automatically filtered and will never be recorded, even if explicitly included:

  • Authorization
  • Cookie
  • Set-Cookie
  • X-API-Key
  • X-Auth-Token
  • Proxy-Authorization
  • WWW-Authenticate

Example:

// Authorization is automatically filtered
handler := tracing.Middleware(tracer,
    tracing.WithHeaders(
        "X-Request-ID",
        "Authorization", // ← Filtered, won't be recorded
        "X-Correlation-ID",
    ),
)(mux)

Query Parameter Recording Options

Default Behavior

By default, all query parameters are recorded as span attributes.

WithRecordParams

func WithRecordParams(params ...string) MiddlewareOption

Specifies which URL query parameters to record as span attributes. Only parameters in this list will be recorded. This provides fine-grained control over which parameters are traced.

If this option is not used, all query parameters are recorded by default (unless WithoutParams is used).

Parameters:

  • params: Parameter names to record

Recorded as: http.request.param.{name}

Example:

handler := tracing.Middleware(tracer,
    tracing.WithRecordParams("user_id", "request_id", "page", "limit"),
)(mux)

Request: GET /api/users?page=2&limit=10&user_id=123&secret=xyz

Span attributes:

  • http.request.param.page: ["2"]
  • http.request.param.limit: ["10"]
  • http.request.param.user_id: ["123"]
  • secret is not recorded (not in whitelist)

WithExcludeParams

func WithExcludeParams(params ...string) MiddlewareOption

Specifies which URL query parameters to exclude from tracing. This is useful for blacklisting sensitive parameters while recording all others.

Parameters in this list will never be recorded, even if WithRecordParams includes them (blacklist takes precedence).

Parameters:

  • params: Parameter names to exclude

Example:

handler := tracing.Middleware(tracer,
    tracing.WithExcludeParams("password", "token", "api_key", "secret"),
)(mux)

Request: GET /api/users?page=2&password=secret123&user_id=123

Span attributes:

  • http.request.param.page: ["2"]
  • http.request.param.user_id: ["123"]
  • password is not recorded (blacklisted)

WithoutParams

func WithoutParams() MiddlewareOption

Disables recording URL query parameters as span attributes. By default, all query parameters are recorded. Use this option if parameters may contain sensitive data.

Example:

handler := tracing.Middleware(tracer,
    tracing.WithoutParams(),
)(mux)

No query parameters will be recorded regardless of the request.

Parameter Recording Precedence

When multiple parameter options are used:

  1. WithoutParams() - If set, no parameters are recorded
  2. WithExcludeParams() - Blacklist takes precedence over whitelist
  3. WithRecordParams() - Only whitelisted parameters are recorded
  4. Default - All parameters are recorded

Example:

// Whitelist with blacklist
handler := tracing.Middleware(tracer,
    tracing.WithRecordParams("page", "limit", "sort", "api_key"),
    tracing.WithExcludeParams("api_key", "token"), // Blacklist overrides
)(mux)

Result: page, limit, and sort are recorded, but api_key is excluded (blacklist wins).

Middleware Functions

Middleware

func Middleware(tracer *Tracer, opts ...MiddlewareOption) func(http.Handler) http.Handler

Creates a middleware function for standalone HTTP integration. Panics if any middleware option is invalid (e.g., invalid regex pattern).

Parameters:

  • tracer: Tracer instance
  • opts: Middleware configuration options

Returns: HTTP middleware function

Panics: If middleware options are invalid

Example:

tracer := tracing.MustNew(
    tracing.WithServiceName("my-api"),
    tracing.WithOTLP("localhost:4317"),
)

handler := tracing.Middleware(tracer,
    tracing.WithExcludePaths("/health", "/metrics"),
    tracing.WithHeaders("X-Request-ID"),
)(mux)

MustMiddleware

func MustMiddleware(tracer *Tracer, opts ...MiddlewareOption) func(http.Handler) http.Handler

Creates a middleware function for standalone HTTP integration. It panics if any middleware option is invalid (e.g., invalid regex pattern). This is a convenience wrapper around Middleware for consistency with MustNew.

Behavior: Identical to Middleware() - both panic on invalid options.

Example:

handler := tracing.MustMiddleware(tracer,
    tracing.WithExcludePaths("/health", "/metrics"),
    tracing.WithHeaders("X-Request-ID"),
)(mux)

Complete Examples

Minimal Middleware

// Trace everything with no filtering
handler := tracing.Middleware(tracer)(mux)

Production Middleware

handler := tracing.Middleware(tracer,
    // Exclude observability endpoints
    tracing.WithExcludePaths("/health", "/metrics", "/ready", "/live"),
    
    // Exclude debug endpoints
    tracing.WithExcludePrefixes("/debug/", "/internal/"),
    
    // Record correlation headers
    tracing.WithHeaders("X-Request-ID", "X-Correlation-ID"),
    
    // Whitelist safe parameters
    tracing.WithRecordParams("page", "limit", "sort", "filter"),
    
    // Blacklist sensitive parameters
    tracing.WithExcludeParams("password", "token", "api_key"),
)(mux)

Development Middleware

handler := tracing.Middleware(tracer,
    // Only exclude metrics
    tracing.WithExcludePaths("/metrics"),
    
    // Record all headers (except sensitive ones)
    tracing.WithHeaders("X-Request-ID", "X-Correlation-ID", "User-Agent"),
)(mux)

High-Security Middleware

handler := tracing.Middleware(tracer,
    // Exclude health checks
    tracing.WithExcludePaths("/health"),
    
    // No headers recorded
    // No query parameters recorded
    tracing.WithoutParams(),
)(mux)

Performance Considerations

Path Exclusion Performance

MethodComplexityPerformance
WithExcludePaths()O(1)~9ns per request (hash lookup)
WithExcludePrefixes()O(n)~9ns per request (n prefixes)
WithExcludePatterns()O(p)~20ns per request (p patterns)

Recommendation: Use exact paths when possible for best performance.

Memory Usage

  • Path exclusion: ~100 bytes per path
  • Header recording: ~50 bytes per header
  • Parameter recording: ~30 bytes per parameter name

Limits

  • Maximum excluded paths: 1000 (enforced by WithExcludePaths)
  • No limit on: Prefixes, patterns, headers, parameters

Validation Errors

Configuration is validated when calling Middleware() or MustMiddleware(). Invalid options cause a panic.

Invalid Regex Pattern

// ✗ Panics: invalid regex
handler := tracing.Middleware(tracer,
    tracing.WithExcludePatterns(`[invalid regex`),
)(mux)
// Panics: "middleware validation errors: excludePatterns: invalid regex..."

Solution: Ensure regex patterns are valid.

Option Reference Table

OptionDescriptionDefault Behavior
WithExcludePaths(paths...)Exclude exact pathsAll paths traced
WithExcludePrefixes(prefixes...)Exclude by prefixAll paths traced
WithExcludePatterns(patterns...)Exclude by regexAll paths traced
WithHeaders(headers...)Record headersNo headers recorded
WithRecordParams(params...)Whitelist paramsAll params recorded
WithExcludeParams(params...)Blacklist paramsNo params excluded
WithoutParams()Disable paramsAll params recorded

Best Practices

Always Exclude Health Checks

tracing.WithExcludePaths("/health", "/metrics", "/ready", "/live")

Health checks are high-frequency and low-value for tracing.

Use Exact Paths for Common Exclusions

// ✓ Good - fastest
tracing.WithExcludePaths("/health", "/metrics")

// ✗ Less optimal - slower
tracing.WithExcludePatterns("^/(health|metrics)$")

Blacklist Sensitive Parameters

tracing.WithExcludeParams(
    "password", "token", "api_key", "secret",
    "credit_card", "ssn", "access_token",
)

Record Correlation Headers

tracing.WithHeaders("X-Request-ID", "X-Correlation-ID", "X-Trace-ID")

Helps correlate traces with logs and other observability data.

Combine Exclusion Methods

handler := tracing.Middleware(tracer,
    tracing.WithExcludePaths("/health", "/metrics"),      // Exact
    tracing.WithExcludePrefixes("/debug/", "/internal/"), // Prefix
    tracing.WithExcludePatterns(`^/v[0-9]+/internal/.*`), // Regex
)(mux)

Next Steps

4 - Troubleshooting

Common issues and solutions for the tracing package

Common issues and solutions when using the tracing package.

Traces Not Appearing

Symptom

No traces appear in your tracing backend (Jaeger, Zipkin, etc.) even though tracing is configured.

Possible Causes & Solutions

1. OTLP Provider Not Started

Problem: OTLP providers require calling Start(ctx) before tracing.

Solution:

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithOTLP("localhost:4317"),
)

// ✓ Required for OTLP providers
if err := tracer.Start(context.Background()); err != nil {
    log.Fatal(err)
}

2. Sampling Rate Too Low

Problem: Sample rate is set too low. For example, 1% sampling means 99% of requests aren’t traced.

Solution: Increase sample rate or remove sampling for testing.

// Development - trace everything
tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithSampleRate(1.0), // 100% sampling
)

3. Wrong Provider Configured

Problem: Using Noop provider (no traces exported).

Solution: Verify provider configuration:

// ✗ Bad - no traces exported
tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithNoop(), // No traces!
)

// ✓ Good - traces exported
tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithOTLP("localhost:4317"),
)

4. Paths Excluded from Tracing

Problem: Paths are excluded via middleware options.

Solution: Check middleware exclusions.

// Check if your paths are excluded
handler := tracing.Middleware(tracer,
    tracing.WithExcludePaths("/health", "/api/users"), // ← Is this excluding your endpoint?
)(mux)

5. Shutdown Called Too Early

Problem: Application exits before spans are exported.

Solution: Ensure proper shutdown with timeout:

defer func() {
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()
    
    if err := tracer.Shutdown(ctx); err != nil {
        log.Printf("Error shutting down tracer: %v", err)
    }
}()

6. OTLP Endpoint Unreachable

Problem: OTLP collector is not running or unreachable.

Solution: Verify collector is running:

# Check if collector is listening
nc -zv localhost 4317  # OTLP gRPC
nc -zv localhost 4318  # OTLP HTTP

Check logs for connection errors:

logger := slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{
    Level: slog.LevelDebug,
}))

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithOTLP("localhost:4317"),
    tracing.WithLogger(logger), // See connection errors
)

Context Propagation Issues

Symptom

Services create separate traces instead of one distributed trace.

Possible Causes & Solutions

1. Context Not Propagated

Problem: Context is not passed through the call chain.

Solution: Always pass context:

// ✓ Good - context propagates
func handler(ctx context.Context) {
    result := doWork(ctx)  // Pass context
}

// ✗ Bad - context lost
func handler(ctx context.Context) {
    result := doWork(context.Background())  // Lost!
}

2. Trace Context Not Injected

Problem: Trace context not injected into outgoing requests.

Solution: Always inject before making requests:

req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)

// ✓ Required - inject trace context
tracer.InjectTraceContext(ctx, req.Header)

resp, _ := http.DefaultClient.Do(req)

3. Trace Context Not Extracted

Problem: Incoming requests don’t extract trace context.

Solution: Middleware automatically extracts, or do it manually:

// Automatic (with middleware)
handler := tracing.Middleware(tracer)(mux)

// Manual (without middleware)
func myHandler(w http.ResponseWriter, r *http.Request) {
    ctx := tracer.ExtractTraceContext(r.Context(), r.Header)
    // Use extracted context...
}

4. Different Propagators

Problem: Services use different propagation formats.

Solution: Ensure all services use the same propagator (default is W3C Trace Context):

// All services should use default (W3C) or same custom propagator
tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    // Default propagator is W3C Trace Context
)

Performance Issues

Symptom

High CPU usage, increased latency, or memory consumption.

Possible Causes & Solutions

1. Too Much Sampling

Problem: Sampling 100% of high-traffic endpoints.

Solution: Reduce sample rate:

// For high-traffic services (> 1000 req/s)
tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithSampleRate(0.1), // 10% sampling
)

2. Not Excluding High-Frequency Endpoints

Problem: Tracing health checks and metrics endpoints.

Solution: Exclude them:

handler := tracing.Middleware(tracer,
    tracing.WithExcludePaths("/health", "/metrics", "/ready", "/live"),
)(mux)

3. Too Many Span Attributes

Problem: Adding excessive attributes to every span.

Solution: Only add essential attributes:

// ✓ Good - essential attributes
tracer.SetSpanAttribute(span, "user.id", userID)
tracer.SetSpanAttribute(span, "request.id", requestID)

// ✗ Bad - too many attributes
for k, v := range req.Header {
    tracer.SetSpanAttribute(span, k, v) // Don't do this!
}

4. Using Regex for Path Exclusion

Problem: Regex patterns are slower than exact paths.

Solution: Prefer exact paths or prefixes:

// ✓ Faster - O(1) hash lookup
tracing.WithExcludePaths("/health", "/metrics")

// ✗ Slower - O(p) regex matching
tracing.WithExcludePatterns("^/(health|metrics)$")

Configuration Errors

Multiple Providers Configured

Error: "validation errors: provider: multiple providers configured"

Problem: Attempting to configure multiple providers.

Solution: Only configure one provider:

// ✗ Error - multiple providers
tracer, err := tracing.New(
    tracing.WithServiceName("my-service"),
    tracing.WithStdout(),
    tracing.WithOTLP("localhost:4317"), // Error!
)

// ✓ Good - one provider
tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithOTLP("localhost:4317"),
)

Empty Service Name

Error: "invalid configuration: serviceName: cannot be empty"

Problem: Service name not provided or empty string.

Solution: Always provide a service name:

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"), // Required
)

Invalid Regex Pattern

Error: Middleware panics with "middleware validation errors: excludePatterns: invalid regex..."

Problem: Invalid regex pattern in WithExcludePatterns.

Solution: Validate regex patterns:

// ✗ Invalid regex
tracing.WithExcludePatterns(`[invalid`)

// ✓ Valid regex
tracing.WithExcludePatterns(`^/v[0-9]+/internal/.*`)

OTLP Connection Issues

TLS Certificate Errors

Problem: TLS certificate verification fails.

Solution: Use insecure connection for local development:

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithOTLP("localhost:4317", tracing.OTLPInsecure()),
)

For production, ensure proper TLS certificates are configured.

Connection Refused

Problem: Cannot connect to OTLP endpoint.

Solution:

  1. Verify collector is running:

    docker ps | grep otel-collector
    
  2. Check endpoint is correct:

    // Correct format: "host:port"
    tracing.WithOTLP("localhost:4317")
    
    // Not: "http://localhost:4317" (no protocol for gRPC)
    
  3. Check network connectivity:

    telnet localhost 4317
    

Wrong Endpoint for HTTP

Problem: Using gRPC endpoint for HTTP or vice versa.

Solution: Use correct provider and endpoint:

// OTLP gRPC (port 4317)
tracing.WithOTLP("localhost:4317")

// OTLP HTTP (port 4318, include protocol)
tracing.WithOTLPHTTP("http://localhost:4318")

Middleware Issues

Spans Not Created

Problem: Middleware doesn’t create spans for requests.

Solution: Ensure middleware is applied:

mux := http.NewServeMux()
mux.HandleFunc("/api/users", handleUsers)

// ✓ Middleware applied
handler := tracing.Middleware(tracer)(mux)
http.ListenAndServe(":8080", handler)

// ✗ Middleware not applied
http.ListenAndServe(":8080", mux) // No tracing!

Context Lost in Handlers

Problem: Context doesn’t contain trace information.

Solution: Use context from request:

func handleUsers(w http.ResponseWriter, r *http.Request) {
    // ✓ Good - use request context
    ctx := r.Context()
    traceID := tracing.TraceID(ctx)
    
    // ✗ Bad - creates new context
    ctx := context.Background() // Lost trace context!
}

Testing Issues

Tests Fail to Clean Up

Problem: Tests hang or don’t complete cleanup.

Solution: Use testing utilities:

func TestSomething(t *testing.T) {
    // ✓ Good - automatic cleanup
    tracer := tracing.TestingTracer(t)
    
    // ✗ Bad - manual cleanup required
    tracer, _ := tracing.New(tracing.WithNoop())
    defer tracer.Shutdown(context.Background())
}

Race Conditions in Tests

Problem: Race detector reports issues in parallel tests.

Solution: Use t.Parallel() correctly:

func TestParallel(t *testing.T) {
    t.Parallel() // Each test gets its own tracer
    
    tracer := tracing.TestingTracer(t)
    // Use tracer...
}

Debugging Tips

Enable Debug Logging

See internal events:

logger := slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{
    Level: slog.LevelDebug,
}))

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithLogger(logger),
)

Use Stdout Provider

See traces immediately:

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithStdout(), // Print traces to console
)

Check Trace IDs

Verify trace context is propagated:

func handleRequest(ctx context.Context) {
    traceID := tracing.TraceID(ctx)
    log.Printf("Processing request [trace=%s]", traceID)
    
    if traceID == "" {
        log.Printf("WARNING: No trace context!")
    }
}

Verify Sampling

Log sampling decisions:

startHook := func(ctx context.Context, span trace.Span, req *http.Request) {
    if span.SpanContext().IsValid() {
        log.Printf("Request sampled: %s", req.URL.Path)
    } else {
        log.Printf("Request not sampled: %s", req.URL.Path)
    }
}

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithSpanStartHook(startHook),
)

Getting Help

Check Documentation

Check Logs

Enable debug logging to see internal events:

tracing.WithLogger(slog.Default())

Verify Configuration

Print configuration at startup:

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithOTLP("localhost:4317"),
    tracing.WithSampleRate(0.1),
)

log.Printf("Tracer configured:")
log.Printf("  Service: %s", tracer.ServiceName())
log.Printf("  Version: %s", tracer.ServiceVersion())
log.Printf("  Provider: %s", tracer.GetProvider())
log.Printf("  Enabled: %v", tracer.IsEnabled())

Common Pitfalls Checklist

  • Called Start() for OTLP providers?
  • Shutdown with proper timeout?
  • Context propagated through call chain?
  • Trace context injected into outgoing requests?
  • Sample rate high enough to see traces?
  • Paths not excluded from tracing?
  • OTLP collector running and reachable?
  • All services using same propagator?
  • Only one provider configured?

Version Compatibility

Go Version

Minimum required: Go 1.25+

Error: go: module requires Go 1.25 or later

Solution: Upgrade Go version:

go version  # Check current version
# Upgrade to Go 1.25+

OpenTelemetry Version

The tracing package uses OpenTelemetry SDK v1.x. If you have conflicts with other dependencies:

go mod tidy
go get -u rivaas.dev/tracing

Next Steps