This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Distributed Tracing

Learn how to implement distributed tracing with Rivaas tracing package

The Rivaas Tracing package provides OpenTelemetry-based distributed tracing. Supports various exporters and integrates with HTTP frameworks. Enables observability best practices with minimal configuration.

Features

  • OpenTelemetry Integration: Full OpenTelemetry tracing support
  • Context Propagation: Automatic trace context propagation across services
  • Span Management: Easy span creation and management with lifecycle hooks
  • HTTP Middleware: Standalone middleware for any HTTP framework
  • Multiple Providers: Stdout, OTLP (gRPC and HTTP), and Noop exporters
  • Path Filtering: Exclude specific paths from tracing via middleware options
  • Consistent API: Same design patterns as the metrics package
  • Thread-Safe: All operations safe for concurrent use

Quick Start

package main

import (
    "context"
    "log"
    "os/signal"
    
    "rivaas.dev/tracing"
)

func main() {
    ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt)
    defer cancel()

    tracer, err := tracing.New(
        tracing.WithServiceName("my-service"),
        tracing.WithServiceVersion("v1.0.0"),
        tracing.WithOTLP("localhost:4317"),
    )
    if err != nil {
        log.Fatal(err)
    }
    
    if err := tracer.Start(ctx); err != nil {
        log.Fatal(err)
    }
    defer tracer.Shutdown(context.Background())

    // Traces exported via OTLP gRPC
    ctx, span := tracer.StartSpan(ctx, "operation")
    defer tracer.FinishSpan(span, 200)
}
package main

import (
    "context"
    "log"
    "os/signal"
    
    "rivaas.dev/tracing"
)

func main() {
    ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt)
    defer cancel()

    tracer, err := tracing.New(
        tracing.WithServiceName("my-service"),
        tracing.WithServiceVersion("v1.0.0"),
        tracing.WithOTLPHTTP("http://localhost:4318"),
    )
    if err != nil {
        log.Fatal(err)
    }
    
    if err := tracer.Start(ctx); err != nil {
        log.Fatal(err)
    }
    defer tracer.Shutdown(context.Background())

    // Traces exported via OTLP HTTP
    ctx, span := tracer.StartSpan(ctx, "operation")
    defer tracer.FinishSpan(span, 200)
}
package main

import (
    "context"
    
    "rivaas.dev/tracing"
)

func main() {
    tracer := tracing.MustNew(
        tracing.WithServiceName("my-service"),
        tracing.WithStdout(),
    )
    defer tracer.Shutdown(context.Background())

    ctx := context.Background()
    
    // Traces printed to stdout
    ctx, span := tracer.StartSpan(ctx, "operation")
    defer tracer.FinishSpan(span, 200)
}

How It Works

  • Providers determine where traces are exported (Stdout, OTLP, Noop)
  • Lifecycle management ensures proper initialization and graceful shutdown
  • HTTP middleware creates spans for requests automatically
  • Custom spans can be created for detailed operation tracing
  • Context propagation enables distributed tracing across services

Learning Path

Follow these guides to learn distributed tracing with Rivaas:

  1. Installation - Get started with the tracing package
  2. Basic Usage - Learn tracer creation and span management
  3. Providers - Understand Stdout, OTLP, and Noop exporters
  4. Configuration - Configure service metadata, sampling, and hooks
  5. Middleware - Integrate HTTP tracing with your application
  6. Context Propagation - Propagate traces across services
  7. Testing - Test your tracing with provided utilities
  8. Examples - See real-world usage patterns

Next Steps

1 - Installation

Install and set up the tracing package

Get started with the Rivaas tracing package by installing it in your Go project.

Requirements

  • Go 1.25 or higher - The tracing package uses modern Go features
  • OpenTelemetry dependencies - Automatically installed via go get

Install the Package

Add the tracing package to your Go module:

go get rivaas.dev/tracing

This will download the package and its OpenTelemetry dependencies.

Verify Installation

Create a simple test file to verify the installation:

package main

import (
    "context"
    "log"
    
    "rivaas.dev/tracing"
)

func main() {
    tracer, err := tracing.New(
        tracing.WithServiceName("test-service"),
        tracing.WithStdout(),
    )
    if err != nil {
        log.Fatal(err)
    }
    defer tracer.Shutdown(context.Background())
    
    log.Println("Tracing initialized successfully!")
}

Run the test:

go run main.go

You should see a success message with no errors.

Dependencies

The tracing package depends on:

  • go.opentelemetry.io/otel - OpenTelemetry API
  • go.opentelemetry.io/otel/sdk - OpenTelemetry SDK
  • go.opentelemetry.io/otel/exporters/stdout/stdouttrace - Stdout exporter
  • go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc - OTLP gRPC exporter
  • go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp - OTLP HTTP exporter

These are automatically installed when you run go get rivaas.dev/tracing.

Module Setup

If you’re starting a new project, initialize a Go module first:

mkdir my-traced-app
cd my-traced-app
go mod init example.com/my-traced-app
go get rivaas.dev/tracing

Next Steps

Now that you have the package installed:

2 - Basic Usage

Learn the fundamentals of creating tracers and managing spans

Learn how to create tracers, manage spans, and add tracing to your Go applications.

Creating a Tracer

The Tracer is the main entry point for distributed tracing. Create one using functional options:

With Error Handling

tracer, err := tracing.New(
    tracing.WithServiceName("my-service"),
    tracing.WithServiceVersion("v1.0.0"),
    tracing.WithStdout(),
)
if err != nil {
    log.Fatalf("Failed to create tracer: %v", err)
}
defer tracer.Shutdown(context.Background())

Panic on Error

For convenience, use MustNew which panics if initialization fails:

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithServiceVersion("v1.0.0"),
    tracing.WithStdout(),
)
defer tracer.Shutdown(context.Background())

Tracer Lifecycle

Starting the Tracer

For OTLP providers (gRPC and HTTP), you must call Start() before tracing:

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithOTLP("localhost:4317"),
)

// Start is required for OTLP providers
if err := tracer.Start(context.Background()); err != nil {
    log.Fatal(err)
}

defer tracer.Shutdown(context.Background())

Shutting Down

Always shut down the tracer to flush pending spans:

defer func() {
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()
    
    if err := tracer.Shutdown(ctx); err != nil {
        log.Printf("Error shutting down tracer: %v", err)
    }
}()

Manual Span Management

Create and manage spans manually for detailed tracing:

Basic Span Creation

func processData(ctx context.Context, tracer *tracing.Tracer) {
    // Start a span
    ctx, span := tracer.StartSpan(ctx, "process-data")
    defer tracer.FinishSpan(span, http.StatusOK)
    
    // Your code here...
}

Adding Attributes

Add attributes to provide context about the operation:

ctx, span := tracer.StartSpan(ctx, "database-query")
defer tracer.FinishSpan(span, http.StatusOK)

// Add attributes
tracer.SetSpanAttribute(span, "db.system", "postgresql")
tracer.SetSpanAttribute(span, "db.query", "SELECT * FROM users")
tracer.SetSpanAttribute(span, "db.rows_returned", 42)

Supported attribute types:

  • string
  • int, int64
  • float64
  • bool
  • Other types (converted to string)

Adding Events

Record significant moments in a span’s lifetime:

import "go.opentelemetry.io/otel/attribute"

ctx, span := tracer.StartSpan(ctx, "cache-lookup")
defer tracer.FinishSpan(span, http.StatusOK)

// Add an event
tracer.AddSpanEvent(span, "cache_hit",
    attribute.String("key", "user:123"),
    attribute.Int("ttl_seconds", 300),
)

Error Handling

Use the status code to indicate span success or failure:

func fetchUser(ctx context.Context, tracer *tracing.Tracer, userID string) error {
    ctx, span := tracer.StartSpan(ctx, "fetch-user")
    defer func() {
        if err != nil {
            tracer.FinishSpan(span, http.StatusInternalServerError)
        } else {
            tracer.FinishSpan(span, http.StatusOK)
        }
    }()
    
    tracer.SetSpanAttribute(span, "user.id", userID)
    
    // Fetch user logic...
    return nil
}

Context Helpers

Work with spans through the context without direct span references:

Set Attributes from Context

func handleRequest(ctx context.Context) {
    // Add attribute to the current span in context
    tracing.SetSpanAttributeFromContext(ctx, "user.role", "admin")
    tracing.SetSpanAttributeFromContext(ctx, "user.id", 12345)
}

Add Events from Context

func processEvent(ctx context.Context) {
    // Add event to the current span in context
    tracing.AddSpanEventFromContext(ctx, "event_processed",
        attribute.String("event_type", "user_login"),
        attribute.String("ip_address", "192.168.1.1"),
    )
}

Get Trace Information

func logWithTraceInfo(ctx context.Context) {
    traceID := tracing.TraceID(ctx)
    spanID := tracing.SpanID(ctx)
    
    log.Printf("Processing request [trace=%s, span=%s]", traceID, spanID)
}

Complete Example

Here’s a complete example showing manual span management:

package main

import (
    "context"
    "log"
    "time"
    
    "go.opentelemetry.io/otel/attribute"
    "rivaas.dev/tracing"
)

func main() {
    tracer := tracing.MustNew(
        tracing.WithServiceName("example-service"),
        tracing.WithStdout(),
    )
    defer tracer.Shutdown(context.Background())
    
    ctx := context.Background()
    
    // Parent span
    ctx, parentSpan := tracer.StartSpan(ctx, "process-order")
    defer tracer.FinishSpan(parentSpan, 200)
    
    tracer.SetSpanAttribute(parentSpan, "order.id", "12345")
    
    // Child span 1
    validateOrder(ctx, tracer)
    
    // Child span 2
    chargePayment(ctx, tracer)
    
    log.Println("Order processed successfully")
}

func validateOrder(ctx context.Context, tracer *tracing.Tracer) {
    ctx, span := tracer.StartSpan(ctx, "validate-order")
    defer tracer.FinishSpan(span, 200)
    
    tracer.SetSpanAttribute(span, "validation.status", "passed")
    tracer.AddSpanEvent(span, "validation_complete")
    
    time.Sleep(10 * time.Millisecond) // Simulate work
}

func chargePayment(ctx context.Context, tracer *tracing.Tracer) {
    ctx, span := tracer.StartSpan(ctx, "charge-payment")
    defer tracer.FinishSpan(span, 200)
    
    tracer.SetSpanAttribute(span, "payment.amount", 99.99)
    tracer.SetSpanAttribute(span, "payment.method", "credit_card")
    
    tracer.AddSpanEvent(span, "payment_authorized",
        attribute.String("authorization_code", "AUTH123"),
    )
    
    time.Sleep(20 * time.Millisecond) // Simulate work
}

Best Practices

Always Close Spans

Use defer to ensure spans are finished even if errors occur:

ctx, span := tracer.StartSpan(ctx, "operation")
defer tracer.FinishSpan(span, http.StatusOK) // Always close

Propagate Context

Always pass the context returned by StartSpan to child operations:

ctx, span := tracer.StartSpan(ctx, "parent")
defer tracer.FinishSpan(span, http.StatusOK)

// Pass the new context to children
childOperation(ctx) // ✓ Correct
childOperation(oldCtx) // ✗ Wrong - breaks trace chain

Use Descriptive Names

Choose clear, consistent span names:

// Good
tracer.StartSpan(ctx, "database-query")
tracer.StartSpan(ctx, "validate-user-input")
tracer.StartSpan(ctx, "send-email")

// Bad
tracer.StartSpan(ctx, "query")
tracer.StartSpan(ctx, "func1")
tracer.StartSpan(ctx, "DoStuff")

Add Meaningful Attributes

Include relevant information as attributes:

ctx, span := tracer.StartSpan(ctx, "api-call")
defer tracer.FinishSpan(span, statusCode)

tracer.SetSpanAttribute(span, "http.method", "POST")
tracer.SetSpanAttribute(span, "http.url", "/api/users")
tracer.SetSpanAttribute(span, "api.endpoint", "create_user")
tracer.SetSpanAttribute(span, "user.role", "admin")

Next Steps

3 - Tracing Providers

Choose and configure trace exporters for your application

The tracing package supports multiple providers for exporting traces. Choose the provider that best fits your environment and infrastructure.

Available Providers

ProviderUse CaseNetwork RequiredBest For
NoopDefault, no tracesNoTesting, disabled tracing
StdoutConsole outputNoDevelopment, debugging
OTLP (gRPC)OpenTelemetry collectorYesProduction (preferred)
OTLP (HTTP)OpenTelemetry collectorYesProduction (alternative)

Basic Configuration

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithNoop(),
)
defer tracer.Shutdown(context.Background())
tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithStdout(),
)
defer tracer.Shutdown(context.Background())
tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithServiceVersion("v1.0.0"),
    tracing.WithOTLP("localhost:4317"),
)
if err := tracer.Start(context.Background()); err != nil {
    log.Fatal(err)
}
defer tracer.Shutdown(context.Background())
tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithServiceVersion("v1.0.0"),
    tracing.WithOTLPHTTP("http://localhost:4318"),
)
if err := tracer.Start(context.Background()); err != nil {
    log.Fatal(err)
}
defer tracer.Shutdown(context.Background())

Noop Provider

The noop provider doesn’t export any traces. It’s the default when no provider is configured.

When to Use

  • Testing environments where tracing isn’t needed
  • Temporarily disabling tracing without code changes
  • Safe default for new projects

Configuration

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithNoop(),
)
defer tracer.Shutdown(context.Background())

Or simply omit the provider option (noop is the default):

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    // No provider = Noop
)

Behavior

  • Spans are created but not recorded
  • No network calls or file I/O
  • Minimal performance overhead
  • Safe for production if tracing is disabled

Stdout Provider

The stdout provider prints traces to standard output in a human-readable format.

When to Use

  • Local development and debugging
  • Troubleshooting span creation and attributes
  • Testing trace propagation
  • Quick validation of tracing logic

Configuration

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithStdout(),
)
defer tracer.Shutdown(context.Background())

Output Format

Traces are printed as pretty-printed JSON to stdout:

{
  "Name": "GET /api/users",
  "SpanContext": {
    "TraceID": "3f3c5e4d...",
    "SpanID": "a1b2c3d4...",
    "TraceFlags": "01"
  },
  "Parent": {
    "TraceID": "3f3c5e4d...",
    "SpanID": "e5f6g7h8..."
  },
  "SpanKind": "Server",
  "StartTime": "2025-01-18T10:15:30.123Z",
  "EndTime": "2025-01-18T10:15:30.456Z",
  "Attributes": [
    {
      "Key": "http.method",
      "Value": {"Type": "STRING", "Value": "GET"}
    }
  ]
}

Limitations

  • Not for production: Output can be noisy and slow
  • No persistence: Traces are only printed, not stored
  • No visualization: Use an actual backend for trace visualization

OTLP Provider (gRPC)

The OTLP gRPC provider exports traces to an OpenTelemetry collector using the gRPC protocol.

When to Use

  • Production environments
  • OpenTelemetry collector infrastructure
  • Jaeger, Zipkin, or other OTLP-compatible backends
  • Best performance and reliability

Basic Configuration

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithServiceVersion("v1.0.0"),
    tracing.WithOTLP("localhost:4317"),
)

// Start is required for OTLP providers
if err := tracer.Start(context.Background()); err != nil {
    log.Fatal(err)
}

defer tracer.Shutdown(context.Background())

Secure Connection (TLS)

By default, OTLP uses TLS:

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithOTLP("collector.example.com:4317"),
    // TLS is enabled by default
)

Insecure Connection (Development)

For local development without TLS:

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithOTLP("localhost:4317", tracing.OTLPInsecure()),
)

Configuration Options

import "rivaas.dev/tracing"

// Secure (production)
tracing.WithOTLP("collector.example.com:4317")

// Insecure (development)
tracing.WithOTLP("localhost:4317", tracing.OTLPInsecure())

OTLP Provider (HTTP)

The OTLP HTTP provider exports traces to an OpenTelemetry collector using the HTTP protocol.

When to Use

  • Alternative to gRPC when firewalls block gRPC
  • Simpler infrastructure without gRPC support
  • HTTP-only environments
  • Debugging with curl/httpie

Configuration

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithServiceVersion("v1.0.0"),
    tracing.WithOTLPHTTP("http://localhost:4318"),
)

// Start is required for OTLP providers
if err := tracer.Start(context.Background()); err != nil {
    log.Fatal(err)
}

defer tracer.Shutdown(context.Background())

HTTPS Endpoint

Use HTTPS for secure connections:

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithOTLPHTTP("https://collector.example.com:4318"),
)

Endpoint Format

The endpoint should include the protocol:

// HTTP (insecure - development only)
tracing.WithOTLPHTTP("http://localhost:4318")

// HTTPS (secure - production)
tracing.WithOTLPHTTP("https://collector.example.com:4318")

Provider Comparison

Performance

ProviderLatencyThroughputCPUMemory
Noop~10nsUnlimitedMinimalMinimal
Stdout~100µsLowLowLow
OTLP (gRPC)~1-2msHighLowMedium
OTLP (HTTP)~2-3msMediumLowMedium

Use Case Matrix

// Development
tracer := tracing.MustNew(
    tracing.WithServiceName("my-api"),
    tracing.WithStdout(), // ← See traces in console
)

// Testing
tracer := tracing.MustNew(
    tracing.WithServiceName("my-api"),
    tracing.WithNoop(), // ← No tracing overhead
)

// Production (recommended)
tracer := tracing.MustNew(
    tracing.WithServiceName("my-api"),
    tracing.WithOTLP("collector:4317"), // ← gRPC to collector
)

// Production (HTTP alternative)
tracer := tracing.MustNew(
    tracing.WithServiceName("my-api"),
    tracing.WithOTLPHTTP("https://collector:4318"), // ← HTTP to collector
)

Switching Providers

Only one provider can be configured at a time. Attempting to configure multiple providers results in a validation error:

// ✗ Error: multiple providers configured
tracer, err := tracing.New(
    tracing.WithServiceName("my-service"),
    tracing.WithStdout(),
    tracing.WithOTLP("localhost:4317"), // Error!
)
// Returns: "validation errors: provider: multiple providers configured"

To switch providers, use environment variables or configuration:

func createTracer(env string) *tracing.Tracer {
    opts := []tracing.Option{
        tracing.WithServiceName("my-service"),
        tracing.WithServiceVersion("v1.0.0"),
    }
    
    switch env {
    case "production":
        opts = append(opts, tracing.WithOTLP("collector:4317"))
    case "development":
        opts = append(opts, tracing.WithStdout())
    default:
        opts = append(opts, tracing.WithNoop())
    }
    
    return tracing.MustNew(opts...)
}

OpenTelemetry Collector Setup

For OTLP providers, you need an OpenTelemetry collector.

Docker Compose Example

version: '3.8'
services:
  otel-collector:
    image: otel/opentelemetry-collector:latest
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP
      - "13133:13133" # health_check

Collector Configuration

Basic otel-collector-config.yaml:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:

exporters:
  logging:
    loglevel: debug
  # Add your backend (Jaeger, Zipkin, etc.)
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, jaeger]

Provider Selection Guide

Choose Noop When:

  • Tracing is disabled via feature flags
  • Running in CI/CD without trace backend
  • Performance testing without observability overhead

Choose Stdout When:

  • Developing locally and need to see traces
  • Debugging span creation and attributes
  • Quick validation of tracing setup

Choose OTLP (gRPC) When:

  • Deploying to production
  • Need high throughput and low latency
  • Using OpenTelemetry collector
  • Standard production setup

Choose OTLP (HTTP) When:

  • gRPC is blocked by firewalls
  • Simpler infrastructure requirements
  • Need HTTP-friendly debugging
  • Backend only supports HTTP

Next Steps

  • Learn Configuration options for service metadata and sampling
  • Set up Middleware for automatic HTTP tracing
  • Explore Examples for production-ready configurations

4 - Configuration

Configure service metadata, sampling, hooks, and logging

Configure your tracer with service information, sampling rates, lifecycle hooks, and logging integration.

Service Configuration

Set service metadata that appears in every span.

Service Name

The service name identifies your application in traces:

tracer := tracing.MustNew(
    tracing.WithServiceName("user-api"),
    tracing.WithStdout(),
)

Best practices:

  • Use descriptive, consistent names across services.
  • Use kebab-case: user-api, order-service, payment-gateway.
  • Avoid generic names like api or service.

Service Version

Track which version of your service created traces:

tracer := tracing.MustNew(
    tracing.WithServiceName("user-api"),
    tracing.WithServiceVersion("v1.2.3"),
    tracing.WithStdout(),
)

Best practices:

  • Use semantic versioning: v1.2.3.
  • Include in CI/CD builds.
  • Track version across deployments.

Combined Example

tracer := tracing.MustNew(
    tracing.WithServiceName("user-api"),
    tracing.WithServiceVersion("v1.2.3"),
    tracing.WithOTLP("collector:4317"),
)

These attributes appear in every span:

  • service.name: "user-api"
  • service.version: "v1.2.3"

Sampling Configuration

Control which requests are traced to reduce overhead and costs.

Sample Rate

Set the percentage of requests to trace (0.0 to 1.0):

tracer := tracing.MustNew(
    tracing.WithServiceName("my-api"),
    tracing.WithSampleRate(0.1), // Trace 10% of requests
    tracing.WithOTLP("collector:4317"),
)

Sample rates:

  • 1.0: 100% sampling. All requests traced.
  • 0.5: 50% sampling.
  • 0.1: 10% sampling.
  • 0.01: 1% sampling.
  • 0.0: 0% sampling (no traces)

Sampling Examples

// Development: trace everything
tracer := tracing.MustNew(
    tracing.WithServiceName("my-api"),
    tracing.WithSampleRate(1.0),
    tracing.WithStdout(),
)

// Production: trace 10% of requests
tracer := tracing.MustNew(
    tracing.WithServiceName("my-api"),
    tracing.WithSampleRate(0.1),
    tracing.WithOTLP("collector:4317"),
)

// High-traffic: trace 1% of requests
tracer := tracing.MustNew(
    tracing.WithServiceName("my-api"),
    tracing.WithSampleRate(0.01),
    tracing.WithOTLP("collector:4317"),
)

Sampling Behavior

  • Probabilistic: Uses deterministic hashing for consistent sampling
  • Request-level: Decision made once per request, all child spans included
  • Zero overhead: Non-sampled requests skip span creation entirely

When to Sample

Traffic LevelRecommended Sample Rate
< 100 req/s1.0 (100%)
100-1000 req/s0.5 (50%)
1000-10000 req/s0.1 (10%)
> 10000 req/s0.01 (1%)

Adjust based on:

  • Trace backend capacity
  • Storage costs
  • Desired trace coverage
  • Debug vs production needs

Span Lifecycle Hooks

Add custom logic when spans start or finish.

Span Start Hook

Execute code when a request span is created:

import (
    "context"
    "net/http"
    
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/trace"
    "rivaas.dev/tracing"
)

startHook := func(ctx context.Context, span trace.Span, req *http.Request) {
    // Add custom attributes
    if tenantID := req.Header.Get("X-Tenant-ID"); tenantID != "" {
        span.SetAttributes(attribute.String("tenant.id", tenantID))
    }
    
    // Add user information
    if userID := req.Header.Get("X-User-ID"); userID != "" {
        span.SetAttributes(attribute.String("user.id", userID))
    }
    
    // Record custom business context
    span.SetAttributes(
        attribute.String("request.region", getRegionFromIP(req)),
        attribute.Bool("request.is_mobile", isMobileRequest(req)),
    )
}

tracer := tracing.MustNew(
    tracing.WithServiceName("my-api"),
    tracing.WithSpanStartHook(startHook),
    tracing.WithOTLP("collector:4317"),
)

Use cases:

  • Add tenant/user identifiers
  • Record business context
  • Integrate with feature flags
  • Custom sampling decisions
  • APM tool integration

Span Finish Hook

Execute code when a request span completes:

import (
    "go.opentelemetry.io/otel/trace"
    "rivaas.dev/tracing"
)

finishHook := func(span trace.Span, statusCode int) {
    // Record custom metrics
    if statusCode >= 500 {
        metrics.IncrementServerErrors()
    }
    
    // Log slow requests
    if span.SpanContext().IsValid() {
        // Calculate duration and log if > threshold
    }
    
    // Send alerts for errors
    if statusCode >= 500 {
        alerting.SendAlert("Server error", statusCode)
    }
}

tracer := tracing.MustNew(
    tracing.WithServiceName("my-api"),
    tracing.WithSpanFinishHook(finishHook),
    tracing.WithOTLP("collector:4317"),
)

Use cases:

  • Record custom metrics
  • Log slow requests
  • Send error alerts
  • Update counters
  • Cleanup resources

Combined Hooks Example

tracer := tracing.MustNew(
    tracing.WithServiceName("my-api"),
    tracing.WithSpanStartHook(func(ctx context.Context, span trace.Span, req *http.Request) {
        // Enrich span with business context
        span.SetAttributes(
            attribute.String("tenant.id", extractTenant(req)),
            attribute.String("feature.flags", getFeatureFlags(req)),
        )
    }),
    tracing.WithSpanFinishHook(func(span trace.Span, statusCode int) {
        // Record completion metrics
        recordRequestMetrics(statusCode)
    }),
    tracing.WithOTLP("collector:4317"),
)

Logging Integration

Integrate tracing with your logging infrastructure.

Using slog

Use Go’s standard log/slog package:

import (
    "log/slog"
    "os"
    
    "rivaas.dev/tracing"
)

logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
    Level: slog.LevelDebug,
}))

tracer := tracing.MustNew(
    tracing.WithServiceName("my-api"),
    tracing.WithLogger(logger),
    tracing.WithOTLP("collector:4317"),
)

The logger receives internal tracing events:

  • Tracer initialization
  • Provider startup/shutdown
  • Configuration warnings
  • Error conditions

Event Levels

Events are logged at appropriate levels:

Event TypeLog LevelExample
ErrorERROR“Failed to export spans”
WarningWARN“OTLP endpoint not specified”
InfoINFO“Tracing initialized”
DebugDEBUG“Request not sampled”

Custom Event Handler

For non-slog logging or custom event handling:

import "rivaas.dev/tracing"

eventHandler := func(e tracing.Event) {
    switch e.Type {
    case tracing.EventError:
        // Send to error tracking (e.g., Sentry)
        sentry.CaptureMessage(e.Message)
        myLogger.Error(e.Message, e.Args...)
    case tracing.EventWarning:
        myLogger.Warn(e.Message, e.Args...)
    case tracing.EventInfo:
        myLogger.Info(e.Message, e.Args...)
    case tracing.EventDebug:
        myLogger.Debug(e.Message, e.Args...)
    }
}

tracer := tracing.MustNew(
    tracing.WithServiceName("my-api"),
    tracing.WithEventHandler(eventHandler),
    tracing.WithOTLP("collector:4317"),
)

Use cases:

  • Integrate with non-slog loggers (zap, zerolog, logrus)
  • Send errors to Sentry/Rollbar
  • Custom alerting
  • Audit logging
  • Metrics from events

No Logging

To disable all internal logging:

tracer := tracing.MustNew(
    tracing.WithServiceName("my-api"),
    // No WithLogger or WithEventHandler = no logging
    tracing.WithOTLP("collector:4317"),
)

Advanced Configuration

Custom Propagator

Use a custom trace context propagation format:

import (
    "go.opentelemetry.io/otel/propagation"
    "rivaas.dev/tracing"
)

// Use B3 propagation format (Zipkin)
b3Propagator := propagation.NewCompositeTextMapPropagator(
    propagation.B3{},
)

tracer := tracing.MustNew(
    tracing.WithServiceName("my-api"),
    tracing.WithCustomPropagator(b3Propagator),
    tracing.WithOTLP("collector:4317"),
)

Custom Tracer Provider

Provide your own OpenTelemetry tracer provider:

import (
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    "rivaas.dev/tracing"
)

// Create custom tracer provider
tp := sdktrace.NewTracerProvider(
    // Your custom configuration
    sdktrace.WithSampler(sdktrace.AlwaysSample()),
)

tracer := tracing.MustNew(
    tracing.WithServiceName("my-api"),
    tracing.WithTracerProvider(tp),
)

// You manage tp.Shutdown() yourself
defer tp.Shutdown(context.Background())

Note: When using WithTracerProvider, you’re responsible for shutting down the provider.

Global Tracer Provider

Register as the global OpenTelemetry tracer provider:

tracer := tracing.MustNew(
    tracing.WithServiceName("my-api"),
    tracing.WithOTLP("collector:4317"),
    tracing.WithGlobalTracerProvider(), // Register globally
)

By default, tracers are not registered globally. Use this option when:

  • You want otel.GetTracerProvider() to return your tracer
  • Integrating with libraries that use the global tracer
  • Single tracer for entire application

Configuration Patterns

Development Configuration

tracer := tracing.MustNew(
    tracing.WithServiceName("my-api"),
    tracing.WithServiceVersion("dev"),
    tracing.WithStdout(),
    tracing.WithSampleRate(1.0), // Trace everything
    tracing.WithLogger(slog.Default()),
)

Production Configuration

tracer := tracing.MustNew(
    tracing.WithServiceName("user-api"),
    tracing.WithServiceVersion(version), // From build
    tracing.WithOTLP(otlpEndpoint),      // From env
    tracing.WithSampleRate(0.1),         // 10% sampling
    tracing.WithSpanStartHook(enrichSpan),
    tracing.WithSpanFinishHook(recordMetrics),
)

Environment-Based Configuration

func createTracer(env string) *tracing.Tracer {
    opts := []tracing.Option{
        tracing.WithServiceName("my-api"),
        tracing.WithServiceVersion(getVersion()),
    }
    
    switch env {
    case "production":
        opts = append(opts,
            tracing.WithOTLP(os.Getenv("OTLP_ENDPOINT")),
            tracing.WithSampleRate(0.1),
        )
    case "staging":
        opts = append(opts,
            tracing.WithOTLP(os.Getenv("OTLP_ENDPOINT")),
            tracing.WithSampleRate(0.5),
        )
    default: // development
        opts = append(opts,
            tracing.WithStdout(),
            tracing.WithSampleRate(1.0),
            tracing.WithLogger(slog.Default()),
        )
    }
    
    return tracing.MustNew(opts...)
}

Next Steps

5 - HTTP Middleware

Automatically trace HTTP requests with middleware

The tracing package provides HTTP middleware for automatic request tracing with any HTTP framework.

Basic Usage

Wrap your HTTP handler with tracing middleware:

import (
    "net/http"
    "rivaas.dev/tracing"
)

func main() {
    tracer := tracing.MustNew(
        tracing.WithServiceName("my-api"),
        tracing.WithOTLP("localhost:4317"),
    )
    tracer.Start(context.Background())
    defer tracer.Shutdown(context.Background())

    mux := http.NewServeMux()
    mux.HandleFunc("/api/users", handleUsers)
    
    // Wrap with middleware
    handler := tracing.Middleware(tracer)(mux)
    
    http.ListenAndServe(":8080", handler)
}

Middleware Functions

Two functions are available for creating middleware:

Middleware (Panics on Error)

handler := tracing.Middleware(tracer,
    tracing.WithExcludePaths("/health"),
)(mux)

Panics if middleware options are invalid (e.g., invalid regex pattern).

MustMiddleware (Alias)

handler := tracing.MustMiddleware(tracer,
    tracing.WithExcludePaths("/health"),
)(mux)

Identical to Middleware() - provided for API consistency with MustNew().

What Gets Traced

The middleware automatically:

  1. Extracts trace context from incoming request headers.
  2. Creates a span for the request with standard attributes.
  3. Propagates context to downstream handlers.
  4. Records HTTP method, URL, status code, and duration.
  5. Finishes the span when the request completes.

Standard Attributes

Every traced request includes:

AttributeDescriptionExample
http.methodHTTP method"GET"
http.urlFull URL"http://localhost:8080/api/users"
http.schemeURL scheme"http"
http.hostHost header"localhost:8080"
http.routeRequest path"/api/users"
http.user_agentUser agent"Mozilla/5.0..."
http.status_codeResponse status200
service.nameService name"my-api"
service.versionService version"v1.0.0"

Path Exclusion

Exclude specific paths from tracing to reduce noise and overhead.

Exact Path Matching

Exclude specific paths exactly:

handler := tracing.Middleware(tracer,
    tracing.WithExcludePaths("/health", "/metrics", "/ready"),
)(mux)

Requests to /health, /metrics, or /ready won’t create spans.

Prefix Matching

Exclude all paths with a given prefix:

handler := tracing.Middleware(tracer,
    tracing.WithExcludePrefixes("/debug/", "/internal/", "/.well-known/"),
)(mux)

Excludes:

  • /debug/pprof
  • /debug/vars
  • /internal/health
  • /.well-known/acme-challenge

Regex Pattern Matching

Exclude paths matching regex patterns:

handler := tracing.Middleware(tracer,
    tracing.WithExcludePatterns(
        `^/v[0-9]+/internal/.*`,  // Version-prefixed internal routes
        `^/api/health.*`,          // Any health-related endpoint
    ),
)(mux)

Important: Invalid regex patterns cause the middleware to panic during initialization.

Combined Exclusions

Use multiple exclusion types together:

handler := tracing.Middleware(tracer,
    // Exact paths
    tracing.WithExcludePaths("/health", "/metrics"),
    
    // Prefixes
    tracing.WithExcludePrefixes("/debug/", "/internal/"),
    
    // Patterns
    tracing.WithExcludePatterns(`^/v[0-9]+/internal/.*`),
)(mux)

Performance

Path exclusion is highly efficient:

  • Exact paths: O(1) hash map lookup
  • Prefixes: O(n) where n = number of prefixes
  • Patterns: O(p) where p = number of patterns

Even with 100+ excluded paths, overhead is negligible (~9ns per request).

Header Recording

Record specific request headers as span attributes.

Basic Header Recording

handler := tracing.Middleware(tracer,
    tracing.WithHeaders("X-Request-ID", "X-Correlation-ID"),
)(mux)

Headers are recorded as: http.request.header.{name}

Example span attributes:

  • http.request.header.x-request-id: "abc123"
  • http.request.header.x-correlation-id: "xyz789"

Security

Sensitive headers are automatically filtered and never recorded:

  • Authorization
  • Cookie
  • Set-Cookie
  • X-API-Key
  • X-Auth-Token
  • Proxy-Authorization
  • WWW-Authenticate

This protects against accidental credential exposure in traces.

// This is safe - Authorization header is filtered
handler := tracing.Middleware(tracer,
    tracing.WithHeaders(
        "X-Request-ID",
        "Authorization", // ← Automatically filtered, won't be recorded
        "X-Correlation-ID",
    ),
)(mux)

Header Name Normalization

Header names are case-insensitive and normalized to lowercase:

tracing.WithHeaders("X-Request-ID", "x-correlation-id", "User-Agent")

All recorded as lowercase:

  • http.request.header.x-request-id
  • http.request.header.x-correlation-id
  • http.request.header.user-agent

Query Parameter Recording

Record URL query parameters as span attributes.

Default Behavior

By default, all query parameters are recorded:

handler := tracing.Middleware(tracer)(mux)
// All params recorded by default

Request: GET /api/users?page=2&limit=10&user_id=123

Span attributes:

  • http.request.param.page: ["2"]
  • http.request.param.limit: ["10"]
  • http.request.param.user_id: ["123"]

Whitelist Parameters

Record only specific parameters:

handler := tracing.Middleware(tracer,
    tracing.WithRecordParams("page", "limit", "user_id"),
)(mux)

Only page, limit, and user_id are recorded. Others are ignored.

Blacklist Parameters

Exclude sensitive parameters while recording all others:

handler := tracing.Middleware(tracer,
    tracing.WithExcludeParams("password", "token", "api_key", "secret"),
)(mux)

All parameters recorded except password, token, api_key, and secret.

Disable Parameter Recording

Don’t record any query parameters:

handler := tracing.Middleware(tracer,
    tracing.WithoutParams(),
)(mux)

Useful when parameters may contain sensitive data.

Combined Parameter Options

// Record only safe parameters, explicitly exclude sensitive ones
handler := tracing.Middleware(tracer,
    tracing.WithRecordParams("page", "limit", "sort"),
    tracing.WithExcludeParams("api_key", "token"), // Takes precedence
)(mux)

Behavior: Blacklist takes precedence. Even if api_key is in the whitelist, it won’t be recorded.

Complete Middleware Example

package main

import (
    "context"
    "log"
    "net/http"
    "os"
    "os/signal"
    
    "rivaas.dev/tracing"
)

func main() {
    ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt)
    defer cancel()

    // Create tracer
    tracer := tracing.MustNew(
        tracing.WithServiceName("user-api"),
        tracing.WithServiceVersion("v1.2.3"),
        tracing.WithOTLP("localhost:4317"),
        tracing.WithSampleRate(0.1), // 10% sampling
    )
    
    if err := tracer.Start(ctx); err != nil {
        log.Fatal(err)
    }
    defer tracer.Shutdown(context.Background())

    // Create HTTP handlers
    mux := http.NewServeMux()
    mux.HandleFunc("/api/users", handleUsers)
    mux.HandleFunc("/api/orders", handleOrders)
    mux.HandleFunc("/health", handleHealth)
    mux.HandleFunc("/metrics", handleMetrics)
    
    // Wrap with tracing middleware
    handler := tracing.MustMiddleware(tracer,
        // Exclude health/metrics endpoints
        tracing.WithExcludePaths("/health", "/metrics", "/ready", "/live"),
        
        // Exclude debug and internal routes
        tracing.WithExcludePrefixes("/debug/", "/internal/"),
        
        // Record correlation headers
        tracing.WithHeaders("X-Request-ID", "X-Correlation-ID", "User-Agent"),
        
        // Whitelist safe parameters
        tracing.WithRecordParams("page", "limit", "sort", "filter"),
        
        // Blacklist sensitive parameters
        tracing.WithExcludeParams("password", "token", "api_key"),
    )(mux)

    log.Println("Server starting on :8080")
    log.Fatal(http.ListenAndServe(":8080", handler))
}

func handleUsers(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "application/json")
    w.Write([]byte(`{"users": []}`))
}

func handleOrders(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "application/json")
    w.Write([]byte(`{"orders": []}`))
}

func handleHealth(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    w.Write([]byte("OK"))
}

func handleMetrics(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "text/plain")
    w.Write([]byte("# Metrics"))
}

Integration with Custom Context

Access the span from within your handlers:

func handleUser(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    
    // Add custom attributes to the current span
    tracing.SetSpanAttributeFromContext(ctx, "user.action", "view_profile")
    tracing.SetSpanAttributeFromContext(ctx, "user.id", getUserID(r))
    
    // Add events
    tracing.AddSpanEventFromContext(ctx, "profile_viewed",
        attribute.String("profile_id", "123"),
    )
    
    // Your handler logic...
}

Comparison with Metrics Middleware

The tracing middleware follows the same pattern as the metrics middleware:

AspectMetricsTracing
Main Functionmetrics.Middleware()tracing.Middleware()
Panic Versionmetrics.MustMiddleware()tracing.MustMiddleware()
Path Exclusionmetrics.WithExcludePaths()tracing.WithExcludePaths()
Prefix Exclusionmetrics.WithExcludePrefixes()tracing.WithExcludePrefixes()
Regex Exclusion✗ Not availabletracing.WithExcludePatterns()
Header Recordingmetrics.WithHeaders()tracing.WithHeaders()
Parameter Recording✗ Not availabletracing.WithRecordParams()

Performance

OperationTimeMemoryAllocations
Request overhead (100% sampling)~1.6 µs2.3 KB23
Path exclusion (100 paths)~9 ns0 B0
Start/Finish span~160 ns240 B3
Set attribute~3 ns0 B0

Best Practices

Always Exclude Health Checks

tracing.WithExcludePaths("/health", "/metrics", "/ready", "/live")

Health checks are high-frequency and low-value for tracing.

Use Sampling for High Traffic

tracer := tracing.MustNew(
    tracing.WithServiceName("my-api"),
    tracing.WithSampleRate(0.1), // 10% sampling
    tracing.WithOTLP("collector:4317"),
)

Reduces overhead and trace storage costs.

Record Correlation IDs

tracing.WithHeaders("X-Request-ID", "X-Correlation-ID", "X-Trace-ID")

Helps correlate traces with logs and other observability data.

Blacklist Sensitive Parameters

tracing.WithExcludeParams("password", "token", "api_key", "secret", "credit_card")

Prevents accidental exposure of credentials in traces.

Combine with Span Hooks

startHook := func(ctx context.Context, span trace.Span, req *http.Request) {
    // Add business context from request
    if tenantID := extractTenant(req); tenantID != "" {
        span.SetAttributes(attribute.String("tenant.id", tenantID))
    }
}

tracer := tracing.MustNew(
    tracing.WithServiceName("my-api"),
    tracing.WithSpanStartHook(startHook),
    tracing.WithOTLP("collector:4317"),
)

Next Steps

6 - Context Propagation

Propagate traces across service boundaries

Learn how to propagate trace context across service boundaries for distributed tracing.

What is Context Propagation?

Context propagation transmits trace information between services. Related operations appear in the same trace, even across network boundaries.

Why It Matters

Without context propagation:

  • Each service creates independent traces.
  • No visibility into end-to-end request flow.
  • Can’t trace requests across microservices.

With context propagation:

  • All services contribute to the same trace.
  • Complete visibility of distributed transactions.
  • Track requests across service boundaries.

W3C Trace Context

The tracing package uses W3C Trace Context format by default. It is:

  • Standard: Widely supported across languages and tools.
  • Propagated via HTTP headers:
    • traceparent: Contains trace ID, span ID, trace flags.
    • tracestate: Contains vendor-specific trace data.
  • Compatible: Works with Jaeger, Zipkin, OpenTelemetry, and more.

Extracting Trace Context

Extract trace context from incoming HTTP requests.

Automatic Extraction (Middleware)

The middleware automatically extracts trace context:

handler := tracing.Middleware(tracer)(mux)
// Context extraction is automatic

No additional code needed - spans automatically become part of the parent trace.

Manual Extraction

For manual span creation or custom HTTP handlers:

func handleRequest(w http.ResponseWriter, r *http.Request) {
    // Extract trace context from request headers
    ctx := tracer.ExtractTraceContext(r.Context(), r.Header)
    
    // Create span with propagated context
    ctx, span := tracer.StartSpan(ctx, "process-request")
    defer tracer.FinishSpan(span, http.StatusOK)
    
    // Span is now part of the distributed trace
}

What Gets Extracted

GET /api/users HTTP/1.1
Host: api.example.com
traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
tracestate: vendor1=value1,vendor2=value2

The ExtractTraceContext method reads these headers and links the new span to the parent trace.

Injecting Trace Context

Inject trace context into outgoing HTTP requests.

Manual Injection

When making HTTP calls to other services:

func callDownstreamService(ctx context.Context, tracer *tracing.Tracer) error {
    // Create outgoing request
    req, err := http.NewRequestWithContext(ctx, "GET", "http://downstream/api", nil)
    if err != nil {
        return err
    }
    
    // Inject trace context into request headers
    tracer.InjectTraceContext(ctx, req.Header)
    
    // Make the request
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    
    return nil
}

What Gets Injected

The InjectTraceContext method adds headers to propagate the trace:

// Before injection
req.Header: {}

// After injection
req.Header: {
    "Traceparent": ["00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01"],
    "Tracestate": ["vendor1=value1"],
}

Complete Distributed Tracing Example

Here’s a complete example showing service-to-service tracing:

Service A (Frontend)

package main

import (
    "context"
    "io"
    "log"
    "net/http"
    
    "rivaas.dev/tracing"
)

func main() {
    tracer := tracing.MustNew(
        tracing.WithServiceName("frontend-api"),
        tracing.WithOTLP("localhost:4317"),
    )
    tracer.Start(context.Background())
    defer tracer.Shutdown(context.Background())

    mux := http.NewServeMux()
    
    // Handler that calls downstream service
    mux.HandleFunc("/api/process", func(w http.ResponseWriter, r *http.Request) {
        ctx := r.Context()
        
        // Create span for this service's work
        ctx, span := tracer.StartSpan(ctx, "frontend-process")
        defer tracer.FinishSpan(span, http.StatusOK)
        
        // Call downstream service with trace propagation
        result, err := callBackendService(ctx, tracer)
        if err != nil {
            http.Error(w, err.Error(), http.StatusInternalServerError)
            return
        }
        
        w.Write([]byte(result))
    })
    
    handler := tracing.Middleware(tracer)(mux)
    log.Fatal(http.ListenAndServe(":8080", handler))
}

func callBackendService(ctx context.Context, tracer *tracing.Tracer) (string, error) {
    // Create span for outgoing call
    ctx, span := tracer.StartSpan(ctx, "call-backend-service")
    defer tracer.FinishSpan(span, http.StatusOK)
    
    // Create HTTP request
    req, err := http.NewRequestWithContext(ctx, "GET", 
        "http://localhost:8081/api/data", nil)
    if err != nil {
        return "", err
    }
    
    // Inject trace context for propagation
    tracer.InjectTraceContext(ctx, req.Header)
    
    // Make the request
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return "", err
    }
    defer resp.Body.Close()
    
    body, _ := io.ReadAll(resp.Body)
    return string(body), nil
}

Service B (Backend)

package main

import (
    "context"
    "log"
    "net/http"
    
    "rivaas.dev/tracing"
)

func main() {
    tracer := tracing.MustNew(
        tracing.WithServiceName("backend-api"),
        tracing.WithOTLP("localhost:4317"),
    )
    tracer.Start(context.Background())
    defer tracer.Shutdown(context.Background())

    mux := http.NewServeMux()
    
    // Handler automatically receives trace context via middleware
    mux.HandleFunc("/api/data", func(w http.ResponseWriter, r *http.Request) {
        ctx := r.Context()
        
        // This span is automatically part of the distributed trace
        ctx, span := tracer.StartSpan(ctx, "fetch-data")
        defer tracer.FinishSpan(span, http.StatusOK)
        
        tracer.SetSpanAttribute(span, "data.source", "database")
        
        // Simulate work
        data := fetchFromDatabase(ctx, tracer)
        
        w.Write([]byte(data))
    })
    
    // Middleware automatically extracts trace context
    handler := tracing.Middleware(tracer)(mux)
    log.Fatal(http.ListenAndServe(":8081", handler))
}

func fetchFromDatabase(ctx context.Context, tracer *tracing.Tracer) string {
    // Nested span - all part of the same trace
    ctx, span := tracer.StartSpan(ctx, "database-query")
    defer tracer.FinishSpan(span, http.StatusOK)
    
    tracer.SetSpanAttribute(span, "db.system", "postgresql")
    tracer.SetSpanAttribute(span, "db.query", "SELECT * FROM data")
    
    return "data from database"
}

Resulting Trace

The trace will show the complete flow:

frontend-api: GET /api/process
├─ frontend-api: frontend-process
│  └─ frontend-api: call-backend-service
│     └─ backend-api: GET /api/data
│        └─ backend-api: fetch-data
│           └─ backend-api: database-query

Context Helper Functions

Work with trace context without direct span references.

Get Trace Information

Retrieve trace and span IDs from context:

func logWithTraceInfo(ctx context.Context) {
    traceID := tracing.TraceID(ctx)
    spanID := tracing.SpanID(ctx)
    
    log.Printf("[trace=%s span=%s] Processing request", traceID, spanID)
}

Returns empty string if no active span.

Set Attributes from Context

Add attributes to the current span:

func processOrder(ctx context.Context, orderID string) {
    // Add attributes to current span in context
    tracing.SetSpanAttributeFromContext(ctx, "order.id", orderID)
    tracing.SetSpanAttributeFromContext(ctx, "order.status", "processing")
}

No-op if no active span.

Add Events from Context

Add events to the current span:

import "go.opentelemetry.io/otel/attribute"

func validatePayment(ctx context.Context, amount float64) {
    // Add event to current span
    tracing.AddSpanEventFromContext(ctx, "payment_validated",
        attribute.Float64("amount", amount),
        attribute.String("currency", "USD"),
    )
}

Get Trace Context

The context already contains trace information:

func passContextToWorker(ctx context.Context) {
    // Context already has trace info - just pass it
    go processInBackground(ctx)
}

func processInBackground(ctx context.Context) {
    // Trace context is preserved
    traceID := tracing.TraceID(ctx)
    log.Printf("Background work [trace=%s]", traceID)
}

Custom Propagators

Use alternative trace context formats.

B3 Propagation (Zipkin)

import "go.opentelemetry.io/contrib/propagators/b3"

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithCustomPropagator(b3.New()),
    tracing.WithOTLP("localhost:4317"),
)

Uses Zipkin’s B3 headers:

  • X-B3-TraceId
  • X-B3-SpanId
  • X-B3-Sampled

Jaeger Propagation

import "go.opentelemetry.io/contrib/propagators/jaeger"

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithCustomPropagator(jaeger.Jaeger{}),
    tracing.WithOTLP("localhost:4317"),
)

Uses Jaeger’s uber-trace-id header.

Composite Propagator

Support multiple formats simultaneously:

import (
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/contrib/propagators/b3"
)

composite := propagation.NewCompositeTextMapPropagator(
    propagation.TraceContext{}, // W3C Trace Context
    propagation.Baggage{},      // W3C Baggage
    b3.New(),                   // B3 (Zipkin)
)

tracer := tracing.MustNew(
    tracing.WithServiceName("my-service"),
    tracing.WithCustomPropagator(composite),
    tracing.WithOTLP("localhost:4317"),
)

Best Practices

Always Propagate Context

Pass context through the entire call chain:

// ✓ Good - context propagates
func handler(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    result := doWork(ctx)  // Pass context
}

func doWork(ctx context.Context) string {
    ctx, span := tracer.StartSpan(ctx, "do-work")
    defer tracer.FinishSpan(span, http.StatusOK)
    
    return doMoreWork(ctx)  // Pass context
}

// ✗ Bad - context lost
func handler(w http.ResponseWriter, r *http.Request) {
    result := doWork(context.Background())  // Lost trace context!
}

Use Context for HTTP Clients

Always use http.NewRequestWithContext:

// ✓ Good
req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
tracer.InjectTraceContext(ctx, req.Header)

// ✗ Bad - no context
req, _ := http.NewRequest("GET", url, nil)
tracer.InjectTraceContext(ctx, req.Header)  // Won't have span info

Inject Before Making Requests

Always inject trace context before sending requests:

req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)

// Inject trace context
tracer.InjectTraceContext(ctx, req.Header)

// Then make request
resp, _ := http.DefaultClient.Do(req)

Extract in Custom Handlers

If not using middleware, extract context manually:

func customHandler(w http.ResponseWriter, r *http.Request) {
    // Extract trace context
    ctx := tracer.ExtractTraceContext(r.Context(), r.Header)
    
    // Use propagated context
    ctx, span := tracer.StartSpan(ctx, "custom-handler")
    defer tracer.FinishSpan(span, http.StatusOK)
}

Troubleshooting

Traces Not Connected Across Services

Problem: Each service shows separate traces instead of one distributed trace.

Solutions:

  1. Ensure both services use the same propagator format (default: W3C Trace Context)
  2. Verify InjectTraceContext is called before making requests
  3. Verify ExtractTraceContext is called when receiving requests
  4. Check that context is passed through the call chain
  5. Verify both services send to the same OTLP collector

Missing Spans in Distributed Trace

Problem: Some spans appear but others are missing.

Solutions:

  1. Check sampling rate - non-sampled requests won’t create spans
  2. Verify all services have tracing enabled
  3. Ensure context is passed to all operations
  4. Check for errors in span creation

Context Lost in Goroutines

Problem: Background goroutines don’t have trace context.

Solution: Pass context explicitly to goroutines:

func handler(ctx context.Context) {
    // ✓ Good - pass context
    go func(ctx context.Context) {
        ctx, span := tracer.StartSpan(ctx, "background-work")
        defer tracer.FinishSpan(span, http.StatusOK)
    }(ctx)
    
    // ✗ Bad - lost context
    go func() {
        ctx := context.Background()  // Lost trace context!
        ctx, span := tracer.StartSpan(ctx, "background-work")
        defer tracer.FinishSpan(span, http.StatusOK)
    }()
}

Next Steps

7 - Testing

Test your tracing implementation with provided utilities

The tracing package provides testing utilities to help you write tests for traced applications.

Testing Utilities

Three helper functions are provided for testing:

FunctionPurposeProvider
TestingTracer()Create tracer for tests.Noop
TestingTracerWithStdout()Create tracer with output.Stdout
TestingMiddleware()Create test middleware.Noop

TestingTracer

Create a tracer configured for unit tests.

Basic Usage

import (
    "testing"
    "rivaas.dev/tracing"
)

func TestSomething(t *testing.T) {
    t.Parallel()
    
    tracer := tracing.TestingTracer(t)
    // Use tracer in test...
}

Features

  • Noop provider: No actual tracing, minimal overhead.
  • Automatic cleanup: Shutdown() called via t.Cleanup().
  • Safe for parallel tests: Each test gets its own tracer.
  • Default configuration:
    • Service name: "test-service".
    • Service version: "v1.0.0".
    • Sample rate: 1.0 (100%).

With Custom Options

Override defaults with your own options.

func TestWithCustomConfig(t *testing.T) {
    tracer := tracing.TestingTracer(t,
        tracing.WithServiceName("my-test-service"),
        tracing.WithSampleRate(0.5),
    )
    // Use tracer...
}

Complete Test Example

func TestProcessOrder(t *testing.T) {
    t.Parallel()
    
    tracer := tracing.TestingTracer(t)
    ctx := context.Background()
    
    // Test your traced function
    result, err := processOrder(ctx, tracer, "order-123")
    
    assert.NoError(t, err)
    assert.Equal(t, "success", result)
}

func processOrder(ctx context.Context, tracer *tracing.Tracer, orderID string) (string, error) {
    ctx, span := tracer.StartSpan(ctx, "process-order")
    defer tracer.FinishSpan(span, 200)
    
    tracer.SetSpanAttribute(span, "order.id", orderID)
    
    return "success", nil
}

TestingTracerWithStdout

Create a tracer that prints traces to stdout for debugging.

When to Use

  • Debugging test failures
  • Verifying span creation
  • Checking span attributes and events
  • Understanding trace structure

Basic Usage

func TestWithDebugOutput(t *testing.T) {
    tracer := tracing.TestingTracerWithStdout(t)
    
    ctx := context.Background()
    ctx, span := tracer.StartSpan(ctx, "test-operation")
    defer tracer.FinishSpan(span, 200)
    
    tracer.SetSpanAttribute(span, "test.value", "debug")
}

Output

When run, you’ll see pretty-printed JSON traces:

{
  "Name": "test-operation",
  "SpanContext": {
    "TraceID": "3f3c5e4d...",
    "SpanID": "a1b2c3d4..."
  },
  "Attributes": [
    {
      "Key": "test.value",
      "Value": {"Type": "STRING", "Value": "debug"}
    }
  ]
}

With Custom Options

func TestDebugWithOptions(t *testing.T) {
    tracer := tracing.TestingTracerWithStdout(t,
        tracing.WithServiceName("debug-service"),
        tracing.WithSampleRate(1.0),
    )
    // Use tracer...
}

TestingMiddleware

Create HTTP middleware for testing traced handlers.

Basic Usage

import (
    "net/http"
    "net/http/httptest"
    "testing"
    
    "rivaas.dev/tracing"
)

func TestHTTPHandler(t *testing.T) {
    t.Parallel()
    
    // Create test middleware
    middleware := tracing.TestingMiddleware(t)
    
    // Wrap your handler
    handler := middleware(http.HandlerFunc(myHandler))
    
    // Test the handler
    req := httptest.NewRequest("GET", "/api/users", nil)
    rec := httptest.NewRecorder()
    
    handler.ServeHTTP(rec, req)
    
    assert.Equal(t, http.StatusOK, rec.Code)
}

func myHandler(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    w.Write([]byte("OK"))
}

With Middleware Options

func TestWithMiddlewareOptions(t *testing.T) {
    middleware := tracing.TestingMiddleware(t,
        tracing.WithExcludePaths("/health"),
        tracing.WithHeaders("X-Request-ID"),
    )
    
    handler := middleware(http.HandlerFunc(myHandler))
    // Test...
}

Testing Path Exclusion

func TestPathExclusion(t *testing.T) {
    middleware := tracing.TestingMiddleware(t,
        tracing.WithExcludePaths("/health"),
    )
    
    handler := middleware(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // This handler should not create a span for /health
        w.WriteHeader(http.StatusOK)
    }))
    
    // Request to excluded path
    req := httptest.NewRequest("GET", "/health", nil)
    rec := httptest.NewRecorder()
    handler.ServeHTTP(rec, req)
    
    assert.Equal(t, http.StatusOK, rec.Code)
}

TestingMiddlewareWithTracer

Use a custom tracer with test middleware.

When to Use

  • Need specific tracer configuration
  • Testing with stdout output
  • Custom sampling rates
  • Specific provider behavior

Basic Usage

func TestWithCustomTracer(t *testing.T) {
    // Create custom tracer
    tracer := tracing.TestingTracer(t,
        tracing.WithSampleRate(0.5),
    )
    
    // Create middleware with custom tracer
    middleware := tracing.TestingMiddlewareWithTracer(t, tracer,
        tracing.WithExcludePaths("/metrics"),
    )
    
    handler := middleware(http.HandlerFunc(myHandler))
    // Test...
}

With Stdout Output

func TestDebugMiddleware(t *testing.T) {
    // Create tracer with stdout
    tracer := tracing.TestingTracerWithStdout(t)
    
    // Create middleware with that tracer
    middleware := tracing.TestingMiddlewareWithTracer(t, tracer)
    
    handler := middleware(http.HandlerFunc(myHandler))
    
    // Test and see trace output
    req := httptest.NewRequest("GET", "/api/users", nil)
    rec := httptest.NewRecorder()
    handler.ServeHTTP(rec, req)
}

Testing Patterns

Table-Driven Tests

func TestHandlers(t *testing.T) {
    tests := []struct {
        name       string
        path       string
        wantStatus int
    }{
        {"users endpoint", "/api/users", http.StatusOK},
        {"orders endpoint", "/api/orders", http.StatusOK},
        {"health check", "/health", http.StatusOK},
    }
    
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            t.Parallel()
            
            middleware := tracing.TestingMiddleware(t)
            handler := middleware(http.HandlerFunc(myHandler))
            
            req := httptest.NewRequest("GET", tt.path, nil)
            rec := httptest.NewRecorder()
            
            handler.ServeHTTP(rec, req)
            
            assert.Equal(t, tt.wantStatus, rec.Code)
        })
    }
}

Testing Span Attributes

func TestSpanAttributes(t *testing.T) {
    t.Parallel()
    
    tracer := tracing.TestingTracer(t)
    ctx := context.Background()
    
    // Create span and add attributes
    ctx, span := tracer.StartSpan(ctx, "test-span")
    tracer.SetSpanAttribute(span, "user.id", "123")
    tracer.SetSpanAttribute(span, "user.role", "admin")
    tracer.FinishSpan(span, 200)
    
    // With noop provider, this doesn't record anything,
    // but ensures the code doesn't panic or error
}

Testing Context Propagation

func TestContextPropagation(t *testing.T) {
    t.Parallel()
    
    tracer := tracing.TestingTracer(t)
    ctx := context.Background()
    
    // Create parent span
    ctx, parentSpan := tracer.StartSpan(ctx, "parent")
    defer tracer.FinishSpan(parentSpan, 200)
    
    // Get trace ID
    traceID := tracing.TraceID(ctx)
    assert.NotEmpty(t, traceID)
    
    // Create child span - should have same trace ID
    ctx, childSpan := tracer.StartSpan(ctx, "child")
    defer tracer.FinishSpan(childSpan, 200)
    
    childTraceID := tracing.TraceID(ctx)
    assert.Equal(t, traceID, childTraceID, "child should have same trace ID")
}

Testing Trace Injection/Extraction

func TestTraceInjection(t *testing.T) {
    t.Parallel()
    
    tracer := tracing.TestingTracer(t)
    ctx := context.Background()
    
    // Create span
    ctx, span := tracer.StartSpan(ctx, "test")
    defer tracer.FinishSpan(span, 200)
    
    // Inject into headers
    headers := http.Header{}
    tracer.InjectTraceContext(ctx, headers)
    
    // Verify headers were set
    assert.NotEmpty(t, headers.Get("Traceparent"))
    
    // Extract from headers
    newCtx := context.Background()
    newCtx = tracer.ExtractTraceContext(newCtx, headers)
    
    // Both contexts should have the same trace ID
    originalTraceID := tracing.TraceID(ctx)
    extractedTraceID := tracing.TraceID(newCtx)
    assert.Equal(t, originalTraceID, extractedTraceID)
}

Integration Test Example

func TestAPIWithTracing(t *testing.T) {
    t.Parallel()
    
    // Create tracer
    tracer := tracing.TestingTracer(t)
    
    // Create test server with tracing
    mux := http.NewServeMux()
    mux.HandleFunc("/api/users", func(w http.ResponseWriter, r *http.Request) {
        ctx := r.Context()
        
        // Add attributes from context
        tracing.SetSpanAttributeFromContext(ctx, "handler", "users")
        
        w.WriteHeader(http.StatusOK)
        w.Write([]byte(`{"users": []}`))
    })
    
    handler := tracing.TestingMiddlewareWithTracer(t, tracer)(mux)
    server := httptest.NewServer(handler)
    defer server.Close()
    
    // Make request
    resp, err := http.Get(server.URL + "/api/users")
    require.NoError(t, err)
    defer resp.Body.Close()
    
    assert.Equal(t, http.StatusOK, resp.StatusCode)
}

Benchmarking

Test tracing overhead in benchmarks:

func BenchmarkTracedHandler(b *testing.B) {
    tracer := tracing.TestingTracer(b)
    
    middleware := tracing.TestingMiddlewareWithTracer(b, tracer)
    handler := middleware(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusOK)
    }))
    
    req := httptest.NewRequest("GET", "/", nil)
    
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        rec := httptest.NewRecorder()
        handler.ServeHTTP(rec, req)
    }
}

func BenchmarkUntracedHandler(b *testing.B) {
    handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusOK)
    })
    
    req := httptest.NewRequest("GET", "/", nil)
    
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        rec := httptest.NewRecorder()
        handler.ServeHTTP(rec, req)
    }
}

Best Practices

Use t.Parallel()

Enable parallel test execution:

func TestSomething(t *testing.T) {
    t.Parallel() // Safe - each test gets its own tracer
    
    tracer := tracing.TestingTracer(t)
    // Test...
}

Don’t Call Shutdown Manually

The test utilities handle cleanup automatically:

// ✓ Good - automatic cleanup
func TestGood(t *testing.T) {
    tracer := tracing.TestingTracer(t)
    // No need to call Shutdown()
}

// ✗ Bad - redundant manual cleanup
func TestBad(t *testing.T) {
    tracer := tracing.TestingTracer(t)
    defer tracer.Shutdown(context.Background()) // Unnecessary
}

Use Stdout for Debugging Only

Don’t use TestingTracerWithStdout for regular tests:

// ✓ Good - stdout only when debugging
func TestDebug(t *testing.T) {
    if testing.Verbose() {
        tracer := tracing.TestingTracerWithStdout(t)
    } else {
        tracer := tracing.TestingTracer(t)
    }
}

// ✗ Bad - noisy test output
func TestRegular(t *testing.T) {
    tracer := tracing.TestingTracerWithStdout(t) // Too verbose
}

Test Error Cases

func TestErrorHandling(t *testing.T) {
    t.Parallel()
    
    tracer := tracing.TestingTracer(t)
    ctx := context.Background()
    
    ctx, span := tracer.StartSpan(ctx, "test-error")
    defer tracer.FinishSpan(span, http.StatusInternalServerError)
    
    tracer.SetSpanAttribute(span, "error", true)
    tracer.SetSpanAttribute(span, "error.message", "test error")
}

Comparison with Other Packages

Testing utilities follow the same pattern:

PackageTesting FunctionProvider
Metricsmetrics.TestingRecorder()Noop
Metricsmetrics.TestingRecorderWithPrometheus()Prometheus
Tracingtracing.TestingTracer()Noop
Tracingtracing.TestingTracerWithStdout()Stdout

Next Steps

8 - Examples

Real-world tracing configurations and patterns

Explore complete examples and best practices for production-ready tracing configurations.

Production Configuration

A production-ready tracing setup with all recommended settings.

package main

import (
    "context"
    "log"
    "log/slog"
    "net/http"
    "os"
    "os/signal"
    "time"
    
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/trace"
    "rivaas.dev/tracing"
)

func main() {
    // Create context for graceful shutdown
    ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt)
    defer cancel()

    // Create logger for internal events
    logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
        Level: slog.LevelInfo,
    }))

    // Create tracer with production settings
    tracer, err := tracing.New(
        tracing.WithServiceName("user-api"),
        tracing.WithServiceVersion(os.Getenv("VERSION")),
        tracing.WithOTLP(os.Getenv("OTLP_ENDPOINT")),
        tracing.WithSampleRate(0.1), // 10% sampling
        tracing.WithLogger(logger),
        tracing.WithSpanStartHook(enrichSpan),
        tracing.WithSpanFinishHook(recordMetrics),
    )
    if err != nil {
        log.Fatalf("Failed to initialize tracing: %v", err)
    }

    // Start tracer (required for OTLP)
    if err := tracer.Start(ctx); err != nil {
        log.Fatalf("Failed to start tracer: %v", err)
    }

    // Ensure graceful shutdown
    defer func() {
        shutdownCtx, shutdownCancel := context.WithTimeout(
            context.Background(), 5*time.Second)
        defer shutdownCancel()
        
        if err := tracer.Shutdown(shutdownCtx); err != nil {
            log.Printf("Error shutting down tracer: %v", err)
        }
    }()

    // Create HTTP handlers
    mux := http.NewServeMux()
    mux.HandleFunc("/api/users", handleUsers)
    mux.HandleFunc("/api/orders", handleOrders)
    mux.HandleFunc("/health", handleHealth)
    mux.HandleFunc("/metrics", handleMetrics)

    // Wrap with tracing middleware
    handler := tracing.MustMiddleware(tracer,
        // Exclude observability endpoints
        tracing.WithExcludePaths("/health", "/metrics", "/ready", "/live"),
        
        // Exclude debug endpoints
        tracing.WithExcludePrefixes("/debug/", "/internal/"),
        
        // Record correlation headers
        tracing.WithHeaders("X-Request-ID", "X-Correlation-ID"),
        
        // Whitelist safe parameters
        tracing.WithRecordParams("page", "limit", "sort"),
        
        // Blacklist sensitive parameters
        tracing.WithExcludeParams("password", "token", "api_key"),
    )(mux)

    // Start server
    log.Printf("Server starting on :8080")
    if err := http.ListenAndServe(":8080", handler); err != nil {
        log.Fatal(err)
    }
}

// enrichSpan adds custom business context to spans
func enrichSpan(ctx context.Context, span trace.Span, req *http.Request) {
    // Add tenant identifier
    if tenantID := req.Header.Get("X-Tenant-ID"); tenantID != "" {
        span.SetAttributes(attribute.String("tenant.id", tenantID))
    }
    
    // Add user information
    if userID := req.Header.Get("X-User-ID"); userID != "" {
        span.SetAttributes(attribute.String("user.id", userID))
    }
    
    // Add deployment information
    span.SetAttributes(
        attribute.String("deployment.region", os.Getenv("REGION")),
        attribute.String("deployment.environment", os.Getenv("ENVIRONMENT")),
    )
}

// recordMetrics records custom metrics based on span completion
func recordMetrics(span trace.Span, statusCode int) {
    // Record error metrics
    if statusCode >= 500 {
        // metrics.IncrementServerErrors()
    }
    
    // Record slow request metrics
    // Could calculate duration and record if above threshold
}

func handleUsers(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    
    // Add custom span attributes
    tracing.SetSpanAttributeFromContext(ctx, "handler", "users")
    tracing.SetSpanAttributeFromContext(ctx, "operation", "list")
    
    w.Header().Set("Content-Type", "application/json")
    w.Write([]byte(`{"users": []}`))
}

func handleOrders(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    
    tracing.SetSpanAttributeFromContext(ctx, "handler", "orders")
    
    w.Header().Set("Content-Type", "application/json")
    w.Write([]byte(`{"orders": []}`))
}

func handleHealth(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    w.Write([]byte("OK"))
}

func handleMetrics(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "text/plain")
    w.Write([]byte("# Metrics"))
}

Development Configuration

A development setup with verbose output for debugging.

package main

import (
    "context"
    "log"
    "log/slog"
    "net/http"
    "os"
    
    "rivaas.dev/tracing"
)

func main() {
    // Create logger with debug level
    logger := slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{
        Level: slog.LevelDebug,
    }))

    // Create tracer with development settings
    tracer := tracing.MustNew(
        tracing.WithServiceName("user-api"),
        tracing.WithServiceVersion("dev"),
        tracing.WithStdout(),          // Print traces to console
        tracing.WithSampleRate(1.0),   // Trace everything
        tracing.WithLogger(logger),    // Verbose logging
    )
    defer tracer.Shutdown(context.Background())

    // Create simple handler
    mux := http.NewServeMux()
    mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("Hello, World!"))
    })

    // Minimal middleware - trace everything
    handler := tracing.MustMiddleware(tracer)(mux)

    log.Println("Development server on :8080")
    log.Fatal(http.ListenAndServe(":8080", handler))
}

Microservices Example

Complete distributed tracing across multiple services.

Service A (API Gateway)

package main

import (
    "context"
    "io"
    "log"
    "net/http"
    
    "rivaas.dev/tracing"
)

func main() {
    tracer := tracing.MustNew(
        tracing.WithServiceName("api-gateway"),
        tracing.WithServiceVersion("v1.0.0"),
        tracing.WithOTLP("localhost:4317"),
    )
    tracer.Start(context.Background())
    defer tracer.Shutdown(context.Background())

    mux := http.NewServeMux()
    mux.HandleFunc("/api/users", func(w http.ResponseWriter, r *http.Request) {
        ctx := r.Context()
        
        // Call user service
        users, err := callUserService(ctx, tracer)
        if err != nil {
            http.Error(w, err.Error(), http.StatusInternalServerError)
            return
        }
        
        w.Header().Set("Content-Type", "application/json")
        w.Write([]byte(users))
    })

    handler := tracing.MustMiddleware(tracer,
        tracing.WithExcludePaths("/health"),
    )(mux)

    log.Fatal(http.ListenAndServe(":8080", handler))
}

func callUserService(ctx context.Context, tracer *tracing.Tracer) (string, error) {
    // Create span for outgoing call
    ctx, span := tracer.StartSpan(ctx, "call-user-service")
    defer tracer.FinishSpan(span, http.StatusOK)
    
    // Create request
    req, err := http.NewRequestWithContext(ctx, "GET", 
        "http://localhost:8081/users", nil)
    if err != nil {
        return "", err
    }
    
    // Inject trace context
    tracer.InjectTraceContext(ctx, req.Header)
    
    // Make request
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return "", err
    }
    defer resp.Body.Close()
    
    body, _ := io.ReadAll(resp.Body)
    return string(body), nil
}

Service B (User Service)

package main

import (
    "context"
    "log"
    "net/http"
    
    "rivaas.dev/tracing"
)

func main() {
    tracer := tracing.MustNew(
        tracing.WithServiceName("user-service"),
        tracing.WithServiceVersion("v1.0.0"),
        tracing.WithOTLP("localhost:4317"),
    )
    tracer.Start(context.Background())
    defer tracer.Shutdown(context.Background())

    mux := http.NewServeMux()
    mux.HandleFunc("/users", func(w http.ResponseWriter, r *http.Request) {
        ctx := r.Context()
        
        // This span is part of the distributed trace
        ctx, span := tracer.StartSpan(ctx, "fetch-users")
        defer tracer.FinishSpan(span, http.StatusOK)
        
        tracer.SetSpanAttribute(span, "db.system", "postgresql")
        
        // Simulate database query
        users := `{"users": [{"id": 1, "name": "Alice"}]}`
        
        w.Header().Set("Content-Type", "application/json")
        w.Write([]byte(users))
    })

    // Middleware automatically extracts trace context
    handler := tracing.MustMiddleware(tracer)(mux)

    log.Fatal(http.ListenAndServe(":8081", handler))
}

Environment-Based Configuration

Configure tracing based on environment.

package main

import (
    "context"
    "log"
    "log/slog"
    "net/http"
    "os"
    
    "rivaas.dev/tracing"
)

func main() {
    tracer := createTracer(os.Getenv("ENVIRONMENT"))
    defer tracer.Shutdown(context.Background())

    // If OTLP, start the tracer
    if tracer.GetProvider() == tracing.OTLPProvider || 
       tracer.GetProvider() == tracing.OTLPHTTPProvider {
        if err := tracer.Start(context.Background()); err != nil {
            log.Fatal(err)
        }
    }

    mux := http.NewServeMux()
    mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("Hello"))
    })

    handler := tracing.MustMiddleware(tracer)(mux)
    log.Fatal(http.ListenAndServe(":8080", handler))
}

func createTracer(env string) *tracing.Tracer {
    serviceName := os.Getenv("SERVICE_NAME")
    if serviceName == "" {
        serviceName = "my-api"
    }

    version := os.Getenv("VERSION")
    if version == "" {
        version = "dev"
    }

    opts := []tracing.Option{
        tracing.WithServiceName(serviceName),
        tracing.WithServiceVersion(version),
    }

    switch env {
    case "production":
        opts = append(opts,
            tracing.WithOTLP(os.Getenv("OTLP_ENDPOINT")),
            tracing.WithSampleRate(0.1), // 10% sampling
        )
    case "staging":
        opts = append(opts,
            tracing.WithOTLP(os.Getenv("OTLP_ENDPOINT")),
            tracing.WithSampleRate(0.5), // 50% sampling
        )
    default: // development
        logger := slog.New(slog.NewTextHandler(os.Stdout, nil))
        opts = append(opts,
            tracing.WithStdout(),
            tracing.WithSampleRate(1.0), // 100% sampling
            tracing.WithLogger(logger),
        )
    }

    return tracing.MustNew(opts...)
}

Database Tracing Example

Trace database operations.

package main

import (
    "context"
    "database/sql"
    "net/http"
    
    "go.opentelemetry.io/otel/attribute"
    "rivaas.dev/tracing"
)

type UserRepository struct {
    db     *sql.DB
    tracer *tracing.Tracer
}

func (r *UserRepository) GetUser(ctx context.Context, userID int) (*User, error) {
    // Create span for database operation
    ctx, span := r.tracer.StartSpan(ctx, "db-get-user")
    defer r.tracer.FinishSpan(span, http.StatusOK)
    
    // Add database attributes
    r.tracer.SetSpanAttribute(span, "db.system", "postgresql")
    r.tracer.SetSpanAttribute(span, "db.operation", "SELECT")
    r.tracer.SetSpanAttribute(span, "db.table", "users")
    r.tracer.SetSpanAttribute(span, "user.id", userID)
    
    // Execute query
    query := "SELECT id, name, email FROM users WHERE id = $1"
    r.tracer.SetSpanAttribute(span, "db.query", query)
    
    var user User
    err := r.db.QueryRowContext(ctx, query, userID).Scan(
        &user.ID, &user.Name, &user.Email)
    if err != nil {
        r.tracer.SetSpanAttribute(span, "error", true)
        r.tracer.SetSpanAttribute(span, "error.message", err.Error())
        return nil, err
    }
    
    // Add event for successful query
    r.tracer.AddSpanEvent(span, "user_found",
        attribute.Int("user.id", user.ID),
    )
    
    return &user, nil
}

type User struct {
    ID    int
    Name  string
    Email string
}

Custom Span Events Example

Record significant events within spans.

func processOrder(ctx context.Context, tracer *tracing.Tracer, order *Order) error {
    ctx, span := tracer.StartSpan(ctx, "process-order")
    defer tracer.FinishSpan(span, http.StatusOK)
    
    tracer.SetSpanAttribute(span, "order.id", order.ID)
    tracer.SetSpanAttribute(span, "order.total", order.Total)
    
    // Event: Order validation started
    tracer.AddSpanEvent(span, "validation_started")
    
    if err := validateOrder(ctx, tracer, order); err != nil {
        tracer.AddSpanEvent(span, "validation_failed",
            attribute.String("error", err.Error()),
        )
        return err
    }
    
    tracer.AddSpanEvent(span, "validation_passed")
    
    // Event: Payment processing started
    tracer.AddSpanEvent(span, "payment_started",
        attribute.Float64("amount", order.Total),
    )
    
    if err := chargePayment(ctx, tracer, order); err != nil {
        tracer.AddSpanEvent(span, "payment_failed",
            attribute.String("error", err.Error()),
        )
        return err
    }
    
    tracer.AddSpanEvent(span, "payment_succeeded",
        attribute.String("transaction_id", "TXN123"),
    )
    
    // Event: Order completed
    tracer.AddSpanEvent(span, "order_completed")
    
    return nil
}

Performance Benchmarks

Actual performance measurements from the tracing package:

// Operation                              Time        Memory      Allocations
// Request overhead (100% sampling)       ~1.6 µs     2.3 KB      23
// Start/Finish span                      ~160 ns     240 B       3
// Set attribute                          ~3 ns       0 B         0
// Path exclusion (100 paths)             ~9 ns       0 B         0

Performance Tips

  1. Use sampling for high-traffic endpoints:

    tracing.WithSampleRate(0.1) // 10% sampling

  2. Exclude health checks:

    tracing.WithExcludePaths("/health", "/metrics", "/ready")

  3. Minimize attributes in hot paths:

    // Only add essential attributes in critical code paths
tracer.SetSpanAttribute(span, "request.id", requestID)

  4. Use path prefixes over regex when possible:

    tracing.WithExcludePrefixes("/debug/") // Faster than regex

Docker Compose Setup

Complete tracing infrastructure with Jaeger:

version: '3.8'
services:
  # Your application
  app:
    build: .
    ports:
      - "8080:8080"
    environment:
      - OTLP_ENDPOINT=otel-collector:4317
      - ENVIRONMENT=development
    depends_on:
      - otel-collector

  # OpenTelemetry Collector
  otel-collector:
    image: otel/opentelemetry-collector:latest
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP
    depends_on:
      - jaeger

  # Jaeger for trace visualization
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686" # Jaeger UI
      - "14250:14250" # Model.proto

OpenTelemetry Collector configuration (otel-collector-config.yaml):

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 10s
    send_batch_size: 1024

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger]

Next Steps