Multiple Sources
Combine configuration from files, environment variables, and remote sources
6 minute read
The config package supports loading configuration from multiple sources simultaneously. This enables powerful patterns like base configuration with environment-specific overrides.
Source Precedence
When multiple sources are configured, later sources override earlier ones:
cfg := config.MustNew(
config.WithFile("config.yaml"), // Base configuration
config.WithFile("config.prod.yaml"), // Production overrides
config.WithEnv("MYAPP_"), // Environment overrides all
)
Priority: Environment variables > config.prod.yaml > config.yaml
Visualization
graph LR
File1[config.yaml] --> Merge1[Merge]
File2[config.json] --> Merge2[Merge]
Merge1 --> Merge2
Env[Environment] --> Merge3[Merge]
Merge2 --> Merge3
Consul[Consul KV] --> Merge4[Merge]
Merge3 --> Merge4
Merge4 --> Final[Final Config]Hierarchical Merging
Sources are merged hierarchically - nested structures are combined intelligently:
config.yaml (base):
server:
host: localhost
port: 8080
timeout: 30s
database:
host: localhost
port: 5432
config.prod.yaml (overrides):
server:
host: 0.0.0.0
database:
host: db.example.com
Merged result:
server:
host: 0.0.0.0 # Overridden.
port: 8080 # From base.
timeout: 30s # From base.
database:
host: db.example.com # Overridden.
port: 5432 # From base.
File Sources
Load configuration from local files:
cfg := config.MustNew(
config.WithFile("config.yaml"),
config.WithFile("config.json"),
config.WithFile("config.toml"),
)
Format Auto-Detection
File formats are detected automatically from extensions:
config.WithFile("app.yaml") // YAML
config.WithFile("app.yml") // YAML
config.WithFile("app.json") // JSON
config.WithFile("app.toml") // TOML
Explicit Format
Use explicit format when the file name doesn’t have an extension:
config.WithFileAs("config.txt", codec.TypeYAML)
config.WithFileAs("data.conf", codec.TypeJSON)
Environment Variable Sources
Load configuration from environment variables:
cfg := config.MustNew(
config.WithFile("config.yaml"),
config.WithEnv("MYAPP_"), // Prefix filter
)
Environment variables override file-based configuration. See Environment Variables for detailed naming conventions.
Content Sources
Load configuration from byte slices. This is useful for testing or when you have dynamic configuration:
configData := []byte(`{
"server": {
"port": 8080,
"host": "localhost"
}
}`)
cfg := config.MustNew(
config.WithContent(configData, codec.TypeJSON),
)
Use Cases for Content Sources
Testing:
func TestConfig(t *testing.T) {
testConfig := []byte(`port: 8080`)
cfg := config.MustNew(
config.WithContent(testConfig, codec.TypeYAML),
)
cfg.Load(context.Background())
assert.Equal(t, 8080, cfg.Int("port"))
}
Dynamic Configuration:
// Configuration from HTTP response
resp, _ := http.Get("https://config-server/config.json")
configBytes, _ := io.ReadAll(resp.Body)
cfg := config.MustNew(
config.WithContent(configBytes, codec.TypeJSON),
)
Remote Sources
Consul
Load configuration from HashiCorp Consul:
cfg := config.MustNew(
config.WithConsul("production/service.yaml"),
)
**Works without Consul:** If `CONSUL_HTTP_ADDR` isn't set, `WithConsul` does nothing. This means you can run your app locally without Consul. When you deploy to production, just set the environment variable and Consul will be used.
Environment variables:
export CONSUL_HTTP_ADDR=consul.example.com:8500
export CONSUL_HTTP_TOKEN=secret-token
Loading from Consul:
cfg := config.MustNew(
config.WithFile("config.yaml"), // Local defaults
config.WithConsul("staging/myapp.json"), // Staging overrides
config.WithEnv("MYAPP_"), // Environment overrides
)
The format is detected from the key path extension (.json, .yaml, .toml).
Custom Sources
Implement custom sources for any data source:
type Source interface {
Load(ctx context.Context) (map[string]any, error)
}
Example: Database Source
type DatabaseSource struct {
db *sql.DB
}
func (s *DatabaseSource) Load(ctx context.Context) (map[string]any, error) {
rows, err := s.db.QueryContext(ctx, "SELECT key, value FROM config")
if err != nil {
return nil, err
}
defer rows.Close()
config := make(map[string]any)
for rows.Next() {
var key, value string
if err := rows.Scan(&key, &value); err != nil {
return nil, err
}
config[key] = value
}
return config, nil
}
// Usage
cfg := config.MustNew(
config.WithSource(&DatabaseSource{db: db}),
)
Example: HTTP Source
type HTTPSource struct {
url string
codec codec.Codec
}
func (s *HTTPSource) Load(ctx context.Context) (map[string]any, error) {
req, _ := http.NewRequestWithContext(ctx, "GET", s.url, nil)
resp, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
data, err := io.ReadAll(resp.Body)
if err != nil {
return nil, err
}
var config map[string]any
if err := s.codec.Decode(data, &config); err != nil {
return nil, err
}
return config, nil
}
// Usage
cfg := config.MustNew(
config.WithSource(&HTTPSource{
url: "https://config-server/config.json",
codec: codec.JSON{},
}),
)
Multi-Environment Pattern
There are two ways to handle environment-specific configuration.
Using Path Expansion (Recommended)
The simplest approach is to use environment variables directly in paths:
cfg := config.MustNew(
config.WithFile("config.yaml"), // Base config
config.WithFile("${APP_ENV}/config.yaml"), // Environment-specific (e.g., "production/config.yaml")
config.WithEnv("MYAPP_"), // Environment variables
)
This is cleaner and works great when your config files are in environment-named folders.
Using String Concatenation
If you need more control or want to set a default, use Go code:
package main
import (
"context"
"log"
"os"
"rivaas.dev/config"
)
func loadConfig() *config.Config {
env := os.Getenv("APP_ENV")
if env == "" {
env = "development"
}
cfg := config.MustNew(
config.WithFile("config.yaml"), // Base config
config.WithFile("config."+env+".yaml"), // Environment-specific
config.WithEnv("MYAPP_"), // Environment variables
)
if err := cfg.Load(context.Background()); err != nil {
log.Fatalf("failed to load config: %v", err)
}
return cfg
}
func main() {
cfg := loadConfig()
// Use configuration
}
File structure:
config.yaml # Base configuration
config.development.yaml
config.staging.yaml
config.production.yaml
Dumping Configuration
Save the effective merged configuration to a file:
cfg := config.MustNew(
config.WithFile("config.yaml"),
config.WithFile("config.prod.yaml"),
config.WithEnv("MYAPP_"),
config.WithFileDumper("effective-config.yaml"),
)
cfg.Load(context.Background())
cfg.Dump(context.Background()) // Writes merged config to effective-config.yaml
Use Cases for Dumping
Debugging:
See the final merged configuration:
config.WithFileDumper("debug-config.json")
Configuration Snapshots:
Save configuration state for auditing:
timestamp := time.Now().Format("20060102-150405")
filename := fmt.Sprintf("config-snapshot-%s.yaml", timestamp)
config.WithFileDumper(filename)
Configuration Templates:
Generate configuration files:
cfg := config.MustNew(
config.WithEnv("MYAPP_"),
config.WithFileDumper("generated-config.yaml"),
)
Custom File Permissions
Control file permissions when dumping:
import "rivaas.dev/config/dumper"
// Default permissions (0644)
fileDumper := dumper.NewFile("config.yaml", encoder)
// Custom permissions (0600 - owner read/write only)
fileDumper := dumper.NewFileWithPermissions("config.yaml", encoder, 0600)
cfg := config.MustNew(
config.WithFile("config.yaml"),
config.WithDumper(fileDumper),
)
Custom Dumpers
Implement custom dumpers for any destination:
type Dumper interface {
Dump(ctx context.Context, data map[string]any) error
}
Example: S3 Dumper
type S3Dumper struct {
bucket string
key string
client *s3.Client
codec codec.Codec
}
func (d *S3Dumper) Dump(ctx context.Context, data map[string]any) error {
bytes, err := d.codec.Encode(data)
if err != nil {
return err
}
_, err = d.client.PutObject(ctx, &s3.PutObjectInput{
Bucket: aws.String(d.bucket),
Key: aws.String(d.key),
Body: bytes.NewReader(bytes),
})
return err
}
// Usage
cfg := config.MustNew(
config.WithFile("config.yaml"),
config.WithDumper(&S3Dumper{
bucket: "my-configs",
key: "app-config.json",
client: s3Client,
codec: codec.JSON{},
}),
)
Error Handling
Errors from sources include context about which source failed:
if err := cfg.Load(context.Background()); err != nil {
// Error format:
// "config error in source[0] during load: file not found: config.yaml"
// "config error in source[2] during load: consul key not found"
log.Printf("Configuration error: %v", err)
}
Complete Example
package main
import (
"context"
"log"
"os"
"rivaas.dev/config"
"rivaas.dev/config/codec"
)
type AppConfig struct {
Server struct {
Host string `config:"host"`
Port int `config:"port"`
} `config:"server"`
Database struct {
Host string `config:"host"`
Port int `config:"port"`
} `config:"database"`
}
func main() {
env := os.Getenv("APP_ENV")
if env == "" {
env = "development"
}
var appConfig AppConfig
cfg := config.MustNew(
// Base configuration
config.WithFile("config.yaml"),
// Environment-specific overrides
config.WithFile("config."+env+".yaml"),
// Remote configuration (production only)
func() config.Option {
if env == "production" {
return config.WithConsul("production/myapp.json")
}
return nil
}(),
// Environment variables (highest priority)
config.WithEnv("MYAPP_"),
// Struct binding
config.WithBinding(&appConfig),
// Dump effective config for debugging
config.WithFileDumper("effective-config.yaml"),
)
if err := cfg.Load(context.Background()); err != nil {
log.Fatalf("failed to load configuration: %v", err)
}
// Save effective configuration
if err := cfg.Dump(context.Background()); err != nil {
log.Printf("warning: failed to dump config: %v", err)
}
log.Printf("Server: %s:%d", appConfig.Server.Host, appConfig.Server.Port)
log.Printf("Database: %s:%d", appConfig.Database.Host, appConfig.Database.Port)
}
Next Steps
- Learn about Custom Codecs for custom formats
- See Examples for real-world multi-source patterns
- Explore Validation for configuration validation
For complete API details, see Options Reference.
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.