Configuration Management Guide

Complete guide to configuring KafkaGuard using configuration files, environment variables, and command-line flags.

Configuration Overview
Configuration Precedence
Configuration File Format
All Configuration Options
Environment Variables
Configuration File Locations
Sensitive Data Handling
Configuration Examples
Configuration Validation

Configuration Overview

KafkaGuard supports three configuration methods that can be used independently or combined:

Configuration File (.kafkaguard.yaml) - YAML file with all settings
Environment Variables (KAFKAGUARD_* prefix) - Shell environment variables
Command-Line Flags - Direct CLI arguments

When to Use Each Method

Method	Best For	Advantages	Disadvantages
Configuration File	Persistent settings, team sharing	Easy to version control, readable	May contain sensitive data if not careful
Environment Variables	Credentials, CI/CD, containerized environments	Secure (no file storage), easy to inject	Less readable, harder to document
Command-Line Flags	One-off scans, quick overrides	Explicit, temporary	Verbose, credentials visible in process list

Best Practice: Use configuration files for general settings, environment variables for credentials, and CLI flags for overrides.

Configuration Precedence

When the same option is specified in multiple places, KafkaGuard uses this precedence order:

1. Command-line flags (highest precedence)
   ↓
2. Environment variables
   ↓
3. Configuration file
   ↓
4. Default values (lowest precedence)

Precedence Example

Given these configurations:

# .kafkaguard.yaml
bootstrap: "kafka-config:9092"
policy: "policies/baseline-dev.yaml"
timeout: 300

# Environment variables
export KAFKAGUARD_BOOTSTRAP="kafka-env:9092"
export KAFKAGUARD_TIMEOUT=600

# Command line
kafkaguard scan --bootstrap kafka-cli:9092

Result:

bootstrap: kafka-cli:9092 (from CLI flag - highest precedence)
policy: policies/baseline-dev.yaml (from config file - no override)
timeout: 600 (from environment variable - overrides config file)

Configuration File Format

KafkaGuard uses YAML format for configuration files (.kafkaguard.yaml).

Basic Configuration File Structure

# .kafkaguard.yaml - KafkaGuard Configuration File

# Kafka Connection Settings
bootstrap: "kafka1:9092,kafka2:9092,kafka3:9092"
timeout: 300

# Policy Settings
policy: "policies/enterprise-default.yaml"

# Report Settings
out: "./reports"
format: ["json", "html", "pdf"]
fail-on: "high"

# Logging
log-level: "info"

# Security Settings
security:
  protocol: "SASL_SSL"
  sasl:
    mechanism: "SCRAM-SHA-512"
    username: "admin"
    # Password should be set via environment variable for security
    # password: "${KAFKA_PASSWORD}"
  tls:
    ca-cert: "/etc/kafkaguard/certs/ca.pem"
    client-cert: "/etc/kafkaguard/certs/client.pem"
    client-key: "/etc/kafkaguard/certs/client-key.pem"
    skip-verify: false

# Performance Settings (optional)
scan:
  parallel: true
  max-collectors: 6

Environment Variable References in Config Files

You can reference environment variables in configuration files using ${VAR_NAME} syntax:

bootstrap: "${KAFKA_BROKER}"
security:
  sasl:
    username: "${KAFKA_USERNAME}"
    password: "${KAFKA_PASSWORD}"

This allows you to keep sensitive data out of configuration files while still using them.

All Configuration Options

Complete reference of all available configuration options:

Connection Options

Option	Type	Default	Description	CLI Flag	Env Var
`bootstrap`	string	(required)	Kafka bootstrap servers (comma-separated)	`--bootstrap`, `-b`	`KAFKAGUARD_BOOTSTRAP`
`timeout`	int	300	Scan timeout in seconds	`--timeout`	`KAFKAGUARD_TIMEOUT`

Example:

bootstrap: "kafka1:9092,kafka2:9092"
timeout: 600

Policy Options

Option	Type	Default	Description	CLI Flag	Env Var
`policy`	string	`policies/baseline-dev.yaml`	Path to policy YAML file	`--policy`, `-p`	`KAFKAGUARD_POLICY`

Example:

policy: "policies/enterprise-default.yaml"

Report Options

Option	Type	Default	Description	CLI Flag	Env Var
`out`	string	`./reports`	Output directory for reports	`--out`, `-o`	`KAFKAGUARD_OUT`
`format`	string[]	`["json"]`	Report formats (json, html, pdf, csv)	`--format`, `-f`	`KAFKAGUARD_FORMAT`
`fail-on`	string	`high`	Fail threshold severity (high, medium, low, none)	`--fail-on`	`KAFKAGUARD_FAIL_ON`

Example:

out: "/var/reports/kafkaguard"
format: ["json", "html", "pdf", "csv"]
fail-on: "medium"

Logging Options

Option	Type	Default	Description	CLI Flag	Env Var
`log-level`	string	`info`	Log level (debug, info, warn, error)	`--log-level`, `-l`	`KAFKAGUARD_LOG_LEVEL`
`no-color`	bool	false	Disable colored output	`--no-color`	`KAFKAGUARD_NO_COLOR`

Example:

log-level: "debug"
no-color: false

Security Options

Protocol Settings

Option	Type	Default	Description	CLI Flag	Env Var
`security.protocol`	string	auto-detect	Security protocol (PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL)	`--security-protocol`	`KAFKAGUARD_SECURITY_PROTOCOL`

Example:

security:
  protocol: "SASL_SSL"

SASL Settings

Option	Type	Default	Description	CLI Flag	Env Var
`security.sasl.mechanism`	string	-	SASL mechanism (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512)	`--sasl-mechanism`	`KAFKAGUARD_SASL_MECHANISM`
`security.sasl.username`	string	-	SASL username	`--sasl-username`	`KAFKAGUARD_SASL_USERNAME`
`security.sasl.password`	string	-	SASL password (use env var recommended)	`--sasl-password`	`KAFKAGUARD_SASL_PASSWORD`

Example:

security:
  sasl:
    mechanism: "SCRAM-SHA-512"
    username: "kafkaguard"
    password: "${KAFKA_PASSWORD}"  # Use env var for security

TLS Settings

Option	Type	Default	Description	CLI Flag	Env Var
`security.tls.ca-cert`	string	-	Path to CA certificate file	`--tls-ca-cert`	`KAFKAGUARD_TLS_CA_CERT`
`security.tls.client-cert`	string	-	Path to client certificate (mTLS)	`--tls-client-cert`	`KAFKAGUARD_TLS_CLIENT_CERT`
`security.tls.client-key`	string	-	Path to client private key (mTLS)	`--tls-client-key`	`KAFKAGUARD_TLS_CLIENT_KEY`
`security.tls.skip-verify`	bool	false	Skip TLS certificate verification (testing only)	`--tls-insecure-skip-verify`	`KAFKAGUARD_TLS_INSECURE_SKIP_VERIFY`

Example:

security:
  tls:
    ca-cert: "/etc/kafkaguard/certs/ca.pem"
    client-cert: "/etc/kafkaguard/certs/client.pem"
    client-key: "/etc/kafkaguard/certs/client-key.pem"
    skip-verify: false

Performance Options

Option	Type	Default	Description	CLI Flag	Env Var
`scan.parallel`	bool	true	Enable parallel data collection	`--parallel`	`KAFKAGUARD_SCAN_PARALLEL`
`scan.max-collectors`	int	6	Maximum concurrent collectors	`--max-collectors`	`KAFKAGUARD_SCAN_MAX_COLLECTORS`

Example:

scan:
  parallel: true
  max-collectors: 10

Environment Variables

All KafkaGuard configuration options can be set via environment variables using the KAFKAGUARD_ prefix.

Environment Variable Naming Convention

Prefix all variables with KAFKAGUARD_
Convert option names to UPPERCASE
Replace hyphens (-) with underscores (_)
For nested options, use underscores to separate levels

Complete Environment Variable Reference

Connection Variables

export KAFKAGUARD_BOOTSTRAP="kafka1:9092,kafka2:9092"
export KAFKAGUARD_TIMEOUT="600"

Policy Variables

export KAFKAGUARD_POLICY="policies/enterprise-default.yaml"

Report Variables

export KAFKAGUARD_OUT="/var/reports"
export KAFKAGUARD_FORMAT="json,html,pdf"
export KAFKAGUARD_FAIL_ON="high"

Logging Variables

export KAFKAGUARD_LOG_LEVEL="debug"
export KAFKAGUARD_NO_COLOR="true"

Security Variables

# Protocol
export KAFKAGUARD_SECURITY_PROTOCOL="SASL_SSL"

# SASL
export KAFKAGUARD_SASL_MECHANISM="SCRAM-SHA-512"
export KAFKAGUARD_SASL_USERNAME="admin"
export KAFKAGUARD_SASL_PASSWORD="secret"

# TLS
export KAFKAGUARD_TLS_CA_CERT="/etc/kafkaguard/certs/ca.pem"
export KAFKAGUARD_TLS_CLIENT_CERT="/etc/kafkaguard/certs/client.pem"
export KAFKAGUARD_TLS_CLIENT_KEY="/etc/kafkaguard/certs/client-key.pem"
export KAFKAGUARD_TLS_INSECURE_SKIP_VERIFY="false"

Performance Variables

export KAFKAGUARD_SCAN_PARALLEL="true"
export KAFKAGUARD_SCAN_MAX_COLLECTORS="10"

Using Environment Files

Create a .env file for easy management:

# .env - KafkaGuard Environment Configuration
KAFKAGUARD_BOOTSTRAP=kafka.prod.example.com:9095
KAFKAGUARD_POLICY=policies/enterprise-default.yaml
KAFKAGUARD_SECURITY_PROTOCOL=SASL_SSL
KAFKAGUARD_SASL_MECHANISM=SCRAM-SHA-512
KAFKAGUARD_SASL_USERNAME=admin
KAFKAGUARD_SASL_PASSWORD=secret
KAFKAGUARD_TLS_CA_CERT=/etc/kafkaguard/certs/ca.pem
KAFKAGUARD_OUT=/var/reports
KAFKAGUARD_FORMAT=json,html,pdf
KAFKAGUARD_LOG_LEVEL=info

Load environment file:

# Source environment file
set -a
source .env
set +a

# Run scan (credentials loaded from environment)
kafkaguard scan

Security Note: Ensure .env files with credentials are:

Not committed to version control (add to .gitignore)
Protected with proper file permissions (chmod 600)
Stored securely (encrypted at rest if possible)

Configuration File Locations

KafkaGuard searches for configuration files in the following order:

Current Directory - ./.kafkaguard.yaml
User Home Directory - ~/.kafkaguard.yaml
System Config Directory - /etc/kafkaguard/config.yaml
Custom Path - Specified via --config flag

Search Order Example

# KafkaGuard searches in this order:
1. ./kafkaguard.yaml (current directory)
2. ~/.kafkaguard.yaml (home directory)
3. /etc/kafkaguard/config.yaml (system config)

# First file found is used. Subsequent files are ignored.

Specifying Custom Config File

# Use custom config file location
kafkaguard scan --config /opt/kafkaguard/custom-config.yaml

# Environment variable for config file
export KAFKAGUARD_CONFIG="/opt/kafkaguard/custom-config.yaml"
kafkaguard scan

Recommended Locations by Use Case

Use Case	Recommended Location	Reason
Development	`./.kafkaguard.yaml` (project directory)	Per-project settings, easy to version control
User-Specific	`~/.kafkaguard.yaml` (home directory)	Personal settings across all projects
System-Wide	`/etc/kafkaguard/config.yaml`	Shared settings for all users
CI/CD	Custom path via `--config`	Explicit, no assumptions about file locations

Sensitive Data Handling

Protect credentials and sensitive configuration data:

Best Practices

✅ DO: Use Environment Variables for Credentials

# .kafkaguard.yaml (safe - no credentials)
security:
  sasl:
    mechanism: "SCRAM-SHA-512"
    username: "${KAFKA_USERNAME}"  # Reference env var
    password: "${KAFKA_PASSWORD}"  # Reference env var

# Set credentials via environment
export KAFKA_USERNAME="admin"
export KAFKA_PASSWORD="secret"
kafkaguard scan

❌ DON'T: Store Credentials in Config Files

# .kafkaguard.yaml (UNSAFE - DO NOT DO THIS)
security:
  sasl:
    username: "admin"
    password: "secret"  # ❌ Credentials in plain text

Secure Credential Storage Options

Option 1: Environment Variables (Simple)

# Set in shell
export KAFKAGUARD_SASL_USERNAME="admin"
export KAFKAGUARD_SASL_PASSWORD="secret"
kafkaguard scan

Option 2: Secret Management Systems (Recommended for Production)

AWS Secrets Manager:

# Fetch from AWS Secrets Manager
export KAFKAGUARD_SASL_PASSWORD=$(aws secretsmanager get-secret-value \
  --secret-id kafka-admin-password \
  --query SecretString \
  --output text)

kafkaguard scan

HashiCorp Vault:

# Fetch from Vault
export KAFKAGUARD_SASL_PASSWORD=$(vault kv get -field=password secret/kafka/admin)
kafkaguard scan

Azure Key Vault:

# Fetch from Azure Key Vault
export KAFKAGUARD_SASL_PASSWORD=$(az keyvault secret show \
  --vault-name my-keyvault \
  --name kafka-admin-password \
  --query value -o tsv)

kafkaguard scan

Option 3: Encrypted Environment Files

# Create encrypted .env file with GPG
gpg --symmetric --cipher-algo AES256 .env

# Decrypt and load when needed
gpg --decrypt .env.gpg | set -a && source /dev/stdin && set +a
kafkaguard scan

File Permissions for Sensitive Files

# Protect configuration files with credentials
chmod 600 .kafkaguard.yaml

# Protect TLS private keys
chmod 600 /etc/kafkaguard/certs/client-key.pem

# Protect environment files
chmod 600 .env

# Verify permissions
ls -la .kafkaguard.yaml
# Expected: -rw------- (600)

Configuration Examples

Example 1: Development Environment

File: .kafkaguard.yaml

# Development cluster configuration
bootstrap: "localhost:9092"
policy: "policies/baseline-dev.yaml"
out: "./reports"
format: ["html", "json"]
log-level: "debug"
timeout: 300

# No security (PLAINTEXT)
security:
  protocol: "PLAINTEXT"

Usage:

kafkaguard scan  # Uses config file

Example 2: Production Environment

File: /etc/kafkaguard/config.yaml

# Production cluster configuration
bootstrap: "kafka1.prod.example.com:9095,kafka2.prod.example.com:9095,kafka3.prod.example.com:9095"
policy: "policies/enterprise-default.yaml"
out: "/var/reports/kafkaguard"
format: ["json", "html", "pdf"]
fail-on: "high"
log-level: "info"
timeout: 600

# Security settings (credentials via environment variables)
security:
  protocol: "SASL_SSL"
  sasl:
    mechanism: "SCRAM-SHA-512"
    username: "${KAFKA_USERNAME}"
    password: "${KAFKA_PASSWORD}"
  tls:
    ca-cert: "/etc/kafkaguard/certs/ca.pem"
    skip-verify: false

# Performance tuning for large cluster
scan:
  parallel: true
  max-collectors: 10

Usage:

# Set credentials via environment
export KAFKA_USERNAME="kafkaguard-prod"
export KAFKA_PASSWORD="$(vault kv get -field=password secret/kafka/prod)"

# Run scan
kafkaguard scan --config /etc/kafkaguard/config.yaml

Example 3: CI/CD Environment

File: .kafkaguard.yaml (in repository)

# CI/CD cluster configuration
policy: "policies/enterprise-default.yaml"
format: ["json"]
fail-on: "high"
log-level: "warn"
no-color: true
timeout: 300

# All other settings via environment variables

Environment Variables (in CI/CD):

# Set via CI/CD secrets
export KAFKAGUARD_BOOTSTRAP="${CI_KAFKA_BROKER}"
export KAFKAGUARD_SASL_USERNAME="${CI_KAFKA_USERNAME}"
export KAFKAGUARD_SASL_PASSWORD="${CI_KAFKA_PASSWORD}"
export KAFKAGUARD_SECURITY_PROTOCOL="SASL_SSL"
export KAFKAGUARD_SASL_MECHANISM="SCRAM-SHA-512"
export KAFKAGUARD_TLS_CA_CERT="${CI_WORKSPACE}/certs/ca.pem"
export KAFKAGUARD_OUT="${CI_WORKSPACE}/reports"

Usage:

kafkaguard scan  # Credentials from CI/CD environment

Example 4: Air-Gapped Environment

File: /opt/kafkaguard/config.yaml

# Air-gapped cluster configuration
bootstrap: "kafka-internal1:9092,kafka-internal2:9092"
policy: "/opt/kafkaguard/policies/enterprise-default.yaml"
out: "/mnt/reports/kafkaguard"
format: ["json", "html", "pdf", "csv"]
fail-on: "medium"
log-level: "info"
timeout: 900

# Internal cluster - PLAINTEXT or SSL (no internet-based auth)
security:
  protocol: "SSL"
  tls:
    ca-cert: "/opt/kafkaguard/certs/ca.pem"
    client-cert: "/opt/kafkaguard/certs/client.pem"
    client-key: "/opt/kafkaguard/certs/client-key.pem"
    skip-verify: false

# Optimize for large cluster
scan:
  parallel: true
  max-collectors: 10

Usage:

kafkaguard scan --config /opt/kafkaguard/config.yaml

Example 5: Multi-Cluster Configuration

Separate config files per cluster:

dev-cluster.yaml:

bootstrap: "kafka-dev:9092"
policy: "policies/baseline-dev.yaml"
out: "./reports/dev"
format: ["json", "html"]
log-level: "debug"

staging-cluster.yaml:

bootstrap: "kafka-staging:9095"
policy: "policies/enterprise-default.yaml"
out: "./reports/staging"
format: ["json", "html"]
security:
  protocol: "SASL_SSL"
  sasl:
    mechanism: "SCRAM-SHA-512"

prod-cluster.yaml:

bootstrap: "kafka-prod:9095"
policy: "policies/enterprise-default.yaml"
out: "./reports/prod"
format: ["json", "html", "pdf"]
fail-on: "high"
security:
  protocol: "SASL_SSL"
  sasl:
    mechanism: "SCRAM-SHA-512"
  tls:
    ca-cert: "/etc/kafkaguard/certs/ca.pem"

Multi-cluster scan script:

#!/bin/bash
# Scan all clusters

for env in dev staging prod; do
  echo "Scanning $env cluster..."
  kafkaguard scan --config "${env}-cluster.yaml"
done

Configuration Validation

Check Active Configuration

Use debug logging to see which configuration is being used:

kafkaguard scan --log-level debug --bootstrap kafka:9092

# Output shows configuration resolution:
# DEBUG Config file: /home/user/.kafkaguard.yaml
# DEBUG Bootstrap: kafka:9092 (from CLI flag)
# DEBUG Policy: policies/baseline-dev.yaml (from config file)
# DEBUG Timeout: 600 (from environment variable)

Validate Configuration File Syntax

# Validate YAML syntax
yamllint .kafkaguard.yaml

# Or use a YAML parser
python3 -c "import yaml; yaml.safe_load(open('.kafkaguard.yaml'))"

# Test configuration with dry run
kafkaguard scan --log-level debug --bootstrap test:9092 --timeout 1

Common Configuration Errors

Error	Cause	Fix
`config file not found`	File doesn't exist in search path	Create file or use `--config` with correct path
`invalid YAML syntax`	YAML parsing error	Check indentation, colons, quotes
`unknown configuration key`	Typo in option name	Check spelling against documentation
`invalid value for 'fail-on'`	Invalid threshold value	Use: `high`, `medium`, `low`, or `none`
`invalid security protocol`	Typo in protocol name	Use: `PLAINTEXT`, `SSL`, `SASL_PLAINTEXT`, `SASL_SSL`

Next Steps

Security & Authentication - Configure SASL/SSL security
CLI Reference - Learn all CLI flags
Usage Patterns - See configuration in action
Troubleshooting - Resolve configuration issues

Document Information

Last Updated: 2025-11-15
Applies to Version: KafkaGuard 1.0.0+
Feedback: Open an issue for corrections or improvements

Configuration Management Guide

Table of Contents

Configuration Overview

When to Use Each Method

Configuration Precedence

Precedence Example

Configuration File Format

Basic Configuration File Structure

Environment Variable References in Config Files

All Configuration Options

Connection Options

Policy Options

Report Options

Logging Options

Security Options

Protocol Settings

SASL Settings

TLS Settings

Performance Options

Environment Variables

Environment Variable Naming Convention

Complete Environment Variable Reference

Connection Variables

Policy Variables

Report Variables

Logging Variables

Security Variables

Performance Variables

Using Environment Files

Configuration File Locations

Search Order Example

Specifying Custom Config File

Recommended Locations by Use Case

Sensitive Data Handling

Best Practices

✅ DO: Use Environment Variables for Credentials

❌ DON'T: Store Credentials in Config Files

Secure Credential Storage Options

Option 1: Environment Variables (Simple)

Option 2: Secret Management Systems (Recommended for Production)

Option 3: Encrypted Environment Files

File Permissions for Sensitive Files

Configuration Examples

Example 1: Development Environment

Example 2: Production Environment

Example 3: CI/CD Environment

Example 4: Air-Gapped Environment

Example 5: Multi-Cluster Configuration

Configuration Validation

Check Active Configuration

Validate Configuration File Syntax

Common Configuration Errors

Next Steps