Security & Authentication Guide

Comprehensive guide to configuring security and authentication for KafkaGuard when scanning secured Kafka clusters.

Security Overview
Security Protocols
SASL Authentication
SSL/TLS Encryption
Mutual TLS (mTLS)
Security Best Practices
Auto-Detection
Authentication Troubleshooting

Security Overview

KafkaGuard supports all standard Kafka security protocols, allowing you to scan clusters with various security configurations:

PLAINTEXT - No encryption, no authentication (dev/sandbox only)
SSL - TLS encryption without authentication
SASL_PLAINTEXT - SASL authentication without encryption (testing only)
SASL_SSL - TLS encryption + SASL authentication (recommended for production)

Security Protocol Selection Matrix

Protocol	Encryption	Authentication	Use Case	Production Ready
PLAINTEXT	❌ None	❌ None	Dev/sandbox	❌ No
SSL	✅ TLS	⚠️ Optional (mTLS)	Internal networks	⚠️ Limited
SASL_PLAINTEXT	❌ None	✅ SASL	Testing only	❌ Never
SASL_SSL	✅ TLS	✅ SASL	Production	✅ Yes

Security Protocols

PLAINTEXT (Port 9092)

Security Level: None Encryption: ❌ No Authentication: ❌ No

Use Cases

Development and sandbox Kafka clusters on trusted networks
Internal testing environments with network-level security (VPN, firewall)
Proof-of-concept and demo environments

Configuration

# Command line
kafkaguard scan --bootstrap kafka-dev:9092

# Configuration file
security:
  protocol: "PLAINTEXT"

# Environment variable
export KAFKAGUARD_SECURITY_PROTOCOL="PLAINTEXT"

Example

kafkaguard scan \
  --bootstrap localhost:9092 \
  --policy policies/baseline-dev.yaml \
  --format html,json \
  --out ./reports

⚠️ Warning: Never use PLAINTEXT for production clusters or clusters handling sensitive data.

SSL (Port 9093)

Security Level: Medium Encryption: ✅ TLS Authentication: ⚠️ Optional (via mTLS)

Use Cases

Encryption-only requirements (no username/password)
Internal networks where client authentication is handled elsewhere
When combined with mutual TLS for certificate-based authentication

Configuration

# Command line
kafkaguard scan \
  --bootstrap kafka:9093 \
  --security-protocol SSL \
  --tls-ca-cert /path/to/ca-cert.pem

# Configuration file
security:
  protocol: "SSL"
  tls:
    ca-cert: "/etc/kafkaguard/certs/ca.pem"

# Environment variables
export KAFKAGUARD_SECURITY_PROTOCOL="SSL"
export KAFKAGUARD_TLS_CA_CERT="/etc/kafkaguard/certs/ca.pem"

Example

kafkaguard scan \
  --bootstrap kafka.internal:9093 \
  --security-protocol SSL \
  --tls-ca-cert /etc/kafkaguard/certs/ca.pem \
  --policy policies/enterprise-default.yaml \
  --format html,json \
  --out ./reports

SASL_PLAINTEXT (Port 9094)

Security Level: Low Encryption: ❌ No Authentication: ✅ SASL

Use Cases

Testing SASL configuration before enabling TLS
Development environments only (never production)

⚠️ WARNING: Credentials are transmitted unencrypted. Never use in production.

Configuration

# Command line
kafkaguard scan \
  --bootstrap kafka:9094 \
  --security-protocol SASL_PLAINTEXT \
  --sasl-mechanism SCRAM-SHA-512 \
  --sasl-username dev-user \
  --sasl-password dev-pass

# Configuration file
security:
  protocol: "SASL_PLAINTEXT"
  sasl:
    mechanism: "SCRAM-SHA-512"
    username: "dev-user"
    password: "${KAFKA_PASSWORD}"  # Use env var

❌ DO NOT USE IN PRODUCTION - Credentials are visible to network sniffers.

SASL_SSL (Port 9095) - RECOMMENDED

Security Level: High Encryption: ✅ TLS Authentication: ✅ SASL

Use Cases

✅ Production Kafka clusters
✅ Clusters handling customer data or PII
✅ SOC2, PCI-DSS, ISO 27001 compliance requirements
✅ Any cluster exposed to untrusted networks

Configuration

# Command line
kafkaguard scan \
  --bootstrap kafka.prod:9095 \
  --security-protocol SASL_SSL \
  --sasl-mechanism SCRAM-SHA-512 \
  --sasl-username admin \
  --sasl-password secret \
  --tls-ca-cert /path/to/ca-cert.pem

# Configuration file
security:
  protocol: "SASL_SSL"
  sasl:
    mechanism: "SCRAM-SHA-512"
    username: "${KAFKA_USERNAME}"
    password: "${KAFKA_PASSWORD}"
  tls:
    ca-cert: "/etc/kafkaguard/certs/ca.pem"

# Environment variables (recommended)
export KAFKAGUARD_SECURITY_PROTOCOL="SASL_SSL"
export KAFKAGUARD_SASL_MECHANISM="SCRAM-SHA-512"
export KAFKAGUARD_SASL_USERNAME="admin"
export KAFKAGUARD_SASL_PASSWORD="secret"
export KAFKAGUARD_TLS_CA_CERT="/etc/kafkaguard/certs/ca.pem"

Example

# Set credentials via environment (secure)
export KAFKAGUARD_SASL_USERNAME="kafkaguard"
export KAFKAGUARD_SASL_PASSWORD="$(vault kv get -field=password secret/kafka/prod)"

# Run scan
kafkaguard scan \
  --bootstrap kafka1.prod:9095,kafka2.prod:9095,kafka3.prod:9095 \
  --security-protocol SASL_SSL \
  --sasl-mechanism SCRAM-SHA-512 \
  --tls-ca-cert /etc/kafkaguard/certs/ca.pem \
  --policy policies/enterprise-default.yaml \
  --format json,html,pdf \
  --out /var/reports

SASL Authentication

SASL (Simple Authentication and Security Layer) provides username/password authentication for Kafka. KafkaGuard supports three SASL mechanisms.

SASL Mechanisms Comparison

Mechanism	Security Level	Use Case	Recommended
PLAIN	Low	Dev/testing only	❌ Not for production
SCRAM-SHA-256	High	Production	✅ Yes
SCRAM-SHA-512	Highest	Production (preferred)	✅ Strongly recommended
GSSAPI (Kerberos)	High	Enterprise SSO	⏭️ Phase 2

SASL/PLAIN

Security: Base64 encoded (easily decoded) Use Case: Development and testing only

Configuration

# NEVER use PLAIN with SASL_PLAINTEXT in production
kafkaguard scan \
  --bootstrap kafka-dev:9094 \
  --security-protocol SASL_PLAINTEXT \
  --sasl-mechanism PLAIN \
  --sasl-username dev-user \
  --sasl-password dev-pass

# If you must use PLAIN, always combine with SSL
kafkaguard scan \
  --bootstrap kafka-dev:9095 \
  --security-protocol SASL_SSL \
  --sasl-mechanism PLAIN \
  --sasl-username dev-user \
  --sasl-password dev-pass \
  --tls-ca-cert /path/to/ca-cert.pem

⚠️ Warning: PLAIN transmits credentials as base64-encoded plaintext. Always use with SASL_SSL, never with SASL_PLAINTEXT.

SASL/SCRAM-SHA-256

Security: Salted Challenge Response Authentication Mechanism with SHA-256 Use Case: Production environments

How SCRAM Works

SCRAM is a challenge-response mechanism that never transmits passwords over the network:

Client sends username
Server sends challenge (salt + iteration count)
Client computes response using password + challenge
Server validates response without ever seeing the password

Configuration

# Command line
kafkaguard scan \
  --bootstrap kafka:9095 \
  --security-protocol SASL_SSL \
  --sasl-mechanism SCRAM-SHA-256 \
  --sasl-username admin \
  --sasl-password secret \
  --tls-ca-cert /path/to/ca-cert.pem

# Configuration file
security:
  protocol: "SASL_SSL"
  sasl:
    mechanism: "SCRAM-SHA-256"
    username: "${KAFKA_USERNAME}"
    password: "${KAFKA_PASSWORD}"
  tls:
    ca-cert: "/etc/kafkaguard/certs/ca.pem"

Example

export KAFKAGUARD_SASL_USERNAME="kafkaguard"
export KAFKAGUARD_SASL_PASSWORD="complex-password-123"

kafkaguard scan \
  --bootstrap kafka.prod:9095 \
  --security-protocol SASL_SSL \
  --sasl-mechanism SCRAM-SHA-256 \
  --tls-ca-cert /etc/kafkaguard/certs/ca.pem \
  --policy policies/enterprise-default.yaml

SASL/SCRAM-SHA-512 (RECOMMENDED)

Security: Strongest SCRAM variant with SHA-512 hashing Use Case: Production (recommended over SHA-256)

Why SCRAM-SHA-512?

Stronger cryptographic hashing (512-bit vs 256-bit)
Better resistance to collision attacks
Recommended by NIST for high-security applications
No performance penalty for KafkaGuard use case

Configuration

# Command line (recommended approach with env vars)
export KAFKAGUARD_SASL_USERNAME="kafkaguard"
export KAFKAGUARD_SASL_PASSWORD="$(vault kv get -field=password secret/kafka/prod)"

kafkaguard scan \
  --bootstrap kafka1.prod:9095,kafka2.prod:9095,kafka3.prod:9095 \
  --security-protocol SASL_SSL \
  --sasl-mechanism SCRAM-SHA-512 \
  --tls-ca-cert /etc/kafkaguard/certs/ca.pem \
  --policy policies/enterprise-default.yaml \
  --format json,html,pdf \
  --out /var/reports

# Configuration file
security:
  protocol: "SASL_SSL"
  sasl:
    mechanism: "SCRAM-SHA-512"
    username: "${KAFKA_USERNAME}"
    password: "${KAFKA_PASSWORD}"
  tls:
    ca-cert: "/etc/kafkaguard/certs/ca.pem"

Example with Secret Manager

# AWS Secrets Manager
export KAFKAGUARD_SASL_PASSWORD=$(aws secretsmanager get-secret-value \
  --secret-id kafka-admin-password \
  --query SecretString \
  --output text)

# HashiCorp Vault
export KAFKAGUARD_SASL_PASSWORD=$(vault kv get -field=password secret/kafka/prod)

# Azure Key Vault
export KAFKAGUARD_SASL_PASSWORD=$(az keyvault secret show \
  --vault-name my-keyvault \
  --name kafka-admin-password \
  --query value -o tsv)

# Run scan with fetched credentials
kafkaguard scan \
  --bootstrap kafka.prod:9095 \
  --security-protocol SASL_SSL \
  --sasl-mechanism SCRAM-SHA-512 \
  --sasl-username kafkaguard \
  --tls-ca-cert /etc/kafkaguard/certs/ca.pem

SSL/TLS Encryption

TLS (Transport Layer Security) encrypts communication between KafkaGuard and Kafka brokers.

TLS Configuration Options

Basic TLS with CA Certificate

kafkaguard scan \
  --bootstrap kafka:9093 \
  --security-protocol SSL \
  --tls-ca-cert /path/to/ca-cert.pem

TLS with SASL Authentication

kafkaguard scan \
  --bootstrap kafka:9095 \
  --security-protocol SASL_SSL \
  --sasl-mechanism SCRAM-SHA-512 \
  --sasl-username admin \
  --sasl-password secret \
  --tls-ca-cert /path/to/ca-cert.pem

Certificate Requirements

CA Certificate

Format: PEM (text-based X.509 certificate)
Purpose: Verify server certificates from Kafka brokers
Required: Yes (for SSL and SASL_SSL protocols)
File Extension: .pem, .crt, or .cert

Example CA Certificate:

-----BEGIN CERTIFICATE-----
MIIDXTCCAkWgAwIBAgIJAKL0UG+mRKfNMA0GCSqGSIb3DQEBCwUAMEUxCzAJBgNV
BAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEwHwYDVQQKDBhJbnRlcm5ldCBX
...
-----END CERTIFICATE-----

Certificate Validation

KafkaGuard automatically validates:

Certificate Expiry - Must not be expired
Certificate Chain - Complete chain from leaf to root CA
Hostname Verification - Certificate CN/SAN must match broker hostname
TLS Protocol Version - Minimum TLS 1.2

Certificate Validation Errors:

Error	Cause	Fix
`certificate has expired`	Certificate expired	Renew certificate
`certificate verify failed`	Invalid certificate chain	Provide correct CA certificate
`hostname mismatch`	CN/SAN doesn't match broker hostname	Update certificate or use correct hostname
`tls: protocol version not supported`	Broker using TLS 1.0/1.1	Upgrade broker to TLS 1.2+

Self-Signed Certificates

For development environments with self-signed certificates:

# Provide CA certificate for self-signed cert
kafkaguard scan \
  --bootstrap kafka-dev:9093 \
  --security-protocol SSL \
  --tls-ca-cert /path/to/self-signed-ca.pem

⚠️ NEVER use --tls-insecure-skip-verify in production - this disables all certificate validation.

Certificate Troubleshooting

Test Certificate Manually

# Test TLS connection with openssl
openssl s_client -connect kafka.example.com:9093 -CAfile /path/to/ca-cert.pem

# Check certificate expiry
openssl x509 -in /path/to/ca-cert.pem -noout -dates

# Verify certificate chain
openssl verify -CAfile /path/to/ca-cert.pem /path/to/broker-cert.pem

Debug TLS Issues

# Enable debug logging
kafkaguard scan \
  --bootstrap kafka:9093 \
  --security-protocol SSL \
  --tls-ca-cert /path/to/ca-cert.pem \
  --log-level debug

Mutual TLS (mTLS)

Mutual TLS provides strong client authentication using X.509 certificates. Both the server and client authenticate each other.

When to Use mTLS

Industry	Use Case	Reason
Financial Services	Banks, payment processors	PCI-DSS requirements, zero-trust networks
Healthcare	HIPAA-compliant systems	PHI protection, regulatory compliance
Government	Classified data systems	FIPS 140-2, high assurance
Enterprise	Zero-trust architectures	Defense in depth, certificate-based auth

mTLS Configuration

Required Files

CA Certificate - Server certificate validation
Client Certificate - Client authentication
Client Private Key - Paired with client certificate

kafkaguard scan \
  --bootstrap kafka:9093 \
  --security-protocol SSL \
  --tls-ca-cert /path/to/ca-cert.pem \
  --tls-client-cert /path/to/client-cert.pem \
  --tls-client-key /path/to/client-key.pem

mTLS with SASL (Defense in Depth)

Combine mTLS with SASL for maximum security:

kafkaguard scan \
  --bootstrap kafka:9095 \
  --security-protocol SASL_SSL \
  --sasl-mechanism SCRAM-SHA-512 \
  --sasl-username admin \
  --sasl-password secret \
  --tls-ca-cert /path/to/ca-cert.pem \
  --tls-client-cert /path/to/client-cert.pem \
  --tls-client-key /path/to/client-key.pem

Certificate Format Requirements

All certificates must be in PEM format (text-based):

CA Certificate (`ca-cert.pem`)

-----BEGIN CERTIFICATE-----
MIIDXTCCAkWgAwIBAgIJAKL0UG+mRKfNMA0GCSqGSIb3DQEBCwUAMEUxCzAJBgNV
...
-----END CERTIFICATE-----

Client Certificate (`client-cert.pem`)

-----BEGIN CERTIFICATE-----
MIIDPzCCAiegAwIBAgIJAKL0UG+mRKfOMA0GCSqGSIb3DQEBCwUAMEUxCzAJBgNV
...
-----END CERTIFICATE-----

Client Private Key (`client-key.pem`)

-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC7VJTUt9Us8cKj
...
-----END PRIVATE KEY-----

Supported Key Types:

RSA (2048+ bits recommended)
ECDSA (256+ bits recommended)

Certificate Generation Example

# Generate CA private key
openssl genrsa -out ca-key.pem 4096

# Generate CA certificate
openssl req -new -x509 -days 365 -key ca-key.pem -out ca-cert.pem

# Generate client private key
openssl genrsa -out client-key.pem 4096

# Generate client certificate signing request
openssl req -new -key client-key.pem -out client.csr

# Sign client certificate with CA
openssl x509 -req -days 365 -in client.csr -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 -out client-cert.pem

mTLS Troubleshooting

Certificate Mismatch Error

Error: tls: private key does not match public key

Cause: Client certificate and key don't match (not generated from same CSR).

Fix: Regenerate certificate and key together.

Missing Certificates Error

Error: both --tls-client-cert and --tls-client-key must be provided for mTLS

Cause: Only one of client cert or key provided.

Fix: Provide both flags.

Certificate Validation Failed

Error: tls: bad certificate

Cause: Client certificate not trusted by server CA.

Fix: Ensure client certificate is signed by CA trusted by Kafka broker.

File Permissions for Private Keys

CRITICAL: Protect private keys with proper file permissions:

# Set restrictive permissions on private key
chmod 600 /path/to/client-key.pem

# Verify permissions
ls -l /path/to/client-key.pem
# Expected: -rw------- (600)

# Certificates can be less restrictive
chmod 644 /path/to/client-cert.pem
chmod 644 /path/to/ca-cert.pem

Security Best Practices

1. Always Use SASL_SSL for Production

✅ DO:

kafkaguard scan \
  --bootstrap kafka.prod:9095 \
  --security-protocol SASL_SSL \
  --sasl-mechanism SCRAM-SHA-512 \
  --tls-ca-cert /path/to/ca-cert.pem

❌ DON'T:

# NEVER use PLAINTEXT in production
kafkaguard scan --bootstrap kafka.prod:9092

# NEVER use SASL_PLAINTEXT (credentials in plaintext)
kafkaguard scan --bootstrap kafka.prod:9094 --security-protocol SASL_PLAINTEXT

2. Prefer SCRAM-SHA-512 Over SCRAM-SHA-256

✅ DO:

--sasl-mechanism SCRAM-SHA-512  # Strongest

⚠️ ACCEPTABLE:

--sasl-mechanism SCRAM-SHA-256  # Still secure

❌ DON'T:

--sasl-mechanism PLAIN  # Weak, easily decoded

3. Use Environment Variables for Passwords

✅ DO:

export KAFKAGUARD_SASL_USERNAME="admin"
export KAFKAGUARD_SASL_PASSWORD="secret"
kafkaguard scan --bootstrap kafka:9095 --security-protocol SASL_SSL

❌ DON'T:

# Credentials visible in process list and shell history
kafkaguard scan --sasl-username admin --sasl-password secret

4. Use Secret Management Systems

✅ DO:

# Fetch from Vault
export KAFKAGUARD_SASL_PASSWORD=$(vault kv get -field=password secret/kafka/prod)

# Fetch from AWS Secrets Manager
export KAFKAGUARD_SASL_PASSWORD=$(aws secretsmanager get-secret-value --secret-id kafka-admin)

# Run scan with fetched credentials
kafkaguard scan

❌ DON'T:

# Hardcoded credentials in config file
security:
  sasl:
    username: "admin"
    password: "secret"  # ❌ Insecure

5. Rotate Credentials Regularly

Recommended Rotation Schedule:

Environment	Rotation Frequency	Reason
Development	90 days	Lower risk, less frequent changes
Staging	60 days	Mirrors production schedule
Production	30-90 days	Industry standard (SOC2, PCI-DSS)
High Security	30 days	Maximum security posture

Rotation Procedure:

Create new credentials on Kafka broker
Update KafkaGuard configuration with new credentials
Test scan with new credentials
Deactivate old credentials after verification
Update all automation (CI/CD, scheduled scans)

6. Validate TLS Certificates

✅ DO:

# Always validate certificates in production
kafkaguard scan \
  --bootstrap kafka:9093 \
  --security-protocol SSL \
  --tls-ca-cert /path/to/ca-cert.pem

❌ DON'T:

# NEVER skip verification in production
kafkaguard scan \
  --bootstrap kafka:9093 \
  --security-protocol SSL \
  --tls-insecure-skip-verify  # ❌ INSECURE

Only skip verification for:

Local development with self-signed certificates
Testing certificate configuration
NEVER in production

7. Use mTLS for Highly Regulated Environments

✅ DO (High Security):

# mTLS + SASL for defense in depth
kafkaguard scan \
  --bootstrap kafka:9095 \
  --security-protocol SASL_SSL \
  --sasl-mechanism SCRAM-SHA-512 \
  --tls-ca-cert /path/to/ca-cert.pem \
  --tls-client-cert /path/to/client-cert.pem \
  --tls-client-key /path/to/client-key.pem

8. Monitor Authentication Failures

Set up alerting for:

Repeated authentication failures (potential attack)
Certificate expiry warnings (KG-005 control)
Deprecated protocol usage (TLS <1.2)
Failed scans due to auth errors

Example monitoring script:

#!/bin/bash
# Monitor scan failures

kafkaguard scan \
  --bootstrap kafka:9095 \
  --policy policies/enterprise-default.yaml \
  --log-level warn \
  2>&1 | grep -i "authentication failed"

if [ $? -eq 0 ]; then
  # Send alert (Slack, email, PagerDuty, etc.)
  echo "Authentication failure detected!" | mail -s "KafkaGuard Alert" security@example.com
fi

9. Implement Least Privilege Access

KafkaGuard only needs read-only permissions on Kafka clusters:

Required Kafka ACLs:

# Minimum required permissions for KafkaGuard
ALLOW principal User:kafkaguard to DESCRIBE on Cluster:kafka-cluster
ALLOW principal User:kafkaguard to DESCRIBE on Topic:*
ALLOW principal User:kafkaguard to DESCRIBE on Group:*

NOT required:

❌ WRITE permissions (KafkaGuard never writes data)
❌ DELETE permissions
❌ ALTER permissions
❌ CREATE permissions

10. Audit and Log All Scans

Enable comprehensive logging:

kafkaguard scan \
  --bootstrap kafka:9095 \
  --log-level info \
  2>&1 | tee /var/log/kafkaguard/scan-$(date +%Y%m%d-%H%M%S).log

Log retention recommendations:

Development: 30 days
Production: 90+ days (required for PCI-DSS)
Compliance: Per regulatory requirements (1-7 years)

Auto-Detection

KafkaGuard can automatically detect the security protocol from broker metadata when --security-protocol is not specified.

How Auto-Detection Works

Connects to broker using PLAINTEXT protocol (most compatible for initial connection)
Queries broker metadata for listener configuration
Extracts listener.security.protocol.map from broker config
Uses detected protocol for subsequent connections
Falls back to PLAINTEXT if detection fails

Auto-Detection Behavior

# Auto-detect protocol (no --security-protocol flag)
kafkaguard scan --bootstrap kafka:9092

# Output shows detected protocol:
# INFO Auto-detected security protocol: SASL_SSL
# INFO Connecting with SASL_SSL...

When Auto-Detection Works Well

✅ Good for:

Development environments with consistent security config
Testing against various cluster types
Environments with dynamic security configuration

⚠️ Not recommended for:

Production clusters (explicit configuration is clearer)
Clusters requiring SASL authentication for metadata queries
Multi-protocol listener configurations

Manual Override

Always override auto-detection in production:

# Explicitly specify protocol (recommended)
kafkaguard scan \
  --bootstrap kafka:9095 \
  --security-protocol SASL_SSL \
  --sasl-mechanism SCRAM-SHA-512

Authentication Troubleshooting

Common SASL Errors

Error: SASL authentication failed

Full Error:

Error: SASL authentication failed: kafka: client has run out of available brokers to talk to

Possible Causes:

Invalid username or password
SASL mechanism mismatch (SHA-256 vs SHA-512)
Wrong security protocol
Broker SASL configuration not enabled

Fix:

# 1. Verify credentials
echo "Username: $KAFKAGUARD_SASL_USERNAME"
echo "Password length: ${#KAFKAGUARD_SASL_PASSWORD}"

# 2. Check broker configuration
# On Kafka broker, verify:
# sasl.enabled.mechanisms=SCRAM-SHA-512
# listeners=SASL_SSL://0.0.0.0:9095

# 3. Test with debug logging
kafkaguard scan \
  --bootstrap kafka:9095 \
  --security-protocol SASL_SSL \
  --sasl-mechanism SCRAM-SHA-512 \
  --log-level debug

# 4. Verify credentials on broker
kafka-configs.sh --bootstrap-server kafka:9095 \
  --describe --entity-type users --entity-name admin

Error: Mechanism mismatch

Full Error:

Error: SASL mechanism 'SCRAM-SHA-256' is not supported

Cause: Broker configured for different SASL mechanism than client.

Fix:

# Check broker's enabled mechanisms
# Look for: sasl.enabled.mechanisms in server.properties

# Match client to broker configuration:
kafkaguard scan \
  --sasl-mechanism SCRAM-SHA-512  # Match broker config

Common TLS Errors

Error: Certificate verification failed

Full Error:

Error: x509: certificate signed by unknown authority

Cause: CA certificate not provided or incorrect.

Fix:

# Provide correct CA certificate
kafkaguard scan \
  --bootstrap kafka:9093 \
  --security-protocol SSL \
  --tls-ca-cert /path/to/correct-ca-cert.pem

# Verify CA certificate contains the signing authority
openssl x509 -in /path/to/ca-cert.pem -text -noout | grep Issuer

Error: TLS handshake failed

Full Error:

Error: tls: protocol version not supported

Cause: Broker using TLS 1.0 or 1.1 (deprecated).

Fix:

# Check broker TLS version
openssl s_client -connect kafka:9093 -tls1_2

# Upgrade broker configuration to TLS 1.2+
# In server.properties:
# ssl.protocol=TLSv1.2

Error: Hostname verification failed

Full Error:

Error: x509: certificate is valid for kafka-broker, not kafka.example.com

Cause: Certificate CN/SAN doesn't match broker hostname.

Fix:

# Option 1: Use correct hostname from certificate
kafkaguard scan --bootstrap kafka-broker:9093

# Option 2: Generate new certificate with correct hostname
# Include hostname in CN or SAN when generating certificate

Debugging Authentication Issues

Enable Debug Logging

kafkaguard scan \
  --bootstrap kafka:9095 \
  --security-protocol SASL_SSL \
  --sasl-mechanism SCRAM-SHA-512 \
  --log-level debug \
  2>&1 | tee debug.log

Test Kafka Connectivity

# Test TCP connection
telnet kafka.example.com 9095

# Test TLS connection
openssl s_client -connect kafka.example.com:9095 -CAfile /path/to/ca-cert.pem

# Test with kafka-console-consumer (validates credentials)
kafka-console-consumer.sh \
  --bootstrap-server kafka:9095 \
  --command-config client.properties \
  --topic __consumer_offsets \
  --max-messages 1

Verify Kafka ACLs

# Check KafkaGuard user ACLs
kafka-acls.sh --bootstrap-server kafka:9095 \
  --command-config admin.properties \
  --list --principal User:kafkaguard

Next Steps

Configuration Guide - Configure security settings in config files
Usage Patterns - See security in action with real workflows
Troubleshooting - Resolve authentication and connection issues
CLI Reference - All security-related flags and options

Document Information

Last Updated: 2025-11-15
Applies to Version: KafkaGuard 1.0.0+
Feedback: Open an issue for corrections or improvements

Security & Authentication Guide

Table of Contents

Security Overview

Security Protocol Selection Matrix

Security Protocols

PLAINTEXT (Port 9092)

Use Cases

Configuration

Example

SSL (Port 9093)

Use Cases

Configuration

Example

SASL_PLAINTEXT (Port 9094)

Use Cases

Configuration

SASL_SSL (Port 9095) - RECOMMENDED

Use Cases

Configuration

Example

SASL Authentication

SASL Mechanisms Comparison

SASL/PLAIN

Configuration

SASL/SCRAM-SHA-256

How SCRAM Works

Configuration

Example

SASL/SCRAM-SHA-512 (RECOMMENDED)

Why SCRAM-SHA-512?

Configuration

Example with Secret Manager

SSL/TLS Encryption

TLS Configuration Options

Basic TLS with CA Certificate

TLS with SASL Authentication

Certificate Requirements

CA Certificate

Certificate Validation

Self-Signed Certificates

Certificate Troubleshooting

Test Certificate Manually

Debug TLS Issues

Mutual TLS (mTLS)

When to Use mTLS

mTLS Configuration

Required Files

mTLS with SASL (Defense in Depth)

Certificate Format Requirements

CA Certificate (ca-cert.pem)

Client Certificate (client-cert.pem)

Client Private Key (client-key.pem)

Certificate Generation Example

mTLS Troubleshooting

Certificate Mismatch Error

Missing Certificates Error

Certificate Validation Failed

File Permissions for Private Keys

Security Best Practices

1. Always Use SASL_SSL for Production

2. Prefer SCRAM-SHA-512 Over SCRAM-SHA-256

3. Use Environment Variables for Passwords

4. Use Secret Management Systems

5. Rotate Credentials Regularly

6. Validate TLS Certificates

7. Use mTLS for Highly Regulated Environments

8. Monitor Authentication Failures

9. Implement Least Privilege Access

10. Audit and Log All Scans

Auto-Detection

How Auto-Detection Works

Auto-Detection Behavior

When Auto-Detection Works Well

Manual Override

Authentication Troubleshooting

Common SASL Errors

Error: SASL authentication failed

Error: Mechanism mismatch

Common TLS Errors

Error: Certificate verification failed

CA Certificate (`ca-cert.pem`)

Client Certificate (`client-cert.pem`)

Client Private Key (`client-key.pem`)