Policy Creation and Customization Guide

This comprehensive guide teaches you how to understand KafkaGuard's policy structure, create custom policies, and manage the policy lifecycle.

Policy Schema Reference
Control Structure Deep Dive
Custom Policy Creation Workflow
CEL Expression Guide
Validation and Testing
Policy Lifecycle Management
Troubleshooting Policy Issues
Advanced Policy Patterns

Policy Schema Reference

KafkaGuard policies are YAML files that define security, reliability, and operational controls for Kafka clusters. This section documents the complete policy schema.

Policy File Structure

# Complete KafkaGuard Policy Schema
version: "1.0"                    # Policy version (required, semantic versioning)
name: "Enterprise Default Policy" # Human-readable name (required)
description: "Production-ready policy with 40 security, reliability, and operational controls"  # Description (required)

metadata:                         # Optional metadata block
  created: "2025-11-15"          # Creation date
  author: "KafkaGuard Team"      # Author information
  tier: "enterprise"             # Policy tier (dev, enterprise, finance)
  changelog:                     # Version history
    - version: "1.0"
      date: "2025-11-15"
      changes: "Initial enterprise policy"

controls:                         # Array of controls (required)
  - id: "KG-001"                 # Control identifier
    title: "Control title"       # Human-readable title
    description: "What this control validates"  # Detailed description
    severity: "HIGH"             # Severity level
    category: "security"         # Control category
    expr: |                      # CEL expression (Common Expression Language)
      broker.config["sasl.enabled"] == "true"
    remediation: |               # Remediation guidance
      Step-by-step fix instructions
    compliance:                  # Compliance framework mappings
      pci_dss: ["4.1", "8.2"]    # PCI-DSS requirements
      soc2: ["CC6.1"]            # SOC2 controls
      iso27001: ["A.9.2.1"]      # ISO 27001 controls

Required Fields

Field	Type	Description	Validation
`version`	string	Semantic version (e.g., "1.0", "2.1.3")	Required, non-empty
`name`	string	Human-readable policy name	Required, non-empty
`description`	string	Policy purpose and scope	Required, non-empty
`controls`	array	Array of control objects	Required, minimum 1 control

Optional Fields

Field	Type	Description	Default
`metadata`	object	Additional policy information	None
`metadata.created`	string	Creation date (ISO 8601)	None
`metadata.author`	string	Policy author	None
`metadata.tier`	string	Policy tier (dev, enterprise, finance)	None
`metadata.changelog`	array	Version history	None

Control Structure Deep Dive

Each control in the controls array defines a specific validation rule. This section explains every control field.

Control Required Fields

Field	Type	Description	Example
`id`	string	Unique identifier (KG-XXX format)	`"KG-001"`
`title`	string	Human-readable title	`"SASL authentication enabled"`
`description`	string	Detailed explanation	`"Validates that SASL authentication is enabled for client connections"`
`severity`	string	Impact level	`"HIGH"`, `"MEDIUM"`, `"LOW"`
`category`	string	Control type	`"security"`, `"reliability"`, `"operational"`, `"compliance"`
`expr`	string	CEL expression	`broker.config["sasl.enabled"] == "true"`
`remediation`	string	Fix instructions	`"Enable SASL authentication in server.properties"`

Control ID Convention

Control IDs follow the format KG-XXX where:

KG = KafkaGuard prefix
XXX = 3-digit number (001-999)

Examples:

KG-001 to KG-040 = Built-in controls
KG-051 to KG-099 = Organization-specific controls
KG-101 to KG-999 = Custom controls

ID Ranges:

KG-001 to KG-050: Reserved for KafkaGuard built-in controls
KG-051 to KG-999: Available for custom controls

Severity Levels

Severity	Description	Use Case	Exit Code Impact
HIGH	Critical security/compliance issue	SASL disabled, expired certificates	Causes `--fail-on high` to fail
MEDIUM	Important reliability/operational issue	Under-replication, configuration drift	Causes `--fail-on medium` to fail
LOW	Minor optimization opportunity	Non-critical settings	Only fails with `--fail-on low`

Control Categories

Category	Description	Examples	Typical Severity
security	Security and authentication controls	SASL, SSL/TLS, ACLs, certificates	HIGH/MEDIUM
reliability	High availability and fault tolerance	Replication, ISR, ZooKeeper	MEDIUM
operational	Configuration and performance	Auto-create, retention, threading	MEDIUM/LOW
compliance	Regulatory and framework requirements	Audit logging, data protection	HIGH

CEL Expression Syntax

Controls use CEL (Common Expression Language) for flexible validation logic.

Available Variables:

Variable	Description	Properties
`broker`	Broker configuration and metadata	`config`, `version`, `listeners`
`topic`	Topic configuration and metadata	`config`, `partitions`, `replicas`
`cluster`	Cluster-wide information	`brokers`, `topics`, `controller`

Common CEL Patterns:

# Broker configuration checks
broker.config["auto.create.topics.enable"] == "false"
broker.config["sasl.enabled"] == "true"
broker.version >= "2.8.0"

# Topic configuration checks
topic.config["retention.ms"] < 604800000  # 7 days in milliseconds
topic.partitions >= 3
topic.replicas >= 3

# Cluster-wide checks
cluster.brokers.size() >= 3
cluster.controller.exists()

CEL Operators:

Comparison: ==, !=, <, <=, >, >=
Logical: && (AND), || (OR), ! (NOT)
String: startsWith(), endsWith(), contains()
Math: +, -, *, /, %

Custom Policy Creation Workflow

Follow this step-by-step workflow to create custom policies for your organization.

Step 1: Plan Your Policy

Define Requirements:

# What are your objectives?
# - Security standards
# - Compliance requirements
# - Operational standards
# - Custom business rules

# Which base tier to start from?
# - baseline-dev: 20 controls (dev environments)
# - enterprise-default: 40 controls (production)
# - Custom: Start from scratch

Step 2: Create Policy File

Clone an existing policy:

# Start with enterprise-default for production
cp policies/enterprise-default.yaml policies/custom-org.yaml

# Or start with baseline-dev for development
cp policies/baseline-dev.yaml policies/custom-dev.yaml

# Or create from scratch
cat > policies/custom-new.yaml << 'EOF'
version: "1.0"
name: "Custom Organization Policy"
description: "Custom policy for our Kafka environments"

controls: []
EOF

Step 3: Add Custom Controls

Add your first custom control:

# Add to policies/custom-org.yaml
controls:
  # Existing controls from base policy...
  # ... (keep all existing controls)

  # Add custom controls starting from KG-051
  - id: "KG-051"
    title: "Custom retention policy validation"
    description: "Validates that topics have retention policies set according to org standards"
    severity: "MEDIUM"
    category: "operational"
    expr: |
      topic.config["retention.ms"].exists() &&
      topic.config["retention.ms"] >= 604800000  # 7 days minimum
    remediation: |
      Set retention.ms to at least 7 days (604800000 ms) for topic:
      kafka-configs.sh --bootstrap-server $KAFKA_BROKERS \
        --alter --entity-type topics --entity-name <topic-name> \
        --add-config retention.ms=604800000
    compliance:
      pci_dss: ["3.1"]  # Data retention requirements
      iso27001: ["A.12.3.1"]  # Information security procedures

Add more custom controls:

  - id: "KG-052"
    title: "Topic naming convention enforcement"
    description: "Validates that topic names follow organizational naming conventions"
    severity: "LOW"
    category: "operational"
    expr: |
      topic.name.startsWith("org.") ||
      topic.name.startsWith("internal.")
    remediation: |
      Rename topic to follow naming convention (org.* or internal.*):
      kafka-topics.sh --bootstrap-server $KAFKA_BROKERS \
        --alter --topic <current-name> \
        --topic <new-name> \
        --replicas <replication-factor> \
        --partitions <partition-count>

      Note: This requires recreating the topic and migrating data.
    compliance: {}

  - id: "KG-053"
    title: "Minimum partition count validation"
    description: "Ensures topics have sufficient partitions for throughput requirements"
    severity: "MEDIUM"
    category: "operational"
    expr: |
      topic.partitions >= 6  # Minimum 6 partitions for production
    remediation: |
      Increase partition count to at least 6:
      kafka-topics.sh --bootstrap-server $KAFKA_BROKERS \
        --alter --topic <topic-name> \
        --partitions 6

      Note: Increasing partitions is irreversible.
    compliance:
      pci_dss: ["2.2"]  # Capacity planning

Step 4: Validate Policy Syntax

Run validation:

# Validate YAML syntax and control structure
kafkaguard validate-policy --policy policies/custom-org.yaml

# Expected output:
# ✅ Policy validation successful
#
# Policy Details:
#   Name:        Custom Organization Policy
#   Version:     1.0
#   Description: Custom policy for our Kafka environments
#   Controls:    43  # 40 built-in + 3 custom

Step 5: Test Policy Against Cluster

Test scan:

# Test against development cluster first
kafkaguard scan \
  --bootstrap kafka-dev:9092 \
  --policy policies/custom-org.yaml \
  --format json \
  --out ./test-reports \
  --log-level info

# Check results
cat test-reports/scan-*.json | jq '.summary'

Step 6: Iterate and Refine

Review custom controls:

# Check if custom controls are working as expected
cat test-reports/scan-*.json | jq '.findings[] | select(.control_id | startswith("KG-05"))'

# Expected: Your custom controls (KG-051, KG-052, KG-053) should appear

Adjust based on results:

# If control KG-051 is too strict, adjust expression
expr: |
  topic.config["retention.ms"].exists() &&
  topic.config["retention.ms"] >= 86400000  # Reduce to 1 day minimum

Step 7: Deploy to Production

Final validation:

# Validate again before production
kafkaguard validate-policy --policy policies/custom-org.yaml

# Test production scan (without failing)
kafkaguard scan \
  --bootstrap kafka-prod:9095 \
  --security-protocol SASL_SSL \
  --sasl-mechanism SCRAM-SHA-512 \
  --tls-ca-cert /etc/kafkaguard/certs/ca.pem \
  --policy policies/custom-org.yaml \
  --fail-on none \
  --format json,html \
  --out ./prod-test-reports

Deploy:

# Copy to production location
sudo cp policies/custom-org.yaml /etc/kafkaguard/policies/

# Update production scans
kafkaguard scan \
  --bootstrap kafka-prod:9095 \
  --policy /etc/kafkaguard/policies/custom-org.yaml \
  --format json,html,pdf \
  --out /var/reports/kafkaguard \
  --fail-on high

CEL Expression Guide

CEL (Common Expression Language) provides powerful validation logic. This section covers advanced CEL usage.

Available Data Types

Broker Data:

// Configuration values (strings, numbers, booleans)
broker.config["auto.create.topics.enable"] == "false"
broker.config["num.network.threads"] >= 8

// Version information
broker.version == "3.0.0"
broker.version >= "2.8.0"

// Listener information
broker.listeners.size() >= 2
broker.listeners.exists(l, l.contains("SASL_SSL"))

Topic Data:

// Configuration
topic.config["cleanup.policy"] == "delete"
topic.config["retention.ms"] <= 604800000  // 7 days

// Partition information
topic.partitions >= 6
topic.replicas >= 3

// Topic name validation
topic.name.startsWith("prod.")
topic.name.contains("sensitive") ? topic.partitions >= 12 : topic.partitions >= 6

Cluster Data:

// Broker count
cluster.brokers.size() >= 3

// Controller existence
cluster.controller.exists()

// Topic count validation
cluster.topics.size() <= 1000  // Maximum topics limit

Advanced CEL Patterns

Conditional Logic:

// If topic contains sensitive data, require more partitions
topic.name.contains("sensitive") ?
  topic.partitions >= 12 :
  topic.partitions >= 6

// Environment-specific validation
broker.config["environment"] == "production" ?
  broker.config["sasl.enabled"] == "true" :
  true  // Allow in non-prod

String Operations:

// Topic naming conventions
topic.name.startsWith("org.")
topic.name.endsWith("-v2")
topic.name.matches("^[a-z][a-z0-9.-]*$")  // Regex validation

// Configuration validation
broker.config["log.dirs"].contains("/var/kafka")

Numeric Operations:

// Threading calculations
broker.config["num.io.threads"] >= broker.config["num.network.threads"] * 2

// Retention validation (convert to days)
topic.config["retention.ms"] / 86400000 <= 30  // Max 30 days

// Size validations
topic.partitions * topic.replicas <= 10000  // Max total replicas

Array Operations:

// Listener validation
broker.listeners.exists(l, l.contains("SASL_SSL"))
broker.listeners.size() >= 2

// ACL validation (future enhancement)
cluster.acls.size() > 0

Common CEL Errors and Fixes

Error	Cause	Fix
`undeclared reference`	Variable doesn't exist	Use `broker`, `topic`, or `cluster`
`no matching overload`	Wrong data type	Check field types in schema
`invalid argument`	Function call error	Check function signatures
`evaluation error`	Null/undefined access	Use `.exists()` or safe navigation

Testing CEL Expressions

Debug CEL expressions:

# Use validate-policy to test syntax
kafkaguard validate-policy --policy policies/custom-policy.yaml

# Test expressions against cluster
kafkaguard scan \
  --bootstrap kafka:9092 \
  --policy policies/custom-policy.yaml \
  --log-level debug \
  2>&1 | grep "CEL"

Validation and Testing

Policy Validation Steps

1. Syntax Validation:

kafkaguard validate-policy --policy policies/custom-policy.yaml

2. YAML Structure Check:

# Check YAML syntax
python3 -c "import yaml; yaml.safe_load(open('policies/custom-policy.yaml'))"

# Or use yamllint
yamllint policies/custom-policy.yaml

3. Control Structure Validation:

# Verify all required fields present
kafkaguard validate-policy --policy policies/custom-policy.yaml --log-level debug

Testing Strategies

Unit Testing (Individual Controls):

# Create minimal policy with one control
cat > test-control.yaml << 'EOF'
version: "1.0"
name: "Test Control"
description: "Testing individual control"

controls:
  - id: "KG-999"
    title: "Test control"
    description: "Test control description"
    severity: "MEDIUM"
    category: "operational"
    expr: |
      broker.config["auto.create.topics.enable"] == "false"
    remediation: "Disable auto.create.topics.enable"
EOF

# Test against cluster
kafkaguard scan --bootstrap kafka:9092 --policy test-control.yaml --format json

Integration Testing:

# Test full policy against development cluster
kafkaguard scan \
  --bootstrap kafka-dev:9092 \
  --policy policies/custom-policy.yaml \
  --format json,html \
  --out ./integration-test \
  --fail-on none  # Don't fail, just report

CI/CD Validation:

# GitHub Actions workflow for policy validation
name: Validate KafkaGuard Policies

on:
  push:
    paths:
      - 'policies/*.yaml'

jobs:
  validate:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - name: Download KafkaGuard
        run: |
          curl -LO https://github.com/aiopsone/kafkaguard-releases/releases/latest/download/kafkaguard_Linux_x86_64.tar.gz && tar -xzf kafkaguard_Linux_x86_64.tar.gz
          chmod +x kafkaguard-linux-amd64
          sudo mv kafkaguard-linux-amd64 /usr/local/bin/kafkaguard

      - name: Validate all policies
        run: |
          for policy in policies/*.yaml; do
            echo "Validating $policy..."
            kafkaguard validate-policy --policy "$policy"
          done

Performance Testing

Benchmark policy evaluation:

# Time policy validation
time kafkaguard validate-policy --policy policies/large-policy.yaml

# Time scan execution
time kafkaguard scan \
  --bootstrap kafka:9092 \
  --policy policies/custom-policy.yaml \
  --format json \
  --timeout 300

Policy Lifecycle Management

Versioning Strategy

Semantic Versioning:

MAJOR.MINOR.PATCH (e.g., 1.2.3)
MAJOR: Breaking changes (control removal, behavior changes)
MINOR: New controls added, severity changes
PATCH: Bug fixes, remediation improvements

Version Examples:

version: "2.1.3"  # Major 2, Minor 1, Patch 3

metadata:
  changelog:
    - version: "2.1.3"
      date: "2025-11-15"
      changes: "Fixed CEL expression in KG-051"
    - version: "2.1.2"
      date: "2025-11-10"
      changes: "Updated remediation guidance for KG-028"
    - version: "2.1.1"
      date: "2025-11-05"
      changes: "Added custom control KG-101"
    - version: "2.1.0"
      date: "2025-11-01"
      changes: "Added 3 new compliance controls"
    - version: "2.0.0"
      date: "2025-10-15"
      changes: "Breaking change: Changed severity of KG-005 from MEDIUM to HIGH"

Environment Promotion

Development → Staging → Production:

# 1. Develop and test in development
kafkaguard scan \
  --bootstrap kafka-dev:9092 \
  --policy policies/v2.1.0-custom-policy.yaml \
  --format json,html

# 2. Validate in staging
kafkaguard scan \
  --bootstrap kafka-staging:9095 \
  --security-protocol SASL_SSL \
  --policy policies/v2.1.0-custom-policy.yaml \
  --format json,html

# 3. Deploy to production with rollback plan
cp policies/v2.1.0-custom-policy.yaml policies/v2.0.0-backup.yaml

# Deploy new version
cp policies/v2.1.0-custom-policy.yaml /etc/kafkaguard/policies/production.yaml

# Test production scan
kafkaguard scan \
  --bootstrap kafka-prod:9095 \
  --policy /etc/kafkaguard/policies/production.yaml \
  --fail-on none \
  --format json \
  --out /tmp/policy-test

Backup and Rollback

Automated Backup Strategy:

#!/bin/bash
# backup-policies.sh - Backup policy files

BACKUP_DIR="/var/backups/kafkaguard/policies"
TIMESTAMP=$(date +%Y%m%d-%H%M%S)

# Create backup directory
mkdir -p "$BACKUP_DIR/$TIMESTAMP"

# Backup all policies
cp policies/*.yaml "$BACKUP_DIR/$TIMESTAMP/"

# Create backup manifest
cat > "$BACKUP_DIR/$TIMESTAMP/manifest.txt" <<EOF
Policy Backup - $TIMESTAMP
===========================

Files backed up:
$(ls -la "$BACKUP_DIR/$TIMESTAMP/")

Created by: $(whoami)
Reason: $(echo "$1" | tr -d '\n' || echo "Scheduled backup")
EOF

# Keep only last 10 backups
cd "$BACKUP_DIR"
ls -t | tail -n +11 | xargs -r rm -rf

echo "Policies backed up to: $BACKUP_DIR/$TIMESTAMP"

Rollback Procedure:

# Identify backup to restore
ls -la /var/backups/kafkaguard/policies/

# Restore from backup
cp /var/backups/kafkaguard/policies/20251115-143000/* policies/

# Validate restored policies
for policy in policies/*.yaml; do
  kafkaguard validate-policy --policy "$policy"
done

# Test scan with restored policies
kafkaguard scan \
  --bootstrap kafka-prod:9095 \
  --policy policies/production.yaml \
  --format json \
  --out /tmp/rollback-test

Change Management

Policy Change Approval Workflow:

Development: Create/modify policy in feature branch
Validation: Run automated validation tests
Peer Review: Code review of policy changes
Integration Testing: Test against staging environment
Security Review: Security team approval for security controls
Compliance Review: Compliance team approval for compliance controls
Production Deployment: Deploy with rollback plan
Monitoring: Monitor scan results for unexpected changes

Change Documentation:

metadata:
  changelog:
    - version: "2.2.0"
      date: "2025-11-20"
      author: "security-team"
      approved_by: "compliance-officer"
      ticket: "SEC-1234"
      changes: |
        Added KG-052: Enhanced topic naming validation
        Changed KG-028 severity from MEDIUM to HIGH
        Updated remediation guidance for KG-005
      testing: |
        - Validated against dev cluster (85% pass rate)
        - Integration test in staging (92% pass rate)
        - No production impact expected
      rollback: "Revert to version 2.1.3"

Troubleshooting Policy Issues

Common Policy Creation Errors

Invalid Control ID Format

Error: invalid control ID format 'KG-1'

Cause: Control ID doesn't use 3-digit format

Fix:

# Incorrect
- id: "KG-1"

# Correct
- id: "KG-051"

Missing Required Fields

Error: missing required field 'expr' in control KG-051

Cause: Required field missing from control

Fix:

# Add missing field
- id: "KG-051"
  title: "Control title"
  description: "Control description"
  severity: "MEDIUM"
  category: "operational"
  expr: "true"  # Required CEL expression
  remediation: "Fix instructions"

CEL Syntax Error

Error: CEL syntax error in control KG-051: undeclared reference to 'borker'

Cause: Typo in CEL expression

Fix:

# Incorrect
expr: borker.config["sasl.enabled"] == "true"

# Correct
expr: broker.config["sasl.enabled"] == "true"

YAML Syntax Error

Error: yaml: line 25: did not find expected key

Cause: Invalid YAML indentation or syntax

Fix:

# Check YAML syntax
yamllint policies/custom-policy.yaml

# Or use online YAML validator

Validation Error Lookup Table

Error Message	Likely Cause	File Section	Fix
`invalid control ID format`	Wrong ID format	`controls[].id`	Use KG-XXX format
`missing required field`	Field missing	`controls[]`	Add required field
`CEL syntax error`	Expression error	`controls[].expr`	Fix CEL syntax
`invalid severity`	Wrong severity	`controls[].severity`	Use HIGH/MEDIUM/LOW
`invalid category`	Wrong category	`controls[].category`	Use security/reliability/operational/compliance
`duplicate control ID`	ID reused	`controls[].id`	Use unique IDs

Debugging Techniques

Isolate Problematic Controls

# Create minimal policy with one control
cat > debug-control.yaml << 'EOF'
version: "1.0"
name: "Debug Control"
description: "Testing single control"

controls:
  - id: "KG-999"
    title: "Debug control"
    description: "Testing control"
    severity: "MEDIUM"
    category: "operational"
    expr: |
      broker.config["auto.create.topics.enable"] == "false"
    remediation: "Set auto.create.topics.enable=false"
EOF

# Test isolated control
kafkaguard scan --bootstrap kafka:9092 --policy debug-control.yaml --format json

Enable Debug Logging

# Debug policy validation
kafkaguard validate-policy \
  --policy policies/custom-policy.yaml \
  --log-level debug

# Debug scan execution
kafkaguard scan \
  --bootstrap kafka:9092 \
  --policy policies/custom-policy.yaml \
  --log-level debug \
  2>&1 | tee scan-debug.log

Test CEL Expressions Individually

# Test CEL syntax (limited validation available)
# Use online CEL evaluator or create test controls

Performance Troubleshooting

Slow Policy Validation

Symptoms: validate-policy takes >30 seconds

Cause: Large policy file, complex CEL expressions

Fix:

Split large policies into smaller files
Simplify CEL expressions
Remove unused controls

Slow Scan Execution

Symptoms: Scan takes >10 minutes on normal cluster

Cause: Complex CEL expressions, large cluster

Fix:

Optimize CEL expressions
Use parallel scanning (--parallel true)
Increase timeout (--timeout 900)

Advanced Policy Patterns

Environment-Specific Policies

# Environment-aware policy
version: "1.0"
name: "Environment-Aware Policy"
description: "Policy that adapts based on environment"

controls:
  - id: "KG-051"
    title: "Environment-specific SASL requirement"
    description: "SASL required in production, optional in dev"
    severity: "HIGH"
    category: "security"
    expr: |
      broker.config["environment"] == "production" ?
        broker.config["sasl.enabled"] == "true" :
        true  # Allow in non-prod
    remediation: |
      For production: Enable SASL authentication
      For development: Set environment=production when ready

Conditional Controls Based on Topic Name

  - id: "KG-052"
    title: "Sensitive data topic validation"
    description: "Topics with sensitive data require enhanced security"
    severity: "HIGH"
    category: "security"
    expr: |
      topic.name.contains("sensitive") || topic.name.contains("pii") ?
        topic.config["cleanup.policy"] == "delete" &&
        topic.partitions >= 12 &&
        topic.replicas >= 3 :
        true  # Normal validation for other topics
    remediation: |
      For sensitive data topics:
      1. Set cleanup.policy=delete
      2. Increase partitions to 12+
      3. Ensure replication factor >= 3

Compliance Framework Integration

  - id: "KG-053"
    title: "PCI-DSS data retention compliance"
    description: "PCI-DSS requirement 3.1: Data retention policies"
    severity: "HIGH"
    category: "compliance"
    expr: |
      topic.config["retention.ms"].exists() &&
      topic.config["retention.ms"] <= 31536000000  # 1 year max for PCI
    remediation: |
      PCI-DSS 3.1 requires limiting data retention to business needs.
      Set retention.ms to <= 1 year (31536000000 ms):

      kafka-configs.sh --bootstrap-server $KAFKA_BROKERS \\
        --alter --entity-type topics --entity-name <topic-name> \\
        --add-config retention.ms=31536000000
    compliance:
      pci_dss: ["3.1"]
      iso27001: ["A.12.3.1"]

Custom Control Libraries

# Company-specific control library
  - id: "KG-101"
    title: "Company naming standard compliance"
    description: "Topics must follow company naming conventions"
    severity: "MEDIUM"
    category: "operational"
    expr: |
      topic.name.matches("^company\\.[a-z][a-z0-9.-]*$")
    remediation: |
      Topic names must follow the pattern: company.<topic-name>
      Examples:
      - company.user-events
      - company.order-processing
      - company.inventory.v2

      Rename topic using:
      kafka-topics.sh --bootstrap-server $KAFKA_BROKERS \\
        --alter --topic <current-name> --topic company.<current-name>

Next Steps

Policy Tiers Guide - Choose the right tier for your environment
CLI Reference - Command-line options for policy management
Validation and Testing - Test your custom policies
Troubleshooting Policy Issues - Debug policy problems

Document Information

Last Updated: 2025-11-15
Applies to Version: KafkaGuard 1.0.0+
Feedback: Open an issue for corrections or improvements

Policy Creation and Customization Guide

Table of Contents

Policy Schema Reference

Policy File Structure

Required Fields

Optional Fields

Control Structure Deep Dive

Control Required Fields

Control ID Convention

Severity Levels

Control Categories

CEL Expression Syntax

Custom Policy Creation Workflow

Step 1: Plan Your Policy

Step 2: Create Policy File

Step 3: Add Custom Controls

Step 4: Validate Policy Syntax

Step 5: Test Policy Against Cluster

Step 6: Iterate and Refine

Step 7: Deploy to Production

CEL Expression Guide

Available Data Types

Advanced CEL Patterns

Common CEL Errors and Fixes

Testing CEL Expressions

Validation and Testing

Policy Validation Steps

Testing Strategies

Performance Testing

Policy Lifecycle Management

Versioning Strategy

Environment Promotion

Backup and Rollback

Change Management

Troubleshooting Policy Issues

Common Policy Creation Errors

Invalid Control ID Format

Missing Required Fields

CEL Syntax Error

YAML Syntax Error

Validation Error Lookup Table

Debugging Techniques

Isolate Problematic Controls

Enable Debug Logging

Test CEL Expressions Individually

Performance Troubleshooting

Slow Policy Validation

Slow Scan Execution

Advanced Policy Patterns

Environment-Specific Policies

Conditional Controls Based on Topic Name

Compliance Framework Integration

Custom Control Libraries

Next Steps