Appearance

Policy Tiers and Custom Policies ​

This guide helps you choose the right KafkaGuard policy tier for your environment and understand how to create custom policies.

Table of Contents ​

Policy Tier Overview ​

KafkaGuard provides three policy tiers designed for different environments and compliance requirements:

TierControlsFocusBest ForFile Location
baseline-dev20Reliability + OperationsDevelopment, testingpolicies/baseline-dev.yaml
enterprise-default40Security + Reliability + OperationsProduction, enterprisepolicies/enterprise-default.yaml
finance-iso50Advanced ComplianceRegulated industries (Phase 2)policies/finance-iso.yaml

Key Differentiators:

  • Control Count: More controls = stricter compliance
  • Security Focus: Baseline has no security controls, others add security requirements
  • Compliance Mapping: Enterprise and Finance tiers include PCI-DSS, SOC2, ISO 27001 mappings
  • Remediation Depth: Higher tiers provide more detailed remediation guidance

Tier Decision Framework ​

Choose your policy tier based on these key factors:

Primary Decision Factors ​

  1. Environment Type (Development vs. Production)
  2. Data Sensitivity (Internal vs. Customer data)
  3. Compliance Requirements (None vs. PCI-DSS/SOC2/ISO 27001)
  4. Security Posture (Basic vs. Enterprise-grade)

Decision Flow Chart ​

┌─────────────────────────┐
│   Is this production?   │
└─────────┬───────────────┘
          │
          ▼
┌─────────────────────────┐    NO     ┌─────────────────────┐
│ Handle customer data?   ├──────────►│   baseline-dev      │
│ or PII?                 │           │   (20 controls)     │
└─────────┬───────────────┘           └─────────────────────┘
          │
          ▼
┌─────────────────────────┐    NO     ┌─────────────────────┐
│ Regulated industry?     ├──────────►│ enterprise-default  │
│ (Finance, Healthcare,   │           │   (40 controls)     │
│ Government)             │           └─────────────────────┘
└─────────┬───────────────┘
          │
          ▼
┌─────────────────────────┐    YES    ┌─────────────────────┐
│ Phase 2 available?      ├──────────►│   finance-iso       │
│ (Current: Phase 1)      │           │   (50 controls)     │
└─────────┬───────────────┘           └─────────────────────┘
          │
          ▼
          NO
    ┌─────────────────────┐
    │ enterprise-default  │
    │   (40 controls)     │
    └─────────────────────┘

Policy Tier Details ​

baseline-dev (20 Controls) ​

Focus: Reliability and operational checks for development environments Best For: Development clusters, sandboxes, testing environments Security Level: None (no security controls)

Control Categories ​

  • Reliability Controls (8): Replication factors, ISR validation, ZooKeeper quorum, broker versions, partition balance, under-replicated partitions, offline partitions
  • Operational Controls (12): Auto-create topics, log directories, retention policies, thread configurations, compression settings, quotas

Sample Scan Commands ​

Development Cluster:

bash
# Basic scan with PLAINTEXT
kafkaguard scan \
  --bootstrap localhost:9092 \
  --policy policies/baseline-dev.yaml \
  --format html,json \
  --out ./reports

Expected Results:

  • ✅ 16-18 passed controls (typical for well-configured dev cluster)
  • ⚠️ 2-4 failed controls (configuration issues)
  • 🔵 0-1 N/A controls
  • 📊 Score: 80-90% (varies by cluster configuration)

enterprise-default (40 Controls) ​

Focus: Security, reliability, and operational validation for production environments Best For: Production clusters, enterprise environments, standard compliance Security Level: High (SASL_SSL required)

Control Categories ​

  • Security Controls (15): SASL authentication, SSL/TLS encryption, certificate validation, inter-broker encryption, ACLs
  • Reliability Controls (12): All baseline-dev reliability controls plus additional checks
  • Operational Controls (13): All baseline-dev operational controls plus additional validation

Sample Scan Commands ​

Production SASL_SSL Scan:

bash
# Production scan with SASL_SSL and SCRAM-SHA-512
export KAFKAGUARD_SASL_USERNAME="kafkaguard"
export KAFKAGUARD_SASL_PASSWORD="$(vault kv get -field=password secret/kafka/prod)"

kafkaguard scan \
  --bootstrap kafka1.prod:9095,kafka2.prod:9095,kafka3.prod:9095 \
  --security-protocol SASL_SSL \
  --sasl-mechanism SCRAM-SHA-512 \
  --tls-ca-cert /etc/kafkaguard/certs/ca.pem \
  --policy policies/enterprise-default.yaml \
  --format json,html,pdf \
  --out /var/reports/kafkaguard \
  --fail-on high \
  --timeout 600

Expected Results:

  • ✅ 32-38 passed controls (typical for compliant production cluster)
  • ❌ 2-8 failed controls (security/compliance gaps)
  • 🔵 0 N/A controls
  • 📊 Score: 80-95% (varies by security posture)

finance-iso (50 Controls - Phase 2) ​

Focus: Advanced compliance controls for regulated industries Best For: Financial services, healthcare, government, highly regulated environments Security Level: Highest (SASL_SSL + mTLS required)

Additional Controls (10 more beyond enterprise-default) ​

  • Enhanced Security (5 controls): Mutual TLS requirements, strong cipher suites, privileged user auditing, encryption at rest validation
  • Advanced Compliance (5 controls): 90+ day audit log retention, admin access restrictions, ISO 27001 specific mappings

Status: Planned for Phase 2 (post-MVP). Currently, use enterprise-default.yaml for regulated environments.

Custom Policy Creation ​

While KafkaGuard provides three built-in policy tiers, you can create custom policies tailored to your organization's specific requirements.

When to Create Custom Policies ​

Create custom policies when you need to:

  • ✅ Enforce organization-specific compliance requirements
  • ✅ Combine controls from multiple tiers
  • ✅ Add custom controls for internal standards
  • ✅ Adjust control severity levels for your environment
  • ✅ Create environment-specific policies (dev, staging, prod)

Quick Policy Structure Reference ​

yaml
version: "1.0"
name: "My Custom Policy"
description: "Custom policy for our environment"
tier: "custom"

controls:
  - id: KG-XXX
    title: "Control title"
    description: "What this control checks"
    severity: HIGH | MEDIUM | LOW
    category: security | reliability | operational | compliance
    expr: |
      # CEL expression that returns true (pass) or false (fail)
      broker.config["auto.create.topics.enable"] == "false"
    remediation: |
      Step-by-step fix instructions
    compliance:
      pci_dss: ["4.1", "8.2"]
      soc2: ["CC6.1"]
      iso27001: ["A.12.6.1"]

Policy Validation ​

Always validate custom policies before deployment:

bash
# Validate policy syntax
kafkaguard validate-policy --policy policies/custom-policy.yaml

# Test policy against dev cluster first
kafkaguard scan \
  --bootstrap kafka-dev:9092 \
  --policy policies/custom-policy.yaml \
  --format json \
  --out ./test-reports

# Review results and iterate
cat test-reports/scan-*.json | jq '.findings[] | select(.status == "FAILED")'

Control ID Convention ​

Control IDs follow the format KG-XXX where:

  • KG = KafkaGuard prefix
  • XXX = 3-digit number (001-999)

ID Ranges:

  • KG-001 to KG-050: Reserved for KafkaGuard built-in controls
  • KG-051 to KG-999: Available for custom controls

CEL Expression Syntax ​

Controls use CEL (Common Expression Language) for flexible validation logic.

Available Variables:

  • broker - Broker configuration and metadata
  • topic - Topic configuration and metadata
  • cluster - Cluster-wide information

Common CEL Patterns:

cel
# Broker configuration checks
broker.config["auto.create.topics.enable"] == "false"
broker.config["sasl.enabled"] == "true"
broker.version >= "2.8.0"

# Topic configuration checks
topic.config["retention.ms"] < 604800000  # 7 days in milliseconds
topic.partitions >= 3
topic.replicas >= 3

# Cluster-wide checks
cluster.brokers.size() >= 3
cluster.controller.exists()

For comprehensive documentation on creating custom policies, see the Policy Creation Guide in the repository.

Next Steps ​

Need Help?

Ready to Get Started?

Book a demo to see KafkaGuard in action and learn how it can help secure your Kafka clusters.

📅 Book a Demo