Kafka

Fully managed Apache Kafka clusters for event streaming and real-time data pipelines.

Overview

Sparbz Cloud Kafka provides production-ready Apache Kafka clusters powered by Strimzi, with automatic scaling, monitoring, and security built-in.

Features

• Managed Clusters: Fully managed Kafka clusters with automatic updates
• Multi-tenant Isolation: Dedicated clusters per organization with network isolation
• Schema Registry: Built-in schema registry for Avro, JSON Schema, and Protobuf
• Kafka Connect: Pre-configured connectors for common data sources and sinks
• Monitoring: Real-time metrics, lag monitoring, and alerting

Cluster Tiers

Tier	Brokers	Storage	Partitions	Retention
Starter	1	10 GB	50	7 days
Pro	3	100 GB	500	30 days
Enterprise	5+	1 TB+	Unlimited	Custom

Getting Started

Create a Cluster

# Create a starter cluster
szc kafka create my-cluster

# Create a pro cluster with 3 brokers
szc kafka create my-cluster --tier pro

# Wait for cluster to be ready
szc kafka create my-cluster --wait

Manage Topics

# List topics
szc kafka topic list <cluster-id>

# Create a topic
szc kafka topic create <cluster-id> orders --partitions 6 --replicas 3

# Create with retention settings
szc kafka topic create <cluster-id> events \
  --partitions 12 \
  --retention 7d \
  --cleanup-policy compact

# Describe a topic
szc kafka topic get <cluster-id> orders

# Delete a topic
szc kafka topic delete <cluster-id> orders

Manage Users and ACLs

# Create a user
szc kafka user create <cluster-id> my-app

# Create with consumer group ACL
szc kafka user create <cluster-id> my-consumer --consumer-group my-group

# List users
szc kafka user list <cluster-id>

# Get user credentials
szc kafka user get <cluster-id> my-app --show-password

# Delete a user
szc kafka user delete <cluster-id> my-app

Connection

Get Connection Details

# Get bootstrap servers
szc kafka get <cluster-id> --json --fields bootstrap_servers

# Get full connection info
szc kafka credentials <cluster-id>

Connect with kafkacat/kcat

# Produce messages
echo "test message" | kcat -b <bootstrap-servers> \
  -X security.protocol=SASL_SSL \
  -X sasl.mechanism=SCRAM-SHA-512 \
  -X sasl.username=<username> \
  -X sasl.password=<password> \
  -t my-topic -P

# Consume messages
kcat -b <bootstrap-servers> \
  -X security.protocol=SASL_SSL \
  -X sasl.mechanism=SCRAM-SHA-512 \
  -X sasl.username=<username> \
  -X sasl.password=<password> \
  -t my-topic -C

Connect with Java

Properties props = new Properties();
props.put("bootstrap.servers", "<bootstrap-servers>");
props.put("security.protocol", "SASL_SSL");
props.put("sasl.mechanism", "SCRAM-SHA-512");
props.put("sasl.jaas.config",
    "org.apache.kafka.common.security.scram.ScramLoginModule required " +
    "username=\"<username>\" password=\"<password>\";");

// Producer
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.send(new ProducerRecord<>("my-topic", "key", "value"));

// Consumer
props.put("group.id", "my-group");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("my-topic"));

Connect with Node.js

const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['<bootstrap-servers>'],
  ssl: true,
  sasl: {
    mechanism: 'scram-sha-512',
    username: '<username>',
    password: '<password>'
  }
});

// Producer
const producer = kafka.producer();
await producer.connect();
await producer.send({
  topic: 'my-topic',
  messages: [{ key: 'key', value: 'value' }]
});

// Consumer
const consumer = kafka.consumer({ groupId: 'my-group' });
await consumer.connect();
await consumer.subscribe({ topic: 'my-topic' });
await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    console.log(message.value.toString());
  }
});

Consumer Groups

List Consumer Groups

szc kafka consumer-group list <cluster-id>

View Consumer Lag

szc kafka consumer-group get <cluster-id> my-group

Monitoring

Cluster Metrics

Available metrics via the Console or API:

• Broker count and health
• Topic partition count
• Consumer lag by group
• Messages per second (in/out)
• Bytes per second (in/out)
• Request latency

Alerts

Configure alerts for:

• Consumer lag exceeding threshold
• Broker offline
• Under-replicated partitions
• Disk usage above threshold

Schema Registry

Schema Registry is automatically deployed with each cluster.

Register a Schema

# Register Avro schema
curl -X POST "https://<schema-registry>/subjects/my-topic-value/versions" \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"schema": "{\"type\":\"record\",\"name\":\"User\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"}]}"}'

Get Schema

curl "https://<schema-registry>/subjects/my-topic-value/versions/latest"

Kafka Connect

Available Connectors

Type	Connector	Description
Source	PostgreSQL CDC	Capture changes from PostgreSQL
Source	MySQL CDC	Capture changes from MySQL
Source	MongoDB CDC	Capture changes from MongoDB
Sink	S3	Write to S3-compatible storage
Sink	Elasticsearch	Index to Elasticsearch
Sink	PostgreSQL	Write to PostgreSQL

Create a Connector

# Example: S3 sink connector
curl -X POST "https://<connect>/connectors" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "s3-sink",
    "config": {
      "connector.class": "io.confluent.connect.s3.S3SinkConnector",
      "tasks.max": "1",
      "topics": "my-topic",
      "s3.bucket.name": "my-bucket",
      "s3.region": "us-east-1",
      "flush.size": "1000"
    }
  }'

Best Practices

Topic Design

• Use descriptive topic names with namespacing (e.g., orders.created, users.updated)
• Choose partition count based on expected throughput (1 partition = ~10 MB/s)
• Set replication factor to 3 for production workloads
• Use compacted topics for changelog/state data

Consumer Design

• Use consumer groups for horizontal scaling
• Implement idempotent consumers for at-least-once delivery
• Handle rebalancing gracefully
• Monitor consumer lag

Security

• Create dedicated users per application
• Use minimal ACL permissions
• Rotate credentials periodically
• Enable audit logging for compliance

Pricing

Tier	Monthly	Included Storage	Overage
Starter	$29	10 GB	$0.10/GB
Pro	$99	100 GB	$0.08/GB
Enterprise	Custom	Custom	Custom

Data transfer within the same region is free. Cross-region transfer is billed at $0.02/GB.