Building a Real-Time Kafka Video Streaming App (Like Netflix) with Python

Mohammed Juyel Haque
Apr 14
2 min read

Ever wondered how video streaming giants like Netflix manage to deliver smooth, real-time content? In this blog, I’ll build a mini Netflix-style video streaming service using Apache Kafka, Python, and OpenCV. You’ll see how video frames can be serialized, transmitted via Kafka, and rendered live on another machine.

What You’ll Learn

Setting up Kafka with Docker Compose
Creating Kafka topics
Streaming video frames as messages
Consuming and displaying video frames in real-time

Project Structure

├── docker-compose.yml
├── producer.py
├── consumer.py
└── song.mp4   # your sample video file

Step 1: Kafka Setup Using Docker Compose

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 2181:2181

  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

Start the Kafka cluster:

docker-compose up -d

Step 2: Create a Kafka Topic

Exec into Kafka Container

docker exec -it <container_id_or_name> bash

Replace <container_id_or_name> with your Kafka container ID or name.

Create a Topic

Inside the container, run:

kafka-topics --create --topic juyel-vd-frames-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

List Topics

kafka-topics --list --bootstrap-server localhost:9092

You should see test-topic listed.

Step 3: Python libraries setup

I will use the kafka-python library to send and receive messages from Kafka.

Install libraries:

pip install -r requirements.txt

requirements.txt

kafka-python
opencv-python
numpy

Step 4: Video Frame Producer (Simulating Netflix's Streaming)

video_producer.py

from kafka import KafkaProducer
import cv2
import pickle
import time

# Initialize Kafka Producer
producer = KafkaProducer(bootstrap_servers='localhost:9092')
topic = 'vd-frames-topic'

# Load video file
cap = cv2.VideoCapture('song.mp4')  # Replace with your video file

frame_count = 0
print("Starting video stream to Kafka...")

while cap.isOpened():
    ret, frame = cap.read()
    if not ret:
        break

    # Serialize the frame using pickle
    data = pickle.dumps(frame)

    # Send frame to Kafka
    producer.send(topic, value=data)
    print(f"Sent frame {frame_count}")
    frame_count += 1

    # Simulate streaming (approx. 30 fps)
    time.sleep(0.03)

cap.release()
producer.flush()
print("Finished sending video frames.")

Step 5: Video Frame Consumer (Your Video Player)

video_consumer.py

from kafka import KafkaConsumer
import cv2
import pickle

# Initialize Kafka Consumer
consumer = KafkaConsumer(
    'vd-frames-topic',
    bootstrap_servers='localhost:9092',
    auto_offset_reset='earliest',
    enable_auto_commit=True,
    group_id='video-stream-player'
)

print("Receiving and playing video stream...")

for message in consumer:
    # Deserialize frame
    frame = pickle.loads(message.value)

    # Display the frame
    cv2.imshow('Kafka Video Stream', frame)

    # Press 'q' to quit
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cv2.destroyAllWindows()
print("Stream closed.")

Real-Time Flow

🎥 Video File (song.mp4)
    ↓ (frame-by-frame)
📤 Python Producer → Kafka Topic → Python Consumer → 🖥️ Live Video Display

Conclusion

I've just built a real-time video streaming system using Kafka that mimics how Netflix might stream content — chunking video into frames and transmitting it as a message stream.

This concept can be extended with: