You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by xi...@apache.org on 2026/04/02 06:07:35 UTC
(pinot) branch master updated: Add developer setup docs and modernize Kafka to official image with KRaft (#18078)
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new ab1f2bc6e55 Add developer setup docs and modernize Kafka to official image with KRaft (#18078)
ab1f2bc6e55 is described below
commit ab1f2bc6e55bf57cb120ee6116cf4a5d447c4838
Author: Xiang Fu <xi...@apache.org>
AuthorDate: Wed Apr 1 23:07:28 2026 -0700
Add developer setup docs and modernize Kafka to official image with KRaft (#18078)
* Add developer setup docs and modernize Kafka to official image with KRaft
Add a "Developer Setup" section to the Docker README documenting how to
run Pinot in an all-in-one container using StartServiceManager instead of
QuickStart. This addresses the common developer pain point where QuickStart
populates 10+ demo tables on every start and deletes all data on restart.
Also migrate docker-compose.yml from confluentinc/cp-kafka to the official
apache/kafka image running in KRaft mode, removing the Kafka dependency on
ZooKeeper (Pinot still requires ZooKeeper for its own cluster coordination).
Co-Authored-By: Claude Opus 4.6 (1M context) <no...@anthropic.com>
* Add link to Pinot docs for running in Docker
Co-Authored-By: Claude Opus 4.6 (1M context) <no...@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <no...@anthropic.com>
---
docker/images/pinot/README.md | 160 ++++++++++++++++++++++++++++++++-
docker/images/pinot/docker-compose.yml | 22 +++--
2 files changed, 170 insertions(+), 12 deletions(-)
diff --git a/docker/images/pinot/README.md b/docker/images/pinot/README.md
index 15778f233fa..8f530c706af 100644
--- a/docker/images/pinot/README.md
+++ b/docker/images/pinot/README.md
@@ -147,7 +147,9 @@ docker run -p 8098:8098 pinot:release-0.1.0 StartServer -zkAddress 172.17.0.2:21
### Use docker compose to bring up Pinot stack
-Please note that this quickstart is based on Kafka 3.0.
+The included `docker-compose.yml` uses the official [apache/kafka](https://hub.docker.com/r/apache/kafka)
+image running in KRaft mode (no ZooKeeper required for Kafka). ZooKeeper is still needed
+for Pinot's own cluster coordination.
Below is a script to use docker compose to bring up zookeeper/kafka/pinot-controller/pinot-broker/pinot-server
```SHELL
@@ -158,12 +160,164 @@ docker-compose -f docker-compose.yml up
Below is the script to create airlineStats table
```SHELL
-docker run --network=pinot_default apachepinot/pinot:release-0.11.0 AddTable -schemaFile examples/stream/airlineStats/airlineStats_schema.json -tableConfigFile examples/stream/airlineStats/docker/airlineStats_realtime_table_config.json -controllerHost pinot-controller -controllerPort 9000 -exec
+docker run --network=pinot_default apachepinot/pinot:latest AddTable -schemaFile examples/stream/airlineStats/airlineStats_schema.json -tableConfigFile examples/stream/airlineStats/docker/airlineStats_realtime_table_config.json -controllerHost pinot-controller -controllerPort 9000 -exec
```
Below is the script to ingest airplane stats data to Kafka
```SHELL
-docker run --network=pinot_default apachepinot/pinot:release-0.11.0 StreamAvroIntoKafka -avroFile examples/stream/airlineStats/rawdata/airlineStats_data.avro -kafkaTopic flights-realtime -kafkaBrokerList kafka:9092 -zkAddress zookeeper:2181
+docker run --network=pinot_default apachepinot/pinot:latest StreamAvroIntoKafka -avroFile examples/stream/airlineStats/rawdata/airlineStats_data.avro -kafkaTopic flights-realtime -kafkaBrokerList kafka:9092
```
In order to query pinot, try to open `localhost:9000` from your browser.
+
+## Developer Setup: All-in-One Without Demo Data
+
+For more deployment options, see the
+[Running Pinot in Docker](https://docs.pinot.apache.org/basics/getting-started/running-pinot-in-docker)
+guide in the Pinot documentation.
+
+The `QuickStart` command is convenient for trying Pinot, but it has two limitations for
+ongoing development work:
+
+1. It populates demo tables (e.g., `baseballStats`, `airlineStats`) on every start.
+2. It deletes all data on container restart.
+
+If you want a lightweight, all-in-one Pinot instance that starts empty and preserves your
+data across restarts, use the `StartServiceManager` command instead. Unlike `QuickStart`,
+`StartServiceManager` requires a separate ZooKeeper instance (it does not embed one).
+
+### All-in-one with docker compose
+
+Below is a minimal `docker-compose.yml` for a persistent, empty Pinot instance with
+ZooKeeper:
+
+```yaml
+version: '3'
+services:
+ zookeeper:
+ image: zookeeper:latest
+ ports:
+ - "2181:2181"
+ environment:
+ ZOO_MY_ID: 1
+ volumes:
+ - ./pinot-data/zookeeper/data:/data
+ - ./pinot-data/zookeeper/datalog:/datalog
+ pinot:
+ image: apachepinot/pinot:latest
+ command:
+ - StartServiceManager
+ - -zkAddress
+ - zookeeper:2181
+ - -clusterName
+ - PinotCluster
+ - -bootstrapServices
+ - CONTROLLER
+ - BROKER
+ - SERVER
+ ports:
+ - "9000:9000" # Controller UI and API
+ - "8099:8099" # Broker query endpoint
+ - "8098:8098" # Server
+ volumes:
+ - ./pinot-data/controller:/tmp/data/controller
+ - ./pinot-data/server:/tmp/data/server
+ depends_on:
+ - zookeeper
+```
+
+If you also need Kafka for real-time stream ingestion, add it using the official
+[apache/kafka](https://hub.docker.com/r/apache/kafka) image in KRaft mode (no ZooKeeper
+needed for Kafka):
+
+```yaml
+ kafka:
+ image: apache/kafka:latest
+ ports:
+ - "9092:9092"
+ environment:
+ KAFKA_NODE_ID: 1
+ KAFKA_PROCESS_ROLES: broker,controller
+ KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:29093
+ KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
+ KAFKA_LISTENERS: PLAINTEXT://kafka:19092,EXTERNAL://0.0.0.0:9092,CONTROLLER://kafka:29093
+ KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:19092,EXTERNAL://kafka:9092
+ KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT
+ KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
+ KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
+ CLUSTER_ID: pinot-kafka-cluster-001
+```
+
+### All-in-one with docker run
+
+First, start ZooKeeper:
+
+```SHELL
+docker run -d --name pinot-zookeeper \
+ -p 2181:2181 \
+ -v $(pwd)/pinot-data/zookeeper/data:/data \
+ -v $(pwd)/pinot-data/zookeeper/datalog:/datalog \
+ zookeeper:latest
+```
+
+Then start Pinot with all services in one container:
+
+```SHELL
+docker run -d --name pinot-dev \
+ --link pinot-zookeeper:zookeeper \
+ -p 9000:9000 -p 8099:8099 -p 8098:8098 \
+ -v $(pwd)/pinot-data/controller:/tmp/data/controller \
+ -v $(pwd)/pinot-data/server:/tmp/data/server \
+ apachepinot/pinot:latest \
+ StartServiceManager \
+ -zkAddress zookeeper:2181 \
+ -clusterName PinotCluster \
+ -bootstrapServices CONTROLLER BROKER SERVER
+```
+
+This starts all services in one process without creating any demo tables. Your data is
+persisted on the host under `./pinot-data/` and survives container restarts.
+
+### Using an existing ZooKeeper (e.g. from a Kafka stack)
+
+If you already run ZooKeeper separately (e.g. as part of a Kafka stack), just point
+`-zkAddress` at your existing instance:
+
+```SHELL
+docker run -d --name pinot-dev \
+ --network my-network \
+ -p 9000:9000 -p 8099:8099 -p 8098:8098 \
+ -v $(pwd)/pinot-data/controller:/tmp/data/controller \
+ -v $(pwd)/pinot-data/server:/tmp/data/server \
+ apachepinot/pinot:latest \
+ StartServiceManager \
+ -zkAddress zookeeper:2181 \
+ -clusterName PinotCluster \
+ -bootstrapServices CONTROLLER BROKER SERVER
+```
+
+### Creating your own tables
+
+Once the cluster is up, create tables via the Controller API or UI at
+`http://localhost:9000`:
+
+```SHELL
+# Add a schema
+curl -F schemaName=@myTable_schema.json http://localhost:9000/schemas
+
+# Add a table
+curl -i -X POST -H 'Content-Type: application/json' \
+ -d @myTable_offline_table_config.json \
+ http://localhost:9000/tables
+```
+
+### QuickStart vs StartServiceManager
+
+| Feature | QuickStart | StartServiceManager |
+|---|---|---|
+| Starts all services in one process | Yes | Yes |
+| Populates demo tables on startup | Yes (10+ tables) | No |
+| Preserves data across restarts | No (deletes on stop) | Yes (with volumes) |
+| Embedded ZooKeeper | Yes | No (requires external ZK) |
+| Suitable for development | Limited | Recommended |
+| Suitable for production | No | No (use separate containers) |
diff --git a/docker/images/pinot/docker-compose.yml b/docker/images/pinot/docker-compose.yml
index d86db11e0f5..9b4a20ed46a 100644
--- a/docker/images/pinot/docker-compose.yml
+++ b/docker/images/pinot/docker-compose.yml
@@ -32,22 +32,26 @@ services:
- ./pinot-docker-demo/zookeeper/data:/data
- ./pinot-docker-demo/zookeeper/datalog:/datalog
kafka:
- image: confluentinc/cp-kafka:latest
+ image: apache/kafka:latest
hostname: kafka
ports:
- "9092:9092"
environment:
- KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka:19092,LISTENER_DOCKER_EXTERNAL://kafka:9092
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
- KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
- KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181/kafka"
- KAFKA_BROKER_ID: 1
- KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
+ # KRaft settings (no ZooKeeper required for Kafka)
+ KAFKA_NODE_ID: 1
+ KAFKA_PROCESS_ROLES: broker,controller
+ KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:29093
+ KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
+ # Listener settings
+ KAFKA_LISTENERS: PLAINTEXT://kafka:19092,EXTERNAL://0.0.0.0:9092,CONTROLLER://kafka:29093
+ KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:19092,EXTERNAL://kafka:9092
+ KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT
+ KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
+ # Cluster settings
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
+ CLUSTER_ID: pinot-kafka-cluster-001
volumes:
- ./pinot-docker-demo/kafka/data:/var/lib/kafka/data
- depends_on:
- - zookeeper
pinot-controller:
image: apachepinot/pinot:latest
hostname: pinot-controller
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org