You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "qswawrq (via GitHub)" <gi...@apache.org> on 2025/11/10 19:14:19 UTC
[I] ZooKeeper Contention During Batch Ingestion [pinot]
qswawrq opened a new issue, #17178:
URL: https://github.com/apache/pinot/issues/17178
# ZooKeeper Contention and Linear Performance Degradation with High-Parallelism Batch Ingestion
## Environment
- **Pinot Version**: apachepinot/pinot:latest
- **Deployment**: Kubernetes
- **Cluster Size**: 1 Controller, 1 ZooKeeper, 3 Brokers, 3 Servers
- **ZooKeeper Config**:
- `ZOO_SNAPCOUNT=100000`
- `ZOO_AUTOPURGE_INTERVAL=1`
- `ZOO_AUTOPURGE_RETAIN_COUNT=5`
- Heap: 1GB
- Storage: 20GB used
## Problem Description
We're experiencing severe ZooKeeper contention and linear performance degradation during batch ingestion of 50,000 parquet files (each around 350 MB) using 100 parallel Kubernetes Job workers with `jobType: SegmentCreationAndMetadataPush`.
### Performance Degradation Timeline
| Time Elapsed | Files/Worker | Total Segments | Time per File | Degradation Factor |
|--------------|--------------|----------------|---------------|-------------------|
| Initial | 0-5 | 500 | 2 minutes | 1x (baseline) |
| +3 hours | 8-9 | 1,002 | 20 minutes | 10x |
| +30 hours | 43-44 | 4,438 | 88 minutes | 44x |
| +50 hours | 73-76 | 6,150 | 175 minutes | 87x |
**Performance continues to degrade linearly as segment count grows.**
**Critical Finding**: We tested with 2, 5, and 100 workers. Linear degradation occurs at **all parallelism levels**, indicating the bottleneck is not just concurrency, but the **O(n) cost of reading/writing the growing segment list**. It seems the ingestion workers take a little time to build out the segments locally but spend 99% time on trying to upload metadata.
## Observed Behavior
### 1. ZooKeeper Metadata Update Failures (17% Error Rate)
Segment uploads frequently fail with optimistic locking errors. In the last 10,000 log lines, we observed **111 ZK version conflicts out of 661 upload attempts** (16.8% failure rate):
```
2025/11/09 15:47:07.055 ERROR [ZKOperator] [jersey-server-managed-async-executor-128]
Failed to update ZK metadata for segment: post_metrics_OFFLINE_20315_20344_000000021514_0,
table: post_metrics_OFFLINE, expected version: 3428
2025/11/09 15:47:07.055 ERROR [PinotSegmentUploadDownloadRestletResource] [jersey-server-managed-async-executor-128]
Exception while uploading segment: Failed to update ZK metadata for segment: post_metrics_OFFLINE_20315_20344_000000021514_0,
table: post_metrics_OFFLINE, expected version: 3428
java.lang.RuntimeException: Failed to update ZK metadata for segment: post_metrics_OFFLINE_20315_20344_000000021514_0,
table: post_metrics_OFFLINE, expected version: 3428
at org.apache.pinot.controller.api.upload.ZKOperator.processExistingSegment(ZKOperator.java:341)
at org.apache.pinot.controller.api.upload.ZKOperator.completeSegmentOperations(ZKOperator.java:120)
at org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource.uploadSegment(PinotSegmentUploadDownloadRestletResource.java:433)
```
### 2. Large ZooKeeper Transaction Logs
ZooKeeper transaction logs have grown to **multi-GB sizes**:
```bash
$ kubectl exec pinot-zookeeper-0 -- ls -lh /bitnami/zookeeper/data/version-2/
-rw-r--r-- 1 1001 1001 129M Nov 10 16:42 log.1c4167f
-rw-r--r-- 1 1001 1001 65M Nov 10 17:00 log.1c58844
-rw-r--r-- 1 1001 1001 1.0G Nov 10 17:28 log.1c78fc6
-rw-r--r-- 1 1001 1001 1.9G Nov 10 17:44 log.1c86b58
-rw-r--r-- 1 1001 1001 2.2G Nov 10 18:32 log.1cc3e55 ← Largest log
Total ZK data directory: 5.4GB
```
Despite autopurge being enabled (`ZOO_AUTOPURGE_INTERVAL=1`, `ZOO_SNAPCOUNT=100000`), individual transaction logs grow to 2.2GB before snapshots are taken. Not sure if it is relevant to the performance degrade.
## Configuration
**Ingestion Job Spec:**
```yaml
jobType: SegmentCreationAndMetadataPush
pushJobSpec:
pushAttempts: 10
pushRetryIntervalMillis: 2000
pushFileNamePattern: 'glob:**post_metrics_OFFLINE_*_*_$FILE_PADDED_*.tar.gz'
```
**Kubernetes Job:**
- `completions: 100`
- `parallelism: 100`
- `completionMode: Indexed`
Each worker processes 500 files sequentially, creating one segment per file.
## Questions
1. **Is this a known limitation** of single-ZK deployments with high-parallelism ingestion?
2. **What is the recommended parallelism architecture** for metadata push operations to avoid ZK contention?
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] ZooKeeper Contention During Batch Ingestion [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #17178:
URL: https://github.com/apache/pinot/issues/17178#issuecomment-4676144632
This issue has been closed because it has been inactive for 180 days. If you believe this issue is still relevant, please feel free to reopen it or create a new issue. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] ZooKeeper Contention During Batch Ingestion [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #17178: ZooKeeper Contention During Batch Ingestion
URL: https://github.com/apache/pinot/issues/17178
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] ZooKeeper Contention During Batch Ingestion [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #17178:
URL: https://github.com/apache/pinot/issues/17178#issuecomment-4317211385
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] ZooKeeper Contention During Batch Ingestion [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #17178:
URL: https://github.com/apache/pinot/issues/17178#issuecomment-3544410313
During our operation of the cluster, we have never run into similar problems. The ZK log size is definitely higher than expected given the segment count (we have large tables with more than 500K segments).
Could you please try a larger instance, or check the ZK settings?
cc @xiangfu0 @KKcorps for extra input
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org