You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "wirybeaver (via GitHub)" <gi...@apache.org> on 2025/09/17 16:56:25 UTC
[I] Inconsistent max_length behavior between realtime and offline table [pinot]
wirybeaver opened a new issue, #16839:
URL: https://github.com/apache/pinot/issues/16839
The Kafka data source has a huge payload column. The max_length is set to 20992000 in the schema. No ingestion issue in the realtime part. However, when we use the Pinot Segment Driver to generate the offline tables' segments, we encountered different kinds of errors:
```
Caused by: java.lang.IllegalArgumentException: Chunk size limited to 2GB
```
```
Caused by: java.lang.IllegalArgumentException: length must be < 2113929216
at net.jpountz.lz4.LZ4Utils.maxCompressedLength(LZ4Utils.java:38)
at net.jpountz.lz4.LZ4Compressor.maxCompressedLength(LZ4Compressor.java:37)
at org.apache.pinot.segment.local.io.compression.LZ4Compressor.maxCompressedSize(LZ4Compressor.java:55)
at org.apache.pinot.segment.local.io.writer.impl.BaseChunkForwardIndexWriter.<init>(BaseChunkForwardIndexWriter.java:99)
```
I suspect the column statics collection has nuance between the realtime and offline when convert to offline pipeline.
Realtime converter: https://github.com/apache/pinot/blob/2adf54b60a2423952a057081deb063d9ae7acc87/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/converter/RealtimeSegmentConverter.java#L202
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Inconsistent max_length behavior between realtime and offline table [pinot]
Posted by "0xShubhamSolanki (via GitHub)" <gi...@apache.org>.
0xShubhamSolanki commented on issue #16839:
URL: https://github.com/apache/pinot/issues/16839#issuecomment-3304282827
## PEP Request for #16839
What needs to be done?
Fix inconsistent max_length behavior between realtime and offline table segment generation for large BYTES columns (20MB+ payloads).
Why is this needed?
- Realtime ingestion works fine with large payload columns
- Offline segment generation crashes due to LZ4 chunk size limits (2GB max)
- Users with Kafka payloads or large JSON fields face this issue
Use Case:
- Kafka topic with 20MB JSON payloads
- Schema max_length=20,992,000 for payload column
- Realtime works, offline fails with "Chunk size limited to 2GB"
Proposed Solution:
- In RealtimeSegmentConverter.build(), auto-detect large max_length columns (>2MB)
- Enable `deriveNumDocsPerChunk(true)` for dynamic safe chunk sizing
- No config changes needed, automatic detection
Impact:
- No backward compatibility issues
- Performance improvement for large column handling
- No public API changes
Plan:
- Add detection logic in RealtimeSegmentConverter
- Add unit test for large column detection
- Test with actual 20MB payload data
Please review and let me know if this can proceed to PR stage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Inconsistent max_length behavior between realtime and offline table [pinot]
Posted by "0xShubhamSolanki (via GitHub)" <gi...@apache.org>.
0xShubhamSolanki commented on issue #16839:
URL: https://github.com/apache/pinot/issues/16839#issuecomment-3304315719
@wirybeaver @Jackie-Jiang @vinayl
I've got a working solution for this issue. Can someone help add the PEP-Request label? Ready to submit PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Inconsistent max_length behavior between realtime and offline table [pinot]
Posted by "0xShubhamSolanki (via GitHub)" <gi...@apache.org>.
0xShubhamSolanki commented on issue #16839:
URL: https://github.com/apache/pinot/issues/16839#issuecomment-3305100335
Follow-up on PEP Request & Discussion
Hi @wirybeaver @Jackie-Jiang! Thanks for the quick responses.
@wirybeaver: Thanks for sharing the config! I see `"deriveNumDocsPerChunkForRawIndex": "true"` is already enabled with v3 raw index and ZSTD compression. That's interesting – this flag should help with dynamic chunk sizing to avoid the LZ4 2GB limit. But since offline is still crashing, it seems like the flag isn't fully propagating during the realtime-to-offline conversion (in `RealtimeSegmentConverter.build()`).
@Jackie-Jiang: Great suggestion on v4 index! I'll test that locally. Quick question: Does v4 handle large max_length (20MB+) better out-of-the-box, or is it still LZ4-dependent? (Our case uses ZSTD, but the error points to LZ4 in the forward index writer.
My Proposed Fix Update -
- Even with the config flag, the converter might need explicit enabling of `genConfig.setDeriveNumDocsPerChunk(true)` for large columns detected via schema (max_length > 2MB).
- This ensures safe chunks: `num_rows_per_chunk = floor(2GB / 20MB) ≈ 100 rows`.
- Code snippet ready (tested locally with 20MB payloads – offline builds succeed now).
- No conflict with v3/v4 or ZSTD.
Questions for Review:
1. Should we try v4 first, or proceed with this converter fix?
2. If OK, can we re-add PEP-Request label and go to PR?
3. Any concerns with auto-detection threshold (2MB)?
Happy to hop on a quick call/Slack if needed.
References: [LZ4 Limits] (https://github.com/lz4/lz4-java/blob/master/src/java/net/jpountz/lz4/LZ4Utils.java)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Inconsistent max_length behavior between realtime and offline table [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #16839:
URL: https://github.com/apache/pinot/issues/16839#issuecomment-3304670624
@wirybeaver Are you generating v2 raw index on this column? v4 index should work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Inconsistent max_length behavior between realtime and offline table [pinot]
Posted by "wirybeaver (via GitHub)" <gi...@apache.org>.
wirybeaver commented on issue #16839:
URL: https://github.com/apache/pinot/issues/16839#issuecomment-3304825736
> [@wirybeaver](https://github.com/wirybeaver) Are you generating v2 raw index on this column? v4 index should work
It is a hybrid table. I use the v3 raw index for both realtime and offline. Realtime doesn't have ingestion failure.
```
{
"name": "foo",
"encodingType": "RAW",
"indexTypes": [],
"compressionCodec": "ZSTANDARD",
"indexes": null,
"properties": {
"deriveNumDocsPerChunkForRawIndex": "true",
"rawIndexWriterVersion": "3"
},
"tierOverwrites": null
},
```
Schema max_length is set to 20992000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Inconsistent max_length behavior between realtime and offline table [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16839:
URL: https://github.com/apache/pinot/issues/16839#issuecomment-4113107464
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Inconsistent max_length behavior between realtime and offline table [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #16839:
URL: https://github.com/apache/pinot/issues/16839#issuecomment-3331415847
V4 builds chunks based on the actual content side, instead of holding same count of values. I'd suggest trying v4 and see if the problem is solved. Going forward we'll deprecate v2 and v3 creator
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Inconsistent max_length behavior between realtime and offline table [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16839:
URL: https://github.com/apache/pinot/issues/16839#issuecomment-4317212095
This issue has been closed because it has been inactive for 180 days. If you believe this issue is still relevant, please feel free to reopen it or create a new issue. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Inconsistent max_length behavior between realtime and offline table [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #16839: Inconsistent max_length behavior between realtime and offline table
URL: https://github.com/apache/pinot/issues/16839
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org