You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/02/02 20:49:57 UTC
[GitHub] [incubator-pinot] pabrahamusa edited a comment on issue #6277: Creating segment larger than 8GB will fail due to gzip limitation
pabrahamusa edited a comment on issue #6277:
URL: https://github.com/apache/incubator-pinot/issues/6277#issuecomment-771973867
@fx19880617 Here is the stack trace ,
I tried these
```
# realtime.segment.flush.desired.size="400M" -- not sure this is working
# "realtime.segment.flush.threshold.size": "400M" - asper docs this should work but still get below error
```
Only thing working for me is rows as follows, But my individual row size changes drastically so I want to fix it based on a fixed size to avoid the GZIP error
"realtime.segment.flush.threshold.rows": "100000",
"realtime.segment.flush.autotune.initialRows": "100000"
```
my-pinot-server-1 server 2021/02/02 19:11:26.153 ERROR [SegmentBuildTimeLeaseExtender] [pool-8-thread-1] Failed to send lease extension for mylog__0__0__20210202T1908Z
my-pinot-server-1 server 2021/02/02 19:12:40.537 ERROR [LLRealtimeSegmentDataManager_mylog__0__0__20210202T1908Z] [mylog__0__0__20210202T1908Z] Exception while in work
my-pinot-server-1 server java.lang.RuntimeException: entry size '19675344496' is too big ( > 8589934591 ).
my-pinot-server-1 server at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.failForBigNumber(TarArchiveOutputStream.java:636) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-b4be817b1f078a4644995ad2be9ccb0eccbf6b29]
my-pinot-server-1 server at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.failForBigNumber(TarArchiveOutputStream.java:624) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-b4be817b1f078a4644995ad2be9ccb0eccbf6b29]
my-pinot-server-1 server at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.failForBigNumbers(TarArchiveOutputStream.java:610) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-b4be817b1f078a4644995ad2be9ccb0eccbf6b29]
my-pinot-server-1 server at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.putArchiveEntry(TarArchiveOutputStream.java:369) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-b4be817b1f078a4644995ad2be9ccb0eccbf6b29]
my-pinot-server-1 server at org.apache.pinot.common.utils.TarGzCompressionUtils.addFileToTarGz(TarGzCompressionUtils.java:78) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-b4be817b1f078a4644995ad2be9ccb0eccbf6b29]
my-pinot-server-1 server at org.apache.pinot.common.utils.TarGzCompressionUtils.addFileToTarGz(TarGzCompressionUtils.java:91) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-b4be817b1f078a4644995ad2be9ccb0eccbf6b29]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org