You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Sotatek-HungNguyenThanh (via GitHub)" <gi...@apache.org> on 2023/08/31 07:01:58 UTC
[GitHub] [pinot] Sotatek-HungNguyenThanh opened a new issue, #11471: Pinot Server crash
Sotatek-HungNguyenThanh opened a new issue, #11471:
URL: https://github.com/apache/pinot/issues/11471
I create a pinot cluster include 1 controller and many server (6 server). I am setup stream table with mode upsert table size 1B records. I run a time I see pinot server offten crash and controller lag when view UI. I received some error below:
2023/08/30 23:12:55.906 ERROR [StatusUpdateUtil] [HelixTaskExecutor-message_handle_thread_7] Exception while logging status update
```
org.apache.helix.HelixException: HelixManager is not connected within retry timeout for cluster PinotCluster
at org.apache.helix.manager.zk.ZKHelixManager.checkConnected(ZKHelixManager.java:417) ~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at org.apache.helix.manager.zk.ZKHelixManager.getHelixDataAccessor(ZKHelixManager.java:681) ~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at org.apache.helix.util.StatusUpdateUtil.logMessageStatusUpdateRecord(StatusUpdateUtil.java:353) [pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at org.apache.helix.util.StatusUpdateUtil.logError(StatusUpdateUtil.java:405) [pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:119) [pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2023/08/30 23:12:59.776 ERROR [HelixTask] [HelixTaskExecutor-message_handle_thread_7] Message execution failed. msgId: 3cc8c5cf-140a-4c93-b0e4-6695d3761868, errorMsg: HelixManager is not connected within retry timeout for cluster PinotCluster
2023/08/30 23:13:11.376 ERROR [ZKHelixManager] [HelixTaskExecutor-message_handle_thread_7] zkClient is not connected after waiting 10000ms., clusterName: PinotCluster, zkAddress: 172.16.198.43:2181
```
```
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Start a Pinot [SERVER]-SendThread(172.16.198.43:2181)"
Exception in thread "transactions__8__69__20230831T0534Z" Exception in thread "HttpServer-0" java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.HashMap.values(HashMap.java:955)
at org.apache.pinot.segment.local.segment.readers.PinotSegmentRecordReader.close(PinotSegmentRecordReader.java:243)
at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:250)
at org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter.build(RealtimeSegmentConverter.java:110)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentInternal(LLRealtimeSegmentDataManager.java:895)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentForCommit(LLRealtimeSegmentDataManager.java:806)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:705)
at java.base/java.lang.Thread.run(Thread.java:829)
java.lang.OutOfMemoryError: Java heap space
Exception in thread "metrics-meter-tick-thread-1" java.lang.OutOfMemoryError: Java heap space
2023/08/31 06:04:02.458 ERROR [SegmentBuildTimeLeaseExtender] [pool-4-thread-1] Failed to send lease extension for transactions__8__69__20230831T0534Z
Exception in thread "transactions__2__74__20230831T0538Z" java.lang.OutOfMemoryError: Java heap space
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #11471: Pinot Server crash
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #11471:
URL: https://github.com/apache/pinot/issues/11471#issuecomment-1712043200
For your transactions, do you have an `id` field (ideally stored as LONG) associated with each transaction? Using hash string of length up to 70 can take a lot of heap memory in the upsert metadata
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #11471: Pinot Server crash
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #11471:
URL: https://github.com/apache/pinot/issues/11471#issuecomment-1702142435
Upsert metadata is held in memory (heap), which is causing servers running out of memory. You may either add more servers, or consider the off-heap upsert feature provided by Star-Tree distribution of Pinot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Pinot Server crash [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang closed issue #11471: Pinot Server crash
URL: https://github.com/apache/pinot/issues/11471
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Sotatek-HungNguyenThanh commented on issue #11471: Pinot Server crash
Posted by "Sotatek-HungNguyenThanh (via GitHub)" <gi...@apache.org>.
Sotatek-HungNguyenThanh commented on issue #11471:
URL: https://github.com/apache/pinot/issues/11471#issuecomment-1703979721
this is my server config
```
pinot.service.role=SERVER
# Pinot Cluster name
pinot.cluster.name=PinotCluster
# Pinot Zookeeper Server
pinot.zk.server=172.16.198.43:2181
# Use hostname as Pinot Instance ID other than IP
pinot.set.instance.id.to.hostname=true
# Pinot Server Netty Port for queris
pinot.server.netty.port=8098
# Pinot Server Admin API port
pinot.server.adminapi.port=8097
# # Pinot Server Data Directory
# pinot.server.instance.dataDir=/opt/pinot/data/server/index
#
# # Pinot Server Temporary Segment Tar Directory
# pinot.server.instance.segmentTarDir=/opt/pinot/data/server/segmentTar
pinot.server.instance.dataDir=/opt/pinot/data/server/index
pinot.server.instance.segmentTarDir=/opt/pinot/data/server/segmentTar
pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.server.segment.fetcher.protocols=file,http,s3
pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.server.storage.factory.s3.region=us-west-2
pinot.server.segment.store.uri=s3://pinot/data/eth_mainnet
pinot.server.instance.segment.store.uri=s3://pinot/data
pinot.server.instance.enable.split.commit=true
realtime.segment.serverUploadToDeepStore=true
pinot.controller.storage.factory.s3.accessKey=admin
pinot.controller.storage.factory.s3.secretKey=Xxxx@2022
pinot.controller.storage.factory.s3.disableAcl=false
```
this is my table schema
```
{
"schemaName": "transactions",
"dimensionFieldSpecs": [
{
"name": "timestamp",
"dataType": "INT"
},
{
"name": "hash",
"dataType": "STRING",
"maxLength": 70
},
{
"name": "nonce",
"dataType": "STRING",
"maxLength": 50
},
{
"name": "from",
"dataType": "STRING",
"maxLength": 50
},
{
"name": "to",
"dataType": "STRING",
"maxLength": 50
},
{
"name": "value",
"dataType": "STRING"
},
],
"dateTimeFieldSpecs": [
{
"name": "block_time",
"dataType": "TIMESTAMP",
"format": "1:MILLISECONDS:EPOCH",
"granularity": "1:MILLISECONDS"
}
],
"primaryKeyColumns": [
"hash"
]
}
```
I have a 6 pinot server (96g ram) and I only store 1.25B logs. If I need to save 25B records, how do I prepare resources? I tried calculating resources according to the guide docs but it's not right
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Pinot Server crash [pinot]
Posted by "hpvd (via GitHub)" <gi...@apache.org>.
hpvd commented on issue #11471:
URL: https://github.com/apache/pinot/issues/11471#issuecomment-2170938148
@Sotatek-HungNguyenThanh Does this idea solves this issue for you?
If so we may close it...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org