You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/11/04 20:35:39 UTC
[GitHub] [pinot] dongxiaoman opened a new issue #7704: Error with Data size larger than 1M, will not write to zk. Data (first 1k)
dongxiaoman opened a new issue #7704:
URL: https://github.com/apache/pinot/issues/7704
This is not critical but shows some problems worth investigating.
Our QA cluster is having trouble with some accumulated data. It has 10k+ real time segments in place, data are not too much.
In logs we see something like below:
```
2021/11/04 20:31:34.166 ERROR [ZkClient] [HelixTaskExecutor-message_handle_thread] Data size larger than 1M, will not write to zk. Data (first 1k): {
"id" : "point_entry_REALTIME",
"simpleFields" : {
"BATCH_MESSAGE_MODE" : "false",
"BUCKET_SIZE" : "0",
"SESSION_ID" : "30069443a0581e1",
"STATE_MODEL_DEF" : "SegmentOnlineOfflineStateModel",
"STATE_MODEL_FACTORY_NAME" : "DEFAULT"
},
"mapFields" : {
"point_entry__0__0__20211030T0056Z" : {
"CURRENT_STATE" : "OFFLINE"
},
"point_entry__0__100__20211102T0746Z" : {
"CURRENT_STATE" : "OFFLINE"
},
"point_entry__0__101__20211102T0817Z" : {
"CURRENT_STATE" : "OFFLINE"
},
"point_entry__0__102__20211102T0909Z" : {
"CURRENT_STATE" : "OFFLINE"
},
"point_entry__0__103__20211102T0946Z" : {
"CURRENT_STATE" : "ONLINE",
"END_TIME" : "1636056441791",
"INFO
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] dongxiaoman commented on issue #7704: Error with Data size larger than 1M, will not write to zk. Data (first 1k)
Posted by GitBox <gi...@apache.org>.
dongxiaoman commented on issue #7704:
URL: https://github.com/apache/pinot/issues/7704#issuecomment-962081829
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #7704: Error with Data size larger than 1M, will not write to zk. Data (first 1k)
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #7704:
URL: https://github.com/apache/pinot/issues/7704#issuecomment-961504359
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] dongxiaoman commented on issue #7704: Error with Data size larger than 1M, will not write to zk. Data (first 1k)
Posted by GitBox <gi...@apache.org>.
dongxiaoman commented on issue #7704:
URL: https://github.com/apache/pinot/issues/7704#issuecomment-962081829
I just realized that it is probably because I have 192 partitions and `"replication": "2"` setting, while I have only 4 hosts to process them. I believe the real time servers is trying to go online for all the 192*2/4 segments at the same time (in one message) which generates a very large JSON file that exceeds the 1MB size.
Right now if I add more hosts it should solve the problem for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #7704: Error with Data size larger than 1M, will not write to zk. Data (first 1k)
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #7704:
URL: https://github.com/apache/pinot/issues/7704#issuecomment-961504359
In the current version of helix that we use in Pinot, data over 1M is automatically compressed by Helix before writing to the zookeeper. I think your compressed data is exceeding this limit (helix provides only one limit in 0.9.x, we have requested to have two limits and they will provide it in 1.x)
https://github.com/apache/helix/blob/master/zookeeper-api/src/main/java/org/apache/helix/zookeeper/datamodel/ZNRecord.java#L63:27
The best way is to remove some segments or set your segment size to be larger so that less number of segments are made.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org