You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@heron.apache.org by GitBox <gi...@apache.org> on 2018/07/18 20:42:49 UTC
[GitHub] nlu90 opened a new pull request #2967: spill state to local disk
when configured
nlu90 opened a new pull request #2967: spill state to local disk when configured
URL: https://github.com/apache/incubator-heron/pull/2967
During our test, we found that if a state is too large (>~64MB), the original checkpointing mechanism fails. The actual state will be packed in the protobuf message and transmitted through the network. Once it's large enough, the receiver will have problems parsing the IncomingPacket, which causes the connection to be closed. And it fails the whole checkpointing process.
This PR solves this problem by introducing the state spilling feature. With the feature turned on, the actual state will be spilled to local disk and only the state location information is transmitted through the network. This ensures the checkpointing works for even very large states.
Tested with internal topologies. A previous related PR is #2962
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services