You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/12/17 14:46:24 UTC
[GitHub] [doris-spark-connector] luolei604 opened a new pull request, #61: [improvement] splits large collections to normal collection to avoid the "Requested array size exceeds VM limit" exception
luolei604 opened a new pull request, #61:
URL: https://github.com/apache/doris-spark-connector/pull/61
# Proposed changes
Splits large collections to normal collection to avoid the "Requested array size exceeds VM limit" exception
1. Avoid the "Requested array size exceeds VM limit "write exception when a batch is large.
2. When the batch data is unbalanced, the same batch takes up more storage, and some take up less storage. We can also increase the batch size to avoid exceptions and reduce the write compaction pressure.
## Problem Summary:
Recursively splits large collections to normal collection and serializes the collection
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
3. Has unit tests been added: (Yes/No/No Need)
4. Has document been added or modified: (Yes/No/No Need)
5. Does it need to update dependencies: (Yes/No)
6. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
The exception :
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris-spark-connector] hf200012 merged pull request #61: [improvement] splits large collections to normal collection to avoid the "Requested array size exceeds VM limit" exception
Posted by GitBox <gi...@apache.org>.
hf200012 merged PR #61:
URL: https://github.com/apache/doris-spark-connector/pull/61
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org