You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2023/06/14 09:44:52 UTC
[GitHub] [incubator-paimon] JingsongLi opened a new issue, #1375: [Feature] Optimize memory for big column
JingsongLi opened a new issue, #1375:
URL: https://github.com/apache/incubator-paimon/issues/1375
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Motivation
Currently Paimon has some problems with memory usage if there is a column with a very large field, more than 2MB.
Problem:
1. Batch size (read.batch-size) not work for sink, currently, Orc writer also create a ColumnarBatch, we should limit its memory too.
2. Orc Dictionary needs much memory, we can introduce a common option and document this to close Dictionary.
3. In the scenario of frequent failover, we should restrict the continuation of new file committing, see #1351
4. `RollingFileWriter` should not refer all writers, or we should set writer and maxKey to null in `SingleFileWriter`s.
5. We should avoid adding statistics to very large columns, which can cause our Manifest file to become very large.
### Solution
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on issue #1375: [Feature] Optimize memory for big column
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on issue #1375:
URL: https://github.com/apache/incubator-paimon/issues/1375#issuecomment-1594023639
> hi,@JingsongLi , do you think is it necessary to separate key and value for big value? like this : https://rocksdb.org/blog/2021/05/26/integrated-blob-db.html
This is a huge topic....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on issue #1375: [Feature] Optimize memory for big column
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on issue #1375:
URL: https://github.com/apache/incubator-paimon/issues/1375#issuecomment-1598066423
> Doris also has similar function, and I think paimon currently may does not have strong user needs. We can discuss this topic when users have needs later.
Hi @zhangjun0x01 , if just for performance improvement, yes Paimon also has the requirement. Can you just create a new issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] zhangjun0x01 commented on issue #1375: [Feature] Optimize memory for big column
Posted by "zhangjun0x01 (via GitHub)" <gi...@apache.org>.
zhangjun0x01 commented on issue #1375:
URL: https://github.com/apache/incubator-paimon/issues/1375#issuecomment-1593300031
hi,@JingsongLi , do you think is it necessary to separate key and value for big value? like this : https://rocksdb.org/blog/2021/05/26/integrated-blob-db.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] zhangjun0x01 commented on issue #1375: [Feature] Optimize memory for big column
Posted by "zhangjun0x01 (via GitHub)" <gi...@apache.org>.
zhangjun0x01 commented on issue #1375:
URL: https://github.com/apache/incubator-paimon/issues/1375#issuecomment-1605586848
> > Doris also has similar function, and I think paimon currently may does not have strong user needs. We can discuss this topic when users have needs later.
>
> Hi @zhangjun0x01 , if just for performance improvement, yes Paimon also has the requirement. Can you just create a new issue?
I create a new issue https://github.com/apache/incubator-paimon/issues/1428
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi closed issue #1375: [Feature] Optimize memory for big column
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed issue #1375: [Feature] Optimize memory for big column
URL: https://github.com/apache/incubator-paimon/issues/1375
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] wg1026688210 commented on issue #1375: [Feature] Optimize memory for big column
Posted by "wg1026688210 (via GitHub)" <gi...@apache.org>.
wg1026688210 commented on issue #1375:
URL: https://github.com/apache/incubator-paimon/issues/1375#issuecomment-1590882374
I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] zhangjun0x01 commented on issue #1375: [Feature] Optimize memory for big column
Posted by "zhangjun0x01 (via GitHub)" <gi...@apache.org>.
zhangjun0x01 commented on issue #1375:
URL: https://github.com/apache/incubator-paimon/issues/1375#issuecomment-1594057692
> > hi,@JingsongLi , do you think is it necessary to separate key and value for big value? like this : https://rocksdb.org/blog/2021/05/26/integrated-blob-db.html
>
> This is a huge topic....
Doris also has similar function, and I think paimon currently may does not have strong user needs. We can discuss this topic when users have needs later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org