You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "wg1026688210 (via GitHub)" <gi...@apache.org> on 2024/03/08 07:37:32 UTC
[I] [Feature] Reduce the access of filesystem when reading paimon table [incubator-paimon]
wg1026688210 opened a new issue, #2978:
URL: https://github.com/apache/incubator-paimon/issues/2978
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Motivation
Currently, the computation engine frequently reads the schema file and the size of the data files from filesystem when reading the Paimon table.This leading to additional access to the file system. The purpose of this issue is to reduce access to the file system by caching the schema ,paimon table and so on.
### Solution
Hive:
- Cache the table into Split to avoid loading Paimon table when create reader #2927
- Cache TableSchema into Configuration to avoid loading read scheme file in PaimonSerDe #2946
Core:
- Obtain the size of the data file from the metadata. #2918
- cache the schema for SchemaManager. #2939
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Reduce the access of filesystem when reading paimon table [paimon]
Posted by "zhangjun0x01 (via GitHub)" <gi...@apache.org>.
zhangjun0x01 commented on issue #2978:
URL: https://github.com/apache/paimon/issues/2978#issuecomment-2029469503
> > I submit an pr to cache table in catalog , #2971 you could help me review it
>
> @zhangjun0x01 This is a separate issue. In theory, we can cache all the files.
yes ，we can open other pr to cache manifest , like this https://github.com/apache/iceberg/pull/4518
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Reduce the access of filesystem when reading paimon table [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on issue #2978:
URL: https://github.com/apache/paimon/issues/2978#issuecomment-2029270309
@wg1026688210 I think we have finished this issue, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Reduce the access of filesystem when reading paimon table [incubator-paimon]
Posted by "zhangjun0x01 (via GitHub)" <gi...@apache.org>.
zhangjun0x01 commented on issue #2978:
URL: https://github.com/apache/incubator-paimon/issues/2978#issuecomment-1990418076
we can also to cache manifest file to improve query speed, after all, `plan()` is also a very heavy operation
https://docs.cloudera.com/cdw-runtime/cloud/iceberg-how-to/topics/iceberg-manifest-caching.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Reduce the access of filesystem when reading paimon table [paimon]
Posted by "Kyofin (via GitHub)" <gi...@apache.org>.
Kyofin commented on issue #2978:
URL: https://github.com/apache/paimon/issues/2978#issuecomment-2488012980
> > > I submit an pr to cache table in catalog , #2971 you could help me review it
> >
> >
> > @zhangjun0x01 This is a separate issue. In theory, we can cache all the files.
>
> yes ，we can open other pr to cache manifest , like this [apache/iceberg#4518](https://github.com/apache/iceberg/pull/4518)
Is there any progress? This is a very important requirement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Reduce the access of filesystem when reading paimon table [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on issue #2978:
URL: https://github.com/apache/paimon/issues/2978#issuecomment-2029270076
> I submit an pr to cache table in catalog , #2971 you could help me review it
@zhangjun0x01 This is a separate issue. In theory, we can cache all the files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Reduce the access of filesystem when reading paimon table [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed issue #2978: [Feature] Reduce the access of filesystem when reading paimon table
URL: https://github.com/apache/paimon/issues/2978
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Reduce the access of filesystem when reading paimon table [incubator-paimon]
Posted by "zhangjun0x01 (via GitHub)" <gi...@apache.org>.
zhangjun0x01 commented on issue #2978:
URL: https://github.com/apache/incubator-paimon/issues/2978#issuecomment-1988399539
I submit an pr to cache table in catalog , https://github.com/apache/incubator-paimon/pull/2971
you could help me review it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org