You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "Akwangg (via GitHub)" <gi...@apache.org> on 2025/07/22 09:24:38 UTC
[PR] [core] Fix scan metric report for extra file-index files [paimon]
Akwangg opened a new pull request, #5937:
URL: https://github.com/apache/paimon/pull/5937
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
Fix scan metric report when using file-index which stored in DataFileMeta#_EXTRA_FILES.
<!-- What is the purpose of the change -->
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Fix scan metric report for extra file-index files [paimon]
Posted by "Akwangg (via GitHub)" <gi...@apache.org>.
Akwangg commented on PR #5937:
URL: https://github.com/apache/paimon/pull/5937#issuecomment-3106712977
> > > @Akwangg Thank you for your contribution! Did you mean evaluating the extra file index during the scanning phase?
> >
> >
> > yes, scan phase only evaluating the embedded file index, the scan result is incorrect for extra file index.
>
> I think it was done on purpose. During the scanning phase, we aim to evaluate as quickly as possible, so we only evaluate the embedding file index, which is stored in the manifest. The extra file index, however, is stored independently and requires additional I/O to load, it may slow down the scanning process.
If so, the reported scan metrics are the total table files, but some files are filtered when the data is actually read, which can easily lead to misunderstand that the index is not effective. Is there a better solution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Fix scan metric report for extra file-index files [paimon]
Posted by "Akwangg (via GitHub)" <gi...@apache.org>.
Akwangg commented on PR #5937:
URL: https://github.com/apache/paimon/pull/5937#issuecomment-3106462785
> @Akwangg Thank you for your contribution! Did you mean evaluating the extra file index during the scanning phase?
yes, scan phase only evaluating the embedded file index, the scan result is incorrect for extra file index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Fix scan metric report for extra file-index files [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed pull request #5937: [core] Fix scan metric report for extra file-index files
URL: https://github.com/apache/paimon/pull/5937
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Fix scan metric report for extra file-index files [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed pull request #5937: [core] Fix scan metric report for extra file-index files
URL: https://github.com/apache/paimon/pull/5937
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Fix scan metric report for extra file-index files [paimon]
Posted by "Tan-JiaLiang (via GitHub)" <gi...@apache.org>.
Tan-JiaLiang commented on PR #5937:
URL: https://github.com/apache/paimon/pull/5937#issuecomment-3106379650
@Akwangg Thank you for your contribution! Did you mean evaluating the extra file index during the scanning phase?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Fix scan metric report for extra file-index files [paimon]
Posted by "Tan-JiaLiang (via GitHub)" <gi...@apache.org>.
Tan-JiaLiang commented on PR #5937:
URL: https://github.com/apache/paimon/pull/5937#issuecomment-3106619106
> > @Akwangg Thank you for your contribution! Did you mean evaluating the extra file index during the scanning phase?感谢您的贡献！您的意思是在扫描阶段评估额外的文件索引吗？
>
> yes, scan phase only evaluating the embedded file index, the scan result is incorrect for extra file index.是的，扫描阶段仅评估嵌入式文件索引，额外文件索引的扫描结果不正确。
I think it was done on purpose. During the scanning phase, we aim to evaluate as quickly as possible, so we only evaluate the embedding file index, which is stored in the manifest. The extra file index, however, is stored independently and requires additional I/O to load, it may slow down the scanning process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Fix scan metric report for extra file-index files [paimon]
Posted by "Tan-JiaLiang (via GitHub)" <gi...@apache.org>.
Tan-JiaLiang commented on PR #5937:
URL: https://github.com/apache/paimon/pull/5937#issuecomment-3106780686
> If so, the reported scan metrics are the total table files, but some files are filtered when the data is actually read, which can easily lead to misunderstand that the index is not effective. Is there a better solution?
I think we can introduce some metrics for the file index evaluation phase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Fix scan metric report for extra file-index files [paimon]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5937:
URL: https://github.com/apache/paimon/pull/5937#issuecomment-4414038220
This pull request has had no activity for 90 days. If you'd like to keep it open, please push a new commit or leave a comment. Thanks for the contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org