You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "njalan (via GitHub)" <gi...@apache.org> on 2023/09/17 15:40:57 UTC
[GitHub] [hudi] njalan opened a new issue, #9736: What is the use of hoodie.meta.sync.metadata_file_listing
njalan opened a new issue, #9736:
URL: https://github.com/apache/hudi/issues/9736
How to use hoodie.meta.sync.metadata_file_listing? If I enable this whether get the file list from metadata for reducing s3 file list ? is it the same as variable DEFAULT_METADATA_ENABLE_FOR_READERS in HoodieMetadataConfig?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9736: What is the use of hoodie.meta.sync.metadata_file_listing
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9736:
URL: https://github.com/apache/hudi/issues/9736#issuecomment-1722774995
@njalan Yes, Your understanding is correct, While doing the metadata sync it needs to list the partitions. For that it can just use the metadata table instead of file listing. When we have a large number of partitions, this can reduce the S3 file list by great instant.
During writes, as metadata table is enable by default it should use metadata table only for most of the file listing operations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] njalan commented on issue #9736: What is the use of hoodie.meta.sync.metadata_file_listing
Posted by "njalan (via GitHub)" <gi...@apache.org>.
njalan commented on issue #9736:
URL: https://github.com/apache/hudi/issues/9736#issuecomment-1725043273
@ad1happy2go I have enabled metadata but list operation is not reduced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] njalan closed issue #9736: What is the use of hoodie.meta.sync.metadata_file_listing
Posted by "njalan (via GitHub)" <gi...@apache.org>.
njalan closed issue #9736: What is the use of hoodie.meta.sync.metadata_file_listing
URL: https://github.com/apache/hudi/issues/9736
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] njalan commented on issue #9736: What is the use of hoodie.meta.sync.metadata_file_listing
Posted by "njalan (via GitHub)" <gi...@apache.org>.
njalan commented on issue #9736:
URL: https://github.com/apache/hudi/issues/9736#issuecomment-1722857559
@ad1happy2go What do you mean for only for most of the file listing operations. is there any threshold ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9736: What is the use of hoodie.meta.sync.metadata_file_listing
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9736:
URL: https://github.com/apache/hudi/issues/9736#issuecomment-1722868107
@njalan I meant there are some code paths which can still do file operations but most of them uses metadata table for that,.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org