You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/12/15 02:55:05 UTC
[GitHub] [iceberg] txdong-sz opened a new issue #1934: remove a list .metadata.json in HadoopCatalog
txdong-sz opened a new issue #1934:
URL: https://github.com/apache/iceberg/issues/1934
in hadoop cos scene this list status may cost a lot of time if there are many files in cos metadata.
and we spend about 10s to list tables .
if we remove (fs.listStatus(metadataPath, TABLE_FILTER).length >= 1) or find a better way to list file like
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jacques-n commented on issue #1934: remove a list .metadata.json in HadoopCatalog
Posted by GitBox <gi...@apache.org>.
jacques-n commented on issue #1934:
URL: https://github.com/apache/iceberg/issues/1934#issuecomment-745028445
I'm guessing that FileSystem.listStatusIterator() could be faster since you only need to check if it is greater than 1 and it doesn't require a full traversal. @txdong-sz do you want to propose a patch? You'd need to implement the filter outside the call since I believe that call doesn't support filters.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] txdong-sz commented on issue #1934: remove a list .metadata.json in HadoopCatalog
Posted by GitBox <gi...@apache.org>.
txdong-sz commented on issue #1934:
URL: https://github.com/apache/iceberg/issues/1934#issuecomment-745713500
@jacques-n
i also think that it is not need to check metadata.json
if namespace with metadata folder in so we can just consider that is a table
or eles
check if that metadata folder have a version-hint.text file in it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org