You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/09/07 17:54:39 UTC
[GitHub] [iceberg] szehon-ho opened a new issue #3086: Allow config for write object storage path to be table location (for remove orphans)
szehon-ho opened a new issue #3086:
URL: https://github.com/apache/iceberg/issues/3086
The current "write.object-storage.path" is a good solution for throttling on object-stores like S3.
However, we like to still use the built-in 'removeOrphan' functionality for tracking storage costs/privacy. Remove Orphan relies on s3 listing under table location to find the list of potential orphan files. Using the recommended 'write.object-storage.path' like 's3://bucket/' breaks removeOrphans, as resultant orphan data files: 's3://bucket/xxxxxxxx/db/table/data/part=a/0000.parquet' are not identified listed under the table location.
The only solution seems to be to set the write.object-storage.path to be the table location manually. This does also allow less throttling (with object store prefixes set to each table location plus some characters of the hash)
But it is not so user friendly:
- Path becomes : 's3://bucket/db/table/xxxxxxxx/db/table/p=a/0000.parquet'. In particular, the 'db/table' are repeated, adding unnecessarily to path length. It is also a bit awkward in general.
- From UX perspective, user needs to be cognizant of the table location when setting this property (table location is optional in any create table statement and many users do not set it)
It will be great to have a new property like 'write.object-storage.table-location', which if enabled to true:
- sets the path to be the more reasonable and efficient 's3://bucket/db/table/xxxxxxxx/p=a/0000.parquet' (removing /data/db/table/ just as /data was removed by the original implementation)
- sets the path to automatically following the default table-location without user having to specify it. Changing table-location will automatically change the write object storage path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho closed issue #3086: Allow config for write object storage path to be table location (for remove orphans)
Posted by GitBox <gi...@apache.org>.
szehon-ho closed issue #3086:
URL: https://github.com/apache/iceberg/issues/3086
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #3086: Allow config for write object storage path to be table location (for remove orphans)
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on issue #3086:
URL: https://github.com/apache/iceberg/issues/3086#issuecomment-1069769469
Closing this issue as the first problem has been addressed in #4349. The second is still kind of annoying to have to do , maybe we can still add the flag if there is interest to default the write-object-store-path to table's location
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #3086: Allow config for write object storage path to be table location (for remove orphans)
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on issue #3086:
URL: https://github.com/apache/iceberg/issues/3086#issuecomment-914513519
Do folks think this will be useful? @jackye1995 for feedback
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org