You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "mohitgargk (via GitHub)" <gi...@apache.org> on 2023/08/08 19:48:39 UTC
[GitHub] [iceberg] mohitgargk opened a new pull request, #8260: relativePath
mohitgargk opened a new pull request, #8260:
URL: https://github.com/apache/iceberg/pull/8260
**Basic changes**
- [ ] Use location provider to relativeize
- [ ] No changes in spec
- [ ] No changes in the value of the objects.
**File Path Changes**
Only there read and write parsers are passed with location-provider
- [ ] metadata-log : TableMetadataParser
- [ ] manifest-list : SnapshotParser
- [ ] manifest :
- [ ] datafile loc
**Ops support**
- [ ] Hadoop
- [ ] BaseMetastoreTableOperations
- [ ] Other ??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] relativePath [wip] [iceberg]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on PR #8260:
URL: https://github.com/apache/iceberg/pull/8260#issuecomment-1836983928
Are we still working on it? Looks like there are some interest from the community also.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] relativePath [wip] [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8260:
URL: https://github.com/apache/iceberg/pull/8260#issuecomment-2347443962
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] relativePath [wip] [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8260:
URL: https://github.com/apache/iceberg/pull/8260#issuecomment-2364773434
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] mohitgargk commented on pull request #8260: relativePath [wip]
Posted by "mohitgargk (via GitHub)" <gi...@apache.org>.
mohitgargk commented on PR #8260:
URL: https://github.com/apache/iceberg/pull/8260#issuecomment-1685354024
> does it handle the case where we pointed the data directory somewhere else ?
Yes.
@manuzhang added the design in the description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] mohitgargk commented on pull request #8260: relativePath [wip]
Posted by "mohitgargk (via GitHub)" <gi...@apache.org>.
mohitgargk commented on PR #8260:
URL: https://github.com/apache/iceberg/pull/8260#issuecomment-1681152456
> Do we have a design doc for this feature?
Should be ready by this week. Will provide link in this thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] manuzhang commented on pull request #8260: relativePath [wip]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #8260:
URL: https://github.com/apache/iceberg/pull/8260#issuecomment-1685611555
@mohitgargk you may send the design doc to dev list for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] relativePath [wip] [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #8260: relativePath [wip]
URL: https://github.com/apache/iceberg/pull/8260
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rustyconover commented on pull request #8260: relativePath [wip]
Posted by "rustyconover (via GitHub)" <gi...@apache.org>.
rustyconover commented on PR #8260:
URL: https://github.com/apache/iceberg/pull/8260#issuecomment-1687313178
Hi All, I love the idea of this spec, I'm especially interested in using Iceberg with S3 replication for disaster recovery with S3 replication.
There will be challenges with the Glue catalog specifically. But I have ideas how to over come them. Additionally with S3 replication there can be time where all of the contents of a table snapshot are not yet replicated, so it would be useful to track which "snapshot" is available at each location where the table is replicated. Again, this is possible to do for S3 but I'm unsure of other storage systems.
From reading the google doc, it is not especially clear what is proposed. There are many "alternatives" proposed and I couldn't divine the actual proposal's TL;DR action points.
Personally, I was thinking of solving this challenge with a map of a "location" string to a regular expression to apply to all table URIs. Each catalog instance could specify their "location" using a URI parameter added to the `metadata.json` current_metadata. An example could be for a table in us-east-2 replicated from us-east-1.
`s3://bucket-us-east-2/database/table_name/data/[uuid].metadata.json?location=aws:us-east-2`
The Iceberg table's metadata would have a property called "location_replacements" which would be a JSON string of a serialized map from each "location" to a regular expression to perform on all URI references.
```
metadata.table.location_replacements = {
"aws:us-east-2": "s/^s3:\/\/bucket-us-east-1\//s3:\/\/bucket-us-east-2/"
}
```
The regular expression substitution would mean paths that normally reference `bucket-us-east-1` would reference `bucket-us-east-2` when reading the catalog from that "location".
The location replacements map would contain a "stringified" regular expression that can be applied all URIs that are contained in the table. The use of regular expressions would allow a lot of flexibility, but bring a moderate amount of complexity. Regular expressions could allow tables to be at different relative paths depending on storage media.
Since this will change almost every client of Iceberg this spec should likely have more eyes on it.
@fokko your thoughts from the Python perspective?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] manuzhang commented on pull request #8260: relativePath [wip]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #8260:
URL: https://github.com/apache/iceberg/pull/8260#issuecomment-1679928589
Do we have a design doc for this feature?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zinking commented on pull request #8260: relativePath [wip]
Posted by "zinking (via GitHub)" <gi...@apache.org>.
zinking commented on PR #8260:
URL: https://github.com/apache/iceberg/pull/8260#issuecomment-1682149541
does it handle the case where we pointed the data directory somewhere else ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org