You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Selinix (via GitHub)" <gi...@apache.org> on 2024/11/21 09:06:52 UTC
[I] Best Practices for Storing and Querying Full History and Latest Versions [iceberg]
Selinix opened a new issue, #11618:
URL: https://github.com/apache/iceberg/issues/11618
### Query engine
Spark for loading, Trino for querying
### Question
Hi,
I’m looking for guidance on the most efficient solution for maintaining full history and querying the latest versions of events without maintaining redundant copies of the data.
A use case is to be able to query either:
1. All versions of an event (e.g., `SELECT * FROM full_hist WHERE id = 'XXX'`)
2. Only the latest version of an event (e.g., `SELECT * FROM latest_slice WHERE id = 'XXX'`)
The latest version is determined by the maximum value in a `version` field for each `id`.
**Questions**
1. Is it better to maintain:
- A single table with full history and periodically deduplicate it into a separate `latest_slice` table?
- Or a single full history table with a view that computes the latest versions dynamically?
2. If the latter, does applying optimization techniques like partitioning, sorting, and ordering on the full history table significantly improve performance for querying the latest versions?
3. Given the preference to store only one copy of the data, what is the most performant and practical solution for this scenario?
Thank you for your guidance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Best Practices for Storing and Querying Full History and Latest Versions [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #11618:
URL: https://github.com/apache/iceberg/issues/11618#issuecomment-2896113671
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Best Practices for Storing and Querying Full History and Latest Versions [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #11618: Best Practices for Storing and Querying Full History and Latest Versions
URL: https://github.com/apache/iceberg/issues/11618
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Best Practices for Storing and Querying Full History and Latest Versions [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #11618:
URL: https://github.com/apache/iceberg/issues/11618#issuecomment-2937825334
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org