You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ravindrabajpai (via GitHub)" <gi...@apache.org> on 2025/03/01 21:07:50 UTC
[I] Varying Partitioning [iceberg]
ravindrabajpai opened a new issue, #12431:
URL: https://github.com/apache/iceberg/issues/12431
### Proposed Change
I have recently started exploring iceberg table format, and I feel hidden partitioning is a very powerful feature. I would like to explore it a bit further to see if following case can be possible. Please ignore this proposal and point me to an alternate solution if there is any.
I would like to explore and propose the following type of partitioning mechanism explained with example -
For example a table with a specific partitioning structure that combines a "status" partition (active/inactive) with a date partition (business_date) for inactive records only.
It solves my small file issue for status=active partition, where if I don't have business date as sub partition, then I can have a few big files right inside the status=active foloder.
1. Proposed/Expected Table Schema and Partitioning Strategy
Let's assume we have a table with columns like:
id (INT): Unique identifier
status (STRING): "active" or "inactive"
business_date (DATE): Date of the business event
data (STRING): Some data
Partitioning strategy will be:
status: Top-level partition (active/inactive)
business_date: Sub-partition only for status = 'inactive'.
Creating the Iceberg Table -
CREATE TABLE my_iceberg_table (
id INT,
status STRING,
business_date DATE,
data STRING
)
USING iceberg
PARTITIONED BY (status, CASE WHEN status = 'inactive' THEN business_date ELSE null END);
Here -
PARTITIONED BY (status, CASE WHEN status = 'inactive' THEN business_date ELSE null END): This is the key part.
It partitions first by status.
The CASE statement creates the sub-partition:
If status is 'inactive', it partitions by business_date.
If status is 'active', it partitions by NULL.
3. Ingesting Records
INSERT INTO my_iceberg_table VALUES
(1, 'active', '2023-10-26', 'Active data 1'),
(2, 'active', '2023-10-27', 'Active data 2'),
(3, 'inactive', '2023-10-26', 'Inactive data 1'),
(4, 'inactive', '2023-10-27', 'Inactive data 2'),
(5, 'inactive', '2023-10-27', 'Inactive data 3');
SELECT * FROM my_iceberg_table WHERE status = 'active';
SELECT * FROM my_iceberg_table WHERE status = 'inactive' AND business_date = '2023-10-27';
### Proposal document
_No response_
### Specifications
- [x] Table
- [ ] View
- [ ] REST
- [ ] Puffin
- [ ] Encryption
- [ ] Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Varying Partitioning [iceberg]
Posted by "ravindrabajpai (via GitHub)" <gi...@apache.org>.
ravindrabajpai commented on issue #12431:
URL: https://github.com/apache/iceberg/issues/12431#issuecomment-2741900475
Hi @RussellSpitzer , Thanks for your reply. "**allowing a limited set of expressions could be possible**" - Even this would address a large number of uses acrosss the industry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Varying Partitioning [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12431:
URL: https://github.com/apache/iceberg/issues/12431#issuecomment-3712558077
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Varying Partitioning [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #12431: Varying Partitioning
URL: https://github.com/apache/iceberg/issues/12431
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Varying Partitioning [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12431:
URL: https://github.com/apache/iceberg/issues/12431#issuecomment-3684619946
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Varying Partitioning [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #12431:
URL: https://github.com/apache/iceberg/issues/12431#issuecomment-2722207110
We've had some discussions about this before and the biggest problem is the same one we are having with Views, Constraints, etc... Any functions we allow need to be universally understood by all engines working with the Iceberg table. This means we need to define a lot more operators or use definitions from some other project. Because of this I think really generic functions are probably always going to be a bit of a non-starter but maybe allowing a limited set of expressions could be possible. We definitely would want to make sure we have specific uses cases we are actually trying to solve though with this that aren't solved better by the engine / config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org