You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "cccs-jc (via GitHub)" <gi...@apache.org> on 2024/03/21 18:48:03 UTC

[I] zorder does not work with sub fields [iceberg]

cccs-jc opened a new issue, #10017:
URL: https://github.com/apache/iceberg/issues/10017

   ### Apache Iceberg version
   
   1.5.0 (latest release)
   
   ### Query engine
   
   Spark
   
   ### Please describe the bug 🐞
   
   The `rewrite_data_files` with zorder does not work on sub-fields.
   
   ```sql
   CALL users.system.rewrite_data_files(
           table => 'users.jcc.flow',
           options => map('max-concurrent-file-group-rewrites', '20',
                          'partial-progress.enabled', 'true',
                          'rewrite-all', 'true'),
           strategy => 'sort',
           sort_order => 'zorder(SRC_IP.v4, DST_IP.v4)',
           where => "END_TIME >= TIMESTAMP '2024-02-12'
               AND END_TIME < TIMESTAMP '2024-02-12' + INTERVAL 1 DAY"
           )
   ```
   I get the error `java.lang.IllegalArgumentException: SRC_IP.v4 does not exist`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] zorder does not work with sub fields [iceberg]

Posted by "cccs-jc (via GitHub)" <gi...@apache.org>.
cccs-jc commented on issue #10017:
URL: https://github.com/apache/iceberg/issues/10017#issuecomment-2040368228

   Seems like it would. I'm not a reviewer but I do want to the fix :-)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] zorder does not work with sub fields [iceberg]

Posted by "cccs-jc (via GitHub)" <gi...@apache.org>.
cccs-jc commented on issue #10017:
URL: https://github.com/apache/iceberg/issues/10017#issuecomment-2186075997

   The issue with nested fields for zorder still exists. Any chance you have time to complete the PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] zorder does not work with sub fields [iceberg]

Posted by "David-N-Perkins (via GitHub)" <gi...@apache.org>.
David-N-Perkins commented on issue #10017:
URL: https://github.com/apache/iceberg/issues/10017#issuecomment-2644062754

   I also need this. Tried to work around this by setting the default sort order of the table, but the table spec doesn't support zordering.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] zorder does not work with sub fields [iceberg]

Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on issue #10017:
URL: https://github.com/apache/iceberg/issues/10017#issuecomment-2040038177

   can this fit for your use case : https://github.com/apache/iceberg/pull/9818/files


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] zorder does not work with sub fields [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10017:
URL: https://github.com/apache/iceberg/issues/10017#issuecomment-3399482543

   This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] zorder does not work with sub fields [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10017:
URL: https://github.com/apache/iceberg/issues/10017#issuecomment-3453899978

   This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] zorder does not work with sub fields [iceberg]

Posted by "cccs-jc (via GitHub)" <gi...@apache.org>.
cccs-jc commented on issue #10017:
URL: https://github.com/apache/iceberg/issues/10017#issuecomment-2231486436

   @singhpk234 just following up on the lack of support for nested fields when applying zordering


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] zorder does not work with sub fields [iceberg]

Posted by "cccs-jc (via GitHub)" <gi...@apache.org>.
cccs-jc commented on issue #10017:
URL: https://github.com/apache/iceberg/issues/10017#issuecomment-2809675040

   @RussellSpitzer  We have created a fork of Iceberg with the PR from @singhpk234 and it works great.
   
   For over a year now, we have been working around the issue by copying our nested columns to the root of the table. We would love for this PR to make it into the code base so we can stop doing that and also not have a custom fork.
   
   Any chance this PR gets looked at and approved.
   Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] zorder does not work with sub fields [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #10017: zorder does not work with sub fields
URL: https://github.com/apache/iceberg/issues/10017


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] zorder does not work with sub fields [iceberg]

Posted by "cccs-jc (via GitHub)" <gi...@apache.org>.
cccs-jc commented on issue #10017:
URL: https://github.com/apache/iceberg/issues/10017#issuecomment-2794311445

   This is still an issue for us as well. The rewrite_data_files procedures says it cannot find the column we specify in our zorder. It only accepts root columns.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] zorder does not work with sub fields [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #10017:
URL: https://github.com/apache/iceberg/issues/10017#issuecomment-2795155788

   We should re-open that PR. I think no one got around to reviewing it before and it just got auto-closed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] zorder does not work with sub fields [iceberg]

Posted by "cccs-jc (via GitHub)" <gi...@apache.org>.
cccs-jc commented on issue #10017:
URL: https://github.com/apache/iceberg/issues/10017#issuecomment-2306898105

   @RussellSpitzer  do you think the PR mentioned by @singhpk234 https://github.com/apache/iceberg/pull/9818/files could get merged ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org