You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ConeyLiu (via GitHub)" <gi...@apache.org> on 2023/07/04 11:57:13 UTC
[GitHub] [iceberg] ConeyLiu opened a new pull request, #7985: Spark 3.3: Remove unnecessary metadata columns reading when merge using Iceberg table
ConeyLiu opened a new pull request, #7985:
URL: https://github.com/apache/iceberg/pull/7985
There are many unnecessary metadata columns reading when merging using the Iceberg table. The problem should be caused by Spark 3.3 [AddMetadataColumns](https://github.com/apache/spark/blob/e9b525e205402ac458db682802771544ced86758/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#L915) rule which has been fixed in Spark 3.4.
In this PR, we add a rule to remove the unnecessary metadata column reading.
Before this PR:
<img width="969" alt="image" src="https://github.com/apache/iceberg/assets/12733256/63055ba0-7ad0-4aa0-8d3f-b395e611f70b">
After this PR:
<img width="913" alt="image" src="https://github.com/apache/iceberg/assets/12733256/2f03b1ec-4dcb-4e6c-a9d2-6dc2e95ca441">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu closed pull request #7985: Spark 3.3: Remove unnecessary metadata columns reading when merge using Iceberg table
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu closed pull request #7985: Spark 3.3: Remove unnecessary metadata columns reading when merge using Iceberg table
URL: https://github.com/apache/iceberg/pull/7985
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on pull request #7985: Spark 3.3: Remove unnecessary metadata columns reading when merge using Iceberg table
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on PR #7985:
URL: https://github.com/apache/iceberg/pull/7985#issuecomment-1620948180
Close this since Spark 3.3 has been fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on pull request #7985: Spark 3.3: Remove unnecessary metadata columns reading when merge using Iceberg table
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on PR #7985:
URL: https://github.com/apache/iceberg/pull/7985#issuecomment-1620115710
Hi @rdblue @szehon-ho @aokolnychyi @RussellSpitzer @Fokko, could you please help to review this when you are free? Thanks a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on pull request #7985: Spark 3.3: Remove unnecessary metadata columns reading when merge using Iceberg table
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #7985:
URL: https://github.com/apache/iceberg/pull/7985#issuecomment-1620368742
I think the problem has already been fixed in Spark 3.3 by this [PR](https://github.com/apache/spark/pull/40889)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on pull request #7985: Spark 3.3: Remove unnecessary metadata columns reading when merge using Iceberg table
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #7985:
URL: https://github.com/apache/iceberg/pull/7985#issuecomment-1620279744
@huaxingao I believe you did the Spark fix for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on pull request #7985: Spark 3.3: Remove unnecessary metadata columns reading when merge using Iceberg table
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on PR #7985:
URL: https://github.com/apache/iceberg/pull/7985#issuecomment-1620466899
Thanks @RussellSpitzer @huaxingao, I see, that's because the Spark 3.3.3 is not released.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org