You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ConeyLiu (via GitHub)" <gi...@apache.org> on 2023/07/24 11:23:02 UTC
[GitHub] [iceberg] ConeyLiu opened a new pull request, #8142: Spark: Evaluate ExpressionUtil.selectsPartitions for partitioned table only
ConeyLiu opened a new pull request, #8142:
URL: https://github.com/apache/iceberg/pull/8142
The `ExpressionUtil.selectsPartitions` is aimed to determine whether the expression can be fully applied to the whole partitions. And it is not necessary for an unpartitioned table. Especially for metadata tables which are always unpartitioned. For example, we get unexpected warnings for the following scan:
```
scala> spark.sql("select file_path, partition, record_count from iceberg.db.table.files where partition.partition_time = 2023012410").show()
23/07/24 14:20:47 WARN SparkScanBuilder: Failed to check if IsNotNull(partition.partition_time) can be pushed down: Cannot find field 'partition.partition_time' in struct: struct<>
23/07/24 14:20:47 WARN SparkScanBuilder: Failed to check if EqualTo(partition.partition_time,2023012410) can be pushed down: Cannot find field 'partition.partition_time' in struct: struct<>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8142: Spark: Evaluate ExpressionUtil.selectsPartitions for partitioned table only
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8142:
URL: https://github.com/apache/iceberg/pull/8142#discussion_r1272920690
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java:
##########
@@ -185,6 +188,10 @@ public Filter[] pushFilters(Filter[] filters) {
return postScanFilters.toArray(new Filter[0]);
}
+ private boolean unPartitioned() {
Review Comment:
Sure, updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on pull request #8142: Spark: Evaluate ExpressionUtil.selectsPartitions for partitioned table only
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #8142:
URL: https://github.com/apache/iceberg/pull/8142#issuecomment-1650320267
Merged, thanks @ConeyLiu . Do you think we need to do the same for other Spark versions like 3.3?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on a diff in pull request #8142: Spark: Evaluate ExpressionUtil.selectsPartitions for partitioned table only
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #8142:
URL: https://github.com/apache/iceberg/pull/8142#discussion_r1272598918
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java:
##########
@@ -185,6 +188,10 @@ public Filter[] pushFilters(Filter[] filters) {
return postScanFilters.toArray(new Filter[0]);
}
+ private boolean unPartitioned() {
Review Comment:
Sorry just noticed, can we make it say 'unpartitioned'? It matches PartitionSpec.unpartitioned()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on pull request #8142: Spark: Evaluate ExpressionUtil.selectsPartitions for partitioned table only
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #8142:
URL: https://github.com/apache/iceberg/pull/8142#issuecomment-1648372793
Yea I think this fixes: https://github.com/apache/iceberg/issues/7774
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on pull request #8142: Spark: Evaluate ExpressionUtil.selectsPartitions for partitioned table only
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on PR #8142:
URL: https://github.com/apache/iceberg/pull/8142#issuecomment-1650898733
Thanks @szehon-ho. I submitted backport for Spark 3.3. The warning only occurs in Spark 3.3/3.4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho merged pull request #8142: Spark: Evaluate ExpressionUtil.selectsPartitions for partitioned table only
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho merged PR #8142:
URL: https://github.com/apache/iceberg/pull/8142
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org