You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "kazuyukitanimura (via GitHub)" <gi...@apache.org> on 2024/12/10 08:52:22 UTC
[PR] fix: Spark 4.0-preview1 SPARK-47120 [datafusion-comet]
kazuyukitanimura opened a new pull request, #1156:
URL: https://github.com/apache/datafusion-comet/pull/1156
## Which issue does this PR close?
Part of https://github.com/apache/datafusion-comet/issues/372 and https://github.com/apache/datafusion-comet/issues/551
## Rationale for this change
To be ready for Spark 4.0
## What changes are included in this PR?
This PR fixes the new test SPARK-47120 added in Spark 4.0
## How are these changes tested?
tests enabled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Spark 4.0-preview1 SPARK-47120 [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #1156:
URL: https://github.com/apache/datafusion-comet/pull/1156#discussion_r1878336544
##########
spark/src/main/scala/org/apache/comet/parquet/ParquetFilters.scala:
##########
@@ -723,20 +723,22 @@ class ParquetFilters(
.lift(nameToParquetField(name).fieldType)
.map(_(nameToParquetField(name).fieldNames, value))
- case sources.LessThan(name, value) if canMakeFilterOn(name, value) =>
+ case sources.LessThan(name, value) if (value != null) && canMakeFilterOn(name, value) =>
Review Comment:
why do we need to change Spark's test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Spark 4.0-preview1 SPARK-47120 [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on code in PR #1156:
URL: https://github.com/apache/datafusion-comet/pull/1156#discussion_r1878839024
##########
spark/src/main/scala/org/apache/comet/parquet/ParquetFilters.scala:
##########
@@ -723,20 +723,22 @@ class ParquetFilters(
.lift(nameToParquetField(name).fieldType)
.map(_(nameToParquetField(name).fieldNames, value))
- case sources.LessThan(name, value) if canMakeFilterOn(name, value) =>
+ case sources.LessThan(name, value) if (value != null) && canMakeFilterOn(name, value) =>
Review Comment:
This is not a test code
This file was copied from Spark for modification. Then there is a logic update for SPARK-47120
https://github.com/apache/spark/pull/45202/files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Spark 4.0-preview1 SPARK-47120 [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura merged PR #1156:
URL: https://github.com/apache/datafusion-comet/pull/1156
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Spark 4.0-preview1 SPARK-47120 [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on PR #1156:
URL: https://github.com/apache/datafusion-comet/pull/1156#issuecomment-2536684461
Merged, thanks @andygrove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Spark 4.0-preview1 SPARK-47120 [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #1156:
URL: https://github.com/apache/datafusion-comet/pull/1156#discussion_r1880144249
##########
spark/src/main/scala/org/apache/comet/parquet/ParquetFilters.scala:
##########
@@ -723,20 +723,22 @@ class ParquetFilters(
.lift(nameToParquetField(name).fieldType)
.map(_(nameToParquetField(name).fieldNames, value))
- case sources.LessThan(name, value) if canMakeFilterOn(name, value) =>
+ case sources.LessThan(name, value) if (value != null) && canMakeFilterOn(name, value) =>
Review Comment:
Ah, sorry, I missed that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org