You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "sunchao (via GitHub)" <gi...@apache.org> on 2024/02/16 00:03:32 UTC
[I] Consider removing `ParquetFilters` [arrow-datafusion-comet]
sunchao opened a new issue, #36:
URL: https://github.com/apache/arrow-datafusion-comet/issues/36
### What is the problem the feature request solves?
The `ParquetFilters` were added originally so we could shade Parquet in Comet. However the shading was removed later as it caused a lot of trouble on the Iceberg side. In addition, we ported several Parquet classes from `parquet-mr` into Comet, so now the boundary between Comet and `parquet-mr` is fairly thin. Therefore, we could consider removing `ParquetFilters` and directly use the one from Spark.
The advantage of this is we are able to absorb changes & improvements in the newer version of Spark. For instance, https://github.com/apache/spark/pull/36696 added Parquet `In`/`NotIn` pushdown from Spark side, which is only available since Spark 3.4. At the moment, as Comet keeps a copy of Spark's `ParquetFilters`, the feature is not added in order to be backward compatible with Spark 3.2 and 3.3.
### Describe the potential solution
Evaluate whether we can remove `ParquetFilters` from Comet.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Consider removing `ParquetFilters` [arrow-datafusion-comet]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on issue #36:
URL: https://github.com/apache/arrow-datafusion-comet/issues/36#issuecomment-2026725164
I will work on this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Consider removing `ParquetFilters` [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #36:
URL: https://github.com/apache/datafusion-comet/issues/36#issuecomment-4490727974
Closed by #4363.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Consider removing `ParquetFilters` [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich closed issue #36: Consider removing `ParquetFilters`
URL: https://github.com/apache/datafusion-comet/issues/36
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Consider removing `ParquetFilters` [arrow-datafusion-comet]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on issue #36:
URL: https://github.com/apache/arrow-datafusion-comet/issues/36#issuecomment-2030128187
I have tried this out. Everything works fine except this fix https://issues.apache.org/jira/browse/SPARK-46092. The fix is in Spark 3.4.3. I will wait for Spark 3.4.3 is released, and then remove the `ParquetFilters`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org