You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/01/16 20:22:08 UTC
[GitHub] [incubator-iceberg] rdblue commented on issue #684: Support Decimal
in Parquet predicates
rdblue commented on issue #684: Support Decimal in Parquet predicates
URL: https://github.com/apache/incubator-iceberg/issues/684#issuecomment-575329875
@chenjunjiedada, I think I was wrong about this issue. I was referring to the note here, where we are converting Iceberg filters to Parquet filters: https://github.com/apache/incubator-iceberg/blob/master/parquet/src/main/java/org/apache/iceberg/parquet/ParquetFilters.java#L207
But that code path is only used when reading Parquet files with a Parquet `ReadSupport`. We don't do that for Spark, we use [the Iceberg reader](https://github.com/apache/incubator-iceberg/blob/master/parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java#L231-L233) now to avoid the overhead of `ReadSupport` and to implement our own row group filters (that support, for example, Decimal and IN).
The new read path already has support for decimal because it [converts Parquet min/max stats](https://github.com/apache/incubator-iceberg/blob/master/parquet/src/main/java/org/apache/iceberg/parquet/ParquetMetricsRowGroupFilter.java#L104) to the objects that Iceberg uses. That [conversion supports decimal](https://github.com/apache/incubator-iceberg/blob/master/parquet/src/main/java/org/apache/iceberg/parquet/ParquetConversions.java#L71).
So the new read path that we actually use already supports decimal predicates. We could go back and fix the old read path, but I don't think that we want to spend the effort since the problem is knowing what physical type to pass to Parquet (int, long, or binary).
I think we should mark this issue completed. What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org