You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/04/07 12:40:59 UTC
[GitHub] [incubator-iceberg] chenjunjiedada commented on issue #830: Support
name mapping resolution for parquet
chenjunjiedada commented on issue #830: Support name mapping resolution for parquet
URL: https://github.com/apache/incubator-iceberg/pull/830#issuecomment-610363057
@rdblue, I addressed some of the comments. The remaining needs a double check.
The ID fallback way cannot work when the iceberg table schema is different from the parquet file schema as the original issue mentioned. Also, the use case in the unit test (import partial data of existing parquet files as new iceberg table) would lead to the following exception which is related to metric row group filters. Therefore I removed the ID fallback way. I 'm not very sure that is the common use case, but if the name mapping way could handle more cases correctly, then it would be better to keep that way unless there is any case that it cannot handle. Does that make sense to you?
```
java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.CharSequence
at org.apache.iceberg.types.Comparators$CharSeqComparator.compare(Comparators.java:180)
at org.apache.iceberg.types.Comparators$NullSafeChainedComparator.compare(Comparators.java:146)
at org.apache.iceberg.parquet.ParquetMetricsRowGroupFilter$MetricsEvalVisitor.lt(ParquetMetricsRowGroupFilter.java:202)
at org.apache.iceberg.parquet.ParquetMetricsRowGroupFilter$MetricsEvalVisitor.lt(ParquetMetricsRowGroupFilter.java:85)
at org.apache.iceberg.expressions.ExpressionVisitors$BoundExpressionVisitor.predicate(ExpressionVisitors.java:122)
at org.apache.iceberg.expressions.ExpressionVisitors.visitEvaluator(ExpressionVisitors.java:321)
at org.apache.iceberg.expressions.ExpressionVisitors.visitEvaluator(ExpressionVisitors.java:336)
at org.apache.iceberg.parquet.ParquetMetricsRowGroupFilter$MetricsEvalVisitor.eval(ParquetMetricsRowGroupFilter.java:108)
at org.apache.iceberg.parquet.ParquetMetricsRowGroupFilter$MetricsEvalVisitor.access$000(ParquetMetricsRowGroupFilter.java:85)
at org.apache.iceberg.parquet.ParquetMetricsRowGroupFilter.shouldRead(ParquetMetricsRowGroupFilter.java:79)
at org.apache.iceberg.parquet.ReadConf.<init>(ReadConf.java:96)
at org.apache.iceberg.parquet.ParquetReader.init(ParquetReader.java:64)
at org.apache.iceberg.parquet.ParquetReader.iterator(ParquetReader.java:74)
at org.apache.iceberg.spark.source.RowDataReader.open(RowDataReader.java:155)
at org.apache.iceberg.spark.source.RowDataReader.open(RowDataReader.java:117)
at org.apache.iceberg.spark.source.BaseDataReader.next(BaseDataReader.java:79)
at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.hasNext(DataSourceRDD.scala:49)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.sort_addToSorter_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:255)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:247)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:123)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org