You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Lin Liu (Jira)" <ji...@apache.org> on 2024/10/09 22:45:00 UTC
[jira] [Commented] (HUDI-8178) Enable partition stats by default
[ https://issues.apache.org/jira/browse/HUDI-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17888081#comment-17888081 ]
Lin Liu commented on HUDI-8178:
-------------------------------
Enabled partition stats for `testReadPathsForOnlyLogFiles`, and the test passed.
Now trying to enable the partition stats by default, and trigger all the tests.
> Enable partition stats by default
> ---------------------------------
>
> Key: HUDI-8178
> URL: https://issues.apache.org/jira/browse/HUDI-8178
> Project: Apache Hudi
> Issue Type: Task
> Reporter: Sagar Sumit
> Assignee: Lin Liu
> Priority: Blocker
> Fix For: 1.0.0
>
>
> There are two issues that need to be fixed before we can enable partition stats by default:
> 1. There is a [class cast exception|https://dev.azure.com/apachehudi/hudi-oss-ci/_build/results?buildId=151&view=logs&j=b1544eb9-7ff1-5db9-0187-3e05abf459bc&t=e0ae894b-41c9-5f4b-7ed2-bdf5243b02e7] under certain curcumstance while collecting stats for log files. This happens even for colstats (run `[testReadPathsForOnlyLogFiles|https://github.com/apache/hudi/blob/33a02987b7fc385253dc0e0efbf112066b8cf190/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestMORDataSource.scala#L885]` test with column stats enabled).
>
> {code:java}
> Caused by: java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.UnsafeRow cannot be cast to org.apache.avro.generic.IndexedRecord
> at org.apache.avro.generic.GenericData.getField(GenericData.java:858)
> at org.apache.avro.generic.GenericData.compare(GenericData.java:1229)
> at org.apache.avro.generic.GenericData.compare(GenericData.java:1148)
> at org.apache.hudi.metadata.HoodieTableMetadataUtil.lambda$null$1(HoodieTableMetadataUtil.java:239)
> at java.util.ArrayList.forEach(ArrayList.java:1259)
> at org.apache.hudi.metadata.HoodieTableMetadataUtil.lambda$collectColumnRangeMetadata$2(HoodieTableMetadataUtil.java:222)
> at java.util.ArrayList.forEach(ArrayList.java:1259)
> at org.apache.hudi.metadata.HoodieTableMetadataUtil.collectColumnRangeMetadata(HoodieTableMetadataUtil.java:219)
> at org.apache.hudi.io.HoodieAppendHandle.processAppendResult(HoodieAppendHandle.java:424)
> at org.apache.hudi.io.HoodieAppendHandle.appendDataAndDeleteBlocks(HoodieAppendHandle.java:475) {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)