You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Raymond Xu (Jira)" <ji...@apache.org> on 2022/01/07 17:42:00 UTC
[jira] [Comment Edited] (HUDI-1850) Read on table fails if the first write to table failed
[ https://issues.apache.org/jira/browse/HUDI-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17470769#comment-17470769 ]
Raymond Xu edited comment on HUDI-1850 at 1/7/22, 5:41 PM:
-----------------------------------------------------------
This is to improve the UX when running Spark SQL read from empty or corrupted table. Empty DF should be returned with proper warn logs.
[~shivnarayan] As discussed, I removed this from 0.10.1
was (Author: xushiyan):
This is to improve the UX when running Spark SQL read from empty or corrupted table. Empty DF should be returned with proper warn logs.
> Read on table fails if the first write to table failed
> ------------------------------------------------------
>
> Key: HUDI-1850
> URL: https://issues.apache.org/jira/browse/HUDI-1850
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Spark Integration
> Affects Versions: 0.8.0
> Reporter: Vaibhav Sinha
> Priority: Major
> Labels: core-flow-ds, pull-request-available, release-blocker, sev:high, spark
> Fix For: 0.11.0
>
> Attachments: Screenshot 2021-04-24 at 7.53.22 PM.png
>
>
> {code:java}
> ava.util.NoSuchElementException: No value present in Option
> at org.apache.hudi.common.util.Option.get(Option.java:88) ~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
> at org.apache.hudi.common.table.TableSchemaResolver.getTableSchemaFromCommitMetadata(TableSchemaResolver.java:215) ~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
> at org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchema(TableSchemaResolver.java:166) ~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
> at org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchema(TableSchemaResolver.java:155) ~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
> at org.apache.hudi.MergeOnReadSnapshotRelation.<init>(MergeOnReadSnapshotRelation.scala:65) ~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
> at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:99) ~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
> at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:63) ~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
> at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:354) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
> at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:326) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
> at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:308) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
> at scala.Option.getOrElse(Option.scala:189) ~[scala-library-2.12.10.jar:?]
> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:308) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:240) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
> {code}
> The screenshot shows the files that got created before the write had failed.
>
> !Screenshot 2021-04-24 at 7.53.22 PM.png!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)