You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 06:25:50 UTC
[I] schema field of EmptyRelation subtype of BaseRelation should not be null [hudi]
hudi-bot opened a new issue, #15750:
URL: https://github.com/apache/hudi/issues/15750
If there are no completed instants in the table, and there is no user defined schema for it as well (as represented by the userSpecifiedSchema field in DataSource.scala), then the EmptyRelation returned by DefaultSource.createRelation sets schema of the EmptyRelation to null. This breaks the contract of Spark's BaseRelation, where the schema is a StructType but is not expected to be null.
Module versions: current apache-hudi master (commit hash abe26d4169c04da05b99941161621876e3569e96) built with spark3.2 and scala-2.12.
Following Hudi session reproduces the above issue:
spark.read.format("hudi")
.option("hoodie.datasource.query.type", "incremental") .load("SOME_HUDI_TABLE_WITH_NO_COMPLETED_INSTANTS_OR_USER_SPECIFIED_SCHEMA")
java.lang.NullPointerException
at org.apache.spark.sql.catalyst.util.CharVarcharUtils$.replaceCharVarcharWithStringInSchema(CharVarcharUtils.scala:41)
at org.apache.spark.sql.execution.datasources.LogicalRelation$.apply(LogicalRelation.scala:76)
at org.apache.spark.sql.SparkSession.baseRelationToDataFrame(SparkSession.scala:440)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:274)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:245)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:245)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:188)
... 50 elided
Find attached a few screenshots which show the code flow and the buggy state of the variables. Also find attached a Java file and pom.xml that can be used to reproduce the same (sorry don't have deanonymized table -to share yet).-
The bug seems to have been introduced in this particular PR change: [https://github.com/apache/hudi/pull/6727/files#diff-4cfd87bb9200170194a633746094de138c3a0e3976d351d0d911ee95651256acR220]
Initial work on that file has happened in this particular Jira (https://issues.apache.org/jira/browse/HUDI-4363) and PR (https://github.com/apache/hudi/pull/6046) respectively.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5688
- Type: Bug
- Fix version(s):
- 1.1.0
- Attachment(s):
- 02/Feb/23 12:29;pramodbiligiri;1-userSpecifiedSchema-is-null.png;https://issues.apache.org/jira/secure/attachment/13055052/1-userSpecifiedSchema-is-null.png
- 02/Feb/23 12:29;pramodbiligiri;2-empty-relation.png;https://issues.apache.org/jira/secure/attachment/13055053/2-empty-relation.png
- 02/Feb/23 12:29;pramodbiligiri;3-table-schema-will-not-resolve.png;https://issues.apache.org/jira/secure/attachment/13055054/3-table-schema-will-not-resolve.png
- 02/Feb/23 12:29;pramodbiligiri;4-resolve-schema-returns-null.png;https://issues.apache.org/jira/secure/attachment/13055055/4-resolve-schema-returns-null.png
- 02/Feb/23 12:41;pramodbiligiri;Main.java;https://issues.apache.org/jira/secure/attachment/13055051/Main.java
- 02/Feb/23 12:41;pramodbiligiri;pom.xml;https://issues.apache.org/jira/secure/attachment/13055050/pom.xml
---
## Comments
06/Feb/23 07:48;pramodbiligiri;A small workaround for the null value, that shows that the bug diagnosis is valid: [https://github.com/apache/hudi/pull/7864]
Not sure if the above change can be considered a fix to the issue.;;;
---
14/Feb/23 00:27;shivnarayan;One of hudi engineer will take over. thanks for reporting. looks like its a valid bug.
I could reproduce w/ latest master when the table does not have any completed commits.
{code:java}
scala> val tripsSnapshotDF = spark.
| read.
| format("hudi").
| load(basePath)
23/02/13 16:21:54 WARN DFSPropertiesConfiguration: Cannot find HUDI_CONF_DIR, please set it as the dir of hudi-defaults.conf
23/02/13 16:21:54 WARN DFSPropertiesConfiguration: Properties file file:/etc/hudi/conf/hudi-defaults.conf not found. Ignoring to load props file
23/02/13 16:21:54 INFO DataSourceUtils: Getting table path..
23/02/13 16:21:54 INFO TablePathUtils: Getting table path from path : file:/tmp/hudi_trips_cow
23/02/13 16:21:54 INFO DefaultSource: Obtained hudi table path: file:/tmp/hudi_trips_cow
23/02/13 16:21:54 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/tmp/hudi_trips_cow
23/02/13 16:21:54 INFO HoodieTableConfig: Loading table properties from file:/tmp/hudi_trips_cow/.hoodie/hoodie.properties
23/02/13 16:21:54 INFO HoodieTableMetaClient: Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from file:/tmp/hudi_trips_cow
23/02/13 16:21:54 INFO DefaultSource: Is bootstrapped table => false, tableType is: COPY_ON_WRITE, queryType is: snapshot
23/02/13 16:21:54 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20230213161934447__commit__INFLIGHT]}
java.lang.NullPointerException
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:417)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:242)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:230)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:197)
... 64 elided {code};;;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] schema field of EmptyRelation subtype of BaseRelation should not be null [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #15750: schema field of EmptyRelation subtype of BaseRelation should not be null
URL: https://github.com/apache/hudi/issues/15750
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] schema field of EmptyRelation subtype of BaseRelation should not be null [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #15750:
URL: https://github.com/apache/hudi/issues/15750#issuecomment-3630124176
### Linked PR(s)
- https://github.com/apache/hudi/pull/6046
- https://github.com/apache/hudi/pull/7864
- https://github.com/apache/hudi/pull/8174
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] schema field of EmptyRelation subtype of BaseRelation should not be null [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #15750:
URL: https://github.com/apache/hudi/issues/15750#issuecomment-4795470927
Closing as resolved. Tracked in [HUDI-4363](https://issues.apache.org/jira/browse/HUDI-4363) (fixVersions: 0.12.1). The change landed in PR [#6046](https://github.com/apache/hudi/pull/6046) (commit `5e624698f78f`). Please reopen if you can still reproduce on the latest release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org