You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/09/07 12:13:53 UTC
[GitHub] [hudi] novakov-alexey opened a new issue #3617: [SUPPORT] Hive Sync to Glue throws Failed to read data schema
novakov-alexey opened a new issue #3617:
URL: https://github.com/apache/hudi/issues/3617
**To Reproduce**
I am getting an exception on Hive Sync to Glue Catalog, when writing empty dataframe with schema to S3.
Steps to reproduce the behavior:
1. Create empty dataframe with schema, for example:
```scala
val schema: StructType = ???
val df = session.createDataFrame(session.sparkContext.emptyRDD[Row], schema)
```
2. Write this dataframe to S3 with Hive sync enabled. For example:
```scala
df.write.format("hudi")
.options(options)
.mode(SaveMode.Overwrite)
.save("s3://my-datalake/my-table...")
```
My Hudi writer options:
```scala
val initLoadConfig = Map(
BULKINSERT_PARALLELISM -> "4",
INSERT_PARALLELISM -> "4",
UPSERT_PARALLELISM -> "4",
DELETE_PARALLELISM -> "4"
)
val unpartitionDataConfig = Map(
HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY -> "org.apache.hudi.hive.NonPartitionedExtractor",
KEYGENERATOR_CLASS_PROP -> "org.apache.hudi.keygen.NonpartitionedKeyGenerator"
)
def writerOptions(
table: String,
primaryKey: String,
database: String,
) = {
Map(
OPERATION_OPT_KEY -> BULK_INSERT_OPERATION_OPT_VAL,
PRECOMBINE_FIELD_PROP -> "some field here",
RECORDKEY_FIELD_OPT_KEY -> primaryKey,
TABLE_NAME -> table,
"hoodie.consistency.check.enabled" -> "true",
ENABLE_ROW_WRITER_OPT_KEY -> "true",
HIVE_USE_JDBC_OPT_KEY -> "true",
HIVE_SYNC_ENABLED_OPT_KEY -> "true",
HIVE_SUPPORT_TIMESTAMP -> "true",
HIVE_DATABASE_OPT_KEY -> database,
HIVE_TABLE_OPT_KEY -> table
) ++ initLoadConfig ++ unpartitionDataConfig
}
```
**Expected behavior**
Hudi table is registered in AWS Glue Catalog as external table.
**Environment Description**
* Hudi version : 0.7
* Spark version : 3.1.1
* Hive version : AWS Glue Catalog
* Hadoop version : EMR 6.3.0
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
**Stacktrace**
```bash
21/09/07 09:48:47 WARN HiveSyncTool: Set partitionFields to empty, since the NonPartitionedExtractor is used
21/09/07 09:48:47 ERROR HiveSyncTool: Got runtime exception when hive syncing
org.apache.hudi.sync.common.HoodieSyncException: Failed to read data schema
at org.apache.hudi.sync.common.AbstractSyncHoodieClient.getDataSchema(AbstractSyncHoodieClient.java:121)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:134)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:94)
at org.apache.hudi.HoodieSparkSqlWriter$.syncHive(HoodieSparkSqlWriter.scala:355)
at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$4(HoodieSparkSqlWriter.scala:403)
at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$4$adapted(HoodieSparkSqlWriter.scala:399)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
at org.apache.hudi.HoodieSparkSqlWriter$.metaSync(HoodieSparkSqlWriter.scala:399)
at org.apache.hudi.HoodieSparkSqlWriter$.bulkInsertAsRow(HoodieSparkSqlWriter.scala:311)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:127)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:134)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:90)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:185)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:223)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:220)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:181)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:134)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:133)
at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:989)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)
at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:110)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:135)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:135)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:253)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:134)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:989)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:438)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:415)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:293)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] novakov-alexey commented on issue #3617: [SUPPORT] Hive Sync to Glue throws Failed to read data schema
Posted by GitBox <gi...@apache.org>.
novakov-alexey commented on issue #3617:
URL: https://github.com/apache/hudi/issues/3617#issuecomment-914252663
@codope this what we discussed in Slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] novakov-alexey commented on issue #3617: [SUPPORT] Hive Sync to Glue throws Failed to read data schema
Posted by GitBox <gi...@apache.org>.
novakov-alexey commented on issue #3617:
URL: https://github.com/apache/hudi/issues/3617#issuecomment-914315442
@codope I have an empty dataset which is used in Spark SQL as a table for joining with other tables. That means I need to have such Hudi table registered in Glue/Hive catalog in advance. Otherwise I get "table not found error" during my query execution.
When I later run complex business query in my Spark SQL code, I do spark.sql("use mydb") where that empty table must be present already. It is a bit weird, but this is current use-case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #3617: [SUPPORT] Hive Sync to Glue throws Failed to read data schema
Posted by GitBox <gi...@apache.org>.
codope commented on issue #3617:
URL: https://github.com/apache/hudi/issues/3617#issuecomment-914303768
@novakov-alexey Thanks for sharing the steps to reproduce. Is there any specific reason you want to register Hudi table in Glue Catalog in advance? Typically, if you write with the data and hive sync is enabled, then the registration in the catalog happens automatically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan edited a comment on issue #3617: [SUPPORT] Hive Sync to Glue throws Failed to read data schema
Posted by GitBox <gi...@apache.org>.
xushiyan edited a comment on issue #3617:
URL: https://github.com/apache/hudi/issues/3617#issuecomment-927522620
@novakov-alexey I checked the behavior is fixed in 0.9.0. Please give release-0.9.0 a try. You can find some guide here to override EMR hudi jars. https://hudi.apache.org/learn/faq#how-to-override-hudi-jars-in-emr
I used this snippet to reproduce your scenario with 0.9.0. After the bulk insert commit, I can see schema value is populated in the commit file. This should allow you to sync with Glue Catalog.
```scala
val opts = Map(
DataSourceWriteOptions.TABLE_NAME.key() -> "language",
DataSourceWriteOptions.TABLE_TYPE.key() -> DataSourceWriteOptions.COW_TABLE_TYPE_OPT_VAL,
DataSourceWriteOptions.OPERATION.key() -> DataSourceWriteOptions.BULK_INSERT_OPERATION_OPT_VAL,
DataSourceWriteOptions.RECORDKEY_FIELD.key() -> "lang",
DataSourceWriteOptions.PRECOMBINE_FIELD.key() -> "score",
DataSourceWriteOptions.KEYGENERATOR_CLASS_NAME.key() -> classOf[NonpartitionedKeyGenerator].getCanonicalName,
DataSourceWriteOptions.HIVE_PARTITION_EXTRACTOR_CLASS.key() -> classOf[NonPartitionedExtractor].getCanonicalName,
DataSourceWriteOptions.HIVE_STYLE_PARTITIONING.key() -> "true",
HoodieWriteConfig.TBL_NAME.key() -> "language",
HoodieWriteConfig.INSERT_PARALLELISM_VALUE.key() -> "1",
HoodieWriteConfig.UPSERT_PARALLELISM_VALUE.key() -> "1",
HoodieWriteConfig.BULKINSERT_PARALLELISM_VALUE.key() -> "1",
HoodieWriteConfig.FINALIZE_WRITE_PARALLELISM_VALUE.key() -> "1",
"spark.default.parallelism" -> "1",
"spark.sql.shuffle.partitions" -> "1"
)
val simpleSchema = StructType(Array(
StructField("lang", StringType, nullable = false),
StructField("score", IntegerType, nullable = false)
))
val emptyDF = spark.createDataFrame(spark.sparkContext.emptyRDD[Row], simpleSchema)
emptyDF.write.format("hudi").options(opts).mode(Overwrite).save(basePath)
val data = spark.read.format("hudi").load(basePath)
data.show()
```
```json
➜ cat .hoodie/20210926212533.commit
{
"partitionToWriteStats" : { },
"compacted" : false,
"extraMetadata" : {
"schema" : "{\"type\":\"record\",\"name\":\"language_record\",\"namespace\":\"hoodie.language\",\"fields\":[{\"name\":\"lang\",\"type\":\"string\"},{\"name\":\"score\",\"type\":\"int\"}]}"
},
"operationType" : "BULK_INSERT",
"fileIdAndRelativePaths" : { },
"totalRecordsDeleted" : 0,
"totalLogRecordsCompacted" : 0,
"totalLogFilesCompacted" : 0,
"totalCompactedRecordsUpdated" : 0,
"totalLogFilesSize" : 0,
"totalScanTime" : 0,
"totalCreateTime" : 0,
"totalUpsertTime" : 0,
"minAndMaxEventTime" : {
"Optional.empty" : {
"val" : null,
"present" : false
}
},
"writePartitionPaths" : [ ]
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan closed issue #3617: [SUPPORT] Hive Sync to Glue throws Failed to read data schema
Posted by GitBox <gi...@apache.org>.
xushiyan closed issue #3617:
URL: https://github.com/apache/hudi/issues/3617
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #3617: [SUPPORT] Hive Sync to Glue throws Failed to read data schema
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #3617:
URL: https://github.com/apache/hudi/issues/3617#issuecomment-927345014
@novakov-alexey what is the schema for this dataset? the snippet in the previous command is the example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #3617: [SUPPORT] Hive Sync to Glue throws Failed to read data schema
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #3617:
URL: https://github.com/apache/hudi/issues/3617#issuecomment-927522620
@novakov-alexey I checked the behavior is fixed in 0.9.0. Please give release-0.9.0 a try. You can find some guide here to override EMR hudi jars. https://hudi.apache.org/learn/faq#how-to-override-hudi-jars-in-emr
I used this snippet to reproduce your scenario with 0.9.0. After the bulk insert commit, I can see schema value is populated in the commit file. This should allow you to sync with Glue Catalog.
```scala
val opts = Map(
DataSourceWriteOptions.TABLE_NAME.key() -> "language",
DataSourceWriteOptions.TABLE_TYPE.key() -> DataSourceWriteOptions.COW_TABLE_TYPE_OPT_VAL,
DataSourceWriteOptions.OPERATION.key() -> DataSourceWriteOptions.BULK_INSERT_OPERATION_OPT_VAL,
DataSourceWriteOptions.RECORDKEY_FIELD.key() -> "lang",
DataSourceWriteOptions.PRECOMBINE_FIELD.key() -> "score",
DataSourceWriteOptions.KEYGENERATOR_CLASS_NAME.key() -> classOf[NonpartitionedKeyGenerator].getCanonicalName,
DataSourceWriteOptions.HIVE_PARTITION_EXTRACTOR_CLASS.key() -> classOf[NonPartitionedExtractor].getCanonicalName,
DataSourceWriteOptions.HIVE_STYLE_PARTITIONING.key() -> "true",
HoodieWriteConfig.TBL_NAME.key() -> "language",
HoodieWriteConfig.INSERT_PARALLELISM_VALUE.key() -> "1",
HoodieWriteConfig.UPSERT_PARALLELISM_VALUE.key() -> "1",
HoodieWriteConfig.BULKINSERT_PARALLELISM_VALUE.key() -> "1",
HoodieWriteConfig.FINALIZE_WRITE_PARALLELISM_VALUE.key() -> "1",
"spark.default.parallelism" -> "1",
"spark.sql.shuffle.partitions" -> "1"
)
val simpleSchema = StructType(Array(
StructField("lang", StringType, nullable = false),
StructField("score", IntegerType, nullable = false)
))
val emptyDF = spark.createDataFrame(spark.sparkContext.emptyRDD[Row], simpleSchema)
emptyDF.write.format("hudi").options(opts).mode(Overwrite).save(basePath)
val data = spark.read.format("hudi").load(basePath)
data.show()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] novakov-alexey commented on issue #3617: [SUPPORT] Hive Sync to Glue throws Failed to read data schema
Posted by GitBox <gi...@apache.org>.
novakov-alexey commented on issue #3617:
URL: https://github.com/apache/hudi/issues/3617#issuecomment-927644242
@xushiyan thank you, it is indeed fixed in Hudi 0.9.
If someone needs an example of building EMR job with own Hudi in SBT, there is one:
https://github.com/novakov-alexey/spark-elt-jobs/blob/main/build.sbt#L207
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] novakov-alexey commented on issue #3617: [SUPPORT] Hive Sync to Glue throws Failed to read data schema
Posted by GitBox <gi...@apache.org>.
novakov-alexey commented on issue #3617:
URL: https://github.com/apache/hudi/issues/3617#issuecomment-927317516
@xushiyan can you advice how to set a json schema string to the commit file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #3617: [SUPPORT] Hive Sync to Glue throws Failed to read data schema
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #3617:
URL: https://github.com/apache/hudi/issues/3617#issuecomment-927234056
@novakov-alexey the problem is caused by 20210907094837.commit having `"schema" : null`. Schema reader is reading from commit file. when it's null it's gonna look for basefile (parquet) to get the schema. But in this case your data is empty.
I think this is current behavior when data is empty, commit metadata are all null or empty. You could try manually set a json serialized schema string to the commit file. Something like this
```json
"extraMetadata" : {
"schema" : "{\"type\":\"record\",\"name\":\"FullName\",\"fields\": []}"
},
```
But I do think it makes sense to make the schema available in commit file even when it's empty data. We may make a fix for this. Meanwhile, let us know if manually setting the schema string works for you.
BTW you don't need to set "hoodie.consistency.check.enabled" -> "true", as S3 has become strongly consistent some time back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan edited a comment on issue #3617: [SUPPORT] Hive Sync to Glue throws Failed to read data schema
Posted by GitBox <gi...@apache.org>.
xushiyan edited a comment on issue #3617:
URL: https://github.com/apache/hudi/issues/3617#issuecomment-927345014
@novakov-alexey what is the schema for this dataset? the snippet in the previous comment is the example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org