You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "SabyasachiDasTR (via GitHub)" <gi...@apache.org> on 2023/04/26 12:58:44 UTC
[GitHub] [hudi] SabyasachiDasTR opened a new issue, #8580: Hudi reads failing on upgrade to Hudi 0.12.2 from Hudi 0.11.1
SabyasachiDasTR opened a new issue, #8580:
URL: https://github.com/apache/hudi/issues/8580
**Describe the problem you faced**
We have a spark streaming job which does only hudi upsert to load data to the partitions. We have 1000’s of collections/partions where data is upserted at high frequncy interval. Lets call it as the main application and it was running on Hudi 0.11.1[emr-6.7.0].
There are nearly 100+ downstream application which reads the data written by this main app based on daily/weekly scheduled run.
These downstream apps uses hudi api to read the source data from main app as well. They range from 0.11 to 0.9 hudi version.
All these downstream apps were able to ready the Hudi 0.11.1 (V4) data.But when we changed our main app version to Hudi 0.12.2[emr-6.10.0] they all started failing with attached stacktrace. Basically an emr job running on older V4 hudi version is not able to read data created by Hudi 0.12.2 (v5) version. It is not backward compatible. This seems to be a basic requirement but is not provided from hudi.
We do want to upgrade to latest hudi 0.12.2 version to be able to fix some of long running issues like duplicate guid issue.
But to be able to do this we cannot ask to change all our downstream apps to upgrade as well.
There are logistics challenges at different levels to do that and doesn’t seem feasible.
**To Reproduce**
Steps to reproduce the behavior:
1. Start emr cluster with 6.10 for hudi .12.2 write to a table using the upsert and config properties attached[Hudiconfigs_and_upsertMethod.txt].Using the hudi.config file and Upsert method (both in the same file hudi.config) for (TNI 6.10 version EMR) and upsert data we can send this information to an S3 bucket. Run this on EMR as a spark job.
2. Spin up a Jupiter notebook emr to replicate Downstream at 6.7 EMR hudi 0.11 which reads the table
3. Then call
val basePath=```s3://<bucket>/<table>/<partition```
val ViewDF = spark.read.format("org.apache.hudi").load(basePath)
ViewDF.createOrReplaceTempView("hudi_doc_table_branch_dest")
4. Reads fail with version 5 failure[ attached stacktrace ]
**Expected behavior**
Downstream job running on hudi 0.11 should be able to read the data generated from main app running on hudo 0.12.2
**Environment Description**
* Hudi version : 0.12.2
* Spark version : 3.3.1
* Hive version : Hive not install on EMR Cluster.
* Hadoop version : 3.3.3
* Storage (HDFS/S3/GCS..) : s3 parquet
* Running on Docker? (yes/no) :no
**Additional context**
Aws ticket on the same : Case ID 12463972981
**Stacktrace**
``` val fs: FileSystem = FileSystem.get(new Configuration())
val maxCommitTime = HoodieDataSourceHelpers.latestCommit(fs, inputPath)
2023-04-07T09:15:37.718+0000 [INFO] [eb423741-4f44-4fea-b76f-912a6b3f0177] [com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem] [S3NativeFileSystem]: Opening 's3://a206760-novusdoc-s3-qa-use1/novusdoc/.hoodie/hoodie.properties' for reading
2023-04-07T09:15:37.896+0000 [ERROR] [eb423741-4f44-4fea-b76f-912a6b3f0177] [org.apache.spark.deploy.yarn.ApplicationMaster] [ApplicationMaster]: User class threw exception: org.apache.hudi.exception.HoodieException: Unknown versionCode:5
org.apache.hudi.exception.HoodieException: Unknown versionCode:5
at org.apache.hudi.common.table.HoodieTableVersion.lambda$versionFromCode$1(HoodieTableVersion.java:58)
at java.util.Optional.orElseThrow(Optional.java:290)
at org.apache.hudi.common.table.HoodieTableVersion.versionFromCode(HoodieTableVersion.java:58)
at org.apache.hudi.common.table.HoodieTableConfig.getTableVersion(HoodieTableConfig.java:472)
at org.apache.hudi.common.table.HoodieTableConfig.fetchConfigs(HoodieTableConfig.java:305)
at org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:244)
at org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:125)
at org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:78)
at org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:668)
at org.apache.hudi.HoodieDataSourceHelpers.allCompletedCommitsCompactions(HoodieDataSourceHelpers.java:77)
at org.apache.hudi.HoodieDataSourceHelpers.latestCommit(HoodieDataSourceHelpers.java:67)
at com.<pkg>.spark.parser.BaseStreamingParser.batchParser(BaseStreamingParser.scala:210)
at com.<pkg>.spark.parser.BaseStreamingParser.batchParser$(BaseStreamingParser.scala:146)
at com.<pkg>.spark.parser.<pkg>$.batchParser(AnzStreamingParser.scala:11)
at com.<pkg>.spark.parser.BaseStreamingParser.main(BaseStreamingParser.scala:129)
at com.<pkg>.spark.parser.BaseStreamingParser.main$(BaseStreamingParser.scala:96)
at com.<pkg>.spark.parser.<pkg>$.main(AnzStreamingParser.scala:11)
at com.<pkg>.spark.parser.<pkg>.main(AnzStreamingParser.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:740)
2023-04-07T09:15:37.899+0000 [INFO] [eb423741-4f44-4fea-b76f-912a6b3f0177] [org.apache.spark.deploy.yarn.ApplicationMaster] [ApplicationMaster]: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.hudi.exception.HoodieException: Unknown versionCode:5
at org.apache.hudi.common.table.HoodieTableVersion.lambda$versionFromCode$1(HoodieTableVersion.java:58)
at java.util.Optional.orElseThrow(Optional.java:290)
at org.apache.hudi.common.table.HoodieTableVersion.versionFromCode(HoodieTableVersion.java:58)
at org.apache.hudi.common.table.HoodieTableConfig.getTableVersion(HoodieTableConfig.java:472)
at org.apache.hudi.common.table.HoodieTableConfig.fetchConfigs(HoodieTableConfig.java:305)
at org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:244)
at org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:125)
at org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:78)
at org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:668)
at org.apache.hudi.HoodieDataSourceHelpers.allCompletedCommitsCompactions(HoodieDataSourceHelpers.java:77)
at org.apache.hudi.HoodieDataSourceHelpers.latestCommit(HoodieDataSourceHelpers.java:67)
at com.<pkg>.spark.parser.BaseStreamingParser.batchParser(BaseStreamingParser.scala:210)
at com.<pkg>.spark.parser.BaseStreamingParser.batchParser$(BaseStreamingParser.scala:146)
at com.<pkg>.spark.parser.<pkg>$.batchParser(AnzStreamingParser.scala:11)
at com.<pkg>.spark.parser.BaseStreamingParser.main(BaseStreamingParser.scala:129)
at com.<pkg>.spark.parser.BaseStreamingParser.main$(BaseStreamingParser.scala:96)
at com.<pkg>.spark.parser.<pkg>$.main(AnzStreamingParser.scala:11)
at com.<pkg>.spark.parser.<pkg>.main(AnzStreamingParser.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:740)
)
2023-04-07T09:15:37.917+0000 [INFO] [eb423741-4f44-4fea-b76f-912a6b3f0177] [org.apache.spark.SparkContext] [SparkContext]: Invoking stop() from shutdown hook
2023-04-07T09:15:37.925+0000 [INFO] [eb423741-4f44-4fea-b76f-912a6b3f0177] [org.sparkproject.jetty.server.AbstractConnector] [AbstractConnector]: Stopped Spark@659dce1b{HTTP/1.1, (http/1.1)}{0.0.0.0:0}
2023-04-07T09:15:37.927+0000 [INFO] [eb423741-4f44-4fea-b76f-912a6b3f0177] [org.apache.spark.ui.SparkUI] [SparkUI]: Stopped Spark web UI at http://ip-100-66-72-129.3175.aws-int.thomsonreuters.com:36067
2023-04-07T09:15:37.973+0000 [INFO] [eb423741-4f44-4fea-b76f-912a6b3f0177] [org.apache.spark.MapOutputTrackerMasterEndpoint] [MapOutputTrackerMasterEndpoint]: MapOutputTrackerMasterEndpoint stopped!
2023-04-07T09:15:38.044+0000 [INFO] [eb423741-4f44-4fea-b76f-912a6b3f0177] [org.apache.spark.SparkContext] [SparkContext]: Successfully stopped SparkContext
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope closed issue #8580: Hudi reads failing on upgrade to Hudi 0.12.2 from Hudi 0.11.1
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed issue #8580: Hudi reads failing on upgrade to Hudi 0.12.2 from Hudi 0.11.1
URL: https://github.com/apache/hudi/issues/8580
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #8580: Hudi reads failing on upgrade to Hudi 0.12.2 from Hudi 0.11.1
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on issue #8580:
URL: https://github.com/apache/hudi/issues/8580#issuecomment-1535910686
Duplicate of https://github.com/apache/hudi/issues/8581. Lets track there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #8580: Hudi reads failing on upgrade to Hudi 0.12.2 from Hudi 0.11.1
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #8580:
URL: https://github.com/apache/hudi/issues/8580#issuecomment-1523549310
yes. unfortunately, the readers has to be upgraded before you can upgrade the writers. this a known limitation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org