You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/10/26 13:51:48 UTC
[GitHub] [hudi] atharvai opened a new issue #3870: [SUPPORT] Hudi v0.8.0 Savepoint rollback failure
atharvai opened a new issue #3870:
URL: https://github.com/apache/hudi/issues/3870
**Describe the problem you faced**
Given a savepoint, rollback fails with no error messages.
**To Reproduce**
Environment: EMR 6.4.0, S3
Steps to reproduce the behavior:
1. create Hudi dataset
2. add some data so there are multiple commits
3. create a savepoint
4. try to rollback savepoint
For detailed instructions see attached PDF
[HudiSavepointsandCommitsdiscovery.pdf](https://github.com/apache/hudi/files/7418645/HudiSavepointsandCommitsdiscovery.pdf)
**Expected behavior**
rollback should be successful and queried data should be reverted to the specific commit.
**Environment Description**
* Hudi version : 0.8.0
* Spark version : 3.1.2
* Hive version : 3.1.2
* Hadoop version : 3.2.1
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
**Additional context**
Spark log doesn't include any `ERROR` log messages
**Stacktrace**
see attached PDF
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] atharvai commented on issue #3870: [SUPPORT] Hudi v0.8.0 Savepoint rollback failure
Posted by GitBox <gi...@apache.org>.
atharvai commented on issue #3870:
URL: https://github.com/apache/hudi/issues/3870#issuecomment-951965505
```
__ __ _
/ /____ _ _ __ ____ _ / /(_)____
__ / // __ `/| | / // __ `// // // __ \
/ /_/ // /_/ / | |/ // /_/ // // // / / /
\____/ \__,_/ |___/ \__,_//_//_//_/ /_/
hudi:phe_london_covid_cases->
https://javalin.io/documentation
hudi:phe_london_covid_cases->
21/10/26 13:51:52 INFO Javalin: Starting Javalin ...
21/10/26 13:51:52 INFO Server: jetty-9.4.20.v20190813; built: 2019-08-13T21:28:18.144Z; git: 84700530e645e812b336747464d6fbbf370c9a20; jvm 1.8.0_302-b08
21/10/26 13:51:52 INFO Server: Started @29666ms
21/10/26 13:51:52 INFO Javalin: Listening on http://localhost:39451/
21/10/26 13:51:52 INFO Javalin: Javalin started in 160ms \o/
21/10/26 13:51:53 INFO S3NativeFileSystem: Opening 's3://<mybucket>/phe_london_covid_cases/.hoodie/hoodie.properties' for reading
21/10/26 13:51:53 INFO MultipartUploadOutputStream: close closed:false s3://<mybucket>/phe_london_covid_cases/.hoodie/20211026135153.restore.inflight
21/10/26 13:51:54 WARN SparkMain: The commit "20211021104635" failed to roll back.
21/10/26 13:51:54 INFO SparkContext: Invoking stop() from shutdown hook
21/10/26 13:51:54 INFO AbstractConnector: Stopped Spark@73877e19{HTTP/1.1, (http/1.1)}{0.0.0.0:4040}
21/10/26 13:51:54 INFO SparkUI: Stopped Spark web UI at http://ip-10-21-176-60.eu-west-1.compute.internal:4040
21/10/26 13:51:54 INFO YarnClientSchedulerBackend: Interrupting monitor thread
21/10/26 13:51:54 INFO YarnClientSchedulerBackend: Shutting down all executors
21/10/26 13:51:54 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
21/10/26 13:51:54 INFO YarnClientSchedulerBackend: YARN client scheduler backend Stopped
21/10/26 13:51:54 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
21/10/26 13:51:54 INFO MemoryStore: MemoryStore cleared
21/10/26 13:51:54 INFO BlockManager: BlockManager stopped
21/10/26 13:51:54 INFO BlockManagerMaster: BlockManagerMaster stopped
21/10/26 13:51:54 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
21/10/26 13:51:54 INFO SparkContext: Successfully stopped SparkContext
21/10/26 13:51:54 INFO ShutdownHookManager: Shutdown hook called
21/10/26 13:51:54 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-5e36e1aa-42d6-43a0-ac12-4bd6d7dfee9f
21/10/26 13:51:54 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-4d065ddb-64c7-451d-a237-de75ccdf0049
hudi:phe_london_covid_cases->2021-10-26 13:51:54,493 INFO table.HoodieTableMetaClient: Loading HoodieTableMetaClient from s3://<mybucket>/phe_london_covid_cases
2021-10-26 13:51:54,495 INFO fs.FSUtils: Hadoop Configuration: fs.defaultFS: [hdfs://ip-.eu-west-1.compute.internal:8020], Config:[Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, emrfs-site.xml], FileSystem: [com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem@4471f8fe]
2021-10-26 13:51:54,559 INFO table.HoodieTableConfig: Loading table properties from s3://<mybucket>/phe_london_covid_cases/.hoodie/hoodie.properties
2021-10-26 13:51:54,576 INFO s3n.S3NativeFileSystem: Opening 's3://<mybucket>/phe_london_covid_cases/.hoodie/hoodie.properties' for reading
2021-10-26 13:51:54,593 INFO table.HoodieTableMetaClient: Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3://<mybucket>/phe_london_covid_cases
Savepoint "20211021104635" failed to roll back
hudi:phe_london_covid_cases->savepoints refresh
2021-10-26 13:52:32,207 INFO table.HoodieTableMetaClient: Loading HoodieTableMetaClient from s3://<mybucket>/phe_london_covid_cases
2021-10-26 13:52:32,208 INFO fs.FSUtils: Hadoop Configuration: fs.defaultFS: [hdfs://ip-.eu-west-1.compute.internal:8020], Config:[Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, emrfs-site.xml], FileSystem: [com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem@4471f8fe]
2021-10-26 13:52:32,254 INFO table.HoodieTableConfig: Loading table properties from s3://<mybucket>/phe_london_covid_cases/.hoodie/hoodie.properties
2021-10-26 13:52:32,254 INFO s3n.S3NativeFileSystem: Opening 's3://<mybucket>/phe_london_covid_cases/.hoodie/hoodie.properties' for reading
2021-10-26 13:52:32,269 INFO table.HoodieTableMetaClient: Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3://<mybucket>/phe_london_covid_cases
Metadata for table phe_london_covid_cases refreshed.
hudi:phe_london_covid_cases->savepoints show
2021-10-26 13:52:36,242 INFO timeline.HoodieActiveTimeline: Loaded instants [[20211021101613__commit__COMPLETED], [20211021101613__savepoint__COMPLETED], [20211021104001__commit__COMPLETED], [20211021104635__commit__COMPLETED], [20211021104635__savepoint__COMPLETED], [20211021110154__commit__COMPLETED], [20211021110154__savepoint__COMPLETED], [==>20211021121705__restore__INFLIGHT], [==>20211021122414__restore__INFLIGHT], [==>20211021124148__restore__INFLIGHT], [==>20211021125331__restore__INFLIGHT], [==>20211021125532__restore__INFLIGHT], [==>20211021140929__restore__INFLIGHT], [20211026134847__restore__COMPLETED], [==>20211026135153__restore__INFLIGHT]]
╔════════════════╗
║ SavepointTime ║
╠════════════════╣
║ 20211021110154 ║
╟────────────────╢
║ 20211021104635 ║
╟────────────────╢
║ 20211021101613 ║
╚════════════════╝
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #3870: [SUPPORT] Hudi v0.8.0 Savepoint rollback failure
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #3870:
URL: https://github.com/apache/hudi/issues/3870
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3870: [SUPPORT] Hudi v0.8.0 Savepoint rollback failure
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3870:
URL: https://github.com/apache/hudi/issues/3870#issuecomment-1003779949
@atharvai : I tried a simple savepoint and rollback and able to get it working. Even the minor fix which I mentioned above is not required. I had to pass in sparkMaster as explicit argument. I have added the details in this [jira](https://issues.apache.org/jira/browse/HUDI-3059).
I tried w/ 0.10.0.
Can you try w/ latest master or 0.10.0 and let me know what do you see. Or if you can go over exact scenario, or steps to reproduce, we can try to reproduce.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] atharvai commented on issue #3870: [SUPPORT] Hudi v0.8.0 Savepoint rollback failure
Posted by GitBox <gi...@apache.org>.
atharvai commented on issue #3870:
URL: https://github.com/apache/hudi/issues/3870#issuecomment-952941884
Commit rollback has similar issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3870: [SUPPORT] Hudi v0.8.0 Savepoint rollback failure
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3870:
URL: https://github.com/apache/hudi/issues/3870#issuecomment-997210444
@atharvai : I was able to get the savepoint rollback working with some minor fix. spark master url parsing had issues.
https://issues.apache.org/jira/browse/HUDI-3059
Let me know if thats not the case for you. or is your use-case is something diff. All I am doing is,
C1, C2, C3, SP1, C4,
do savepoint rollback for SP1. and count matched until C3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3870: [SUPPORT] Hudi v0.8.0 Savepoint rollback failure
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3870:
URL: https://github.com/apache/hudi/issues/3870#issuecomment-1020607736
Closing this due to inactivity. we have savepoint rollback feature added to both tables. Unless we have further info, we can't help triage. thanks for reporting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3870: [SUPPORT] Hudi v0.8.0 Savepoint rollback failure
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3870:
URL: https://github.com/apache/hudi/issues/3870#issuecomment-1020607736
Closing this due to inactivity. we have savepoint rollback feature added to both tables. Unless we have further info, we can't help triage. thanks for reporting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3870: [SUPPORT] Hudi v0.8.0 Savepoint rollback failure
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3870:
URL: https://github.com/apache/hudi/issues/3870#issuecomment-1015342619
@atharvai : hey do you have any updates for us.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #3870: [SUPPORT] Hudi v0.8.0 Savepoint rollback failure
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #3870:
URL: https://github.com/apache/hudi/issues/3870
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3870: [SUPPORT] Hudi v0.8.0 Savepoint rollback failure
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3870:
URL: https://github.com/apache/hudi/issues/3870#issuecomment-991847923
@yihua : Can you take this up please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] atharvai commented on issue #3870: [SUPPORT] Hudi v0.8.0 Savepoint rollback failure
Posted by GitBox <gi...@apache.org>.
atharvai commented on issue #3870:
URL: https://github.com/apache/hudi/issues/3870#issuecomment-952312361
Spark history server shows the app has no jobs or stages. seems odd that it looks like a no-op
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org