You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/07/30 07:33:18 UTC
[GitHub] [hudi] nowordofalie opened a new issue #1887: [SUPPORT] docker-demo : Earliest write inflight instant time must be later than compaction time
nowordofalie opened a new issue #1887:
URL: https://github.com/apache/hudi/issues/1887
How do I prevent `20/07/29 21:25:48 ERROR utilities.HoodieCompactor: java.lang.IllegalArgumentException: Earliest write inflight instant time must be later than compaction time. Earliest :[==>20200729211339__deltacommit__INFLIGHT], Compaction scheduled at 20200729212535` when using the docker demo
**To Reproduce**
use docker demo : current git SHA 0cb24e4a
* docker exec -it adhoc-1 /bin/bash
* /var/hoodie/ws/hudi-cli/hudi-cli.sh
* connect --path /user/hive/warehouse/stock_ticks_mor
* compactions show all
* compaction schedule
**Expected behavior**
* compaction of MOR log to produce read optimised parquet
**Environment Description**
docker demo : current git SHA 0cb24e4a
* Running on Docker? : yes
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```
20/07/29 21:25:48 INFO javalin.Javalin: Starting Javalin ...
20/07/29 21:25:48 INFO javalin.Javalin: Listening on http://localhost:38763/
20/07/29 21:25:48 INFO javalin.Javalin: Javalin started in 459ms \o/
20/07/29 21:25:48 INFO service.TimelineService: Starting Timeline server on port :38763
20/07/29 21:25:48 INFO embedded.EmbeddedTimelineService: Started embedded timeline server at adhoc-2:38763
20/07/29 21:25:48 INFO client.HoodieWriteClient: Scheduling compaction at instant time :20200729212535
20/07/29 21:25:48 INFO table.HoodieTableMetaClient: Loading HoodieTableMetaClient from /user/hive/warehouse/stock_ticks_mor
20/07/29 21:25:48 INFO fs.FSUtils: Hadoop Configuration: fs.defaultFS: [hdfs://namenode:8020], Config:[Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, __spark_hadoop_conf__.xml], FileSystem: [DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1402254799_1, ugi=root (auth:SIMPLE)]]]
20/07/29 21:25:48 INFO table.HoodieTableConfig: Loading table properties from /user/hive/warehouse/stock_ticks_mor/.hoodie/hoodie.properties
20/07/29 21:25:48 INFO table.HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from /user/hive/warehouse/stock_ticks_mor
20/07/29 21:25:48 INFO table.HoodieTableMetaClient: Loading Active commit timeline for /user/hive/warehouse/stock_ticks_mor
20/07/29 21:25:48 INFO timeline.HoodieActiveTimeline: Loaded instants [[20200729205850__deltacommit__COMPLETED], [==>20200729211339__deltacommit__INFLIGHT]]
20/07/29 21:25:48 INFO view.FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
20/07/29 21:25:48 INFO view.FileSystemViewManager: Creating remote first table view
20/07/29 21:25:48 ERROR utilities.HoodieCompactor: java.lang.IllegalArgumentException: Earliest write inflight instant time must be later than compaction time. Earliest :[==>20200729211339__deltacommit__INFLIGHT], Compaction scheduled at 20200729212535
20/07/29 21:25:48 INFO spark.SparkContext: Invoking stop() from shutdown hook
20/07/29 21:25:48 INFO server.AbstractConnector: Stopped Spark@51751e5f{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
20/07/29 21:25:48 INFO ui.SparkUI: Stopped Spark web UI at http://adhoc-2:4040
20/07/29 21:25:48 INFO cluster.StandaloneSchedulerBackend: Shutting down all executors
20/07/29 21:25:48 INFO cluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
20/07/29 21:25:48 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
20/07/29 21:25:49 INFO memory.MemoryStore: MemoryStore cleared
20/07/29 21:25:49 INFO storage.BlockManager: BlockManager stopped
20/07/29 21:25:49 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
20/07/29 21:25:49 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
20/07/29 21:25:49 INFO spark.SparkContext: Successfully stopped SparkContext
20/07/29 21:25:49 INFO util.ShutdownHookManager: Shutdown hook called
20/07/29 21:25:49 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-cbea1789-1054-4d76-95d0-6bacbf450a06
20/07/29 21:25:49 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-a18ee635-d5aa-4774-97e9-5f06518b093f
Failed to run compaction for 20200729212535
hudi:stock_ticks_mor->
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nowordofalie commented on issue #1887: [SUPPORT] docker-demo : Earliest write inflight instant time must be later than compaction time
Posted by GitBox <gi...@apache.org>.
nowordofalie commented on issue #1887:
URL: https://github.com/apache/hudi/issues/1887#issuecomment-666557905
Thanks @bvaradar for your help, will take a look
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nowordofalie closed issue #1887: [SUPPORT] docker-demo : Earliest write inflight instant time must be later than compaction time
Posted by GitBox <gi...@apache.org>.
nowordofalie closed issue #1887:
URL: https://github.com/apache/hudi/issues/1887
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #1887: [SUPPORT] docker-demo : Earliest write inflight instant time must be later than compaction time
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #1887:
URL: https://github.com/apache/hudi/issues/1887#issuecomment-666515458
this usually means you are running multiple writers at the same time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org