You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Balaji Varadarajan (Jira)" <ji...@apache.org> on 2019/12/28 20:03:00 UTC
[jira] [Commented] (HUDI-474) Delta Streamer is not able to read
the commit files
[ https://issues.apache.org/jira/browse/HUDI-474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17004569#comment-17004569 ]
Balaji Varadarajan commented on HUDI-474:
-----------------------------------------
[~srkhan] As per your comment in the email thread - "when I have checked, folder .aux was empty ... ", this means that you have upgraded deltastreamer to latest master (which includes [https://github.com/apache/incubator-hudi/pull/1009]) when running earlier clean operations but from the stack trace, it looks like older layout version is used.
Is it possible you are using 0.5.0 version of deltastreamer or have you set the configuration. hoodie.timeline.layout.version. In any case, can you attach the complete logs of your deltastreamer run (where you saw the problem) in this Jira ? I did not see the attachment in your email to the dev ML.
Balaji.V
> Delta Streamer is not able to read the commit files
> ---------------------------------------------------
>
> Key: HUDI-474
> URL: https://issues.apache.org/jira/browse/HUDI-474
> Project: Apache Hudi (incubating)
> Issue Type: Bug
> Components: DeltaStreamer
> Reporter: Shahida Khan
> Assignee: Balaji Varadarajan
> Priority: Major
> Fix For: 0.5.1
>
> Attachments: Gmail - Commit time issue in DeltaStreamer (Real-Time).pdf
>
>
> DeltaStreamer is not to able to read the correct commit files under when job is deployed realtime.
> below is the stack trace:
> ava.util.concurrent.ExecutionException:
> org.apache.hudi.exception.HoodieException: Could not read commit
> details from hdfs:/user/hive/warehouse/hudi.db/tbltest/.hoodie/.aux/20191226153400.clean.requested at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at
> org.apache.hudi.utilities.deltastreamer.AbstractDeltaStreamerService.waitForShutdown(AbstractDeltaStreamerService.java:72) at
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:117) at
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:297) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at
> java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:688)Caused
> by: org.apache.hudi.exception.HoodieException: Could not read commit
> details from hdfs:/user/hive/warehouse/hudi.db/tbltest/.hoodie/.aux/20191226153400.clean.requested at
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:411) at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at
> java.lang.Thread.run(Thread.java:748)
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)