You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "vinoyang (Jira)" <ji...@apache.org> on 2020/08/06 16:05:00 UTC
[jira] [Closed] (HUDI-1151) Fix NPE when no new data in kafka using
HoodieDeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-1151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
vinoyang closed HUDI-1151.
--------------------------
Resolution: Fixed
Fixed via master branch: b51646dcc76acc68e97dd6a67cc7557e362b590d
> Fix NPE when no new data in kafka using HoodieDeltaStreamer
> -----------------------------------------------------------
>
> Key: HUDI-1151
> URL: https://issues.apache.org/jira/browse/HUDI-1151
> Project: Apache Hudi
> Issue Type: Bug
> Components: DeltaStreamer
> Reporter: wangxianghu
> Assignee: wangxianghu
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.6.0
>
>
> I get this NPE when using HoodieDeltaStreamer to etl kafka data to hudi table. the topic is newly created, and there is no data in the topic when the job is started
> {code:java}
> 20/08/05 21:18:00 INFO helpers.KafkaOffsetGen: SourceLimit not configured, set numEvents to default value : 5000000
> 20/08/05 21:18:00 INFO sources.JsonKafkaSource: About to read 0 from Kafka for topic :hudi_test_callback
> 20/08/05 21:18:00 INFO deltastreamer.DeltaSync: No new data, source checkpoint has not changed. Nothing to commit. Old checkpoint=(Option{val=hudi_test_callback,0:0}). New Checkpoint=(hudi_test_callback,0:0)
> 20/08/05 21:18:00 ERROR deltastreamer.HoodieDeltaStreamer: Shutting down delta-sync due to exception
> java.lang.NullPointerException
> at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:570)
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 20/08/05 21:18:00 INFO deltastreamer.HoodieDeltaStreamer: Delta Sync shutdown. Error ?true
> 20/08/05 21:18:00 ERROR async.AbstractAsyncService: Monitor noticed one or more threads failed. Requesting graceful shutdown of other threads
> java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieException
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at org.apache.hudi.async.AbstractAsyncService.lambda$monitorThreads$0(AbstractAsyncService.java:136)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hudi.exception.HoodieException
> at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:585)
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
> ... 3 more
> Caused by: java.lang.NullPointerException
> at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:570)
> ... 4 more
> 20/08/05 21:18:00 ERROR async.AbstractAsyncService: Service shutdown with error
> java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieException
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at org.apache.hudi.async.AbstractAsyncService.waitForShutdown(AbstractAsyncService.java:72)
> at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.lambda$sync$1(HoodieDeltaStreamer.java:152)
> at org.apache.hudi.common.util.Option.ifPresent(Option.java:96)
> at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:149)
> at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:454)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:673)
> Caused by: org.apache.hudi.exception.HoodieException
> at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:585)
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
> at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:570)
> ... 4 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)