You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/10/06 06:48:18 UTC
[GitHub] [hudi] fengjian428 opened a new issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
fengjian428 opened a new issue #3755:
URL: https://github.com/apache/hudi/issues/3755
Environment： Hudi 0.9 ，Hbase 1.4.12
when I run delta streamer(version 0.9) to ingest data from kafka to a Hbase indexed mor table , after few commits, met this error when compaction running
In hdfs there is a file has same fileId and commit instant but different in the middle: hdfs://tl5/projects/data_vite/mysql_ingestion/rti_vite/shopee_item_v4_db__item_v4_tab_newHbase/BR/2021-10/813800cd-1aaf-43ea-829f-4feef4a51cb3-0_19-2672-4427765_20211006051032.parquet
below is 20211006051032.commit's content,
What does 2672-4427765 and 2657-4368242 mean? why they are mismatch and how can I fix this error?
I tried recreate table , it happens again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-1002290377
@fengjian428 : do you have any updates for us in this regard. If issue is resolved, feel free to close the github issue. if not, let us know how we can be of help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
codope commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-951918490
@fengjian428 How frequently are you facing this issue? You mentioned earlier that:
> this table was create by Delta streamer's SqlSource from another table, but when ingest real-time data from kafka with kafkasource, the compaction dose not work, I need shutdown Delta streamer and start it again, then compaction works
Seems like a dangling file due to failed commit. Given the symptoms, would it be possible for you to write a unit test? Or even just list the steps to help me reproduce?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-1018631431
@fengjian428 : gentle ping. Interested to know any progress. or if you got past the issue, how did you get it resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-991852883
We made some fix around illegal arg exception you have posted above. Can you try with 0.10.0.
Also, interested to understand more details around "after running few hours, the compaction plan was unreasonable". May I know how do you say that? is it bcoz of the stacktrace posted above or due to something else you noticed. can you throw some light please.
and for the timeout issue you posted above, do we know if job was making progress ? can you post us spark staged screen shots if have any.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-1039433744
@yihua : can you take a look at this issue and see if this could be due to the repeated deferrencing of RDD, which got fixed recently?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-1053172311
> #4753
could this happens when using cow table?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 edited a comment on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
fengjian428 edited a comment on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-940179806
> It seems that the file left in reconcile stage is different with commit meta. Could you kindly share relevant logs and file status about marker file?
before I got error above, this rpc timeout error happen, and when I restart delta streamer, above error happen, should I change unpersist method non-blocking and try again?
`Caused by: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieException: Futures timed out after [600 seconds]. This timeout is controlled by spark.rpc.askTimeout
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at org.apache.hudi.async.HoodieAsyncService.waitForShutdown(HoodieAsyncService.java:90)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.lambda$sync$1(HoodieDeltaStreamer.java:171)
... 8 more
Caused by: org.apache.hudi.exception.HoodieException: Futures timed out after [600 seconds]. This timeout is controlled by spark.rpc.askTimeout
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:657)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [600 seconds]. This timeout is controlled by spark.rpc.askTimeout
at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:47)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:62)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:58)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:76)
at org.apache.spark.storage.BlockManagerMaster.removeRdd(BlockManagerMaster.scala:131)
at org.apache.spark.SparkContext.unpersistRDD(SparkContext.scala:1821)
at org.apache.spark.rdd.RDD.unpersist(RDD.scala:217)
at org.apache.spark.api.java.JavaRDD.unpersist(JavaRDD.scala:53)
at org.apache.hudi.client.SparkRDDWriteClient.lambda$releaseResources$5(SparkRDDWriteClient.java:499)
at java.lang.Iterable.forEach(Iterable.java:75)
at org.apache.hudi.client.SparkRDDWriteClient.releaseResources(SparkRDDWriteClient.java:499)
at org.apache.hudi.client.AbstractHoodieWriteClient.commitStats(AbstractHoodieWriteClient.java:193)
at org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:124)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:525)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:304)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:633)
... 4 more
Caused by: java.util.concurrent.TimeoutException: Futures timed out after [600 seconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:223)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:227)
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:220)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
... 16 more`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-951963942
> streamer
this should only happens when table has massive update. In my case, it's a 10 TB size table, millions records in every batch from kafka，and those records cause TBs file written.
I found if use mor mode with default configuration, after running few hours, the compaction plan was unreasonable, then this error happen, there also a lot of small file in HDFS. If I change async compaction to sync, it work fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-935750802
this table was create by Delta streamer's SqlSource from another table, but when ingest real-time data from kafka with kafkasource, the compaction dose not work, I need shutdown Delta streamer and start it again, then compaction works
I 'm not sure if that can effect following processing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-1073074228
@fengjian428 : was clustering enabled in your case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-942709709
@guanziyue Do you know what this error means?
`21/10/14 04:36:22 ERROR RequestHandler: Got runtime exception servicing request partition=TH%2F2021-01&maxinstant=20211014043535&basepath=hdfs%3A%2F%2Ftl5%2Fprojects%2Fdata_vite%2Fmysql_ingestion%2Frti_vite%2Fshopee_item_v4_db__item_v4_tab_new6&lastinstantts=20211014043614&timelinehash=5d50a0189abbb1e122f7a838ac389bb21ae27ef6db6428821c908be8f566e032
java.lang.IllegalArgumentException: Last known instant from client was 20211014043614 but server has the following timeline [[20211014042315__deltacommit__COMPLETED], [20211014042356__deltacommit__COMPLETED], [20211014042430__deltacommit__COMPLETED], [20211014042509__deltacommit__COMPLETED], [20211014042534__deltacommit__COMPLETED], [20211014042558__commit__COMPLETED], [20211014042607__deltacommit__COMPLETED], [20211014042648__deltacommit__COMPLETED], [20211014042713__deltacommit__COMPLETED], [20211014042736__deltacommit__COMPLETED], [20211014042758__deltacommit__COMPLETED], [20211014042820__commit__COMPLETED], [20211014042824__deltacommit__COMPLETED], [20211014042905__clean__COMPLETED], [20211014042918__deltacommit__COMPLETED], [20211014042937__clean__COMPLETED], [20211014042948__deltacommit__COMPLETED], [20211014043012__clean__COMPLETED], [20211014043022__deltacommit__COMPLETED], [20211014043047__clean__COMPLETED], [20211014043056__deltacommit__COMPLETED], [20211014043115__clean
__COMPLETED], [20211014043124__commit__COMPLETED], [20211014043127__deltacommit__COMPLETED], [20211014043145__clean__COMPLETED], [20211014043313__deltacommit__COMPLETED], [20211014043351__clean__COMPLETED], [20211014043419__deltacommit__COMPLETED], [20211014043443__clean__COMPLETED], [20211014043454__deltacommit__COMPLETED], [20211014043525__clean__COMPLETED], [20211014043535__deltacommit__COMPLETED], [20211014043605__clean__COMPLETED], [20211014043614__commit__COMPLETED]]
at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:40)
at org.apache.hudi.timeline.service.RequestHandler$ViewHandler.handle(RequestHandler.java:510)
at io.javalin.security.SecurityUtil.noopAccessManager(SecurityUtil.kt:22)
at io.javalin.Javalin.lambda$addHandler$0(Javalin.java:606)
at io.javalin.core.JavalinServlet$service$2$1.invoke(JavalinServlet.kt:46)
at io.javalin.core.JavalinServlet$service$2$1.invoke(JavalinServlet.kt:17)
at io.javalin.core.JavalinServlet$service$1.invoke(JavalinServlet.kt:143)
at io.javalin.core.JavalinServlet$service$2.invoke(JavalinServlet.kt:41)
at io.javalin.core.JavalinServlet.service(JavalinServlet.kt:107)
at io.javalin.core.util.JettyServerUtil$initialize$httpHandler$1.doHandle(JettyServerUtil.kt:72)
at org.apache.hudi.org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at org.apache.hudi.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
at org.apache.hudi.org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1668)
at org.apache.hudi.org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at org.apache.hudi.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
at org.apache.hudi.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at org.apache.hudi.org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61)
at org.apache.hudi.org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174)
at org.apache.hudi.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.apache.hudi.org.eclipse.jetty.server.Server.handle(Server.java:502)
at org.apache.hudi.org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
at org.apache.hudi.org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
at org.apache.hudi.org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.apache.hudi.org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.apache.hudi.org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.apache.hudi.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at org.apache.hudi.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at org.apache.hudi.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.apache.hudi.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.apache.hudi.org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at org.apache.hudi.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at org.apache.hudi.org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.lang.Thread.run(Thread.java:748)
21/10/14 04:36:22 WARN ExceptionMapper: Uncaught exception
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-991894043
> We made some fix around illegal arg exception you have posted above. Can you try with 0.10.0. Also, interested to understand more details around "after running few hours, the compaction plan was unreasonable". May I know how do you say that? is it bcoz of the stacktrace posted above or due to something else you noticed. can you throw some light please.
>
> and for the timeout issue you posted above, do we know if job was making progress ? can you post us spark staged screen shots if have any.
OK let me try it with 0.10 and see how it goes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-1073167451
> @fengjian428 : was clustering enabled in your case.
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-1039796580
> > > past
> > > think this is cause by meta table, I enabled metatable then got this error
>
> @fengjian428 did you mean you disabled metadata table then it's resolved?
yes I think so
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-991852883
We made some fix around illegal arg exception you have posted above. Can you try with 0.10.0.
Also, interested to understand more details around "after running few hours, the compaction plan was unreasonable". May I know how do you say that? is it bcoz of the stacktrace posted above or due to something else you noticed. can you throw some light please.
and for the timeout issue you posted above, do we know if job was making progress ? can you post us spark stages screen shots if have any.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-1039796580
> > > past
> > > think this is cause by meta table, I enabled metatable then got this error
>
> @fengjian428 did you mean you disabled metadata table then it's resolved?
yes I think so
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-1019280565
> past
think this is cause by meta table, I enabled metatable then got this error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-1063736219
> > #4753
>
> could this happens when using cow table? @yihua
If clustering is enabled for the COW table, that commit metadata issue can happen on the COW table as well since the clustering commit metadata was also affected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 edited a comment on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
fengjian428 edited a comment on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-1053172311
> #4753
could this happens when using cow table? @yihua
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-1025291114
> > past
> > think this is cause by meta table, I enabled metatable then got this error
@fengjian428 did you mean you disabled metadata table then it's resolved?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] guanziyue commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
guanziyue commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-938281439
It seems that the file left in reconcile stage is different with commit meta. Could you kindly share relevant logs and file status about marker file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-993918620
sure, thanks a bunch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-945612669
I found two sparkHoodieBloomIndex were running, is that means two writers ran parallelism？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 edited a comment on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
fengjian428 edited a comment on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-945089345
> Are you having concurrent writers? If yes, I have come across a similar issue reported by someone else. let me know.
I have a question, when Hudi does delta commit, if data is new , it need append them to exist parquet file. meanwhile may cause concurrent issue with async compaction thread if compaction plan contains same parquet file，how Hudi avoid that？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-945089345
> Are you having concurrent writers? If yes, I have come across a similar issue reported by someone else. let me know.
I have a question, when Hudi does delta commit, if data is new , it need append them to exist parquet file. meanwhile may cause concurrent issue with async compaction thread，how Hudi avoid that？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-940179806
> It seems that the file left in reconcile stage is different with commit meta. Could you kindly share relevant logs and file status about marker file?
before I got error above, this rpc timeout error happen, and when I restart delta streamer, above error happen
Caused by: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieException: Futures timed out after [600 seconds]. This timeout is controlled by spark.rpc.askTimeout
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at org.apache.hudi.async.HoodieAsyncService.waitForShutdown(HoodieAsyncService.java:90)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.lambda$sync$1(HoodieDeltaStreamer.java:171)
... 8 more
Caused by: org.apache.hudi.exception.HoodieException: Futures timed out after [600 seconds]. This timeout is controlled by spark.rpc.askTimeout
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:657)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [600 seconds]. This timeout is controlled by spark.rpc.askTimeout
at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:47)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:62)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:58)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:76)
at org.apache.spark.storage.BlockManagerMaster.removeRdd(BlockManagerMaster.scala:131)
at org.apache.spark.SparkContext.unpersistRDD(SparkContext.scala:1821)
at org.apache.spark.rdd.RDD.unpersist(RDD.scala:217)
at org.apache.spark.api.java.JavaRDD.unpersist(JavaRDD.scala:53)
at org.apache.hudi.client.SparkRDDWriteClient.lambda$releaseResources$5(SparkRDDWriteClient.java:499)
at java.lang.Iterable.forEach(Iterable.java:75)
at org.apache.hudi.client.SparkRDDWriteClient.releaseResources(SparkRDDWriteClient.java:499)
at org.apache.hudi.client.AbstractHoodieWriteClient.commitStats(AbstractHoodieWriteClient.java:193)
at org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:124)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:525)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:304)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:633)
... 4 more
Caused by: java.util.concurrent.TimeoutException: Futures timed out after [600 seconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:223)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:227)
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:220)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
... 16 more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-1039433744
@yihua : can you take a look at this issue and see if this could be due to the repeated deferrencing of RDD, which got fixed recently?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-1052473596
@fengjian428 Sorry for the late reply. Recently we've triaged an issue related to wrong commit metadata that affects metadata table, caused by RDD caching and the old logic of deriving commit metadata. The fix is #4753 . I believe that caused your issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
codope commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-951914894
> I found two sparkHoodieBloomIndex were running, is that means two writers ran parallelism？
I believe those are part of the same writer process. Hudi performs index lookup to get existing location of records. As part of that, it will tag the incoming records as inserts or updates, by joining with existing record keys. So you see two mapToPair calls. Check [SparkHoodieBloomIndex#tagLocation](https://github.com/apache/hudi/blob/e3fc74668fc43fefd73087ff725245b8ed85b4a1/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/bloom/SparkHoodieBloomIndex.java#L70) method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running
Posted by GitBox <gi...@apache.org>.
codope commented on issue #3755:
URL: https://github.com/apache/hudi/issues/3755#issuecomment-951901867
> I have a question, when Hudi does delta commit, if data is new , it need append them to exist parquet file. meanwhile may cause concurrent issue with async compaction thread if compaction plan contains same parquet file，how Hudi avoid that？
The compactor does filter out file groups in pending compaction while generating compaction plan. [Check this](https://github.com/apache/hudi/blob/e3fc74668fc43fefd73087ff725245b8ed85b4a1/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/HoodieCompactor.java#L262).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org