You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "sivabalan narayanan (Jira)" <ji...@apache.org> on 2023/03/13 19:33:00 UTC
[jira] [Assigned] (HUDI-5928) PostWriteTermination Strategy does not shutdown the spark job completely
[ https://issues.apache.org/jira/browse/HUDI-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
sivabalan narayanan reassigned HUDI-5928:
-----------------------------------------
Assignee: sivabalan narayanan
> PostWriteTermination Strategy does not shutdown the spark job completely
> ------------------------------------------------------------------------
>
> Key: HUDI-5928
> URL: https://issues.apache.org/jira/browse/HUDI-5928
> Project: Apache Hudi
> Issue Type: Bug
> Components: writer-core
> Reporter: sivabalan narayanan
> Assignee: sivabalan narayanan
> Priority: Critical
>
> When post write termination strategy is used, spark job is not fully shutdown.
> {code:java}
> 23/03/13 11:16:06 INFO DFSPathSelector: Root path => /tmp/input_dataset/ source limit => 30000000
> 23/03/13 11:16:06 INFO DeltaSync: No new data, source checkpoint has not changed. Nothing to commit. Old checkpoint=(Option{val=1678730775000}). New Checkpoint=(1678730775000)
> 23/03/13 11:16:06 INFO NoNewDataTerminationStrategy: Shutting down on continuous mode as there is no new data for 3
> 23/03/13 11:16:12 INFO HoodieAsyncService: Waiting for next instant up to 10 seconds
> 23/03/13 11:16:22 INFO HoodieAsyncService: Waiting for next instant up to 10 seconds
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 410
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 638
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 567
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 618
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 591
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 577
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 620
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 539
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 424
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 429
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 516
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 622
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 503
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 446
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 460
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 547
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 606
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 435
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 625
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 601
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 630
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 405
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 495
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 500
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 559
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 520
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 621
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 588
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 431
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 423
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 534
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 603
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 631
> 23/03/13 11:16:31 INFO BlockManagerInfo: Removed broadcast_20_piece0 on 192.168.1.7:60400 in memory (size: 94.3 KB, free: 4.1 GB)
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 537
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 568
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 597
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 562
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 535
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 481
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 611
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 585
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 640
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 515
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 645
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 609
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 581
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 530
> 23/03/13 11:16:31 INFO BlockManagerInfo: Removed broadcast_23_piece0 on 192.168.1.7:60400 in memory (size: 25.7 KB, free: 4.1 GB)
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 602
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 650
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 469
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 517
> 23/03/13 11:16:31 INFO ContextCleaner: Cleaned accumulator 542 {code}
> Job is just stuck there.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)