You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 08:52:59 UTC

[I] UT for Job module 4 is stuck for 4 hours [hudi]

hudi-bot opened a new issue, #16202:
URL: https://github.com/apache/hudi/issues/16202

   Of late, we are incurring timeouts for job module 4 in azure for UT module. 
   
   Its very strange from the logs. There are only 15 tests triggered and all 15 of them are complete. And the test is just stuck in there. after 3 hour 40 mins or so, entire job run is cancelled and we don't see any logs in between. 
   
    
   
   Excerpt from logs:
   {code:java}
   2023-08-23T08:16:30.2299637Z [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 s - in org.apache.hudi.hadoop.TestAnnotation
   2023-08-23T08:16:30.2319408Z [INFO] Running org.apache.hudi.hadoop.hive.TestHoodieCombineHiveInputFormat
   2023-08-23T08:16:30.2421972Z Formatting using clusterid: testClusterID
   2023-08-23T08:16:30.2570394Z 31120 [Listener at localhost/41109] WARN  org.apache.hadoop.conf.Configuration [] - No unit for dfs.heartbeat.interval(3) assuming SECONDS
   2023-08-23T08:16:30.2574429Z 31121 [Listener at localhost/41109] WARN  org.apache.hadoop.conf.Configuration [] - No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS
   2023-08-23T08:16:30.5588638Z 31422 [Listener at localhost/41109] WARN  org.apache.hadoop.conf.Configuration [] - No unit for dfs.heartbeat.interval(3) assuming SECONDS
   2023-08-23T08:16:30.5589693Z 31422 [Listener at localhost/41109] WARN  org.apache.hadoop.conf.Configuration [] - No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS
   2023-08-23T08:16:30.6633485Z 31526 [Listener at localhost/33189] WARN  org.apache.hadoop.conf.Configuration [] - No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS
   2023-08-23T08:16:34.4630382Z 35326 [Listener at localhost/41177] WARN  org.apache.hadoop.hdfs.server.datanode.DirectoryScanner [] - DirectoryScanner: shutdown has been called
   2023-08-23T08:16:34.5845964Z 35447 [BP-118616970-10.1.148.0-1692778590260 heartbeating to localhost/127.0.0.1:33189] WARN  org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager [] - IncrementalBlockReportManager interrupted
   2023-08-23T08:16:34.5895718Z 35447 [BP-118616970-10.1.148.0-1692778590260 heartbeating to localhost/127.0.0.1:33189] WARN  org.apache.hadoop.hdfs.server.datanode.DataNode [] - Ending block pool service for: Block pool BP-118616970-10.1.148.0-1692778590260 (Datanode Uuid eaf09d45-eb56-49ca-bec3-2da8a8f7bd14) service to localhost/127.0.0.1:33189
   2023-08-23T08:16:34.5952216Z 35458 [refreshUsed-/tmp/hdfs-test-service16927785902315510072575960611105/data/data2/current/BP-118616970-10.1.148.0-1692778590260] WARN  org.apache.hadoop.fs.CachingGetSpaceUsed [] - Thread Interrupted waiting to refresh disk information: sleep interrupted
   2023-08-23T08:16:34.5971425Z 35458 [refreshUsed-/tmp/hdfs-test-service16927785902315510072575960611105/data/data1/current/BP-118616970-10.1.148.0-1692778590260] WARN  org.apache.hadoop.fs.CachingGetSpaceUsed [] - Thread Interrupted waiting to refresh disk information: sleep interrupted
   2023-08-23T08:16:34.6255412Z 35488 [2085456234@qtp-936129838-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45383] WARN  org.apache.hadoop.http.HttpServer2 [] - HttpServer Acceptor: isRunning is false. Rechecking.
   2023-08-23T08:16:34.6289510Z 35492 [2085456234@qtp-936129838-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45383] WARN  org.apache.hadoop.http.HttpServer2 [] - HttpServer Acceptor: isRunning is false
   2023-08-23T08:16:34.7378002Z [WARNING] Tests run: 4, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 4.503 s - in org.apache.hudi.hadoop.hive.TestHoodieCombineHiveInputFormat
   2023-08-23T12:00:25.4660529Z ##[error]The operation was canceled.
   2023-08-23T12:00:25.4674798Z ##[section]Finishing: UT other modules {code}
   Just check for log timing for last 3 lines. 
   
    
   
   Ref:  https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/19423/logs/21
   
    
   
    
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-6755
   - Type: Bug
   - Epic: https://issues.apache.org/jira/browse/HUDI-4302


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] UT for Job module 4 is stuck for 4 hours [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #16202: UT for Job module 4 is stuck for 4 hours
URL: https://github.com/apache/hudi/issues/16202


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] UT for Job module 4 is stuck for 4 hours [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #16202:
URL: https://github.com/apache/hudi/issues/16202#issuecomment-4795553841

   Triage update: this Azure CI hang is no longer reproducible on current `master`. Commit [`688d6c07`](https://github.com/apache/hudi/commit/688d6c07a2110a2dba0286f8277cfa8cb4bdb881) (HUDI-6820, PR #9661, "Fixing CI stability issues") moved the flaky `hudi-hadoop-mr` module — which contains `TestHoodieCombineHiveInputFormat`, the test where module 4 was getting stuck — and `hudi-java-client` out of Azure CI onto GitHub Actions specifically to address this flakiness. The current `azure-pipelines-20230430.yml` (lines 42-43) still excludes `hudi-hadoop-mr`, and `.github/workflows/bot.yml` (line 558) runs these modules in a dedicated `test-hudi-hadoop-mr-and-hudi-java-client` job. Closing as fixed; please reopen if the underlying test flakiness still surfaces on GitHub Actions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org