[GitHub] [incubator-druid] jon-wei opened a new pull request #7673: Add option to use YARN RM as fallback for JobHistory failure

[GitBox <gi...@apache.org>]

jon-wei opened a new pull request #7673: Add option to use YARN RM as fallback for JobHistory failure
URL: https://github.com/apache/incubator-druid/pull/7673
 
 
   When a Hadoop job run by the HadoopIndexTask completes,`job.waitForCompletion()` expects the JobHistory server to be up and functioning correctly.
   
   It's possible for a job to succeed but later have the HadoopIndexTask fail because it had a problem when contacting the JobHistory server. 
   
   This PR adds a `useYarnRMJobStatusFallback` option to the Hadoop tuning config that allows the HadoopIndexTask to check for the application status from the YARN ResourceManager if an IOException occurs at `job.waitForCompletion()`. If the YARN RM check succeeds, the task proceeds normally. The new option is false by default.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org