[GitHub] [incubator-druid] clintropolis commented on a change in pull request #7673: Add option to use YARN RM as fallback for JobHistory failure

[GitBox <gi...@apache.org>]

clintropolis commented on a change in pull request #7673: Add option to use YARN RM as fallback for JobHistory failure
URL: https://github.com/apache/incubator-druid/pull/7673#discussion_r284857037
 
 

 ##########
 File path: docs/content/ingestion/hadoop.md
 ##########
 @@ -198,6 +198,7 @@ The tuningConfig is optional and default parameters will be used if no tuningCon
 |useExplicitVersion|Boolean|Forces HadoopIndexTask to use version.|no (default = false)|
 |logParseExceptions|Boolean|If true, log an error message when a parsing exception occurs, containing information about the row where the error occurred.|false|no|
 |maxParseExceptions|Integer|The maximum number of parse exceptions that can occur before the task halts ingestion and fails. Overrides `ignoreInvalidRows` if `maxParseExceptions` is defined.|unlimited|no|
+|useYarnRMJobStatusFallback|Boolean|If the Hadoop jobs created by the indexing task are unable to retrieve their completion status from the JobHistory server, and this parameter is true, the indexing task will try to fetch the application status from `http://<yarn-rm-address>/ws/v1/cluster/apps/<application-id>`, where `<yarn-rm-address>` is the value of `yarn.resourcemanager.webapp.address` in your Hadoop configuration. This flag is intended as a fallback for cases where an indexing task's jobs succeed, but the JobHistory server is unavailable, causing the indexing task to fail because it cannot determine the job statuses.|no (default = true)|
 
 Review comment:
   If this defaults to true, any reason to have it be an option at all? Would anyone _not_ want this to happen on fallback?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org