You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Amar Kamat (JIRA)" <ji...@apache.org> on 2008/06/23 11:25:45 UTC
[jira] Created: (HADOOP-3618) JobClient should keep on retrying if
the jobtracker is still initializing
JobClient should keep on retrying if the jobtracker is still initializing
-------------------------------------------------------------------------
Key: HADOOP-3618
URL: https://issues.apache.org/jira/browse/HADOOP-3618
Project: Hadoop Core
Issue Type: Improvement
Components: mapred
Reporter: Amar Kamat
Assignee: Amar Kamat
When the user submits the job while the jobtracker is still initializing, the jobclient comes out with an exception. ideally the jobclient should keep on retrying until the jobtracker is up and ready. This will also take care of HADOOP-3289.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3618) JobClient should keep on retrying
if the jobtracker is still initializing
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12607542#action_12607542 ]
Amar Kamat commented on HADOOP-3618:
------------------------------------
bq. 1. If the thread is interrupted ...
We can do that.
bq. in other projects, .....
Steve, here the jobs that wait are few. The jobs that are immediately submitted as the jobtracker comes up are the ones that will see this problem. Once the jobtracker is up, there is no waiting. If you say that there are lots and lots of jobs that (can) get fired as soon as the JT comes up then yeah its is a problem. But adding a random part to the sleep sounds ok. I purposefully didnt keep it as a configurable param since the jt restart takes few seconds (on a decent enough cluster) and hence the wait time is in the order of millisecs. I had no strong reason to make it configurable. Let me know if thats not the case.
bq. If this sleep-until-ready pattern is common ..
Yup. If its a common pattern it can be factored out.
----
Thanks for the review.
> JobClient should keep on retrying if the jobtracker is still initializing
> -------------------------------------------------------------------------
>
> Key: HADOOP-3618
> URL: https://issues.apache.org/jira/browse/HADOOP-3618
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Attachments: HADOOP-3618.patch
>
>
> When the user submits the job while the jobtracker is still initializing, the jobclient comes out with an exception. ideally the jobclient should keep on retrying until the jobtracker is up and ready. This will also take care of HADOOP-3289.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3618) JobClient should keep on retrying
if the jobtracker is still initializing
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12607533#action_12607533 ]
Steve Loughran commented on HADOOP-3618:
----------------------------------------
the wait loop spins and sleeps:
+ // check if the jobtracker is ready
+ while (true) {
+ if (jobSubmitClient.isReady()) {
+ break;
+ }
+ try {
+ Thread.sleep(JOBTRACKER_POLL_INTERVAL);
+ } catch (InterruptedException ie){}
+ }
1. If the thread is interrupted, it implies somebody wanted to stop it. why not listen to that request by ending the thread, rather than spinning indefinately. This loop will make a job client thread impossible to kill in-process until the tracker is live.
2. in other projects, we've found problems if a few hundred machines have just come up fully synchronised, as they can do when a site's power gets toggled. They all poll simultaneously, flood the network and then wait..even with exponential back-off they are all in sync. So: a bit of random jitter on the sleep is good; likewise, the poll interval may be a configuration point.
If this sleep-until-ready pattern is common, it should be factored out into a method of its own and shared across things. I've been stubbing out (for my deployment use) a simple lifecycle interface (start/stop/getstatus/ping)...if that were adopted then we this patch could poll the getStatus() method.
> JobClient should keep on retrying if the jobtracker is still initializing
> -------------------------------------------------------------------------
>
> Key: HADOOP-3618
> URL: https://issues.apache.org/jira/browse/HADOOP-3618
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Attachments: HADOOP-3618.patch
>
>
> When the user submits the job while the jobtracker is still initializing, the jobclient comes out with an exception. ideally the jobclient should keep on retrying until the jobtracker is up and ready. This will also take care of HADOOP-3289.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3618) JobClient should keep on retrying if
the jobtracker is still initializing
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-3618:
-------------------------------
Attachment: HADOOP-3618.patch
Attaching a patch for review while I continue testing it.
> JobClient should keep on retrying if the jobtracker is still initializing
> -------------------------------------------------------------------------
>
> Key: HADOOP-3618
> URL: https://issues.apache.org/jira/browse/HADOOP-3618
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Attachments: HADOOP-3618.patch
>
>
> When the user submits the job while the jobtracker is still initializing, the jobclient comes out with an exception. ideally the jobclient should keep on retrying until the jobtracker is up and ready. This will also take care of HADOOP-3289.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3618) JobClient should keep on retrying
if the jobtracker is still initializing
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12607622#action_12607622 ]
Steve Loughran commented on HADOOP-3618:
----------------------------------------
I think it makes sense to not worry too much about making the sleep configurable. The jitter is sometimes useful for dealing with mass numbers of callers, though the only time we had a big problem it was with embedded hardware whose RNGs all initialised the same way. Randomness is sometimes hard to find.
When a full cluster reboots, its the data nodes that come up first; their boot time depends on the state of their disks. The name node ought to come up faster if its a RAID5 FS, but as it has to do playback it will take a while to go live. What happens to the job and task trackers in this situation? Will they just sit around? Because if we arent saving a job list over a cluster-crash there wont be a big set of jobs trying get restarted, not unless there are external clients hitting the site hard.
> JobClient should keep on retrying if the jobtracker is still initializing
> -------------------------------------------------------------------------
>
> Key: HADOOP-3618
> URL: https://issues.apache.org/jira/browse/HADOOP-3618
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Attachments: HADOOP-3618.patch
>
>
> When the user submits the job while the jobtracker is still initializing, the jobclient comes out with an exception. ideally the jobclient should keep on retrying until the jobtracker is up and ready. This will also take care of HADOOP-3289.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3618) JobClient should keep on retrying
if the jobtracker is still initializing
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12607756#action_12607756 ]
Doug Cutting commented on HADOOP-3618:
--------------------------------------
Shouldn't this be implemented using a RetryPolicy?
http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/io/retry/package-summary.html
> JobClient should keep on retrying if the jobtracker is still initializing
> -------------------------------------------------------------------------
>
> Key: HADOOP-3618
> URL: https://issues.apache.org/jira/browse/HADOOP-3618
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Attachments: HADOOP-3618.patch
>
>
> When the user submits the job while the jobtracker is still initializing, the jobclient comes out with an exception. ideally the jobclient should keep on retrying until the jobtracker is up and ready. This will also take care of HADOOP-3289.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.