You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Uma Maheswara Rao G (Created) (JIRA)" <ji...@apache.org> on 2011/12/21 16:29:30 UTC
[jira] [Created] (HDFS-2713) HA : An alternative approach to
clients handling Namenode failover.
HA : An alternative approach to clients handling Namenode failover.
--------------------------------------------------------------------
Key: HDFS-2713
URL: https://issues.apache.org/jira/browse/HDFS-2713
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ha, hdfs client
Affects Versions: HA branch (HDFS-1623)
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
This is the approach for client failover which we adopted when we developed HA for Hadoop. I would like to propose thia approach for others to review & include in the HA implementation, if found useful.
This is similar to the ConfiguredProxyProvider in the sense that the it takes the address of both the Namenodes as the input. The major differences I can see from the current implementation are
1) During failover, user threads can be controlled very accurately about *the time they wait for active namenode* to be available, awaiting the retry. Beyond this, the threads will not be made to wait; DFS Client will throw an Exception indicating that the operation has failed.
2) Failover happens in a seperate thread, not in the client application threads. The thread will keep trying to find the Active Namenode until it succeeds.
3) This also means that irrespective of whether the operation's RetryAction is RETRY_FAILOVER or FAIL, the user thread can trigger the client's failover.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2713) HA : An alternative approach to
clients handling Namenode failover.
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191791#comment-13191791 ]
Aaron T. Myers commented on HDFS-2713:
--------------------------------------
Hey Uma, can we either close this JIRA, or re-purpose it as I previously suggested? Thanks a lot.
> HA : An alternative approach to clients handling Namenode failover.
> --------------------------------------------------------------------
>
> Key: HDFS-2713
> URL: https://issues.apache.org/jira/browse/HDFS-2713
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, hdfs client
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
>
> This is the approach for client failover which we adopted when we developed HA for Hadoop. I would like to propose thia approach for others to review & include in the HA implementation, if found useful.
> This is similar to the ConfiguredProxyProvider in the sense that the it takes the address of both the Namenodes as the input. The major differences I can see from the current implementation are
> 1) During failover, user threads can be controlled very accurately about *the time they wait for active namenode* to be available, awaiting the retry. Beyond this, the threads will not be made to wait; DFS Client will throw an Exception indicating that the operation has failed.
> 2) Failover happens in a seperate thread, not in the client application threads. The thread will keep trying to find the Active Namenode until it succeeds.
> 3) This also means that irrespective of whether the operation's RetryAction is RETRY_FAILOVER or FAIL, the user thread can trigger the client's failover.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2713) HA : An alternative approach to
clients handling Namenode failover.
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174148#comment-13174148 ]
Uma Maheswara Rao G commented on HDFS-2713:
-------------------------------------------
While writing a file if Active Namenode goes down, the thread cannot wait forever until the Standby Namenode gets fully functional as Active. We place a maximum limit on how long the user thread can wait for a Namenode to be available. For futher discussions, can term this limit as namenodeAwaitTimeout. If Namenode is not available even after namenodeAwaitTimeout, DFS Client will throw an Exception back to the caller.
Corresponding to Namenode pair, there is a HASwitchAgent which is responsible for managing connections to multiple Namenode addresses configured. The HASwitchAgent exposes a performFailover() API, to trigger the switching logic. During any DFS operation, if RetryInvocationHandler finds that Active Namenode is down, the performFailover() API will be invoked. Then the HASwitchAgent will execute the switching procedure in a seperate thread, by trying to connect to the two Namenode addresses configured. This search for finding the Active Namenode will continue until it successfully connects to an Active Namenode.
The thread which invoked the performFailover() API will go to a TIMED WAIT and will be notified when the Active Namenode is available. The WAIT will happen for the time configured as 'namenode.await.timeout'. Once this time has elapsed, an IOException will be thrown from performFailover() API of HASwitchAgent. The subsequent invocations of the same API may already find that the switching logic is in progress and just make the calling thread to wait for 'namenode.await.timeout'.
> HA : An alternative approach to clients handling Namenode failover.
> --------------------------------------------------------------------
>
> Key: HDFS-2713
> URL: https://issues.apache.org/jira/browse/HDFS-2713
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, hdfs client
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
>
> This is the approach for client failover which we adopted when we developed HA for Hadoop. I would like to propose thia approach for others to review & include in the HA implementation, if found useful.
> This is similar to the ConfiguredProxyProvider in the sense that the it takes the address of both the Namenodes as the input. The major differences I can see from the current implementation are
> 1) During failover, user threads can be controlled very accurately about *the time they wait for active namenode* to be available, awaiting the retry. Beyond this, the threads will not be made to wait; DFS Client will throw an Exception indicating that the operation has failed.
> 2) Failover happens in a seperate thread, not in the client application threads. The thread will keep trying to find the Active Namenode until it succeeds.
> 3) This also means that irrespective of whether the operation's RetryAction is RETRY_FAILOVER or FAIL, the user thread can trigger the client's failover.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2713) HA : An alternative approach to
clients handling Namenode failover.
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192370#comment-13192370 ]
Aaron T. Myers commented on HDFS-2713:
--------------------------------------
bq. We cached the active connection in client in our internal version. So, that in the same JVM, every new client need not perform failover.
Presumably with some static map or static reference to a client connection? That sort of design terrifies me, with the potential for subtle bugs that are tough to reproduce and write tests for. So, without really, really good justification, I'd be opposed to such a change.
bq. Do you think this will help us to solve below case in this HA proposal( atleast per JVM level) ?
Maybe, but only for a rather limited set of circumstances, i.e. when a single JVM has multiple DFS clients, and a failover has occurred during the lifetime of the JVM. It will do nothing for what I suspect will be a far more common case - a brand new JVM (e.g. `hadoop fs -ls') that has exactly one client connection to the NN, and has to try the first-listed NN before attempting a connection to the second.
I honestly believe that optimizing client side failover time to this degree will be mostly wasted effort. We already do a client failover immediately when an error is detected (i.e. we don't retry to the first-listed NN at all) so the slow-down should be largely unnoticeable for most users. Doing a client failover once per client connection per NN failover event just isn't that big a deal.
If you really want to try to optimize client failover, I strongly recommend you look into implementing "speculative RPC," i.e. for a given RPC call, try connecting to both in parallel and see which one returns first. Regardless of which NN is the active, one NN should either throw a StandbyException or other error, while the other NN will return a correct result, which the client library can return to the DFSClient. This seems way safer, and has the advantage of benefiting even the fresh JVM case.
> HA : An alternative approach to clients handling Namenode failover.
> --------------------------------------------------------------------
>
> Key: HDFS-2713
> URL: https://issues.apache.org/jira/browse/HDFS-2713
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, hdfs client
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
>
> This is the approach for client failover which we adopted when we developed HA for Hadoop. I would like to propose thia approach for others to review & include in the HA implementation, if found useful.
> This is similar to the ConfiguredProxyProvider in the sense that the it takes the address of both the Namenodes as the input. The major differences I can see from the current implementation are
> 1) During failover, user threads can be controlled very accurately about *the time they wait for active namenode* to be available, awaiting the retry. Beyond this, the threads will not be made to wait; DFS Client will throw an Exception indicating that the operation has failed.
> 2) Failover happens in a seperate thread, not in the client application threads. The thread will keep trying to find the Active Namenode until it succeeds.
> 3) This also means that irrespective of whether the operation's RetryAction is RETRY_FAILOVER or FAIL, the user thread can trigger the client's failover.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2713) HA : An alternative approach to
clients handling Namenode failover.
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192030#comment-13192030 ]
Uma Maheswara Rao G commented on HDFS-2713:
-------------------------------------------
Thanks a lot Aaron for your time. Will check the real improvements on the loads with this background thread once we release the initial HA, if it gives really benefit then i will upload patch later. For now i am closing it ( since background thread may make the the code little complex, don't want to include this in initial release).
> HA : An alternative approach to clients handling Namenode failover.
> --------------------------------------------------------------------
>
> Key: HDFS-2713
> URL: https://issues.apache.org/jira/browse/HDFS-2713
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, hdfs client
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
>
> This is the approach for client failover which we adopted when we developed HA for Hadoop. I would like to propose thia approach for others to review & include in the HA implementation, if found useful.
> This is similar to the ConfiguredProxyProvider in the sense that the it takes the address of both the Namenodes as the input. The major differences I can see from the current implementation are
> 1) During failover, user threads can be controlled very accurately about *the time they wait for active namenode* to be available, awaiting the retry. Beyond this, the threads will not be made to wait; DFS Client will throw an Exception indicating that the operation has failed.
> 2) Failover happens in a seperate thread, not in the client application threads. The thread will keep trying to find the Active Namenode until it succeeds.
> 3) This also means that irrespective of whether the operation's RetryAction is RETRY_FAILOVER or FAIL, the user thread can trigger the client's failover.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2713) HA : An alternative approach to
clients handling Namenode failover.
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192032#comment-13192032 ]
Uma Maheswara Rao G commented on HDFS-2713:
-------------------------------------------
Lastly i forgot to mention other point.i.e, We cached the active connection in client in our internal version. So, that in the same JVM, every new client need not perform failover. First client will perform failover and others will use the same proxy.
Do you think this will help us to solve below case in this HA proposal( atleast per JVM level) ?
{quote}
>From HA doc section 8.3
o Failover
time
is
longer
since
clients
always
talks
to
the
first
NN
(who
may
be
dead)
before
looking
up
new
address
of
NN.
{quote}
If yes, i will file separate JIRA and start work on that (also this point should not block initial release).
Am i missing something here?
> HA : An alternative approach to clients handling Namenode failover.
> --------------------------------------------------------------------
>
> Key: HDFS-2713
> URL: https://issues.apache.org/jira/browse/HDFS-2713
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, hdfs client
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
>
> This is the approach for client failover which we adopted when we developed HA for Hadoop. I would like to propose thia approach for others to review & include in the HA implementation, if found useful.
> This is similar to the ConfiguredProxyProvider in the sense that the it takes the address of both the Namenodes as the input. The major differences I can see from the current implementation are
> 1) During failover, user threads can be controlled very accurately about *the time they wait for active namenode* to be available, awaiting the retry. Beyond this, the threads will not be made to wait; DFS Client will throw an Exception indicating that the operation has failed.
> 2) Failover happens in a seperate thread, not in the client application threads. The thread will keep trying to find the Active Namenode until it succeeds.
> 3) This also means that irrespective of whether the operation's RetryAction is RETRY_FAILOVER or FAIL, the user thread can trigger the client's failover.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2713) HA : An alternative approach to
clients handling Namenode failover.
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174419#comment-13174419 ]
Aaron T. Myers commented on HDFS-2713:
--------------------------------------
bq. Yes , i agree existing implementation also supports.But the main difference is, lets take a case with existing implementation, after a finite period of time, if it is not able to get active node proxy instance it will throw exception and fail. After some time if other call comes, then again it will do failover. But with my proposed one, Background thread will continue failover indefinitely until it finds active proxy instance. So, by the time next call come this background thread may make ready of active node proxy.
I still don't see what benefit the background thread has. In the case you describe, with the current implementation, the second client request (after the failed one which had timed out retrying/failing over) would just simply succeed, or fail over immediately and then succeed. So, the background thread won't have saved much if any work, and instead may indefinitely be doing (potentially unnecessary) work in the background.
What am I missing?
> HA : An alternative approach to clients handling Namenode failover.
> --------------------------------------------------------------------
>
> Key: HDFS-2713
> URL: https://issues.apache.org/jira/browse/HDFS-2713
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, hdfs client
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
>
> This is the approach for client failover which we adopted when we developed HA for Hadoop. I would like to propose thia approach for others to review & include in the HA implementation, if found useful.
> This is similar to the ConfiguredProxyProvider in the sense that the it takes the address of both the Namenodes as the input. The major differences I can see from the current implementation are
> 1) During failover, user threads can be controlled very accurately about *the time they wait for active namenode* to be available, awaiting the retry. Beyond this, the threads will not be made to wait; DFS Client will throw an Exception indicating that the operation has failed.
> 2) Failover happens in a seperate thread, not in the client application threads. The thread will keep trying to find the Active Namenode until it succeeds.
> 3) This also means that irrespective of whether the operation's RetryAction is RETRY_FAILOVER or FAIL, the user thread can trigger the client's failover.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2713) HA : An alternative approach to
clients handling Namenode failover.
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174862#comment-13174862 ]
Uma Maheswara Rao G commented on HDFS-2713:
-------------------------------------------
Thanks a lot, Aaron for the patience in review. :-)
{quote}
still don't see what benefit the background thread has. In the case you describe, with the current implementation, the second client request (after the failed one which had timed out retrying/failing over) would just simply succeed, or fail over immediately and then succeed. So, the background thread won't have saved much if any work, and instead may indefinitely be doing (potentially unnecessary) work in the background.
{quote}
After any DFSClient operation fails due to Namenode unavailability, the most important thing to do is to detect when the Active Namenode becomes available again.
So the background thread is not doing any unnecessary work, it is doing the high priority work.
The difference in our approaches is that, the importance given to the failover till it succeeds.
In the approch I described, it is considered very important in the sense that one thread is dedicated to find the Active Namenode and after finding only it will exit.
If the RetryDecision is FAILOVER_AND_RETRY,then only the failover is done.
If there are many issued to Namenode whose RetryDecision is FAIL, failover won't happen.
My intention is that, when one client call finds failover is required and not able to complete the failover within the wait time, then why do I need to wait till next call comes to try again and failover after mindealy wait?
Even though the first call fails, this background thread will ensure to find the active proxy instance. If next call comes now(this is user thread), it need not wait to connect and failover again. Immediately it can make use of that proxy instance and goahead.
I will try to integrate the logic with *ConfiguredFailverProxyProvider* and upload a temp patch for more understanding.
Thanks
Uma
> HA : An alternative approach to clients handling Namenode failover.
> --------------------------------------------------------------------
>
> Key: HDFS-2713
> URL: https://issues.apache.org/jira/browse/HDFS-2713
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, hdfs client
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
>
> This is the approach for client failover which we adopted when we developed HA for Hadoop. I would like to propose thia approach for others to review & include in the HA implementation, if found useful.
> This is similar to the ConfiguredProxyProvider in the sense that the it takes the address of both the Namenodes as the input. The major differences I can see from the current implementation are
> 1) During failover, user threads can be controlled very accurately about *the time they wait for active namenode* to be available, awaiting the retry. Beyond this, the threads will not be made to wait; DFS Client will throw an Exception indicating that the operation has failed.
> 2) Failover happens in a seperate thread, not in the client application threads. The thread will keep trying to find the Active Namenode until it succeeds.
> 3) This also means that irrespective of whether the operation's RetryAction is RETRY_FAILOVER or FAIL, the user thread can trigger the client's failover.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2713) HA : An alternative approach to
clients handling Namenode failover.
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174366#comment-13174366 ]
Uma Maheswara Rao G commented on HDFS-2713:
-------------------------------------------
Thanks a lot, Aaron for taking a look.
{quote}
1) During failover, user threads can be controlled very accurately about the time they wait for active namenode to be available, awaiting the retry. Beyond this, the threads will not be made to wait; DFS Client will throw an Exception indicating that the operation has failed.
The current system already supports this. Clients will failover and retry with some random, exponential backoff for a finite period of time, after which the operation will fail, throwing an exception.
{quote}
Yes , i agree existing implementation also supports.But the main difference is, lets take a case with existing implementation, after a finite period of time, if it is not able to get active node proxy instance it will throw exception and fail. After some time if other call comes, then again it will do failover. But with my proposed one, Background thread will continue failover indefinitely until it finds active proxy instance. So, by the time next call come this background thread may make ready of active node proxy.
This is the main difference i wanted to explain.
{quote}
The other thing that's not clear to me is how you'd propose to incorporate it into HDFS. Would it be an alternative to the current implementation? Or done as an enhancement to the current implementation?
{quote}
here two ways, one way is to enhance the existing ConfiguredFailOverProxyProvider implementation to incorporate the this proposal. (separating the failover logic into separate background thread)
Other way is to keep the new proposal as separete FailOverProxyProvider implementation.
Which one is preferable for you?
> HA : An alternative approach to clients handling Namenode failover.
> --------------------------------------------------------------------
>
> Key: HDFS-2713
> URL: https://issues.apache.org/jira/browse/HDFS-2713
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, hdfs client
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
>
> This is the approach for client failover which we adopted when we developed HA for Hadoop. I would like to propose thia approach for others to review & include in the HA implementation, if found useful.
> This is similar to the ConfiguredProxyProvider in the sense that the it takes the address of both the Namenodes as the input. The major differences I can see from the current implementation are
> 1) During failover, user threads can be controlled very accurately about *the time they wait for active namenode* to be available, awaiting the retry. Beyond this, the threads will not be made to wait; DFS Client will throw an Exception indicating that the operation has failed.
> 2) Failover happens in a seperate thread, not in the client application threads. The thread will keep trying to find the Active Namenode until it succeeds.
> 3) This also means that irrespective of whether the operation's RetryAction is RETRY_FAILOVER or FAIL, the user thread can trigger the client's failover.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2713) HA : An alternative approach to
clients handling Namenode failover.
Posted by "Uma Maheswara Rao G (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G resolved HDFS-2713.
---------------------------------------
Resolution: Won't Fix
> HA : An alternative approach to clients handling Namenode failover.
> --------------------------------------------------------------------
>
> Key: HDFS-2713
> URL: https://issues.apache.org/jira/browse/HDFS-2713
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, hdfs client
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
>
> This is the approach for client failover which we adopted when we developed HA for Hadoop. I would like to propose thia approach for others to review & include in the HA implementation, if found useful.
> This is similar to the ConfiguredProxyProvider in the sense that the it takes the address of both the Namenodes as the input. The major differences I can see from the current implementation are
> 1) During failover, user threads can be controlled very accurately about *the time they wait for active namenode* to be available, awaiting the retry. Beyond this, the threads will not be made to wait; DFS Client will throw an Exception indicating that the operation has failed.
> 2) Failover happens in a seperate thread, not in the client application threads. The thread will keep trying to find the Active Namenode until it succeeds.
> 3) This also means that irrespective of whether the operation's RetryAction is RETRY_FAILOVER or FAIL, the user thread can trigger the client's failover.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2713) HA : An alternative approach to
clients handling Namenode failover.
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174347#comment-13174347 ]
Aaron T. Myers commented on HDFS-2713:
--------------------------------------
Hey Uma, there are several things I don't understand about this proposal.
bq. 1) During failover, user threads can be controlled very accurately about the time they wait for active namenode to be available, awaiting the retry. Beyond this, the threads will not be made to wait; DFS Client will throw an Exception indicating that the operation has failed.
The current system already supports this. Clients will failover and retry with some random, exponential backoff for a finite period of time, after which the operation will fail, throwing an exception.
bq. 2) Failover happens in a seperate thread, not in the client application threads. The thread will keep trying to find the Active Namenode until it succeeds.
What's the point of doing this in a separate thread? Given that client operations still block while the failover is attempted, it doesn't seem like this difference will be tangible to the user.
3) This also means that irrespective of whether the operation's RetryAction is RETRY_FAILOVER or FAIL, the user thread can trigger the client's failover.
This confuses me. How does this work?
In short, this proposal just seems _different_ and not necessarily _better_ than the current implementation. This implementation also seems like a more complex design to me, so without tangible user benefits I don't see much point in doing it.
The other thing that's not clear to me is how you'd propose to incorporate it into HDFS. Would it be an alternative to the current implementation? Or done as an enhancement to the current implementation?
> HA : An alternative approach to clients handling Namenode failover.
> --------------------------------------------------------------------
>
> Key: HDFS-2713
> URL: https://issues.apache.org/jira/browse/HDFS-2713
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, hdfs client
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
>
> This is the approach for client failover which we adopted when we developed HA for Hadoop. I would like to propose thia approach for others to review & include in the HA implementation, if found useful.
> This is similar to the ConfiguredProxyProvider in the sense that the it takes the address of both the Namenodes as the input. The major differences I can see from the current implementation are
> 1) During failover, user threads can be controlled very accurately about *the time they wait for active namenode* to be available, awaiting the retry. Beyond this, the threads will not be made to wait; DFS Client will throw an Exception indicating that the operation has failed.
> 2) Failover happens in a seperate thread, not in the client application threads. The thread will keep trying to find the Active Namenode until it succeeds.
> 3) This also means that irrespective of whether the operation's RetryAction is RETRY_FAILOVER or FAIL, the user thread can trigger the client's failover.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2713) HA : An alternative approach to
clients handling Namenode failover.
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174390#comment-13174390 ]
Uma Maheswara Rao G commented on HDFS-2713:
-------------------------------------------
{quote}
IMO it seems preferable to enhance (or replace) the existing code rather than introduce a new option.
{quote}
Make sense to me.
> HA : An alternative approach to clients handling Namenode failover.
> --------------------------------------------------------------------
>
> Key: HDFS-2713
> URL: https://issues.apache.org/jira/browse/HDFS-2713
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, hdfs client
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
>
> This is the approach for client failover which we adopted when we developed HA for Hadoop. I would like to propose thia approach for others to review & include in the HA implementation, if found useful.
> This is similar to the ConfiguredProxyProvider in the sense that the it takes the address of both the Namenodes as the input. The major differences I can see from the current implementation are
> 1) During failover, user threads can be controlled very accurately about *the time they wait for active namenode* to be available, awaiting the retry. Beyond this, the threads will not be made to wait; DFS Client will throw an Exception indicating that the operation has failed.
> 2) Failover happens in a seperate thread, not in the client application threads. The thread will keep trying to find the Active Namenode until it succeeds.
> 3) This also means that irrespective of whether the operation's RetryAction is RETRY_FAILOVER or FAIL, the user thread can trigger the client's failover.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2713) HA : An alternative approach to
clients handling Namenode failover.
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174317#comment-13174317 ]
Uma Maheswara Rao G commented on HDFS-2713:
-------------------------------------------
Hi Todd/ ATM, Before moving, i wanted to know your opinions on this proposal.
> HA : An alternative approach to clients handling Namenode failover.
> --------------------------------------------------------------------
>
> Key: HDFS-2713
> URL: https://issues.apache.org/jira/browse/HDFS-2713
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, hdfs client
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
>
> This is the approach for client failover which we adopted when we developed HA for Hadoop. I would like to propose thia approach for others to review & include in the HA implementation, if found useful.
> This is similar to the ConfiguredProxyProvider in the sense that the it takes the address of both the Namenodes as the input. The major differences I can see from the current implementation are
> 1) During failover, user threads can be controlled very accurately about *the time they wait for active namenode* to be available, awaiting the retry. Beyond this, the threads will not be made to wait; DFS Client will throw an Exception indicating that the operation has failed.
> 2) Failover happens in a seperate thread, not in the client application threads. The thread will keep trying to find the Active Namenode until it succeeds.
> 3) This also means that irrespective of whether the operation's RetryAction is RETRY_FAILOVER or FAIL, the user thread can trigger the client's failover.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2713) HA : An alternative approach to
clients handling Namenode failover.
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174980#comment-13174980 ]
Aaron T. Myers commented on HDFS-2713:
--------------------------------------
{quote}
After any DFSClient operation fails due to Namenode unavailability, the most important thing to do is to detect when the Active Namenode becomes available again.
So the background thread is not doing any unnecessary work, it is doing the high priority work.
{quote}
That's not necessarily true. It's only important *if* the DFSClient will indeed be used later for another client operation after some client operation has timed out. If it's not reused, then any work the background thread has done will in fact have been unnecessary. I would guess (pure conjecture) that most client programs will not be structured this way. That is, once a single call times out for a given DFSClient, that DFSClient will not be reused.
{quote}
My intention is that, when one client call finds failover is required and not able to complete the failover within the wait time, then why do I need to wait till next call comes to try again and failover after mindealy wait?
Even though the first call fails, this background thread will ensure to find the active proxy instance. If next call comes now(this is user thread), it need not wait to connect and failover again. Immediately it can make use of that proxy instance and goahead.
{quote}
But the delay on the subsequent call should be minimal (less than a second), assuming the NN is indeed back up. Given that failover of the NN will likely take tens of seconds, this particular aspect of client failover doesn't seem to me like it's something worth optimizing aggressively.
Two things:
# This isn't so much "an alternative approach to clients handling NN failover" as it is a potential performance improvement for subsequent calls after a failed call due to the active NN being down. Would you mind changing the JIRA summary and description to better reflect this?
# Given that it's a performance improvement, could you provide some benchmarks? Do you have an actual workload which benefits from this change?
> HA : An alternative approach to clients handling Namenode failover.
> --------------------------------------------------------------------
>
> Key: HDFS-2713
> URL: https://issues.apache.org/jira/browse/HDFS-2713
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, hdfs client
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
>
> This is the approach for client failover which we adopted when we developed HA for Hadoop. I would like to propose thia approach for others to review & include in the HA implementation, if found useful.
> This is similar to the ConfiguredProxyProvider in the sense that the it takes the address of both the Namenodes as the input. The major differences I can see from the current implementation are
> 1) During failover, user threads can be controlled very accurately about *the time they wait for active namenode* to be available, awaiting the retry. Beyond this, the threads will not be made to wait; DFS Client will throw an Exception indicating that the operation has failed.
> 2) Failover happens in a seperate thread, not in the client application threads. The thread will keep trying to find the Active Namenode until it succeeds.
> 3) This also means that irrespective of whether the operation's RetryAction is RETRY_FAILOVER or FAIL, the user thread can trigger the client's failover.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2713) HA : An alternative approach to
clients handling Namenode failover.
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174373#comment-13174373 ]
Todd Lipcon commented on HDFS-2713:
-----------------------------------
IMO it seems preferable to enhance (or replace) the existing code rather than introduce a new option. There's no sense in supporting both if one has clear advantages.
If it's easier to write as new code, though, we could implement it as a new provider, then remove the old one when it gets committed.
> HA : An alternative approach to clients handling Namenode failover.
> --------------------------------------------------------------------
>
> Key: HDFS-2713
> URL: https://issues.apache.org/jira/browse/HDFS-2713
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, hdfs client
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
>
> This is the approach for client failover which we adopted when we developed HA for Hadoop. I would like to propose thia approach for others to review & include in the HA implementation, if found useful.
> This is similar to the ConfiguredProxyProvider in the sense that the it takes the address of both the Namenodes as the input. The major differences I can see from the current implementation are
> 1) During failover, user threads can be controlled very accurately about *the time they wait for active namenode* to be available, awaiting the retry. Beyond this, the threads will not be made to wait; DFS Client will throw an Exception indicating that the operation has failed.
> 2) Failover happens in a seperate thread, not in the client application threads. The thread will keep trying to find the Active Namenode until it succeeds.
> 3) This also means that irrespective of whether the operation's RetryAction is RETRY_FAILOVER or FAIL, the user thread can trigger the client's failover.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira