You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Deshui Yu (JIRA)" <ji...@apache.org> on 2019/02/27 10:07:00 UTC
[jira] [Comment Edited] (HDFS-14316) RBF: Support unavailable
subclusters for mount points with multiple destinations
[ https://issues.apache.org/jira/browse/HDFS-14316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779089#comment-16779089 ]
Deshui Yu edited comment on HDFS-14316 at 2/27/19 10:06 AM:
------------------------------------------------------------
Hi, [~elgoiri], in RouterClientProtocol.java, line 238~251:
238 try {
239 RemoteLocation createLocation = rpcServer.getCreateLocation(src);
240 return (HdfsFileStatus) rpcClient.invokeSingle(createLocation, method);
241 } catch (Exception e) {
242 if (!rpcServer.isPathFaultTolerant(src)) {
243 throw e;
244 }
245 LOG.error("Cannot create {}: {}", src, e.getMessage());
246 List<RemoteLocation> locations =
247 rpcServer.getLocationsForPath(src, true);
248 LOG.info("{} supports failed subclusters, trying: {}", src, locations);
249 return rpcClient.invokeSequential(
250 locations, method, HdfsFileStatus.class, null);
251 }
We use try-fail-retry mode to handle subcluster failure and continue to write in next available(maybe) subcluster. Here're some concerns from my perspective:
1. Can we make sure the RemoteLocation list returned in line 246 will not contain the failed subcluster anymore (especially in Random Order), or another failed subcluster? If not, maybe line 249 will also throw exception in catch{} clause.
2. try-fail-retry mode is straight-forward and simple to implement, but the latency is increased once a subcluster fail since the create operation requires 2-rounds rpc call to namenodes. Since router has a global view about the availability of subclusters, can we utilize this global knowledge to skip the failed subcluster rather than try-fail-retry?
was (Author: fishyds):
Hi, [~elgoiri], in RouterClientProtocol.java, line 238~251:
238 try {
239 RemoteLocation createLocation = rpcServer.getCreateLocation(src);
240 return (HdfsFileStatus) rpcClient.invokeSingle(createLocation, method);
241 } catch (Exception e) {
242 if (!rpcServer.isPathFaultTolerant(src)) {
243 throw e;
244 }
245 LOG.error("Cannot create {}: {}", src, e.getMessage());
246 List<RemoteLocation> locations =
247 rpcServer.getLocationsForPath(src, true);
248 LOG.info("{} supports failed subclusters, trying: {}", src, locations);
249 return rpcClient.invokeSequential(
250 locations, method, HdfsFileStatus.class, null);
251 }
We use try-fail-retry mode to handle subcluster failure and continue to write in next available(maybe) subcluster. Here're some concerns from my perspective:
1. Can we make sure the RemoteLocation list returned in line 246 will not contain the failed subcluster anymore (especially in Random Order), or another failed subcluster? If not, maybe line 249 will also throw exception in catch{} clause.
2. try-fail-retry mode is straight-forward and simple to implement, but the latency is increased once a subclass fail since the create operation requires 2-rounds rpc call to namenodes. Since router has a global view about the availability of subclusters, can we utilize this global knowledge to skip the failed subcluster rather than try-fail-retry?
> RBF: Support unavailable subclusters for mount points with multiple destinations
> --------------------------------------------------------------------------------
>
> Key: HDFS-14316
> URL: https://issues.apache.org/jira/browse/HDFS-14316
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Íñigo Goiri
> Assignee: Íñigo Goiri
> Priority: Major
> Attachments: HDFS-14316-HDFS-13891.000.patch, HDFS-14316-HDFS-13891.001.patch
>
>
> Currently mount points with multiple destinations (e.g., HASH_ALL) fail writes when the destination subcluster is down. We need an option to allow writing in other subclusters when one is down.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org