You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by "Sowmya Ramesh (JIRA)" <ji...@apache.org> on 2016/06/06 17:58:21 UTC
[jira] [Comment Edited] (FALCON-1894) HDFS Data replication cannot
be initiated independent of Oozie server location
[ https://issues.apache.org/jira/browse/FALCON-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313342#comment-15313342 ]
Sowmya Ramesh edited comment on FALCON-1894 at 6/6/16 5:57 PM:
---------------------------------------------------------------
[~abush]: Issue will be seen only if HDFS DR is initiated from Falcon UI as it doesn't use the logic in FalconClient.
Recipes submitted through Falcon CLI will not hit this issue.
Below code in HdfsReplicationRecipeTool takes care of generating the absolute path
{noformat}
// Construct fully qualified hdfs src path
String srcPaths = recipeProperties.getProperty(HdfsReplicationRecipeToolOptions
.REPLICATION_SOURCE_DIR.getName());
StringBuilder absoluteSrcPaths = new StringBuilder();
String srcFsPath = recipeProperties.getProperty(
HdfsReplicationRecipeToolOptions.REPLICATION_SOURCE_CLUSTER_FS_WRITE_ENDPOINT.getName());
if (StringUtils.isNotEmpty(srcFsPath)) {
srcFsPath = StringUtils.removeEnd(srcFsPath, File.separator);
}
if (StringUtils.isNotEmpty(srcPaths)) {
String[] paths = srcPaths.split(COMMA_SEPARATOR);
for (String path : paths) {
StringBuilder srcpath = new StringBuilder(srcFsPath);
srcpath.append(path.trim());
srcpath.append(COMMA_SEPARATOR);
absoluteSrcPaths.append(srcpath);
}
}
{noformat}
Fix you have recommended will not work if user wants to replicate multiple comma separated directories.
Better workaround will be for the suer to provide the fully qualified source path to replicate instead of modifying the workflow template as below
{noformat}
drSourceDir=hftp://c6402.ambari.apache.org:50070/user/ambari-qa/test1/, hftp://c6402.ambari.apache.org:50070/user/ambari-qa/test2/
{noformat}
Full qualified paths should be specified only for drSourceDir and not for drTargetDir.
With https://issues.apache.org/jira/browse/FALCON-1107 recipe is renamed as extensions and processing is moved to server side. Logic to construct the fully qualified path is on the server side and hence will work for both UI and cmd line.
I am closing this as fixed as it will not be sen after 0.10 release.
Thanks!
was (Author: sowmyaramesh):
[~abush]: Issue will be seen only if HDFS DR is initiated from Falcon UI as it doesn't use the logic in FalconClient.
Recipes submitted through Falcon CLI will not hit this issue.
Below code in HdfsReplicationRecipeTool takes care of generating the absolute path
{noformat}
// Construct fully qualified hdfs src path
String srcPaths = recipeProperties.getProperty(HdfsReplicationRecipeToolOptions
.REPLICATION_SOURCE_DIR.getName());
StringBuilder absoluteSrcPaths = new StringBuilder();
String srcFsPath = recipeProperties.getProperty(
HdfsReplicationRecipeToolOptions.REPLICATION_SOURCE_CLUSTER_FS_WRITE_ENDPOINT.getName());
if (StringUtils.isNotEmpty(srcFsPath)) {
srcFsPath = StringUtils.removeEnd(srcFsPath, File.separator);
}
if (StringUtils.isNotEmpty(srcPaths)) {
String[] paths = srcPaths.split(COMMA_SEPARATOR);
for (String path : paths) {
StringBuilder srcpath = new StringBuilder(srcFsPath);
srcpath.append(path.trim());
srcpath.append(COMMA_SEPARATOR);
absoluteSrcPaths.append(srcpath);
}
}
{noformat}
Fix you have recommended will not work if user wants to replicate multiple comma separated directories.
Better workaround will be for the suer to provide the fully qualified source path to replicate instead of modifying the workflow template as below
{noformat}
drSourceDir=hftp://c6402.ambari.apache.org:50070/user/ambari-qa/test1/, hftp://c6402.ambari.apache.org:50070/user/ambari-qa/test2/
{noformat}
With https://issues.apache.org/jira/browse/FALCON-1107 recipe is renamed as extensions and processing is moved to server side. Logic to construct the fully qualified path is on the server side and hence will work for both UI and cmd line.
I am closing this as fixed as it will not be sen after 0.10 release.
Thanks!
> HDFS Data replication cannot be initiated independent of Oozie server location
> ------------------------------------------------------------------------------
>
> Key: FALCON-1894
> URL: https://issues.apache.org/jira/browse/FALCON-1894
> Project: Falcon
> Issue Type: Bug
> Components: general
> Affects Versions: trunk
> Reporter: Alex Bush
> Assignee: Sowmya Ramesh
> Priority: Minor
> Fix For: trunk, 0.10
>
>
> The HDFS mirroring scripts allow replication between two clusters.
> Currently, even though the UI allows the replication in any direction between clusters independent of which cluster the Falcon and Oozie servers belong to this is not observed and the source cluster is always the cluster with Oozie/Falcon server.
> Steps to reproduce:
> 1) Define both clusters in Falcon server on cluster 2
> 2) Set up HDFS mirroring in Falcon server on cluster 2 from Cluster 1 to Cluster 2 and set to run on Oozie server of Cluster 2
> Result:
> Falcon will replicate data from Cluster 2 to Cluster 2
> Cause:
> In hdfs-replication-workflow.xml, the source dir should be defined like the target dir by including clusterfs:
> https://github.com/apache/falcon/blob/master/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml#L63
> <arg>${drSourceDir}</arg>
> should be
> <arg>${drSourceClusterFS}${drSourceDir}</arg>
>
> like
> https://github.com/apache/falcon/blob/master/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml#L65
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)