[jira] [Updated] (OOZIE-2) Oozie 'move' fs action is inconsistent

["Mona Chitnis (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/OOZIE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mona Chitnis updated OOZIE-2:
-----------------------------


For the patch committed in trunk with this JIRA ID (also available under the same JIRA ID in Apache ReviewBoard system), I grant license to ASF for inclusion in ASF works (as per the Apache License §5) 
                
> Oozie 'move' fs action is inconsistent
> --------------------------------------
>
>                 Key: OOZIE-2
>                 URL: https://issues.apache.org/jira/browse/OOZIE-2
>             Project: Oozie
>          Issue Type: Improvement
>            Reporter: Mona Chitnis
>              Labels: oozie
>             Fix For: pre-Apache
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> > Oozie 'move' fs action is inconsistent
> > --------------------------------------
> >
> >                 Key: OOZIE-133
> >                 URL: http://h12.grid.sp2.yahoo.net/browse/OOZIE-133
> >             Project: oozie
> >          Issue Type: New Feature
> >          Components: workflow
> >    Affects Versions: 3.0.2
> >            Reporter: Mona Chitnis
> >            Assignee: Oozie
> >   Original Estimate: 1 week
> >  Remaining Estimate: 1 week
> >
> > I'm using the 'move' fs action and I first got the following error:
> > FS001: Missing scheme in path [/projects/ngdstone/user/ogg_oozie/intermediate/tmp_price_feats_uniq/.pig_header]
> > when I had the following in my workflow.xml :
> > <fs>
> >         <move source='${OUT}/intermediate/tmp_price_feats_uniq/.pig_header'
> > target='${OUT}/intermediate/tmp_predict_supply_feats/'/>
> >         <move source='${OUT}/intermediate/tmp_price_feats_uniq/.pig_header'
> > target='${OUT}/intermediate/tmp_predict_supply_feats/'/>
> > </fs>
> > I then prefixed the namenode URI to the paths (like I did for the <prepare> paths), as such:
> > <fs>
> >         <move source='${nameNode}${OUT}/intermediate/tmp_price_feats_uniq/.pig_header'
> > target='${nameNode}${OUT}/intermediate/tmp_predict_supply_feats/'/>
> >         <move source='${nameNode}${OUT}/intermediate/tmp_price_feats_uniq/.pig_header'
> > target='${nameNode}${OUT}/intermediate/tmp_predict_supply_feats/'/>
> > </fs>
> > However, I now get this error:
> > FS003: Scheme [hdfs] not allowed in path
> > [hdfs://mithrilblue-nn1.blue.ygrid.yahoo.com:8020/projects/ngdstone/user/ogg_oozie/intermediate/tmp_predict_supply_feats]
> > it seems the 'scheme' is only needed for the source path, but not the target.  This is inconsistent.
> > Finally, if the source path is a file and the target path is a directory, Oozie will complain that the target already
> > exists.  I feel it should be consistent with the Hadoop CLI (and Unix) and simply understand that the source should be
> > placed under the target directory.
> --
>      [ http://h12.grid.sp2.yahoo.net/browse/OOZIE-133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
> Alejandro Abdelnur resolved OOZIE-133.
> --------------------------------------
>     Resolution: Won't Fix
> This is not an bug, it is like that by design. The reason is to make clear it is a move within the current filesystem; no actual data movement.
> --
> Mona Chitnis commented on OOZIE-133:
> ------------------------------------
> There are two parts to this issue.
> 1. target not to mention scheme
> 2. if target is an existing directory, exception thrown
> For part 1, if the target does include scheme, we can allow it but only if target's hdfs scheme is the same as source's (since move essentially incorporates a hadoop fs rename). This way users who have typed source and target paths both having the namenode parameter for the sake of consistency, do not face an exception.
> For part 2, hadoop can care of placing the source dir or file as a child of the target dir, if target dir exists. Is there any reason why oozie should not be consistent with this?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira