[jira] [Commented] (FALCON-107) Adding extensions

["Srikanth Sundarrajan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FALCON-107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768112#comment-13768112 ] 

Srikanth Sundarrajan commented on FALCON-107:
---------------------------------------------

Perhaps we should call this path.libext....., as they are paths and it is better to be explicit
{code}
+debug.libext.feed.retention=${falcon.libext}
{code}

It might make sense to consider other non-standard actions which might similarly include lib files such as mr/pig/java. 
{code}
+            for(Object obj: wf.getDecisionOrForkOrJoin()) {
+                if(!(obj instanceof ACTION))
+                    continue;
+                ACTION action = (ACTION) obj;
+                List<String> files = null;
+                if (action.getJava() != null) {
+                    files = action.getJava().getFile();
+                } else if (action.getPig() != null) {
+                    files = action.getPig().getFile();
+                } else if(action.getMapReduce() != null) {
+                    files = action.getMapReduce().getFile();
+                }
+                if (files != null) {
+                    files.add(lib.getPath().toString());
+                }
+            }
{code}

Signature can be modified to source,target paths instead of local & cluster. That would be more explicit and be more descriptive.
{code}
+    public static void pushLibsToHDFS(String localPaths, Path clusterPath, Cluster cluster, FalconPathFilter pathFilter)
{code}

Increasingly there is case for lifecycle to be handled explicitly. [~svenkat] was attempting to add this as part of FALCON-36, perhaps it might be better to expedite this, as work on that has been parked right now. I would like schedulable entities and their lifecycles to be enumerated and libs pushed. May be we can move this to a separate JIRA.
{code}
+            pushLibsToHDFS(properties.getProperty("libext"), libext, cluster, null);
+            pushLibsToHDFS(properties.getProperty("libext.feed"), new Path(libext, EntityType.FEED.name()) , cluster, null);
+            pushLibsToHDFS(properties.getProperty("libext.feed.replication"), new Path(libext, EntityType.FEED.name() + "/replication"), cluster, null);
+            pushLibsToHDFS(properties.getProperty("libext.feed.retention"), new Path(libext, EntityType.FEED.name() + "/retention"), cluster, null);
+            pushLibsToHDFS(properties.getProperty("libext.process"), new Path(libext, EntityType.PROCESS.name()) , cluster, null);
{code}

* Would like to see an assertion in EntityManagerJerseyIT to ensure that extension libs are indeed copied to cluster post submit
* More tests (of Mappers) to ensure that extension libs are indeed included in workflows would be nice.
                
> Adding extensions
> -----------------
>
>                 Key: FALCON-107
>                 URL: https://issues.apache.org/jira/browse/FALCON-107
>             Project: Falcon
>          Issue Type: New Feature
>            Reporter: Shwetha G S
>            Assignee: Shwetha G S
>             Fix For: 0.4
>
>         Attachments: FALCON-107-v1.patch
>
>
> Falcon should support ability to add extensions like new filesystems(if not part of hadoop release). In general, to add extra libraries that are available for falcon service, and feed/process workflows

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira