[jira] [Updated] (CRUNCH-46) Scrunch jobs launched from repl using PipelineLike#done are not shipped with jar of repl code.

["Kiyan Ahmadizadeh (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CRUNCH-46?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kiyan Ahmadizadeh updated CRUNCH-46:
------------------------------------

    Attachment: CRUNCH-46.patch

This patch fixes the problem by adding code to PipelineLike#done that ships the repl jar with the job.
                
> Scrunch jobs launched from repl using PipelineLike#done are not shipped with jar of repl code.
> ----------------------------------------------------------------------------------------------
>
>                 Key: CRUNCH-46
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-46
>             Project: Crunch
>          Issue Type: Bug
>            Reporter: Kiyan Ahmadizadeh
>         Attachments: CRUNCH-46.patch
>
>
> Suppose the following example code is run in the scrunch/scala repl:
> val pipeline = Pipeline()
> val textLines = pipeline.read(From.textFile("shakes.txt"))
> val alphaNumericTextLines = textLines.map(line => line.toLowerCase().replaceAll("[^A-Za-z ]", ""))
> val words = alphaNumericTextLines.flatMap(line => line.split("""\W+"""))
> counts = words.count()
> counts.write(To.textFile("/user/kiyan/counts"))
> pipeline.done()
>  
> This code results in a ClassNotFoundException in MapReduce tasks.  However, changing the last line to pipeline.run() produces no errors.
> The problem is that the method org.apache.crunch.scrunch.PipelineLike#run adds a jar of generated repl code to the job's running tasks, but org.apache.crunch.scrunch.PipelineLike#done does not.  The done method should be modified to take the same actions as the run method when launching a job from the Scala repl.  This will ensure users can launch jobs from the repl regardless of how they conclude their pipelines.
>  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira