You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Jeremy Custenborder (JIRA)" <ji...@apache.org> on 2011/08/01 17:28:09 UTC

[jira] [Created] (PIG-2198) pig.additional.jars does not support s3 paths

pig.additional.jars does not support s3 paths
---------------------------------------------

                 Key: PIG-2198
                 URL: https://issues.apache.org/jira/browse/PIG-2198
             Project: Pig
          Issue Type: Bug
    Affects Versions: 0.8.1
            Reporter: Jeremy Custenborder
            Priority: Minor


The delimiter for additional jars is ":" which is part of the uri for a S3 path. 

*Repro*
Add the following to the command line.
{quote}
-Dpig.additional.jars=s3://bucket/udf.jar
{quote}

{quote}
Error before Pig is launched
----------------------------
ERROR 101: Could not copy s3 to local destination /mnt/var/lib/hadoop/steps/1/tmp/s3

org.apache.pig.backend.executionengine.ExecException: ERROR 4010: Failed to register jar :s3. Caught exception.
	at org.apache.pig.PigServer.addJarsFromProperties(PigServer.java:253)
	at org.apache.pig.PigServer.<init>(PigServer.java:239)
	at org.apache.pig.PigServer.<init>(PigServer.java:215)
	at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
	at org.apache.pig.Main.run(Main.java:492)
	at org.apache.pig.Main.main(Main.java:107)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 101: Could not copy s3 to local destination /mnt/var/lib/hadoop/steps/1/tmp/s3
	at org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:765)
	at org.apache.pig.PigServer.registerJar(PigServer.java:483)
	at org.apache.pig.PigServer.addJarsFromProperties(PigServer.java:248)
	... 10 more
Caused by: java.io.FileNotFoundException: File does not exist: s3
	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:457)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1256)
	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1237)
	at org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:763)
	... 12 more
================================================================================
{quote} 






--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (PIG-2198) pig.additional.jars does not support s3 paths

Posted by "Jeremy Custenborder (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073983#comment-13073983 ] 

Jeremy Custenborder commented on PIG-2198:
------------------------------------------

Good catch. I have no clue why I thought of ;. I was thinking about using the same delimiter that is used for FileInputFormat.addInputPaths which is actually a comma. I think this parameter will be much more useful with the support for dfs locations. 

> pig.additional.jars does not support s3 paths
> ---------------------------------------------
>
>                 Key: PIG-2198
>                 URL: https://issues.apache.org/jira/browse/PIG-2198
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.8.1
>            Reporter: Jeremy Custenborder
>            Priority: Minor
>         Attachments: PIG-2198.patch
>
>
> The delimiter for additional jars is ":" which is part of the uri for a S3 path. 
> *Repro*
> Add the following to the command line.
> {quote}
> -Dpig.additional.jars=s3://bucket/udf.jar
> {quote}
> {quote}
> Error before Pig is launched
> ----------------------------
> ERROR 101: Could not copy s3 to local destination /mnt/var/lib/hadoop/steps/1/tmp/s3
> org.apache.pig.backend.executionengine.ExecException: ERROR 4010: Failed to register jar :s3. Caught exception.
> 	at org.apache.pig.PigServer.addJarsFromProperties(PigServer.java:253)
> 	at org.apache.pig.PigServer.<init>(PigServer.java:239)
> 	at org.apache.pig.PigServer.<init>(PigServer.java:215)
> 	at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
> 	at org.apache.pig.Main.run(Main.java:492)
> 	at org.apache.pig.Main.main(Main.java:107)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 101: Could not copy s3 to local destination /mnt/var/lib/hadoop/steps/1/tmp/s3
> 	at org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:765)
> 	at org.apache.pig.PigServer.registerJar(PigServer.java:483)
> 	at org.apache.pig.PigServer.addJarsFromProperties(PigServer.java:248)
> 	... 10 more
> Caused by: java.io.FileNotFoundException: File does not exist: s3
> 	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:457)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1256)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1237)
> 	at org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:763)
> 	... 12 more
> ================================================================================
> {quote} 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (PIG-2198) pig.additional.jars does not support s3 paths

Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alan Gates updated PIG-2198:
----------------------------

    Status: Open  (was: Patch Available)

Canceling patch pending addition of tests.

> pig.additional.jars does not support s3 paths
> ---------------------------------------------
>
>                 Key: PIG-2198
>                 URL: https://issues.apache.org/jira/browse/PIG-2198
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.8.1
>            Reporter: Jeremy Custenborder
>            Priority: Minor
>         Attachments: PIG-2198-delimiter.patch, PIG-2198.patch
>
>
> The delimiter for additional jars is ":" which is part of the uri for a S3 path. 
> *Repro*
> Add the following to the command line.
> {quote}
> -Dpig.additional.jars=s3://bucket/udf.jar
> {quote}
> {quote}
> Error before Pig is launched
> ----------------------------
> ERROR 101: Could not copy s3 to local destination /mnt/var/lib/hadoop/steps/1/tmp/s3
> org.apache.pig.backend.executionengine.ExecException: ERROR 4010: Failed to register jar :s3. Caught exception.
> 	at org.apache.pig.PigServer.addJarsFromProperties(PigServer.java:253)
> 	at org.apache.pig.PigServer.<init>(PigServer.java:239)
> 	at org.apache.pig.PigServer.<init>(PigServer.java:215)
> 	at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
> 	at org.apache.pig.Main.run(Main.java:492)
> 	at org.apache.pig.Main.main(Main.java:107)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 101: Could not copy s3 to local destination /mnt/var/lib/hadoop/steps/1/tmp/s3
> 	at org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:765)
> 	at org.apache.pig.PigServer.registerJar(PigServer.java:483)
> 	at org.apache.pig.PigServer.addJarsFromProperties(PigServer.java:248)
> 	... 10 more
> Caused by: java.io.FileNotFoundException: File does not exist: s3
> 	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:457)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1256)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1237)
> 	at org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:763)
> 	... 12 more
> ================================================================================
> {quote} 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (PIG-2198) pig.additional.jars does not support s3 paths

Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073649#comment-13073649 ] 

Thejas M Nair commented on PIG-2198:
------------------------------------

':' is what is used for separating path names in linux (including java class path), so that is what is intuitive to users on linux.  Also, ';' is interpreted by (bash) shell as end of command, users will have to use quotes to avoid that interpretation.  

I think it is better to make the delimiter configurable. Another option is to not use ':' within quoted strings as delimiters, but getting those quotes through the shell to pig would require escaping the quotes, and that would be ugly.


> pig.additional.jars does not support s3 paths
> ---------------------------------------------
>
>                 Key: PIG-2198
>                 URL: https://issues.apache.org/jira/browse/PIG-2198
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.8.1
>            Reporter: Jeremy Custenborder
>            Priority: Minor
>         Attachments: PIG-2198.patch
>
>
> The delimiter for additional jars is ":" which is part of the uri for a S3 path. 
> *Repro*
> Add the following to the command line.
> {quote}
> -Dpig.additional.jars=s3://bucket/udf.jar
> {quote}
> {quote}
> Error before Pig is launched
> ----------------------------
> ERROR 101: Could not copy s3 to local destination /mnt/var/lib/hadoop/steps/1/tmp/s3
> org.apache.pig.backend.executionengine.ExecException: ERROR 4010: Failed to register jar :s3. Caught exception.
> 	at org.apache.pig.PigServer.addJarsFromProperties(PigServer.java:253)
> 	at org.apache.pig.PigServer.<init>(PigServer.java:239)
> 	at org.apache.pig.PigServer.<init>(PigServer.java:215)
> 	at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
> 	at org.apache.pig.Main.run(Main.java:492)
> 	at org.apache.pig.Main.main(Main.java:107)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 101: Could not copy s3 to local destination /mnt/var/lib/hadoop/steps/1/tmp/s3
> 	at org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:765)
> 	at org.apache.pig.PigServer.registerJar(PigServer.java:483)
> 	at org.apache.pig.PigServer.addJarsFromProperties(PigServer.java:248)
> 	... 10 more
> Caused by: java.io.FileNotFoundException: File does not exist: s3
> 	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:457)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1256)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1237)
> 	at org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:763)
> 	... 12 more
> ================================================================================
> {quote} 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (PIG-2198) pig.additional.jars does not support s3 paths

Posted by "Jeremy Custenborder (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073578#comment-13073578 ] 

Jeremy Custenborder commented on PIG-2198:
------------------------------------------

I'm not sure if we want to change the delimiter because this will break environments that are already using this parameter.I added a patch that uses a different delimiter but I'm not sure if that is the best action.  

> pig.additional.jars does not support s3 paths
> ---------------------------------------------
>
>                 Key: PIG-2198
>                 URL: https://issues.apache.org/jira/browse/PIG-2198
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.8.1
>            Reporter: Jeremy Custenborder
>            Priority: Minor
>         Attachments: PIG-2198.patch
>
>
> The delimiter for additional jars is ":" which is part of the uri for a S3 path. 
> *Repro*
> Add the following to the command line.
> {quote}
> -Dpig.additional.jars=s3://bucket/udf.jar
> {quote}
> {quote}
> Error before Pig is launched
> ----------------------------
> ERROR 101: Could not copy s3 to local destination /mnt/var/lib/hadoop/steps/1/tmp/s3
> org.apache.pig.backend.executionengine.ExecException: ERROR 4010: Failed to register jar :s3. Caught exception.
> 	at org.apache.pig.PigServer.addJarsFromProperties(PigServer.java:253)
> 	at org.apache.pig.PigServer.<init>(PigServer.java:239)
> 	at org.apache.pig.PigServer.<init>(PigServer.java:215)
> 	at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
> 	at org.apache.pig.Main.run(Main.java:492)
> 	at org.apache.pig.Main.main(Main.java:107)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 101: Could not copy s3 to local destination /mnt/var/lib/hadoop/steps/1/tmp/s3
> 	at org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:765)
> 	at org.apache.pig.PigServer.registerJar(PigServer.java:483)
> 	at org.apache.pig.PigServer.addJarsFromProperties(PigServer.java:248)
> 	... 10 more
> Caused by: java.io.FileNotFoundException: File does not exist: s3
> 	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:457)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1256)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1237)
> 	at org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:763)
> 	... 12 more
> ================================================================================
> {quote} 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (PIG-2198) pig.additional.jars does not support s3 paths

Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085425#comment-13085425 ] 

Alan Gates commented on PIG-2198:
---------------------------------

I'll be reviewing this patch.

> pig.additional.jars does not support s3 paths
> ---------------------------------------------
>
>                 Key: PIG-2198
>                 URL: https://issues.apache.org/jira/browse/PIG-2198
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.8.1
>            Reporter: Jeremy Custenborder
>            Priority: Minor
>         Attachments: PIG-2198-delimiter.patch, PIG-2198.patch
>
>
> The delimiter for additional jars is ":" which is part of the uri for a S3 path. 
> *Repro*
> Add the following to the command line.
> {quote}
> -Dpig.additional.jars=s3://bucket/udf.jar
> {quote}
> {quote}
> Error before Pig is launched
> ----------------------------
> ERROR 101: Could not copy s3 to local destination /mnt/var/lib/hadoop/steps/1/tmp/s3
> org.apache.pig.backend.executionengine.ExecException: ERROR 4010: Failed to register jar :s3. Caught exception.
> 	at org.apache.pig.PigServer.addJarsFromProperties(PigServer.java:253)
> 	at org.apache.pig.PigServer.<init>(PigServer.java:239)
> 	at org.apache.pig.PigServer.<init>(PigServer.java:215)
> 	at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
> 	at org.apache.pig.Main.run(Main.java:492)
> 	at org.apache.pig.Main.main(Main.java:107)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 101: Could not copy s3 to local destination /mnt/var/lib/hadoop/steps/1/tmp/s3
> 	at org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:765)
> 	at org.apache.pig.PigServer.registerJar(PigServer.java:483)
> 	at org.apache.pig.PigServer.addJarsFromProperties(PigServer.java:248)
> 	... 10 more
> Caused by: java.io.FileNotFoundException: File does not exist: s3
> 	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:457)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1256)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1237)
> 	at org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:763)
> 	... 12 more
> ================================================================================
> {quote} 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (PIG-2198) pig.additional.jars does not support s3 paths

Posted by "Jeremy Custenborder (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jeremy Custenborder updated PIG-2198:
-------------------------------------

    Attachment: PIG-2198-delimiter.patch

Here is another patch that will check a property for a different delimiter.

> pig.additional.jars does not support s3 paths
> ---------------------------------------------
>
>                 Key: PIG-2198
>                 URL: https://issues.apache.org/jira/browse/PIG-2198
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.8.1
>            Reporter: Jeremy Custenborder
>            Priority: Minor
>         Attachments: PIG-2198-delimiter.patch, PIG-2198.patch
>
>
> The delimiter for additional jars is ":" which is part of the uri for a S3 path. 
> *Repro*
> Add the following to the command line.
> {quote}
> -Dpig.additional.jars=s3://bucket/udf.jar
> {quote}
> {quote}
> Error before Pig is launched
> ----------------------------
> ERROR 101: Could not copy s3 to local destination /mnt/var/lib/hadoop/steps/1/tmp/s3
> org.apache.pig.backend.executionengine.ExecException: ERROR 4010: Failed to register jar :s3. Caught exception.
> 	at org.apache.pig.PigServer.addJarsFromProperties(PigServer.java:253)
> 	at org.apache.pig.PigServer.<init>(PigServer.java:239)
> 	at org.apache.pig.PigServer.<init>(PigServer.java:215)
> 	at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
> 	at org.apache.pig.Main.run(Main.java:492)
> 	at org.apache.pig.Main.main(Main.java:107)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 101: Could not copy s3 to local destination /mnt/var/lib/hadoop/steps/1/tmp/s3
> 	at org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:765)
> 	at org.apache.pig.PigServer.registerJar(PigServer.java:483)
> 	at org.apache.pig.PigServer.addJarsFromProperties(PigServer.java:248)
> 	... 10 more
> Caused by: java.io.FileNotFoundException: File does not exist: s3
> 	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:457)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1256)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1237)
> 	at org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:763)
> 	... 12 more
> ================================================================================
> {quote} 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (PIG-2198) pig.additional.jars does not support s3 paths

Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086519#comment-13086519 ] 

Alan Gates commented on PIG-2198:
---------------------------------

The changes look fine.  The patch should include tests that check that the specified delimiter is used properly and that URIs (you should be able to use file:) work.

> pig.additional.jars does not support s3 paths
> ---------------------------------------------
>
>                 Key: PIG-2198
>                 URL: https://issues.apache.org/jira/browse/PIG-2198
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.8.1
>            Reporter: Jeremy Custenborder
>            Priority: Minor
>         Attachments: PIG-2198-delimiter.patch, PIG-2198.patch
>
>
> The delimiter for additional jars is ":" which is part of the uri for a S3 path. 
> *Repro*
> Add the following to the command line.
> {quote}
> -Dpig.additional.jars=s3://bucket/udf.jar
> {quote}
> {quote}
> Error before Pig is launched
> ----------------------------
> ERROR 101: Could not copy s3 to local destination /mnt/var/lib/hadoop/steps/1/tmp/s3
> org.apache.pig.backend.executionengine.ExecException: ERROR 4010: Failed to register jar :s3. Caught exception.
> 	at org.apache.pig.PigServer.addJarsFromProperties(PigServer.java:253)
> 	at org.apache.pig.PigServer.<init>(PigServer.java:239)
> 	at org.apache.pig.PigServer.<init>(PigServer.java:215)
> 	at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
> 	at org.apache.pig.Main.run(Main.java:492)
> 	at org.apache.pig.Main.main(Main.java:107)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 101: Could not copy s3 to local destination /mnt/var/lib/hadoop/steps/1/tmp/s3
> 	at org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:765)
> 	at org.apache.pig.PigServer.registerJar(PigServer.java:483)
> 	at org.apache.pig.PigServer.addJarsFromProperties(PigServer.java:248)
> 	... 10 more
> Caused by: java.io.FileNotFoundException: File does not exist: s3
> 	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:457)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1256)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1237)
> 	at org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:763)
> 	... 12 more
> ================================================================================
> {quote} 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (PIG-2198) pig.additional.jars does not support s3 paths

Posted by "Jeremy Custenborder (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jeremy Custenborder updated PIG-2198:
-------------------------------------

    Attachment: PIG-2198.patch

Changed the delimiter from : to ;

> pig.additional.jars does not support s3 paths
> ---------------------------------------------
>
>                 Key: PIG-2198
>                 URL: https://issues.apache.org/jira/browse/PIG-2198
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.8.1
>            Reporter: Jeremy Custenborder
>            Priority: Minor
>         Attachments: PIG-2198.patch
>
>
> The delimiter for additional jars is ":" which is part of the uri for a S3 path. 
> *Repro*
> Add the following to the command line.
> {quote}
> -Dpig.additional.jars=s3://bucket/udf.jar
> {quote}
> {quote}
> Error before Pig is launched
> ----------------------------
> ERROR 101: Could not copy s3 to local destination /mnt/var/lib/hadoop/steps/1/tmp/s3
> org.apache.pig.backend.executionengine.ExecException: ERROR 4010: Failed to register jar :s3. Caught exception.
> 	at org.apache.pig.PigServer.addJarsFromProperties(PigServer.java:253)
> 	at org.apache.pig.PigServer.<init>(PigServer.java:239)
> 	at org.apache.pig.PigServer.<init>(PigServer.java:215)
> 	at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
> 	at org.apache.pig.Main.run(Main.java:492)
> 	at org.apache.pig.Main.main(Main.java:107)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 101: Could not copy s3 to local destination /mnt/var/lib/hadoop/steps/1/tmp/s3
> 	at org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:765)
> 	at org.apache.pig.PigServer.registerJar(PigServer.java:483)
> 	at org.apache.pig.PigServer.addJarsFromProperties(PigServer.java:248)
> 	... 10 more
> Caused by: java.io.FileNotFoundException: File does not exist: s3
> 	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:457)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1256)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1237)
> 	at org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:763)
> 	... 12 more
> ================================================================================
> {quote} 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (PIG-2198) pig.additional.jars does not support s3 paths

Posted by "Jeremy Custenborder (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jeremy Custenborder updated PIG-2198:
-------------------------------------

    Release Note: Changed the delimiter from : to ;
          Status: Patch Available  (was: Open)

> pig.additional.jars does not support s3 paths
> ---------------------------------------------
>
>                 Key: PIG-2198
>                 URL: https://issues.apache.org/jira/browse/PIG-2198
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.8.1
>            Reporter: Jeremy Custenborder
>            Priority: Minor
>
> The delimiter for additional jars is ":" which is part of the uri for a S3 path. 
> *Repro*
> Add the following to the command line.
> {quote}
> -Dpig.additional.jars=s3://bucket/udf.jar
> {quote}
> {quote}
> Error before Pig is launched
> ----------------------------
> ERROR 101: Could not copy s3 to local destination /mnt/var/lib/hadoop/steps/1/tmp/s3
> org.apache.pig.backend.executionengine.ExecException: ERROR 4010: Failed to register jar :s3. Caught exception.
> 	at org.apache.pig.PigServer.addJarsFromProperties(PigServer.java:253)
> 	at org.apache.pig.PigServer.<init>(PigServer.java:239)
> 	at org.apache.pig.PigServer.<init>(PigServer.java:215)
> 	at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
> 	at org.apache.pig.Main.run(Main.java:492)
> 	at org.apache.pig.Main.main(Main.java:107)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 101: Could not copy s3 to local destination /mnt/var/lib/hadoop/steps/1/tmp/s3
> 	at org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:765)
> 	at org.apache.pig.PigServer.registerJar(PigServer.java:483)
> 	at org.apache.pig.PigServer.addJarsFromProperties(PigServer.java:248)
> 	... 10 more
> Caused by: java.io.FileNotFoundException: File does not exist: s3
> 	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:457)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1256)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1237)
> 	at org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:763)
> 	... 12 more
> ================================================================================
> {quote} 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira