You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Ankur (JIRA)" <ji...@apache.org> on 2011/02/25 04:58:38 UTC
[jira] Created: (PIG-1869) Job conf parameters not passed to
Streaming
Job conf parameters not passed to Streaming
-------------------------------------------
Key: PIG-1869
URL: https://issues.apache.org/jira/browse/PIG-1869
Project: Pig
Issue Type: Bug
Affects Versions: 0.8.0
Reporter: Ankur
Pig Streaming does not set the job conf parameters as environment variables for the streaming binary. This is unlike Hadoop streaming where all the job conf parameters are available as environment variables.
Here is a small test script along with the streaming binary
=== stream.pig ===
A = LOAD 'dat' as (f1:chararray);
DEFINE MY_BIN `./stream.sh` SHIP('./stream.sh');
B = STREAM A THROUGH MY_BIN;
dump B;
=== stream.sh ===
#!/bin/bash
export
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (PIG-1869) Job conf parameters not passed to
Streaming
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-1869:
--------------------------------
Fix Version/s: 0.10
> Job conf parameters not passed to Streaming
> -------------------------------------------
>
> Key: PIG-1869
> URL: https://issues.apache.org/jira/browse/PIG-1869
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Ankur
> Fix For: 0.10
>
>
> Pig Streaming does not set the job conf parameters as environment variables for the streaming binary. This is unlike Hadoop streaming where all the job conf parameters are available as environment variables.
> Here is a small test script along with the streaming binary
> === stream.pig ===
> A = LOAD 'dat' as (f1:chararray);
> DEFINE MY_BIN `./stream.sh` SHIP('./stream.sh');
> B = STREAM A THROUGH MY_BIN;
> dump B;
> === stream.sh ===
> #!/bin/bash
> export
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-1869) Job conf parameters not passed to
Streaming
Posted by "Olga Natkovich (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-1869:
--------------------------------
Fix Version/s: (was: 0.10)
> Job conf parameters not passed to Streaming
> -------------------------------------------
>
> Key: PIG-1869
> URL: https://issues.apache.org/jira/browse/PIG-1869
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Ankur
>
> Pig Streaming does not set the job conf parameters as environment variables for the streaming binary. This is unlike Hadoop streaming where all the job conf parameters are available as environment variables.
> Here is a small test script along with the streaming binary
> === stream.pig ===
> A = LOAD 'dat' as (f1:chararray);
> DEFINE MY_BIN `./stream.sh` SHIP('./stream.sh');
> B = STREAM A THROUGH MY_BIN;
> dump B;
> === stream.sh ===
> #!/bin/bash
> export
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (PIG-1869) Job conf parameters not passed to
Streaming
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy resolved PIG-1869.
------------------------------------
Resolution: Duplicate
Fixed in PIG-2900
Not following what exactly you want to happen with input files, please file a separate ticket.
> Job conf parameters not passed to Streaming
> -------------------------------------------
>
> Key: PIG-1869
> URL: https://issues.apache.org/jira/browse/PIG-1869
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Ankur
>
> Pig Streaming does not set the job conf parameters as environment variables for the streaming binary. This is unlike Hadoop streaming where all the job conf parameters are available as environment variables.
> Here is a small test script along with the streaming binary
> === stream.pig ===
> A = LOAD 'dat' as (f1:chararray);
> DEFINE MY_BIN `./stream.sh` SHIP('./stream.sh');
> B = STREAM A THROUGH MY_BIN;
> dump B;
> === stream.sh ===
> #!/bin/bash
> export
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (PIG-1869) Job conf parameters not passed to
Streaming
Posted by "Ankur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999209#comment-12999209 ]
Ankur commented on PIG-1869:
----------------------------
In my streaming binary, I would like to get hold of the parameter "map.input.file" that give information about the split file being processed. Also, starting with release 0.8, pig automatically combines multiple files into a single split, it is then responsibility of Pig to set this parameter correctly while reading individual split in combined split.
> Job conf parameters not passed to Streaming
> -------------------------------------------
>
> Key: PIG-1869
> URL: https://issues.apache.org/jira/browse/PIG-1869
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Ankur
>
> Pig Streaming does not set the job conf parameters as environment variables for the streaming binary. This is unlike Hadoop streaming where all the job conf parameters are available as environment variables.
> Here is a small test script along with the streaming binary
> === stream.pig ===
> A = LOAD 'dat' as (f1:chararray);
> DEFINE MY_BIN `./stream.sh` SHIP('./stream.sh');
> B = STREAM A THROUGH MY_BIN;
> dump B;
> === stream.sh ===
> #!/bin/bash
> export
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira