You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org> on 2012/09/01 00:00:07 UTC
[jira] [Created] (PIG-2900) Streaming should provide conf settings
in the environment
Dmitriy V. Ryaboy created PIG-2900:
--------------------------------------
Summary: Streaming should provide conf settings in the environment
Key: PIG-2900
URL: https://issues.apache.org/jira/browse/PIG-2900
Project: Pig
Issue Type: New Feature
Reporter: Dmitriy V. Ryaboy
Assignee: Dmitriy V. Ryaboy
Hadoop Streaming converts jobconf properties into environment variables; Pig streaming does not. This is a useful feature that Pig streaming should provide.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2900) Streaming should provide conf settings
in the environment
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy updated PIG-2900:
-----------------------------------
Release Note:
The STREAM operator now makes all jobconf properties available to the programs processing streaming input via environment variables, consistend with Hadoop Streaming behavior.
All "." characters in the jobconf properties are replaced with underscores, "_".
> Streaming should provide conf settings in the environment
> ---------------------------------------------------------
>
> Key: PIG-2900
> URL: https://issues.apache.org/jira/browse/PIG-2900
> Project: Pig
> Issue Type: New Feature
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.11
>
> Attachments: PIG-2900.1.patch, PIG-2900.patch
>
>
> Hadoop Streaming converts jobconf properties into environment variables; Pig streaming does not. This is a useful feature that Pig streaming should provide.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2900) Streaming should provide conf
settings in the environment
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454665#comment-13454665 ]
Dmitriy V. Ryaboy commented on PIG-2900:
----------------------------------------
Alan,
I'll add the release notes.
That's interesting about dfs_data_dir .. are you using hadoop 23? Either way, I guess some other value should be used; I didn't know dfs.data.dir can be absent. Do you think we can rely on hadoop.tmp.dir existing in the default conf?
> Streaming should provide conf settings in the environment
> ---------------------------------------------------------
>
> Key: PIG-2900
> URL: https://issues.apache.org/jira/browse/PIG-2900
> Project: Pig
> Issue Type: New Feature
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Attachments: PIG-2900.1.patch, PIG-2900.patch
>
>
> Hadoop Streaming converts jobconf properties into environment variables; Pig streaming does not. This is a useful feature that Pig streaming should provide.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2900) Streaming should provide conf settings
in the environment
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy updated PIG-2900:
-----------------------------------
Status: Patch Available (was: Open)
> Streaming should provide conf settings in the environment
> ---------------------------------------------------------
>
> Key: PIG-2900
> URL: https://issues.apache.org/jira/browse/PIG-2900
> Project: Pig
> Issue Type: New Feature
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Attachments: PIG-2900.1.patch, PIG-2900.patch
>
>
> Hadoop Streaming converts jobconf properties into environment variables; Pig streaming does not. This is a useful feature that Pig streaming should provide.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2900) Streaming should provide conf
settings in the environment
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454818#comment-13454818 ]
Alan Gates commented on PIG-2900:
---------------------------------
I'm just building Pig with default options on my mac. I didn't know it could be missing either. hadoop.tmp.dir seems to be shared across platforms at the moment.
I'm +1 for this patch.
> Streaming should provide conf settings in the environment
> ---------------------------------------------------------
>
> Key: PIG-2900
> URL: https://issues.apache.org/jira/browse/PIG-2900
> Project: Pig
> Issue Type: New Feature
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Attachments: PIG-2900.1.patch, PIG-2900.patch
>
>
> Hadoop Streaming converts jobconf properties into environment variables; Pig streaming does not. This is a useful feature that Pig streaming should provide.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2900) Streaming should provide conf
settings in the environment
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451472#comment-13451472 ]
Dmitriy V. Ryaboy commented on PIG-2900:
----------------------------------------
Bump for review.
> Streaming should provide conf settings in the environment
> ---------------------------------------------------------
>
> Key: PIG-2900
> URL: https://issues.apache.org/jira/browse/PIG-2900
> Project: Pig
> Issue Type: New Feature
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Attachments: PIG-2900.1.patch, PIG-2900.patch
>
>
> Hadoop Streaming converts jobconf properties into environment variables; Pig streaming does not. This is a useful feature that Pig streaming should provide.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2900) Streaming should provide conf settings
in the environment
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy updated PIG-2900:
-----------------------------------
Attachment: PIG-2900.1.patch
Now with tests. Ready for review.
> Streaming should provide conf settings in the environment
> ---------------------------------------------------------
>
> Key: PIG-2900
> URL: https://issues.apache.org/jira/browse/PIG-2900
> Project: Pig
> Issue Type: New Feature
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Attachments: PIG-2900.1.patch, PIG-2900.patch
>
>
> Hadoop Streaming converts jobconf properties into environment variables; Pig streaming does not. This is a useful feature that Pig streaming should provide.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2900) Streaming should provide conf
settings in the environment
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454636#comment-13454636 ]
Alan Gates commented on PIG-2900:
---------------------------------
In general it looks good. I had a couple of questions/comments:
We should add a note to the release notes section of the JIRA noting the new
feature and how the mapping of env var names will be handled (e.g. a.b.c will
be mapped to a_b_c).
It would be nice to have an e2e test that checks that the environment variable ends up on the remote side. I'll take a look at adding that.
The unit test you provided fails on my mac. It seems dfs_data_dir isn't in the created configuration. A lot of other values are, like hadoop_tmp_dir. I didn't run it on Linux to see if it works ok there.
> Streaming should provide conf settings in the environment
> ---------------------------------------------------------
>
> Key: PIG-2900
> URL: https://issues.apache.org/jira/browse/PIG-2900
> Project: Pig
> Issue Type: New Feature
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Attachments: PIG-2900.1.patch, PIG-2900.patch
>
>
> Hadoop Streaming converts jobconf properties into environment variables; Pig streaming does not. This is a useful feature that Pig streaming should provide.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2900) Streaming should provide conf settings
in the environment
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy updated PIG-2900:
-----------------------------------
Resolution: Fixed
Fix Version/s: 0.11
Status: Resolved (was: Patch Available)
Committed to trunk.
Thanks for the review, Alan!
> Streaming should provide conf settings in the environment
> ---------------------------------------------------------
>
> Key: PIG-2900
> URL: https://issues.apache.org/jira/browse/PIG-2900
> Project: Pig
> Issue Type: New Feature
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.11
>
> Attachments: PIG-2900.1.patch, PIG-2900.patch
>
>
> Hadoop Streaming converts jobconf properties into environment variables; Pig streaming does not. This is a useful feature that Pig streaming should provide.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2900) Streaming should provide conf settings
in the environment
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy updated PIG-2900:
-----------------------------------
Attachment: PIG-2900.patch
No tests, but all the code is ripped out straight from Hadoop Streaming. Tested on the cluster.
Will add tests.
> Streaming should provide conf settings in the environment
> ---------------------------------------------------------
>
> Key: PIG-2900
> URL: https://issues.apache.org/jira/browse/PIG-2900
> Project: Pig
> Issue Type: New Feature
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Attachments: PIG-2900.patch
>
>
> Hadoop Streaming converts jobconf properties into environment variables; Pig streaming does not. This is a useful feature that Pig streaming should provide.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira