You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org> on 2012/09/20 01:35:07 UTC
[jira] [Created] (PIG-2925) Extremely long JobConf values should
not be added to Streaming environment
Dmitriy V. Ryaboy created PIG-2925:
--------------------------------------
Summary: Extremely long JobConf values should not be added to Streaming environment
Key: PIG-2925
URL: https://issues.apache.org/jira/browse/PIG-2925
Project: Pig
Issue Type: Bug
Reporter: Dmitriy V. Ryaboy
PIG-2900 made Pig match Hadoop's practice of making all JobConf properties available as environment variables for pig streaming.
As it happens, ProcessBuilder creates a long string with all environment variables and their values on it, then runs it; having some of the weightier properties like serialized plans in the environment can make the total command length exceed OS limits.
We propose adding a blacklist of properties which should not be added to streaming environment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (PIG-2925) Extremely long JobConf values should
not be added to Streaming environment
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy reassigned PIG-2925:
--------------------------------------
Assignee: Dmitriy V. Ryaboy
> Extremely long JobConf values should not be added to Streaming environment
> --------------------------------------------------------------------------
>
> Key: PIG-2925
> URL: https://issues.apache.org/jira/browse/PIG-2925
> Project: Pig
> Issue Type: Bug
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
>
> PIG-2900 made Pig match Hadoop's practice of making all JobConf properties available as environment variables for pig streaming.
> As it happens, ProcessBuilder creates a long string with all environment variables and their values on it, then runs it; having some of the weightier properties like serialized plans in the environment can make the total command length exceed OS limits.
> We propose adding a blacklist of properties which should not be added to streaming environment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2925) Extremely long JobConf values should
not be added to Streaming environment
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy updated PIG-2925:
-----------------------------------
Status: Patch Available (was: Open)
> Extremely long JobConf values should not be added to Streaming environment
> --------------------------------------------------------------------------
>
> Key: PIG-2925
> URL: https://issues.apache.org/jira/browse/PIG-2925
> Project: Pig
> Issue Type: Bug
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Attachments: PIG-2925.patch
>
>
> PIG-2900 made Pig match Hadoop's practice of making all JobConf properties available as environment variables for pig streaming.
> As it happens, ProcessBuilder creates a long string with all environment variables and their values on it, then runs it; having some of the weightier properties like serialized plans in the environment can make the total command length exceed OS limits.
> We propose adding a blacklist of properties which should not be added to streaming environment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2925) Extremely long JobConf values should
not be added to Streaming environment
Posted by "Jonathan Coveney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Coveney updated PIG-2925:
----------------------------------
Resolution: Won't Fix
Status: Resolved (was: Patch Available)
We are instead using the method in PIG-2971
> Extremely long JobConf values should not be added to Streaming environment
> --------------------------------------------------------------------------
>
> Key: PIG-2925
> URL: https://issues.apache.org/jira/browse/PIG-2925
> Project: Pig
> Issue Type: Bug
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Attachments: PIG-2925.patch
>
>
> PIG-2900 made Pig match Hadoop's practice of making all JobConf properties available as environment variables for pig streaming.
> As it happens, ProcessBuilder creates a long string with all environment variables and their values on it, then runs it; having some of the weightier properties like serialized plans in the environment can make the total command length exceed OS limits.
> We propose adding a blacklist of properties which should not be added to streaming environment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2925) Extremely long JobConf values should
not be added to Streaming environment
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy updated PIG-2925:
-----------------------------------
Attachment: PIG-2925.patch
Decided it's easier to set the max length, and allow users to bump it up or down as needed, than to mess with explicit blacklists. Too hard to control what everyone might decide to shove into their config.
> Extremely long JobConf values should not be added to Streaming environment
> --------------------------------------------------------------------------
>
> Key: PIG-2925
> URL: https://issues.apache.org/jira/browse/PIG-2925
> Project: Pig
> Issue Type: Bug
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Attachments: PIG-2925.patch
>
>
> PIG-2900 made Pig match Hadoop's practice of making all JobConf properties available as environment variables for pig streaming.
> As it happens, ProcessBuilder creates a long string with all environment variables and their values on it, then runs it; having some of the weightier properties like serialized plans in the environment can make the total command length exceed OS limits.
> We propose adding a blacklist of properties which should not be added to streaming environment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira