You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Rohini Palaniswamy (JIRA)" <ji...@apache.org> on 2012/09/07 23:38:08 UTC
[jira] [Created] (PIG-2912) Pig should clone JobConf while creating
JobContextImpl and TaskAttemptContextImpl in Hadoop23
Rohini Palaniswamy created PIG-2912:
---------------------------------------
Summary: Pig should clone JobConf while creating JobContextImpl and TaskAttemptContextImpl in Hadoop23
Key: PIG-2912
URL: https://issues.apache.org/jira/browse/PIG-2912
Project: Pig
Issue Type: Bug
Affects Versions: 0.9.3, 0.10.1
Reporter: Rohini Palaniswamy
Assignee: Rohini Palaniswamy
Fix For: 0.9.3, 0.11, 0.10.1
There is change in the semantics of
JobContext::JobContext(Configuration, JobID). While in .20, the Config was
cloned, in .23 the Config is adopted (if it's a JobConf). This causes the same
Configuration instance to be written-to for different tables in the same job.
It would affect multi store commands in pig on Hadoop 23/2.0. The
cloning in HadoopShims was part of PIG-2578 but was reverted to other issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2912) Pig should clone JobConf while creating
JobContextImpl and TaskAttemptContextImpl in Hadoop23
Posted by "Rohini Palaniswamy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rohini Palaniswamy updated PIG-2912:
------------------------------------
Status: Patch Available (was: Open)
> Pig should clone JobConf while creating JobContextImpl and TaskAttemptContextImpl in Hadoop23
> ---------------------------------------------------------------------------------------------
>
> Key: PIG-2912
> URL: https://issues.apache.org/jira/browse/PIG-2912
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.3, 0.10.1
> Reporter: Rohini Palaniswamy
> Assignee: Rohini Palaniswamy
> Fix For: 0.9.3, 0.11, 0.10.1
>
> Attachments: PIG-2912-branch09.patch, PIG-2912-branch10.patch, PIG-2912-trunk.patch
>
>
> There is change in the semantics of
> JobContext::JobContext(Configuration, JobID). While in .20, the Config was
> cloned, in .23 the Config is adopted (if it's a JobConf). This causes the same
> Configuration instance to be written-to for different tables in the same job.
> It would affect multi store commands in pig on Hadoop 23/2.0. The
> cloning in HadoopShims was part of PIG-2578 but was reverted to other issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2912) Pig should clone JobConf while
creating JobContextImpl and TaskAttemptContextImpl in Hadoop23
Posted by "Rohini Palaniswamy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452265#comment-13452265 ]
Rohini Palaniswamy commented on PIG-2912:
-----------------------------------------
Dmitriy,
This patch is mainly to address a behaviour change in hadoop between 20 and 23 in the way instantiation of new JobContext is done and to deal with it in HadoopShims so that JobContext objects in the backend do not get overwritten in case of multiple stores. PIG-2578 main problem was with JobControlCompiler and it changed frontend behaviour. I will create a separate jira to add test case for PIG-2578.
Hadoop 20:
{code}
JobContext(JobConf conf, org.apache.hadoop.mapreduce.JobID jobId,
Progressable progress) {
super(conf, jobId); //Gets cloned
this.job = conf;
this.progress = progress;
}
{code}
Hadoop 23:
{code}
public JobContextImpl(Configuration conf, JobID jobId) {
if (conf instanceof JobConf) {
this.conf = (JobConf)conf; //Gets assigned
} else {
this.conf = new JobConf(conf);
}
{code}
> Pig should clone JobConf while creating JobContextImpl and TaskAttemptContextImpl in Hadoop23
> ---------------------------------------------------------------------------------------------
>
> Key: PIG-2912
> URL: https://issues.apache.org/jira/browse/PIG-2912
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.3, 0.10.1
> Reporter: Rohini Palaniswamy
> Assignee: Rohini Palaniswamy
> Fix For: 0.9.3, 0.11, 0.10.1
>
> Attachments: PIG-2912-branch09.patch, PIG-2912-branch10.patch, PIG-2912-trunk.patch
>
>
> There is change in the semantics of
> JobContext::JobContext(Configuration, JobID). While in .20, the Config was
> cloned, in .23 the Config is adopted (if it's a JobConf). This causes the same
> Configuration instance to be written-to for different tables in the same job.
> It would affect multi store commands in pig on Hadoop 23/2.0. The
> cloning in HadoopShims was part of PIG-2578 but was reverted to other issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2912) Pig should clone JobConf while
creating JobContextImpl and TaskAttemptContextImpl in Hadoop23
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452455#comment-13452455 ]
Daniel Dai commented on PIG-2912:
---------------------------------
+1. We shall keep the same behavior between 20 and 23. Put the abstraction in shims is the right approach. Will commit soon.
> Pig should clone JobConf while creating JobContextImpl and TaskAttemptContextImpl in Hadoop23
> ---------------------------------------------------------------------------------------------
>
> Key: PIG-2912
> URL: https://issues.apache.org/jira/browse/PIG-2912
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.3, 0.10.1
> Reporter: Rohini Palaniswamy
> Assignee: Rohini Palaniswamy
> Fix For: 0.9.3, 0.11, 0.10.1
>
> Attachments: PIG-2912-branch09.patch, PIG-2912-branch10.patch, PIG-2912-trunk.patch
>
>
> There is change in the semantics of
> JobContext::JobContext(Configuration, JobID). While in .20, the Config was
> cloned, in .23 the Config is adopted (if it's a JobConf). This causes the same
> Configuration instance to be written-to for different tables in the same job.
> It would affect multi store commands in pig on Hadoop 23/2.0. The
> cloning in HadoopShims was part of PIG-2578 but was reverted to other issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2912) Pig should clone JobConf while creating
JobContextImpl and TaskAttemptContextImpl in Hadoop23
Posted by "Rohini Palaniswamy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rohini Palaniswamy updated PIG-2912:
------------------------------------
Attachment: PIG-2912-trunk.patch
PIG-2912-branch10.patch
PIG-2912-branch09.patch
The patch creates a clone if JobConf is passed.
Testcase added in TestMultiQueryLocal ensures that if new settings are added in the backend by a store, they are not passed to other stores in a multi store script.
> Pig should clone JobConf while creating JobContextImpl and TaskAttemptContextImpl in Hadoop23
> ---------------------------------------------------------------------------------------------
>
> Key: PIG-2912
> URL: https://issues.apache.org/jira/browse/PIG-2912
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.3, 0.10.1
> Reporter: Rohini Palaniswamy
> Assignee: Rohini Palaniswamy
> Fix For: 0.9.3, 0.11, 0.10.1
>
> Attachments: PIG-2912-branch09.patch, PIG-2912-branch10.patch, PIG-2912-trunk.patch
>
>
> There is change in the semantics of
> JobContext::JobContext(Configuration, JobID). While in .20, the Config was
> cloned, in .23 the Config is adopted (if it's a JobConf). This causes the same
> Configuration instance to be written-to for different tables in the same job.
> It would affect multi store commands in pig on Hadoop 23/2.0. The
> cloning in HadoopShims was part of PIG-2578 but was reverted to other issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2912) Pig should clone JobConf while creating
JobContextImpl and TaskAttemptContextImpl in Hadoop23
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2912:
----------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Patch committed to 0.9/0.10/trunk. Thanks Rohini!
> Pig should clone JobConf while creating JobContextImpl and TaskAttemptContextImpl in Hadoop23
> ---------------------------------------------------------------------------------------------
>
> Key: PIG-2912
> URL: https://issues.apache.org/jira/browse/PIG-2912
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.3, 0.10.1
> Reporter: Rohini Palaniswamy
> Assignee: Rohini Palaniswamy
> Fix For: 0.9.3, 0.11, 0.10.1
>
> Attachments: PIG-2912-branch09.patch, PIG-2912-branch10.patch, PIG-2912-trunk.patch
>
>
> There is change in the semantics of
> JobContext::JobContext(Configuration, JobID). While in .20, the Config was
> cloned, in .23 the Config is adopted (if it's a JobConf). This causes the same
> Configuration instance to be written-to for different tables in the same job.
> It would affect multi store commands in pig on Hadoop 23/2.0. The
> cloning in HadoopShims was part of PIG-2578 but was reverted to other issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2912) Pig should clone JobConf while
creating JobContextImpl and TaskAttemptContextImpl in Hadoop23
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452137#comment-13452137 ]
Dmitriy V. Ryaboy commented on PIG-2912:
----------------------------------------
Could you create a test in which the storage or loader function uses UDFContext to ship around information? I don't think the current test probes the issue that caused us to roll back PIG-2578
> Pig should clone JobConf while creating JobContextImpl and TaskAttemptContextImpl in Hadoop23
> ---------------------------------------------------------------------------------------------
>
> Key: PIG-2912
> URL: https://issues.apache.org/jira/browse/PIG-2912
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.3, 0.10.1
> Reporter: Rohini Palaniswamy
> Assignee: Rohini Palaniswamy
> Fix For: 0.9.3, 0.11, 0.10.1
>
> Attachments: PIG-2912-branch09.patch, PIG-2912-branch10.patch, PIG-2912-trunk.patch
>
>
> There is change in the semantics of
> JobContext::JobContext(Configuration, JobID). While in .20, the Config was
> cloned, in .23 the Config is adopted (if it's a JobConf). This causes the same
> Configuration instance to be written-to for different tables in the same job.
> It would affect multi store commands in pig on Hadoop 23/2.0. The
> cloning in HadoopShims was part of PIG-2578 but was reverted to other issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira