You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Mark Wagner <wa...@gmail.com> on 2014/02/11 20:42:54 UTC
Review Request 17958: PIG-3762 Propagate properties to edges and vertices
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17958/
-----------------------------------------------------------
Review request for pig, Alex Bain, Cheolsoo Park, Daniel Dai, and Rohini Palaniswamy.
Bugs: PIG-3768
https://issues.apache.org/jira/browse/PIG-3768
Repository: pig-git
Description
-------
Instead of creating empty configurations, initialize configurations with the properties provided by the user.
Diffs
-----
src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java b3e74d9
Diff: https://reviews.apache.org/r/17958/diff/
Testing
-------
Ran test-commit, and ran a script that wasn't respecting memory setting before, which succeeded after the patch.
Thanks,
Mark Wagner
Re: Review Request 17958: PIG-3762 Propagate properties to edges and vertices
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17958/#review34242
-----------------------------------------------------------
Ship it!
Please just make the minor change and go ahead with the commit. Just upload the final patch in the jira, in case someone wants to take it and apply later.
src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java
<https://reviews.apache.org/r/17958/#comment64268>
new Job(ConfigurationUtil.toConfiguration(pc.getProperties(), false));
src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java
<https://reviews.apache.org/r/17958/#comment64269>
This can be removed and conf directly passed when doing new Job() as above comment.
- Rohini Palaniswamy
On Feb. 11, 2014, 7:42 p.m., Mark Wagner wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17958/
> -----------------------------------------------------------
>
> (Updated Feb. 11, 2014, 7:42 p.m.)
>
>
> Review request for pig, Alex Bain, Cheolsoo Park, Daniel Dai, and Rohini Palaniswamy.
>
>
> Bugs: PIG-3768
> https://issues.apache.org/jira/browse/PIG-3768
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Instead of creating empty configurations, initialize configurations with the properties provided by the user.
>
>
> Diffs
> -----
>
> src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java b3e74d9
>
> Diff: https://reviews.apache.org/r/17958/diff/
>
>
> Testing
> -------
>
> Ran test-commit, and ran a script that wasn't respecting memory setting before, which succeeded after the patch.
>
>
> Thanks,
>
> Mark Wagner
>
>
Re: Review Request 17958: PIG-3762 Propagate properties to edges and vertices
Posted by Mark Wagner <wa...@gmail.com>.
> On Feb. 11, 2014, 7:55 p.m., Rohini Palaniswamy wrote:
> > src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java, line 402
> > <https://reviews.apache.org/r/17958/diff/1/?file=482887#file482887line402>
> >
> > Doesn't this take care of the memory settings for you?
That will take care of it I specify "mapreduce.task.io.sort.mb", but if I use the proper tez property ("tez.runtime.io.sort.mb"), then nothing gets passed. Any other properties like pig.cachedbag.type also don't make it. So I think both of these are necessary. The current code provides backwards compatibility for MR settings, and this patch supports the current Tez settings and unrelated properties.
- Mark
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17958/#review34223
-----------------------------------------------------------
On Feb. 11, 2014, 7:42 p.m., Mark Wagner wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17958/
> -----------------------------------------------------------
>
> (Updated Feb. 11, 2014, 7:42 p.m.)
>
>
> Review request for pig, Alex Bain, Cheolsoo Park, Daniel Dai, and Rohini Palaniswamy.
>
>
> Bugs: PIG-3768
> https://issues.apache.org/jira/browse/PIG-3768
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Instead of creating empty configurations, initialize configurations with the properties provided by the user.
>
>
> Diffs
> -----
>
> src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java b3e74d9
>
> Diff: https://reviews.apache.org/r/17958/diff/
>
>
> Testing
> -------
>
> Ran test-commit, and ran a script that wasn't respecting memory setting before, which succeeded after the patch.
>
>
> Thanks,
>
> Mark Wagner
>
>
Re: Review Request 17958: PIG-3762 Propagate properties to edges and vertices
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17958/#review34223
-----------------------------------------------------------
src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java
<https://reviews.apache.org/r/17958/#comment64255>
Doesn't this take care of the memory settings for you?
- Rohini Palaniswamy
On Feb. 11, 2014, 7:42 p.m., Mark Wagner wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17958/
> -----------------------------------------------------------
>
> (Updated Feb. 11, 2014, 7:42 p.m.)
>
>
> Review request for pig, Alex Bain, Cheolsoo Park, Daniel Dai, and Rohini Palaniswamy.
>
>
> Bugs: PIG-3768
> https://issues.apache.org/jira/browse/PIG-3768
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Instead of creating empty configurations, initialize configurations with the properties provided by the user.
>
>
> Diffs
> -----
>
> src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java b3e74d9
>
> Diff: https://reviews.apache.org/r/17958/diff/
>
>
> Testing
> -------
>
> Ran test-commit, and ran a script that wasn't respecting memory setting before, which succeeded after the patch.
>
>
> Thanks,
>
> Mark Wagner
>
>