You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Cheolsoo Park <pi...@gmail.com> on 2013/10/11 23:56:56 UTC
Review Request 14601: PIG-3502 Initial implementation of TezLauncher
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14601/
-----------------------------------------------------------
Review request for pig, Daniel Dai, Mark Wagner, and Rohini Palaniswamy.
Bugs: PIG-3502
https://issues.apache.org/jira/browse/PIG-3502
Repository: pig-git
Description
-------
Initial implementation of TezLauncher. The patch includes the following changes:
- Refactors Launcher and MapReduceLauncher. Basically, I moved what's MR-specific to MapReduceLauncher and kept what's common in Launcher.
- Removes kill() method from ExecutionEngine interface. It was used merely by Launcher, and it didn't seem useful.
- Implements the DAG submission of TezLauncher. Basically, I overrode the submit() method in TezJob (extension of ControlledJob) that calls DAGClient.submit().
- Adds a skeleton TezStats class that extends PigStats. This class needs to be implemented to monitor job status after submission.
- Adds a unit test called TestTezLauncher. For now, all it does is to compile queries and submit a DAG. More meaningful unit tests will be added after TezStats is implemented.
- Adds a TezLocalExecType. To start up a YARN mini cluster, I needed non-local Tez mode. So I created "tez_local" and "tez" modes.
- Whitespace and indentation fixes.
Diffs
-----
build.xml 0439082
src/META-INF/services/org.apache.pig.ExecType e995cd5
src/org/apache/pig/backend/executionengine/ExecutionEngine.java 7d480d7
src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java b886f9d
src/org/apache/pig/backend/hadoop/executionengine/Launcher.java 77543c2
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java f3be738
src/org/apache/pig/backend/hadoop/executionengine/tez/TezExecType.java 8315e76
src/org/apache/pig/backend/hadoop/executionengine/tez/TezJob.java 04ffba0
src/org/apache/pig/backend/hadoop/executionengine/tez/TezJobControlCompiler.java b7d4d98
src/org/apache/pig/backend/hadoop/executionengine/tez/TezLauncher.java ff9e950
src/org/apache/pig/backend/hadoop/executionengine/tez/TezLocalExecType.java e69de29
src/org/apache/pig/backend/hadoop/executionengine/tez/TezStats.java e69de29
src/org/apache/pig/impl/PigContext.java 225526f
src/org/apache/pig/impl/util/Utils.java 3720c94
test/org/apache/pig/tez/TestTezCompiler.java b9e6b48
test/org/apache/pig/tez/TestTezJobControlCompiler.java f14d49d
test/org/apache/pig/tez/TestTezLauncher.java e69de29
Diff: https://reviews.apache.org/r/14601/diff/
Testing
-------
TestTezLauncher is added.
Thanks,
Cheolsoo Park
Re: Review Request 14601: PIG-3502 Initial implementation of TezLauncher
Posted by Mark Wagner <wa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14601/#review26956
-----------------------------------------------------------
I played around with this this afternoon and noticed a few things. I haven't gone through all of it yet.
src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java
<https://reviews.apache.org/r/14601/#comment52506>
We'll also need to set some environment variables (I think CLASSPATH at a minimal). Check out MRHelpers
src/org/apache/pig/backend/hadoop/executionengine/tez/TezLauncher.java
<https://reviews.apache.org/r/14601/#comment52504>
This returns a configuration w/o defaults, but the TezJobControlCompiler turns it into a TezConfiguration, so it needs the tez-site items as well. I added a toConfiguration(Props, bool) that optionally includes defaults.
I'm also getting error messages when I close the session about killing the ExecutionEngine. I haven't looked into that, though.
- Mark Wagner
On Oct. 12, 2013, 1:19 a.m., Cheolsoo Park wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14601/
> -----------------------------------------------------------
>
> (Updated Oct. 12, 2013, 1:19 a.m.)
>
>
> Review request for pig, Daniel Dai, Mark Wagner, and Rohini Palaniswamy.
>
>
> Bugs: PIG-3502
> https://issues.apache.org/jira/browse/PIG-3502
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Initial implementation of TezLauncher. The patch includes the following changes:
> - Refactors Launcher and MapReduceLauncher. Basically, I moved what's MR-specific to MapReduceLauncher and kept what's common in Launcher.
> - Removes kill() method from ExecutionEngine interface. It was used merely by Launcher, and it didn't seem useful.
> - Implements the DAG submission of TezLauncher. Basically, I overrode the submit() method in TezJob (extension of ControlledJob) that calls DAGClient.submit().
> - Adds a skeleton TezStats class that extends PigStats. This class needs to be implemented to monitor job status after submission.
> - Adds a unit test called TestTezLauncher. For now, all it does is to compile queries and submit a DAG. More meaningful unit tests will be added after TezStats is implemented.
> - Adds a TezLocalExecType. To start up a YARN mini cluster, I needed non-local Tez mode. So I created "tez_local" and "tez" modes.
> - Whitespace and indentation fixes.
>
>
> Diffs
> -----
>
> build.xml 0439082
> src/META-INF/services/org.apache.pig.ExecType e995cd5
> src/org/apache/pig/backend/executionengine/ExecutionEngine.java 7d480d7
> src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java b886f9d
> src/org/apache/pig/backend/hadoop/executionengine/Launcher.java 77543c2
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java f3be738
> src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 5414906
> src/org/apache/pig/backend/hadoop/executionengine/tez/TezExecType.java 8315e76
> src/org/apache/pig/backend/hadoop/executionengine/tez/TezJob.java 04ffba0
> src/org/apache/pig/backend/hadoop/executionengine/tez/TezJobControlCompiler.java b7d4d98
> src/org/apache/pig/backend/hadoop/executionengine/tez/TezLauncher.java ff9e950
> src/org/apache/pig/backend/hadoop/executionengine/tez/TezLocalExecType.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/tez/TezStats.java PRE-CREATION
> src/org/apache/pig/impl/PigContext.java 225526f
> src/org/apache/pig/impl/util/Utils.java 3720c94
> test/org/apache/pig/tez/TestTezCompiler.java b9e6b48
> test/org/apache/pig/tez/TestTezJobControlCompiler.java f14d49d
> test/org/apache/pig/tez/TestTezLauncher.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/14601/diff/
>
>
> Testing
> -------
>
> TestTezLauncher is added.
>
>
> Thanks,
>
> Cheolsoo Park
>
>
Re: Review Request 14601: PIG-3502 Initial implementation of TezLauncher
Posted by Cheolsoo Park <pi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14601/
-----------------------------------------------------------
(Updated Oct. 12, 2013, 1:19 a.m.)
Review request for pig, Daniel Dai, Mark Wagner, and Rohini Palaniswamy.
Changes
-------
Looking at the mini cluster log, I found that DAG was failing with NPE because local resources were not set for each vertex. I updated DAG construction code to set local resources in vertex.
Now DAG is successfully launched by AM, but it fails due to YARN-1229 in hadoop 2.1.1-beta. The test case doesn't fail because it doesn't check the job status for now. I will follow up with Tez devs about this YARN AM failure.
Btw, there was an incorrect statement in my description:
>> Removes kill() method from ExecutionEngine interface. It was used merely by Launcher, and it didn't seem useful.
kill() was used by ExecutionEngineKiller, but since I moved it to Launcher and renamed it as HangingJobKiller, kill() is no longer needed in ExecutionEngine interface.
Bugs: PIG-3502
https://issues.apache.org/jira/browse/PIG-3502
Repository: pig-git
Description
-------
Initial implementation of TezLauncher. The patch includes the following changes:
- Refactors Launcher and MapReduceLauncher. Basically, I moved what's MR-specific to MapReduceLauncher and kept what's common in Launcher.
- Removes kill() method from ExecutionEngine interface. It was used merely by Launcher, and it didn't seem useful.
- Implements the DAG submission of TezLauncher. Basically, I overrode the submit() method in TezJob (extension of ControlledJob) that calls DAGClient.submit().
- Adds a skeleton TezStats class that extends PigStats. This class needs to be implemented to monitor job status after submission.
- Adds a unit test called TestTezLauncher. For now, all it does is to compile queries and submit a DAG. More meaningful unit tests will be added after TezStats is implemented.
- Adds a TezLocalExecType. To start up a YARN mini cluster, I needed non-local Tez mode. So I created "tez_local" and "tez" modes.
- Whitespace and indentation fixes.
Diffs (updated)
-----
build.xml 0439082
src/META-INF/services/org.apache.pig.ExecType e995cd5
src/org/apache/pig/backend/executionengine/ExecutionEngine.java 7d480d7
src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java b886f9d
src/org/apache/pig/backend/hadoop/executionengine/Launcher.java 77543c2
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java f3be738
src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 5414906
src/org/apache/pig/backend/hadoop/executionengine/tez/TezExecType.java 8315e76
src/org/apache/pig/backend/hadoop/executionengine/tez/TezJob.java 04ffba0
src/org/apache/pig/backend/hadoop/executionengine/tez/TezJobControlCompiler.java b7d4d98
src/org/apache/pig/backend/hadoop/executionengine/tez/TezLauncher.java ff9e950
src/org/apache/pig/backend/hadoop/executionengine/tez/TezLocalExecType.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/tez/TezStats.java PRE-CREATION
src/org/apache/pig/impl/PigContext.java 225526f
src/org/apache/pig/impl/util/Utils.java 3720c94
test/org/apache/pig/tez/TestTezCompiler.java b9e6b48
test/org/apache/pig/tez/TestTezJobControlCompiler.java f14d49d
test/org/apache/pig/tez/TestTezLauncher.java PRE-CREATION
Diff: https://reviews.apache.org/r/14601/diff/
Testing
-------
TestTezLauncher is added.
Thanks,
Cheolsoo Park