You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Vinod Kone <vi...@gmail.com> on 2013/01/15 23:29:22 UTC
Review Request: Improved Hadoop port for mesos
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8965/
-----------------------------------------------------------
Review request for mesos, Benjamin Hindman, Matei Zaharia, and Ben Mahler.
Description
-------
Features:
--> Mesos scheduler is a thin shim that delegates the actual scheduling of task trackers to the underlying scheduler.
--> Mesos executor is a thin shim that starts/stops task tracker properly.
--> Doesn't depend on mesos specific patches (status updates, task tracker instrumentation) for hadoop.
--> Builds mesos executor bundle that can be used when running on a real mesos cluster.
Diffs
-----
hadoop/Makefile.am 46d16f7c03cda217d1851d78515cad96f60cba2b
hadoop/NOTES PRE-CREATION
hadoop/TUTORIAL.sh fca22c5b7bd485a5d18c99cb6666fab6bb0a06bb
hadoop/hadoop-0.20.2-cdh3u3.patch 84a948f23122a25fb41dfe497a73bff0c8e0867d
hadoop/hadoop-0.20.2-cdh3u3_hadoop-env.sh.patch b1731e27b1f59d8dc913d259d8f32a69f46ac0b0
hadoop/hadoop-0.20.2-cdh3u3_mesos.patch f93e53e286525829bdc2a3da71eefab8f093e5b8
hadoop/hadoop-0.20.205.0.patch cab488e402343bdbd5d0c4101785b9a48331f85e
hadoop/hadoop-0.20.205.0_hadoop-env.sh.patch 1b8f4d06323a3ca3f624d9374066e29be5a18acd
hadoop/mapred-site.xml.patch c2d6397d8fc1d662afd606e1069ff469c3556e19
hadoop/mesos-executor ce8dcccf7eaecd35dfa041a78aa9760dc2ce451c
hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkExecutor.java 9238fe53a718c47ff4bbe2d555d1c072fa8bd6e2
hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java 754b198a5c7baacc92ce2adc96da15aa97c47ad1
hadoop/mesos/src/java/org/apache/hadoop/mapred/HadoopFrameworkMessage.java 1e9510873bab962250afbaf81a5c63a6b191e299
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java PRE-CREATION
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 1408f300a43eab043b40e4d313e6632cb21fd7b2
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosTaskTrackerInstrumentation.java 8c517d80e54555e12f906a219b12eb572ed7e3a3
Diff: https://reviews.apache.org/r/8965/diff/
Testing
-------
make hadoop-0.20.2-cdh3u3
make hadoop-0.20.205.0
Also, verified that the generator executor bundle is self-contained and works as expected for non-local runs.
Thanks,
Vinod Kone
Re: Review Request: Improved Hadoop port for mesos
Posted by Vinod Kone <vi...@gmail.com>.
> On Jan. 16, 2013, 10:27 p.m., Ben Mahler wrote:
> > Any line limit rules for Java? I've seen 100 and 120 as rules in the past.
this is 80 per sun's conventions which is what hadoop apparently follows, save for the indentation length. http://wiki.apache.org/hadoop/CodeReviewChecklist
> On Jan. 16, 2013, 10:27 p.m., Ben Mahler wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java, line 28
> > <https://reviews.apache.org/r/8965/diff/1/?file=248619#file248619line28>
> >
> > argument formatting
that format is per sun's conventions. i'm going to send a formatting only diff later, to make sure our code conforms to hadoop/sun's guidelines.
> On Jan. 16, 2013, 10:27 p.m., Ben Mahler wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java, line 33
> > <https://reviews.apache.org/r/8965/diff/1/?file=248619#file248619line33>
> >
> > fits on one line?
nope, the line width is 80 chars.
> On Jan. 16, 2013, 10:27 p.m., Ben Mahler wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java, line 59
> > <https://reviews.apache.org/r/8965/diff/1/?file=248619#file248619line59>
> >
> > LOG calls take a throwable as a second argument, so replace with:
> >
> > LOG.fatal("Failed to start task tracker", e);
> > // Kill the e.printStackTrace();
sweet.
> On Jan. 16, 2013, 10:27 p.m., Ben Mahler wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java, line 74
> > <https://reviews.apache.org/r/8965/diff/1/?file=248619#file248619line74>
> >
> > Strange line wrapping.
?
> On Jan. 16, 2013, 10:27 p.m., Ben Mahler wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, line 145
> > <https://reviews.apache.org/r/8965/diff/1/?file=248620#file248620line145>
> >
> > This log message doesn't make sense, given the status task ID is completed, not pending..
agreed. killed it
> On Jan. 16, 2013, 10:27 p.m., Ben Mahler wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, line 199
> > <https://reviews.apache.org/r/8965/diff/1/?file=248620#file248620line199>
> >
> > How is this different than not setting the user?
this is a required field.
> On Jan. 16, 2013, 10:27 p.m., Ben Mahler wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, line 200
> > <https://reviews.apache.org/r/8965/diff/1/?file=248620#file248620line200>
> >
> > Do we need the identifier?
i'm indifferent, but killed it
> On Jan. 16, 2013, 10:27 p.m., Ben Mahler wrote:
> > hadoop/mapred-site.xml.patch, line 41
> > <https://reviews.apache.org/r/8965/diff/1/?file=248614#file248614line41>
> >
> > what is this file:///? Add an explanation?
axed this altogether
> On Jan. 16, 2013, 10:27 p.m., Ben Mahler wrote:
> > hadoop/TUTORIAL.sh, line 629
> > <https://reviews.apache.org/r/8965/diff/1/?file=248608#file248608line629>
> >
> > We should include a comment with respect to the slaves needing to have >= 5 cores, >= X RAM, >= Y DISK. And how they can change that by reducing the slots. Even with reducing, we still will require 3 cores.
> >
> > This might be problematic for people. So a comment here and/or in our config patch would be great.
this info is now taken from the config file. i've included comments there.
- Vinod
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8965/#review15406
-----------------------------------------------------------
On Jan. 15, 2013, 10:29 p.m., Vinod Kone wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8965/
> -----------------------------------------------------------
>
> (Updated Jan. 15, 2013, 10:29 p.m.)
>
>
> Review request for mesos, Benjamin Hindman, Matei Zaharia, and Ben Mahler.
>
>
> Description
> -------
>
> Features:
>
> --> Mesos scheduler is a thin shim that delegates the actual scheduling of task trackers to the underlying scheduler.
> --> Mesos executor is a thin shim that starts/stops task tracker properly.
> --> Doesn't depend on mesos specific patches (status updates, task tracker instrumentation) for hadoop.
> --> Builds mesos executor bundle that can be used when running on a real mesos cluster.
>
>
> Diffs
> -----
>
> hadoop/Makefile.am 46d16f7c03cda217d1851d78515cad96f60cba2b
> hadoop/NOTES PRE-CREATION
> hadoop/TUTORIAL.sh fca22c5b7bd485a5d18c99cb6666fab6bb0a06bb
> hadoop/hadoop-0.20.2-cdh3u3.patch 84a948f23122a25fb41dfe497a73bff0c8e0867d
> hadoop/hadoop-0.20.2-cdh3u3_hadoop-env.sh.patch b1731e27b1f59d8dc913d259d8f32a69f46ac0b0
> hadoop/hadoop-0.20.2-cdh3u3_mesos.patch f93e53e286525829bdc2a3da71eefab8f093e5b8
> hadoop/hadoop-0.20.205.0.patch cab488e402343bdbd5d0c4101785b9a48331f85e
> hadoop/hadoop-0.20.205.0_hadoop-env.sh.patch 1b8f4d06323a3ca3f624d9374066e29be5a18acd
> hadoop/mapred-site.xml.patch c2d6397d8fc1d662afd606e1069ff469c3556e19
> hadoop/mesos-executor ce8dcccf7eaecd35dfa041a78aa9760dc2ce451c
> hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkExecutor.java 9238fe53a718c47ff4bbe2d555d1c072fa8bd6e2
> hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java 754b198a5c7baacc92ce2adc96da15aa97c47ad1
> hadoop/mesos/src/java/org/apache/hadoop/mapred/HadoopFrameworkMessage.java 1e9510873bab962250afbaf81a5c63a6b191e299
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java PRE-CREATION
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 1408f300a43eab043b40e4d313e6632cb21fd7b2
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosTaskTrackerInstrumentation.java 8c517d80e54555e12f906a219b12eb572ed7e3a3
>
> Diff: https://reviews.apache.org/r/8965/diff/
>
>
> Testing
> -------
>
> make hadoop-0.20.2-cdh3u3
> make hadoop-0.20.205.0
>
> Also, verified that the generator executor bundle is self-contained and works as expected for non-local runs.
>
>
> Thanks,
>
> Vinod Kone
>
>
Re: Review Request: Improved Hadoop port for mesos
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8965/#review15406
-----------------------------------------------------------
Any line limit rules for Java? I've seen 100 and 120 as rules in the past.
hadoop/NOTES
<https://reviews.apache.org/r/8965/#comment33178>
s/JAVA/Java
hadoop/TUTORIAL.sh
<https://reviews.apache.org/r/8965/#comment33179>
Reword:
Build mesos executor package to be downloaded and executed on mesos slaves.
hadoop/TUTORIAL.sh
<https://reviews.apache.org/r/8965/#comment33182>
$ rm -rf cloudera # Only for cdh3.
hadoop/TUTORIAL.sh
<https://reviews.apache.org/r/8965/#comment33180>
end with a period
hadoop/TUTORIAL.sh
<https://reviews.apache.org/r/8965/#comment33181>
s/spacei/space
Also mention that this is only for cdh3
hadoop/TUTORIAL.sh
<https://reviews.apache.org/r/8965/#comment33183>
end with period
hadoop/TUTORIAL.sh
<https://reviews.apache.org/r/8965/#comment33206>
We should include a comment with respect to the slaves needing to have >= 5 cores, >= X RAM, >= Y DISK. And how they can change that by reducing the slots. Even with reducing, we still will require 3 cores.
This might be problematic for people. So a comment here and/or in our config patch would be great.
hadoop/mapred-site.xml.patch
<https://reviews.apache.org/r/8965/#comment33212>
what is this file:///? Add an explanation?
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java
<https://reviews.apache.org/r/8965/#comment33227>
Can you kill this member variable?
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java
<https://reviews.apache.org/r/8965/#comment33218>
argument formatting
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java
<https://reviews.apache.org/r/8965/#comment33217>
Can you add a log line to each callback?
namely: registered, reregistered, disconnected, frameworkMessage, shutdown
This should prove useful for debugging.
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java
<https://reviews.apache.org/r/8965/#comment33219>
fits on one line?
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java
<https://reviews.apache.org/r/8965/#comment33220>
Can you add a comment as to why we need this?
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java
<https://reviews.apache.org/r/8965/#comment33221>
LOG calls take a throwable as a second argument, so replace with:
LOG.fatal("Failed to start task tracker", e);
// Kill the e.printStackTrace();
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java
<https://reviews.apache.org/r/8965/#comment33222>
ditto
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java
<https://reviews.apache.org/r/8965/#comment33223>
Capitalize: TaskTracker Run Thread
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java
<https://reviews.apache.org/r/8965/#comment33224>
Inline the @Override for anonymous classes
@Override public void run() {
}
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java
<https://reviews.apache.org/r/8965/#comment33225>
Strange line wrapping.
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java
<https://reviews.apache.org/r/8965/#comment33226>
Pass e as an argument instead, and use "Failed to sleep" style string.
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java
<https://reviews.apache.org/r/8965/#comment33228>
ditto from above, kill the member variable and just do:
new Thread() {
...
}.start();
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java
<https://reviews.apache.org/r/8965/#comment33229>
Ditto for passing e.
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java
<https://reviews.apache.org/r/8965/#comment33230>
Ditto for passing e.
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/8965/#comment33231>
This never gets set to true, is it needed?
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/8965/#comment33233>
Revise my TODO:
s/to run a TaskTracker/to run a default (4 slot) TaskTracker
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/8965/#comment33234>
Maybe just say "corresponding idle task trackers." instead?
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/8965/#comment33235>
I think this 'if' statement no longer makes sense, can you remove it?
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/8965/#comment33236>
This log message doesn't make sense, given the status task ID is completed, not pending..
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/8965/#comment33237>
How is this different than not setting the user?
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/8965/#comment33240>
Do we need the identifier?
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/8965/#comment33232>
running is always false here
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/8965/#comment33242>
Can you update my TODO here to be just:
TODO(bmahler): Consider allowing non-mesos TaskTrackers.
I'm pretty sure the second sentence is no longer applicable, since I refactored.
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/8965/#comment33244>
What a beautiful, informative, and eloquent log message!! I wonder who added this ;)
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/8965/#comment33245>
Kill my TODO here, I ended up fixing this.
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/8965/#comment33246>
Can you move this try/catch into the if local block?
Also, pass e as an argument and use "Failed to find.."
- Ben Mahler
On Jan. 15, 2013, 10:29 p.m., Vinod Kone wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8965/
> -----------------------------------------------------------
>
> (Updated Jan. 15, 2013, 10:29 p.m.)
>
>
> Review request for mesos, Benjamin Hindman, Matei Zaharia, and Ben Mahler.
>
>
> Description
> -------
>
> Features:
>
> --> Mesos scheduler is a thin shim that delegates the actual scheduling of task trackers to the underlying scheduler.
> --> Mesos executor is a thin shim that starts/stops task tracker properly.
> --> Doesn't depend on mesos specific patches (status updates, task tracker instrumentation) for hadoop.
> --> Builds mesos executor bundle that can be used when running on a real mesos cluster.
>
>
> Diffs
> -----
>
> hadoop/Makefile.am 46d16f7c03cda217d1851d78515cad96f60cba2b
> hadoop/NOTES PRE-CREATION
> hadoop/TUTORIAL.sh fca22c5b7bd485a5d18c99cb6666fab6bb0a06bb
> hadoop/hadoop-0.20.2-cdh3u3.patch 84a948f23122a25fb41dfe497a73bff0c8e0867d
> hadoop/hadoop-0.20.2-cdh3u3_hadoop-env.sh.patch b1731e27b1f59d8dc913d259d8f32a69f46ac0b0
> hadoop/hadoop-0.20.2-cdh3u3_mesos.patch f93e53e286525829bdc2a3da71eefab8f093e5b8
> hadoop/hadoop-0.20.205.0.patch cab488e402343bdbd5d0c4101785b9a48331f85e
> hadoop/hadoop-0.20.205.0_hadoop-env.sh.patch 1b8f4d06323a3ca3f624d9374066e29be5a18acd
> hadoop/mapred-site.xml.patch c2d6397d8fc1d662afd606e1069ff469c3556e19
> hadoop/mesos-executor ce8dcccf7eaecd35dfa041a78aa9760dc2ce451c
> hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkExecutor.java 9238fe53a718c47ff4bbe2d555d1c072fa8bd6e2
> hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java 754b198a5c7baacc92ce2adc96da15aa97c47ad1
> hadoop/mesos/src/java/org/apache/hadoop/mapred/HadoopFrameworkMessage.java 1e9510873bab962250afbaf81a5c63a6b191e299
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java PRE-CREATION
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 1408f300a43eab043b40e4d313e6632cb21fd7b2
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosTaskTrackerInstrumentation.java 8c517d80e54555e12f906a219b12eb572ed7e3a3
>
> Diff: https://reviews.apache.org/r/8965/diff/
>
>
> Testing
> -------
>
> make hadoop-0.20.2-cdh3u3
> make hadoop-0.20.205.0
>
> Also, verified that the generator executor bundle is self-contained and works as expected for non-local runs.
>
>
> Thanks,
>
> Vinod Kone
>
>
Re: Review Request: Improved Hadoop port for mesos
Posted by Vinod Kone <vi...@gmail.com>.
> On Jan. 16, 2013, 6:34 p.m., Ben Mahler wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, line 126
> > <https://reviews.apache.org/r/8965/diff/1/?file=248620#file248620line126>
> >
> > Here, we should only consider those MesosTrackers with active = true, otherwise we will kill trackers that have yet to heartbeat.
done
> On Jan. 16, 2013, 6:34 p.m., Ben Mahler wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, line 529
> > <https://reviews.apache.org/r/8965/diff/1/?file=248620#file248620line529>
> >
> > We'll also need to remove the tracker here on any terminal status.
done
- Vinod
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8965/#review15396
-----------------------------------------------------------
On Jan. 15, 2013, 10:29 p.m., Vinod Kone wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8965/
> -----------------------------------------------------------
>
> (Updated Jan. 15, 2013, 10:29 p.m.)
>
>
> Review request for mesos, Benjamin Hindman, Matei Zaharia, and Ben Mahler.
>
>
> Description
> -------
>
> Features:
>
> --> Mesos scheduler is a thin shim that delegates the actual scheduling of task trackers to the underlying scheduler.
> --> Mesos executor is a thin shim that starts/stops task tracker properly.
> --> Doesn't depend on mesos specific patches (status updates, task tracker instrumentation) for hadoop.
> --> Builds mesos executor bundle that can be used when running on a real mesos cluster.
>
>
> Diffs
> -----
>
> hadoop/Makefile.am 46d16f7c03cda217d1851d78515cad96f60cba2b
> hadoop/NOTES PRE-CREATION
> hadoop/TUTORIAL.sh fca22c5b7bd485a5d18c99cb6666fab6bb0a06bb
> hadoop/hadoop-0.20.2-cdh3u3.patch 84a948f23122a25fb41dfe497a73bff0c8e0867d
> hadoop/hadoop-0.20.2-cdh3u3_hadoop-env.sh.patch b1731e27b1f59d8dc913d259d8f32a69f46ac0b0
> hadoop/hadoop-0.20.2-cdh3u3_mesos.patch f93e53e286525829bdc2a3da71eefab8f093e5b8
> hadoop/hadoop-0.20.205.0.patch cab488e402343bdbd5d0c4101785b9a48331f85e
> hadoop/hadoop-0.20.205.0_hadoop-env.sh.patch 1b8f4d06323a3ca3f624d9374066e29be5a18acd
> hadoop/mapred-site.xml.patch c2d6397d8fc1d662afd606e1069ff469c3556e19
> hadoop/mesos-executor ce8dcccf7eaecd35dfa041a78aa9760dc2ce451c
> hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkExecutor.java 9238fe53a718c47ff4bbe2d555d1c072fa8bd6e2
> hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java 754b198a5c7baacc92ce2adc96da15aa97c47ad1
> hadoop/mesos/src/java/org/apache/hadoop/mapred/HadoopFrameworkMessage.java 1e9510873bab962250afbaf81a5c63a6b191e299
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java PRE-CREATION
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 1408f300a43eab043b40e4d313e6632cb21fd7b2
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosTaskTrackerInstrumentation.java 8c517d80e54555e12f906a219b12eb572ed7e3a3
>
> Diff: https://reviews.apache.org/r/8965/diff/
>
>
> Testing
> -------
>
> make hadoop-0.20.2-cdh3u3
> make hadoop-0.20.205.0
>
> Also, verified that the generator executor bundle is self-contained and works as expected for non-local runs.
>
>
> Thanks,
>
> Vinod Kone
>
>
Re: Review Request: Improved Hadoop port for mesos
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8965/#review15396
-----------------------------------------------------------
Thanks for taking on the review for this.
Just some comments for issues I introduced when refactoring the scheduler.
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/8965/#comment33140>
Here, we should only consider those MesosTrackers with active = true, otherwise we will kill trackers that have yet to heartbeat.
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/8965/#comment33138>
We'll also need to remove the tracker here on any terminal status.
- Ben Mahler
On Jan. 15, 2013, 10:29 p.m., Vinod Kone wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8965/
> -----------------------------------------------------------
>
> (Updated Jan. 15, 2013, 10:29 p.m.)
>
>
> Review request for mesos, Benjamin Hindman, Matei Zaharia, and Ben Mahler.
>
>
> Description
> -------
>
> Features:
>
> --> Mesos scheduler is a thin shim that delegates the actual scheduling of task trackers to the underlying scheduler.
> --> Mesos executor is a thin shim that starts/stops task tracker properly.
> --> Doesn't depend on mesos specific patches (status updates, task tracker instrumentation) for hadoop.
> --> Builds mesos executor bundle that can be used when running on a real mesos cluster.
>
>
> Diffs
> -----
>
> hadoop/Makefile.am 46d16f7c03cda217d1851d78515cad96f60cba2b
> hadoop/NOTES PRE-CREATION
> hadoop/TUTORIAL.sh fca22c5b7bd485a5d18c99cb6666fab6bb0a06bb
> hadoop/hadoop-0.20.2-cdh3u3.patch 84a948f23122a25fb41dfe497a73bff0c8e0867d
> hadoop/hadoop-0.20.2-cdh3u3_hadoop-env.sh.patch b1731e27b1f59d8dc913d259d8f32a69f46ac0b0
> hadoop/hadoop-0.20.2-cdh3u3_mesos.patch f93e53e286525829bdc2a3da71eefab8f093e5b8
> hadoop/hadoop-0.20.205.0.patch cab488e402343bdbd5d0c4101785b9a48331f85e
> hadoop/hadoop-0.20.205.0_hadoop-env.sh.patch 1b8f4d06323a3ca3f624d9374066e29be5a18acd
> hadoop/mapred-site.xml.patch c2d6397d8fc1d662afd606e1069ff469c3556e19
> hadoop/mesos-executor ce8dcccf7eaecd35dfa041a78aa9760dc2ce451c
> hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkExecutor.java 9238fe53a718c47ff4bbe2d555d1c072fa8bd6e2
> hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java 754b198a5c7baacc92ce2adc96da15aa97c47ad1
> hadoop/mesos/src/java/org/apache/hadoop/mapred/HadoopFrameworkMessage.java 1e9510873bab962250afbaf81a5c63a6b191e299
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java PRE-CREATION
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 1408f300a43eab043b40e4d313e6632cb21fd7b2
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosTaskTrackerInstrumentation.java 8c517d80e54555e12f906a219b12eb572ed7e3a3
>
> Diff: https://reviews.apache.org/r/8965/diff/
>
>
> Testing
> -------
>
> make hadoop-0.20.2-cdh3u3
> make hadoop-0.20.205.0
>
> Also, verified that the generator executor bundle is self-contained and works as expected for non-local runs.
>
>
> Thanks,
>
> Vinod Kone
>
>
Re: Review Request: Improved Hadoop port for mesos
Posted by Vinod Kone <vi...@gmail.com>.
> On Jan. 25, 2013, 4:04 a.m., Ben Mahler wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, line 236
> > <https://reviews.apache.org/r/8965/diff/4/?file=251921#file251921line236>
> >
> > Could also be expired.
if its expired, this call wouldn't be made by the jobtracker!
> On Jan. 25, 2013, 4:04 a.m., Ben Mahler wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, line 237
> > <https://reviews.apache.org/r/8965/diff/4/?file=251921#file251921line237>
> >
> > so null is handled?
> >
> > seems like it would expect an empty list returned intead..
no, null is handled correctly by the jobtracker.
> On Jan. 25, 2013, 4:04 a.m., Ben Mahler wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, line 587
> > <https://reviews.apache.org/r/8965/diff/4/?file=251921#file251921line587>
> >
> > I think a switch (with all the states) is clearer here, but up to you.
'if' seems more appropriate here.
> On Jan. 25, 2013, 4:04 a.m., Ben Mahler wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, line 304
> > <https://reviews.apache.org/r/8965/diff/4/?file=251921#file251921line304>
> >
> > I think the logic here is incorrect on this latest review.
> >
> > We want to know about the "available" slots. That is, slots that are not currently in use. (status.getAvailableMapSlots()).
> >
> > Whereas here we consider all launched slots as available, regardless of whether or not they are in use.
> >
> > So.. to fix this:
> > kill the launched vars here
> >
> > and the loop above becomes:
> >
> > idleMapSlots = 0;
> > idleReduceSlots = 0;
> > for (TaskTrackerStatus status : jobTracker.taskTrackers()) {
> > HttpHost host = new HttpHost(status.getHost(), status.getHttpPort());
> > if (mesosTrackers.containsKey(host)) {
> > mesosTrackers.get(host).active = true;
> > idleMapSlots += status.getAvailableMapSlots();
> > idleReduceSlots += status.getAvailableReduceSlots();
> > }
> > }
> >
> > make sense?
makes total sense. thank you.
- Vinod
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8965/#review15674
-----------------------------------------------------------
On Jan. 25, 2013, 2:32 a.m., Vinod Kone wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8965/
> -----------------------------------------------------------
>
> (Updated Jan. 25, 2013, 2:32 a.m.)
>
>
> Review request for mesos, Benjamin Hindman and Ben Mahler.
>
>
> Description
> -------
>
> Features:
>
> --> Mesos scheduler is a thin shim that delegates the actual scheduling of task trackers to the underlying scheduler.
> --> Mesos executor is a thin shim that starts/stops task tracker properly.
> --> Doesn't depend on mesos specific patches (status updates, task tracker instrumentation) for hadoop.
> --> Builds mesos executor bundle that can be used when running on a real mesos cluster.
>
>
> Diffs
> -----
>
> hadoop/Makefile.am 46d16f7c03cda217d1851d78515cad96f60cba2b
> hadoop/NOTES PRE-CREATION
> hadoop/TUTORIAL.sh fca22c5b7bd485a5d18c99cb6666fab6bb0a06bb
> hadoop/hadoop-0.20.2-cdh3u3.patch 84a948f23122a25fb41dfe497a73bff0c8e0867d
> hadoop/hadoop-0.20.2-cdh3u3_hadoop-env.sh.patch b1731e27b1f59d8dc913d259d8f32a69f46ac0b0
> hadoop/hadoop-0.20.2-cdh3u3_mesos.patch f93e53e286525829bdc2a3da71eefab8f093e5b8
> hadoop/hadoop-0.20.205.0.patch cab488e402343bdbd5d0c4101785b9a48331f85e
> hadoop/hadoop-0.20.205.0_hadoop-env.sh.patch 1b8f4d06323a3ca3f624d9374066e29be5a18acd
> hadoop/hadoop-7698-1.patch PRE-CREATION
> hadoop/mapred-site.xml.patch c2d6397d8fc1d662afd606e1069ff469c3556e19
> hadoop/mesos-executor ce8dcccf7eaecd35dfa041a78aa9760dc2ce451c
> hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkExecutor.java 9238fe53a718c47ff4bbe2d555d1c072fa8bd6e2
> hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java 754b198a5c7baacc92ce2adc96da15aa97c47ad1
> hadoop/mesos/src/java/org/apache/hadoop/mapred/HadoopFrameworkMessage.java 1e9510873bab962250afbaf81a5c63a6b191e299
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java PRE-CREATION
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 1408f300a43eab043b40e4d313e6632cb21fd7b2
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosTaskTrackerInstrumentation.java 8c517d80e54555e12f906a219b12eb572ed7e3a3
>
> Diff: https://reviews.apache.org/r/8965/diff/
>
>
> Testing
> -------
>
> make hadoop-0.20.2-cdh3u3
> make hadoop-0.20.205.0
>
> Also, verified that the generator executor bundle is self-contained and works as expected for non-local runs.
>
>
> Thanks,
>
> Vinod Kone
>
>
Re: Review Request: Improved Hadoop port for mesos
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8965/#review15674
-----------------------------------------------------------
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java
<https://reviews.apache.org/r/8965/#comment33805>
This reads strangely:
Setting config option : mesos.master : local
How about:
Setting config option 'mesos.master' to 'local'
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java
<https://reviews.apache.org/r/8965/#comment33814>
fatal, and did you want to use the "Failed to start..." format?
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java
<https://reviews.apache.org/r/8965/#comment33806>
kill this unnecessary e.printStackTrace()?
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java
<https://reviews.apache.org/r/8965/#comment33813>
fatal
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java
<https://reviews.apache.org/r/8965/#comment33807>
ditto
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/8965/#comment33808>
You can kill my TODO here, now that you've added the slot options.
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/8965/#comment33809>
Can you update this comment with the correct numbers?
Also, you can change this from a TODO to a NOTE.
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/8965/#comment33810>
These can now become final again, since we no longer change them for local runs.
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/8965/#comment33811>
kill the '.'
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/8965/#comment33812>
Given the amount of nested scope here (8 close braces), can you re-structure this a bit?
jobUpdated(...) {
JobInProgress job = event.getJobInProgress();
if (!job.isComplete()) return;
LOG.info("Completed job : " + job.getJobID());
synchronized (MesosScheduler.this) {
// the kill logic
}
}
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/8965/#comment33815>
Could also be expired.
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/8965/#comment33816>
so null is handled?
seems like it would expect an empty list returned intead..
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/8965/#comment33817>
s/Compute/Mark
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/8965/#comment33818>
launch excessive trackers
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/8965/#comment33819>
I think the logic here is incorrect on this latest review.
We want to know about the "available" slots. That is, slots that are not currently in use. (status.getAvailableMapSlots()).
Whereas here we consider all launched slots as available, regardless of whether or not they are in use.
So.. to fix this:
kill the launched vars here
and the loop above becomes:
idleMapSlots = 0;
idleReduceSlots = 0;
for (TaskTrackerStatus status : jobTracker.taskTrackers()) {
HttpHost host = new HttpHost(status.getHost(), status.getHttpPort());
if (mesosTrackers.containsKey(host)) {
mesosTrackers.get(host).active = true;
idleMapSlots += status.getAvailableMapSlots();
idleReduceSlots += status.getAvailableReduceSlots();
}
}
make sense?
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/8965/#comment33820>
I think a switch (with all the states) is clearer here, but up to you.
- Ben Mahler
On Jan. 25, 2013, 2:32 a.m., Vinod Kone wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8965/
> -----------------------------------------------------------
>
> (Updated Jan. 25, 2013, 2:32 a.m.)
>
>
> Review request for mesos, Benjamin Hindman and Ben Mahler.
>
>
> Description
> -------
>
> Features:
>
> --> Mesos scheduler is a thin shim that delegates the actual scheduling of task trackers to the underlying scheduler.
> --> Mesos executor is a thin shim that starts/stops task tracker properly.
> --> Doesn't depend on mesos specific patches (status updates, task tracker instrumentation) for hadoop.
> --> Builds mesos executor bundle that can be used when running on a real mesos cluster.
>
>
> Diffs
> -----
>
> hadoop/Makefile.am 46d16f7c03cda217d1851d78515cad96f60cba2b
> hadoop/NOTES PRE-CREATION
> hadoop/TUTORIAL.sh fca22c5b7bd485a5d18c99cb6666fab6bb0a06bb
> hadoop/hadoop-0.20.2-cdh3u3.patch 84a948f23122a25fb41dfe497a73bff0c8e0867d
> hadoop/hadoop-0.20.2-cdh3u3_hadoop-env.sh.patch b1731e27b1f59d8dc913d259d8f32a69f46ac0b0
> hadoop/hadoop-0.20.2-cdh3u3_mesos.patch f93e53e286525829bdc2a3da71eefab8f093e5b8
> hadoop/hadoop-0.20.205.0.patch cab488e402343bdbd5d0c4101785b9a48331f85e
> hadoop/hadoop-0.20.205.0_hadoop-env.sh.patch 1b8f4d06323a3ca3f624d9374066e29be5a18acd
> hadoop/hadoop-7698-1.patch PRE-CREATION
> hadoop/mapred-site.xml.patch c2d6397d8fc1d662afd606e1069ff469c3556e19
> hadoop/mesos-executor ce8dcccf7eaecd35dfa041a78aa9760dc2ce451c
> hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkExecutor.java 9238fe53a718c47ff4bbe2d555d1c072fa8bd6e2
> hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java 754b198a5c7baacc92ce2adc96da15aa97c47ad1
> hadoop/mesos/src/java/org/apache/hadoop/mapred/HadoopFrameworkMessage.java 1e9510873bab962250afbaf81a5c63a6b191e299
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java PRE-CREATION
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 1408f300a43eab043b40e4d313e6632cb21fd7b2
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosTaskTrackerInstrumentation.java 8c517d80e54555e12f906a219b12eb572ed7e3a3
>
> Diff: https://reviews.apache.org/r/8965/diff/
>
>
> Testing
> -------
>
> make hadoop-0.20.2-cdh3u3
> make hadoop-0.20.205.0
>
> Also, verified that the generator executor bundle is self-contained and works as expected for non-local runs.
>
>
> Thanks,
>
> Vinod Kone
>
>
Re: Review Request: Improved Hadoop port for mesos
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8965/
-----------------------------------------------------------
(Updated Jan. 28, 2013, 1:42 a.m.)
Review request for mesos, Benjamin Hindman and Ben Mahler.
Changes
-------
oops. sorry. attached
Description
-------
Features:
--> Mesos scheduler is a thin shim that delegates the actual scheduling of task trackers to the underlying scheduler.
--> Mesos executor is a thin shim that starts/stops task tracker properly.
--> Doesn't depend on mesos specific patches (status updates, task tracker instrumentation) for hadoop.
--> Builds mesos executor bundle that can be used when running on a real mesos cluster.
Diffs (updated)
-----
hadoop/Makefile.am 46d16f7c03cda217d1851d78515cad96f60cba2b
hadoop/NOTES PRE-CREATION
hadoop/TUTORIAL.sh fca22c5b7bd485a5d18c99cb6666fab6bb0a06bb
hadoop/hadoop-0.20.2-cdh3u3.patch 84a948f23122a25fb41dfe497a73bff0c8e0867d
hadoop/hadoop-0.20.2-cdh3u3_hadoop-env.sh.patch b1731e27b1f59d8dc913d259d8f32a69f46ac0b0
hadoop/hadoop-0.20.2-cdh3u3_mesos.patch f93e53e286525829bdc2a3da71eefab8f093e5b8
hadoop/hadoop-0.20.205.0.patch cab488e402343bdbd5d0c4101785b9a48331f85e
hadoop/hadoop-0.20.205.0_hadoop-env.sh.patch 1b8f4d06323a3ca3f624d9374066e29be5a18acd
hadoop/hadoop-7698-1.patch PRE-CREATION
hadoop/hadoop-gridmix.patch PRE-CREATION
hadoop/mapred-site.xml.patch c2d6397d8fc1d662afd606e1069ff469c3556e19
hadoop/mesos-executor ce8dcccf7eaecd35dfa041a78aa9760dc2ce451c
hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkExecutor.java 9238fe53a718c47ff4bbe2d555d1c072fa8bd6e2
hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java 754b198a5c7baacc92ce2adc96da15aa97c47ad1
hadoop/mesos/src/java/org/apache/hadoop/mapred/HadoopFrameworkMessage.java 1e9510873bab962250afbaf81a5c63a6b191e299
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java PRE-CREATION
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 1408f300a43eab043b40e4d313e6632cb21fd7b2
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosTaskTrackerInstrumentation.java 8c517d80e54555e12f906a219b12eb572ed7e3a3
Diff: https://reviews.apache.org/r/8965/diff/
Testing
-------
make hadoop-0.20.2-cdh3u3
make hadoop-0.20.205.0
Also, verified that the generator executor bundle is self-contained and works as expected for non-local runs.
Thanks,
Vinod Kone
Re: Review Request: Improved Hadoop port for mesos
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8965/#review15710
-----------------------------------------------------------
Ship it!
Ship It!
- Ben Mahler
On Jan. 25, 2013, 7:20 p.m., Vinod Kone wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8965/
> -----------------------------------------------------------
>
> (Updated Jan. 25, 2013, 7:20 p.m.)
>
>
> Review request for mesos, Benjamin Hindman and Ben Mahler.
>
>
> Description
> -------
>
> Features:
>
> --> Mesos scheduler is a thin shim that delegates the actual scheduling of task trackers to the underlying scheduler.
> --> Mesos executor is a thin shim that starts/stops task tracker properly.
> --> Doesn't depend on mesos specific patches (status updates, task tracker instrumentation) for hadoop.
> --> Builds mesos executor bundle that can be used when running on a real mesos cluster.
>
>
> Diffs
> -----
>
> hadoop/Makefile.am 46d16f7c03cda217d1851d78515cad96f60cba2b
> hadoop/NOTES PRE-CREATION
> hadoop/TUTORIAL.sh fca22c5b7bd485a5d18c99cb6666fab6bb0a06bb
> hadoop/hadoop-0.20.2-cdh3u3.patch 84a948f23122a25fb41dfe497a73bff0c8e0867d
> hadoop/hadoop-0.20.2-cdh3u3_hadoop-env.sh.patch b1731e27b1f59d8dc913d259d8f32a69f46ac0b0
> hadoop/hadoop-0.20.2-cdh3u3_mesos.patch f93e53e286525829bdc2a3da71eefab8f093e5b8
> hadoop/hadoop-0.20.205.0.patch cab488e402343bdbd5d0c4101785b9a48331f85e
> hadoop/hadoop-0.20.205.0_hadoop-env.sh.patch 1b8f4d06323a3ca3f624d9374066e29be5a18acd
> hadoop/hadoop-7698-1.patch PRE-CREATION
> hadoop/mapred-site.xml.patch c2d6397d8fc1d662afd606e1069ff469c3556e19
> hadoop/mesos-executor ce8dcccf7eaecd35dfa041a78aa9760dc2ce451c
> hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkExecutor.java 9238fe53a718c47ff4bbe2d555d1c072fa8bd6e2
> hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java 754b198a5c7baacc92ce2adc96da15aa97c47ad1
> hadoop/mesos/src/java/org/apache/hadoop/mapred/HadoopFrameworkMessage.java 1e9510873bab962250afbaf81a5c63a6b191e299
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java PRE-CREATION
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 1408f300a43eab043b40e4d313e6632cb21fd7b2
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosTaskTrackerInstrumentation.java 8c517d80e54555e12f906a219b12eb572ed7e3a3
>
> Diff: https://reviews.apache.org/r/8965/diff/
>
>
> Testing
> -------
>
> make hadoop-0.20.2-cdh3u3
> make hadoop-0.20.205.0
>
> Also, verified that the generator executor bundle is self-contained and works as expected for non-local runs.
>
>
> Thanks,
>
> Vinod Kone
>
>
Re: Review Request: Improved Hadoop port for mesos
Posted by Benjamin Hindman <be...@berkeley.edu>.
> On Jan. 28, 2013, 12:51 a.m., Benjamin Hindman wrote:
> > Ship It!
Looks like you're actually missing 'hadoop-gridmix.patch'.
- Benjamin
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8965/#review15740
-----------------------------------------------------------
On Jan. 25, 2013, 7:20 p.m., Vinod Kone wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8965/
> -----------------------------------------------------------
>
> (Updated Jan. 25, 2013, 7:20 p.m.)
>
>
> Review request for mesos, Benjamin Hindman and Ben Mahler.
>
>
> Description
> -------
>
> Features:
>
> --> Mesos scheduler is a thin shim that delegates the actual scheduling of task trackers to the underlying scheduler.
> --> Mesos executor is a thin shim that starts/stops task tracker properly.
> --> Doesn't depend on mesos specific patches (status updates, task tracker instrumentation) for hadoop.
> --> Builds mesos executor bundle that can be used when running on a real mesos cluster.
>
>
> Diffs
> -----
>
> hadoop/Makefile.am 46d16f7c03cda217d1851d78515cad96f60cba2b
> hadoop/NOTES PRE-CREATION
> hadoop/TUTORIAL.sh fca22c5b7bd485a5d18c99cb6666fab6bb0a06bb
> hadoop/hadoop-0.20.2-cdh3u3.patch 84a948f23122a25fb41dfe497a73bff0c8e0867d
> hadoop/hadoop-0.20.2-cdh3u3_hadoop-env.sh.patch b1731e27b1f59d8dc913d259d8f32a69f46ac0b0
> hadoop/hadoop-0.20.2-cdh3u3_mesos.patch f93e53e286525829bdc2a3da71eefab8f093e5b8
> hadoop/hadoop-0.20.205.0.patch cab488e402343bdbd5d0c4101785b9a48331f85e
> hadoop/hadoop-0.20.205.0_hadoop-env.sh.patch 1b8f4d06323a3ca3f624d9374066e29be5a18acd
> hadoop/hadoop-7698-1.patch PRE-CREATION
> hadoop/mapred-site.xml.patch c2d6397d8fc1d662afd606e1069ff469c3556e19
> hadoop/mesos-executor ce8dcccf7eaecd35dfa041a78aa9760dc2ce451c
> hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkExecutor.java 9238fe53a718c47ff4bbe2d555d1c072fa8bd6e2
> hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java 754b198a5c7baacc92ce2adc96da15aa97c47ad1
> hadoop/mesos/src/java/org/apache/hadoop/mapred/HadoopFrameworkMessage.java 1e9510873bab962250afbaf81a5c63a6b191e299
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java PRE-CREATION
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 1408f300a43eab043b40e4d313e6632cb21fd7b2
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosTaskTrackerInstrumentation.java 8c517d80e54555e12f906a219b12eb572ed7e3a3
>
> Diff: https://reviews.apache.org/r/8965/diff/
>
>
> Testing
> -------
>
> make hadoop-0.20.2-cdh3u3
> make hadoop-0.20.205.0
>
> Also, verified that the generator executor bundle is self-contained and works as expected for non-local runs.
>
>
> Thanks,
>
> Vinod Kone
>
>
Re: Review Request: Improved Hadoop port for mesos
Posted by Benjamin Hindman <be...@berkeley.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8965/#review15740
-----------------------------------------------------------
Ship it!
Ship It!
- Benjamin Hindman
On Jan. 25, 2013, 7:20 p.m., Vinod Kone wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8965/
> -----------------------------------------------------------
>
> (Updated Jan. 25, 2013, 7:20 p.m.)
>
>
> Review request for mesos, Benjamin Hindman and Ben Mahler.
>
>
> Description
> -------
>
> Features:
>
> --> Mesos scheduler is a thin shim that delegates the actual scheduling of task trackers to the underlying scheduler.
> --> Mesos executor is a thin shim that starts/stops task tracker properly.
> --> Doesn't depend on mesos specific patches (status updates, task tracker instrumentation) for hadoop.
> --> Builds mesos executor bundle that can be used when running on a real mesos cluster.
>
>
> Diffs
> -----
>
> hadoop/Makefile.am 46d16f7c03cda217d1851d78515cad96f60cba2b
> hadoop/NOTES PRE-CREATION
> hadoop/TUTORIAL.sh fca22c5b7bd485a5d18c99cb6666fab6bb0a06bb
> hadoop/hadoop-0.20.2-cdh3u3.patch 84a948f23122a25fb41dfe497a73bff0c8e0867d
> hadoop/hadoop-0.20.2-cdh3u3_hadoop-env.sh.patch b1731e27b1f59d8dc913d259d8f32a69f46ac0b0
> hadoop/hadoop-0.20.2-cdh3u3_mesos.patch f93e53e286525829bdc2a3da71eefab8f093e5b8
> hadoop/hadoop-0.20.205.0.patch cab488e402343bdbd5d0c4101785b9a48331f85e
> hadoop/hadoop-0.20.205.0_hadoop-env.sh.patch 1b8f4d06323a3ca3f624d9374066e29be5a18acd
> hadoop/hadoop-7698-1.patch PRE-CREATION
> hadoop/mapred-site.xml.patch c2d6397d8fc1d662afd606e1069ff469c3556e19
> hadoop/mesos-executor ce8dcccf7eaecd35dfa041a78aa9760dc2ce451c
> hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkExecutor.java 9238fe53a718c47ff4bbe2d555d1c072fa8bd6e2
> hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java 754b198a5c7baacc92ce2adc96da15aa97c47ad1
> hadoop/mesos/src/java/org/apache/hadoop/mapred/HadoopFrameworkMessage.java 1e9510873bab962250afbaf81a5c63a6b191e299
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java PRE-CREATION
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 1408f300a43eab043b40e4d313e6632cb21fd7b2
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosTaskTrackerInstrumentation.java 8c517d80e54555e12f906a219b12eb572ed7e3a3
>
> Diff: https://reviews.apache.org/r/8965/diff/
>
>
> Testing
> -------
>
> make hadoop-0.20.2-cdh3u3
> make hadoop-0.20.205.0
>
> Also, verified that the generator executor bundle is self-contained and works as expected for non-local runs.
>
>
> Thanks,
>
> Vinod Kone
>
>
Re: Review Request: Improved Hadoop port for mesos
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8965/
-----------------------------------------------------------
(Updated Jan. 25, 2013, 7:20 p.m.)
Review request for mesos, Benjamin Hindman and Ben Mahler.
Changes
-------
benm's comments
Description
-------
Features:
--> Mesos scheduler is a thin shim that delegates the actual scheduling of task trackers to the underlying scheduler.
--> Mesos executor is a thin shim that starts/stops task tracker properly.
--> Doesn't depend on mesos specific patches (status updates, task tracker instrumentation) for hadoop.
--> Builds mesos executor bundle that can be used when running on a real mesos cluster.
Diffs (updated)
-----
hadoop/Makefile.am 46d16f7c03cda217d1851d78515cad96f60cba2b
hadoop/NOTES PRE-CREATION
hadoop/TUTORIAL.sh fca22c5b7bd485a5d18c99cb6666fab6bb0a06bb
hadoop/hadoop-0.20.2-cdh3u3.patch 84a948f23122a25fb41dfe497a73bff0c8e0867d
hadoop/hadoop-0.20.2-cdh3u3_hadoop-env.sh.patch b1731e27b1f59d8dc913d259d8f32a69f46ac0b0
hadoop/hadoop-0.20.2-cdh3u3_mesos.patch f93e53e286525829bdc2a3da71eefab8f093e5b8
hadoop/hadoop-0.20.205.0.patch cab488e402343bdbd5d0c4101785b9a48331f85e
hadoop/hadoop-0.20.205.0_hadoop-env.sh.patch 1b8f4d06323a3ca3f624d9374066e29be5a18acd
hadoop/hadoop-7698-1.patch PRE-CREATION
hadoop/mapred-site.xml.patch c2d6397d8fc1d662afd606e1069ff469c3556e19
hadoop/mesos-executor ce8dcccf7eaecd35dfa041a78aa9760dc2ce451c
hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkExecutor.java 9238fe53a718c47ff4bbe2d555d1c072fa8bd6e2
hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java 754b198a5c7baacc92ce2adc96da15aa97c47ad1
hadoop/mesos/src/java/org/apache/hadoop/mapred/HadoopFrameworkMessage.java 1e9510873bab962250afbaf81a5c63a6b191e299
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java PRE-CREATION
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 1408f300a43eab043b40e4d313e6632cb21fd7b2
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosTaskTrackerInstrumentation.java 8c517d80e54555e12f906a219b12eb572ed7e3a3
Diff: https://reviews.apache.org/r/8965/diff/
Testing
-------
make hadoop-0.20.2-cdh3u3
make hadoop-0.20.205.0
Also, verified that the generator executor bundle is self-contained and works as expected for non-local runs.
Thanks,
Vinod Kone
Re: Review Request: Improved Hadoop port for mesos
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8965/
-----------------------------------------------------------
(Updated Jan. 25, 2013, 2:32 a.m.)
Review request for mesos, Benjamin Hindman and Ben Mahler.
Changes
-------
Formatted the code according to hadoop's (== eclipse java profile with indent=2 instead of 4) formatting style.
adjusted some formatted lines, to read better.
also consistent capitalization for Mesos, Hadoop, TaskTracker and JobTracker in comments and logs.
Description
-------
Features:
--> Mesos scheduler is a thin shim that delegates the actual scheduling of task trackers to the underlying scheduler.
--> Mesos executor is a thin shim that starts/stops task tracker properly.
--> Doesn't depend on mesos specific patches (status updates, task tracker instrumentation) for hadoop.
--> Builds mesos executor bundle that can be used when running on a real mesos cluster.
Diffs (updated)
-----
hadoop/Makefile.am 46d16f7c03cda217d1851d78515cad96f60cba2b
hadoop/NOTES PRE-CREATION
hadoop/TUTORIAL.sh fca22c5b7bd485a5d18c99cb6666fab6bb0a06bb
hadoop/hadoop-0.20.2-cdh3u3.patch 84a948f23122a25fb41dfe497a73bff0c8e0867d
hadoop/hadoop-0.20.2-cdh3u3_hadoop-env.sh.patch b1731e27b1f59d8dc913d259d8f32a69f46ac0b0
hadoop/hadoop-0.20.2-cdh3u3_mesos.patch f93e53e286525829bdc2a3da71eefab8f093e5b8
hadoop/hadoop-0.20.205.0.patch cab488e402343bdbd5d0c4101785b9a48331f85e
hadoop/hadoop-0.20.205.0_hadoop-env.sh.patch 1b8f4d06323a3ca3f624d9374066e29be5a18acd
hadoop/hadoop-7698-1.patch PRE-CREATION
hadoop/mapred-site.xml.patch c2d6397d8fc1d662afd606e1069ff469c3556e19
hadoop/mesos-executor ce8dcccf7eaecd35dfa041a78aa9760dc2ce451c
hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkExecutor.java 9238fe53a718c47ff4bbe2d555d1c072fa8bd6e2
hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java 754b198a5c7baacc92ce2adc96da15aa97c47ad1
hadoop/mesos/src/java/org/apache/hadoop/mapred/HadoopFrameworkMessage.java 1e9510873bab962250afbaf81a5c63a6b191e299
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java PRE-CREATION
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 1408f300a43eab043b40e4d313e6632cb21fd7b2
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosTaskTrackerInstrumentation.java 8c517d80e54555e12f906a219b12eb572ed7e3a3
Diff: https://reviews.apache.org/r/8965/diff/
Testing
-------
make hadoop-0.20.2-cdh3u3
make hadoop-0.20.205.0
Also, verified that the generator executor bundle is self-contained and works as expected for non-local runs.
Thanks,
Vinod Kone
Re: Review Request: Improved Hadoop port for mesos
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8965/
-----------------------------------------------------------
(Updated Jan. 25, 2013, 1:42 a.m.)
Review request for mesos, Benjamin Hindman and Ben Mahler.
Changes
-------
benh's offline comments:
--> deleted hadoop-0.20.2-cdh3u3.patch and hadoop-0.20.205.0.patch
--> gridmix specific patch is called hadoop-gridmix.patch
--> jsvc patch is called hadoop-7698-1.patch (the same name it was given on apache's jira ticket)
--> explained why we fake resources in the tutorial.
i will send another diff with formatting-only changes.
Description
-------
Features:
--> Mesos scheduler is a thin shim that delegates the actual scheduling of task trackers to the underlying scheduler.
--> Mesos executor is a thin shim that starts/stops task tracker properly.
--> Doesn't depend on mesos specific patches (status updates, task tracker instrumentation) for hadoop.
--> Builds mesos executor bundle that can be used when running on a real mesos cluster.
Diffs (updated)
-----
hadoop/Makefile.am 46d16f7c03cda217d1851d78515cad96f60cba2b
hadoop/NOTES PRE-CREATION
hadoop/TUTORIAL.sh fca22c5b7bd485a5d18c99cb6666fab6bb0a06bb
hadoop/hadoop-0.20.2-cdh3u3.patch 84a948f23122a25fb41dfe497a73bff0c8e0867d
hadoop/hadoop-0.20.2-cdh3u3_hadoop-env.sh.patch b1731e27b1f59d8dc913d259d8f32a69f46ac0b0
hadoop/hadoop-0.20.2-cdh3u3_mesos.patch f93e53e286525829bdc2a3da71eefab8f093e5b8
hadoop/hadoop-0.20.205.0.patch cab488e402343bdbd5d0c4101785b9a48331f85e
hadoop/hadoop-0.20.205.0_hadoop-env.sh.patch 1b8f4d06323a3ca3f624d9374066e29be5a18acd
hadoop/hadoop-7698-1.patch PRE-CREATION
hadoop/mapred-site.xml.patch c2d6397d8fc1d662afd606e1069ff469c3556e19
hadoop/mesos-executor ce8dcccf7eaecd35dfa041a78aa9760dc2ce451c
hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkExecutor.java 9238fe53a718c47ff4bbe2d555d1c072fa8bd6e2
hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java 754b198a5c7baacc92ce2adc96da15aa97c47ad1
hadoop/mesos/src/java/org/apache/hadoop/mapred/HadoopFrameworkMessage.java 1e9510873bab962250afbaf81a5c63a6b191e299
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java PRE-CREATION
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 1408f300a43eab043b40e4d313e6632cb21fd7b2
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosTaskTrackerInstrumentation.java 8c517d80e54555e12f906a219b12eb572ed7e3a3
Diff: https://reviews.apache.org/r/8965/diff/
Testing
-------
make hadoop-0.20.2-cdh3u3
make hadoop-0.20.205.0
Also, verified that the generator executor bundle is self-contained and works as expected for non-local runs.
Thanks,
Vinod Kone
Re: Review Request: Improved Hadoop port for mesos
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8965/
-----------------------------------------------------------
(Updated Jan. 23, 2013, 8:24 p.m.)
Review request for mesos, Benjamin Hindman and Ben Mahler.
Changes
-------
Bens' comments.
Rebased off trunk.
Description
-------
Features:
--> Mesos scheduler is a thin shim that delegates the actual scheduling of task trackers to the underlying scheduler.
--> Mesos executor is a thin shim that starts/stops task tracker properly.
--> Doesn't depend on mesos specific patches (status updates, task tracker instrumentation) for hadoop.
--> Builds mesos executor bundle that can be used when running on a real mesos cluster.
Diffs (updated)
-----
hadoop/Makefile.am 46d16f7c03cda217d1851d78515cad96f60cba2b
hadoop/NOTES PRE-CREATION
hadoop/TUTORIAL.sh fca22c5b7bd485a5d18c99cb6666fab6bb0a06bb
hadoop/hadoop-0.20.2-cdh3u3.patch 84a948f23122a25fb41dfe497a73bff0c8e0867d
hadoop/hadoop-0.20.2-cdh3u3_hadoop-env.sh.patch b1731e27b1f59d8dc913d259d8f32a69f46ac0b0
hadoop/hadoop-0.20.2-cdh3u3_mesos.patch f93e53e286525829bdc2a3da71eefab8f093e5b8
hadoop/hadoop-0.20.205.0.patch cab488e402343bdbd5d0c4101785b9a48331f85e
hadoop/hadoop-0.20.205.0_hadoop-env.sh.patch 1b8f4d06323a3ca3f624d9374066e29be5a18acd
hadoop/mapred-site.xml.patch c2d6397d8fc1d662afd606e1069ff469c3556e19
hadoop/mesos-executor ce8dcccf7eaecd35dfa041a78aa9760dc2ce451c
hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkExecutor.java 9238fe53a718c47ff4bbe2d555d1c072fa8bd6e2
hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java 754b198a5c7baacc92ce2adc96da15aa97c47ad1
hadoop/mesos/src/java/org/apache/hadoop/mapred/HadoopFrameworkMessage.java 1e9510873bab962250afbaf81a5c63a6b191e299
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java PRE-CREATION
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 1408f300a43eab043b40e4d313e6632cb21fd7b2
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosTaskTrackerInstrumentation.java 8c517d80e54555e12f906a219b12eb572ed7e3a3
Diff: https://reviews.apache.org/r/8965/diff/
Testing
-------
make hadoop-0.20.2-cdh3u3
make hadoop-0.20.205.0
Also, verified that the generator executor bundle is self-contained and works as expected for non-local runs.
Thanks,
Vinod Kone
Re: Review Request: Improved Hadoop port for mesos
Posted by Vinod Kone <vi...@gmail.com>.
> On Jan. 17, 2013, 11:30 p.m., Benjamin Hindman wrote:
> > hadoop/TUTORIAL.sh, line 168
> > <https://reviews.apache.org/r/8965/diff/1/?file=248608#file248608line168>
> >
> > Can we please make this fit in 80 characters?
apparently i can tell vim to highlight when text width exceeds certain chars!
highlight OverLength ctermbg=red ctermfg=white guibg=#592929
match OverLength /\%81v.\+/
- Vinod
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8965/#review15467
-----------------------------------------------------------
On Jan. 15, 2013, 10:29 p.m., Vinod Kone wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8965/
> -----------------------------------------------------------
>
> (Updated Jan. 15, 2013, 10:29 p.m.)
>
>
> Review request for mesos, Benjamin Hindman, Matei Zaharia, and Ben Mahler.
>
>
> Description
> -------
>
> Features:
>
> --> Mesos scheduler is a thin shim that delegates the actual scheduling of task trackers to the underlying scheduler.
> --> Mesos executor is a thin shim that starts/stops task tracker properly.
> --> Doesn't depend on mesos specific patches (status updates, task tracker instrumentation) for hadoop.
> --> Builds mesos executor bundle that can be used when running on a real mesos cluster.
>
>
> Diffs
> -----
>
> hadoop/Makefile.am 46d16f7c03cda217d1851d78515cad96f60cba2b
> hadoop/NOTES PRE-CREATION
> hadoop/TUTORIAL.sh fca22c5b7bd485a5d18c99cb6666fab6bb0a06bb
> hadoop/hadoop-0.20.2-cdh3u3.patch 84a948f23122a25fb41dfe497a73bff0c8e0867d
> hadoop/hadoop-0.20.2-cdh3u3_hadoop-env.sh.patch b1731e27b1f59d8dc913d259d8f32a69f46ac0b0
> hadoop/hadoop-0.20.2-cdh3u3_mesos.patch f93e53e286525829bdc2a3da71eefab8f093e5b8
> hadoop/hadoop-0.20.205.0.patch cab488e402343bdbd5d0c4101785b9a48331f85e
> hadoop/hadoop-0.20.205.0_hadoop-env.sh.patch 1b8f4d06323a3ca3f624d9374066e29be5a18acd
> hadoop/mapred-site.xml.patch c2d6397d8fc1d662afd606e1069ff469c3556e19
> hadoop/mesos-executor ce8dcccf7eaecd35dfa041a78aa9760dc2ce451c
> hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkExecutor.java 9238fe53a718c47ff4bbe2d555d1c072fa8bd6e2
> hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java 754b198a5c7baacc92ce2adc96da15aa97c47ad1
> hadoop/mesos/src/java/org/apache/hadoop/mapred/HadoopFrameworkMessage.java 1e9510873bab962250afbaf81a5c63a6b191e299
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java PRE-CREATION
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 1408f300a43eab043b40e4d313e6632cb21fd7b2
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosTaskTrackerInstrumentation.java 8c517d80e54555e12f906a219b12eb572ed7e3a3
>
> Diff: https://reviews.apache.org/r/8965/diff/
>
>
> Testing
> -------
>
> make hadoop-0.20.2-cdh3u3
> make hadoop-0.20.205.0
>
> Also, verified that the generator executor bundle is self-contained and works as expected for non-local runs.
>
>
> Thanks,
>
> Vinod Kone
>
>
Re: Review Request: Improved Hadoop port for mesos
Posted by Benjamin Hindman <be...@berkeley.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8965/#review15467
-----------------------------------------------------------
hadoop/NOTES
<https://reviews.apache.org/r/8965/#comment33378>
s/jsvc/'jsvc'/
hadoop/TUTORIAL.sh
<https://reviews.apache.org/r/8965/#comment33370>
s/to one of hadoop's contributions (GridMix)./ to GridMix, a contribution in Hadoop./
hadoop/TUTORIAL.sh
<https://reviews.apache.org/r/8965/#comment33377>
Can we please make this fit in 80 characters?
hadoop/TUTORIAL.sh
<https://reviews.apache.org/r/8965/#comment33371>
/mesos/Mesos/g
/hadoop/Hadoop/g
/hdfs/HDFS/g
hadoop/TUTORIAL.sh
<https://reviews.apache.org/r/8965/#comment33380>
Wrap as much as possible.
hadoop/TUTORIAL.sh
<https://reviews.apache.org/r/8965/#comment33385>
Kill.
hadoop/TUTORIAL.sh
<https://reviews.apache.org/r/8965/#comment33390>
Kill line.
hadoop/mapred-site.xml.patch
<https://reviews.apache.org/r/8965/#comment33409>
Let's comment this out in the patch and put a "dummy" value that includes hdfs (maybe with a comment that says don't forget to copy the executor yourself!).
hadoop/mapred-site.xml.patch
<https://reviews.apache.org/r/8965/#comment33396>
Unnecessary.
- Benjamin Hindman
On Jan. 15, 2013, 10:29 p.m., Vinod Kone wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8965/
> -----------------------------------------------------------
>
> (Updated Jan. 15, 2013, 10:29 p.m.)
>
>
> Review request for mesos, Benjamin Hindman, Matei Zaharia, and Ben Mahler.
>
>
> Description
> -------
>
> Features:
>
> --> Mesos scheduler is a thin shim that delegates the actual scheduling of task trackers to the underlying scheduler.
> --> Mesos executor is a thin shim that starts/stops task tracker properly.
> --> Doesn't depend on mesos specific patches (status updates, task tracker instrumentation) for hadoop.
> --> Builds mesos executor bundle that can be used when running on a real mesos cluster.
>
>
> Diffs
> -----
>
> hadoop/Makefile.am 46d16f7c03cda217d1851d78515cad96f60cba2b
> hadoop/NOTES PRE-CREATION
> hadoop/TUTORIAL.sh fca22c5b7bd485a5d18c99cb6666fab6bb0a06bb
> hadoop/hadoop-0.20.2-cdh3u3.patch 84a948f23122a25fb41dfe497a73bff0c8e0867d
> hadoop/hadoop-0.20.2-cdh3u3_hadoop-env.sh.patch b1731e27b1f59d8dc913d259d8f32a69f46ac0b0
> hadoop/hadoop-0.20.2-cdh3u3_mesos.patch f93e53e286525829bdc2a3da71eefab8f093e5b8
> hadoop/hadoop-0.20.205.0.patch cab488e402343bdbd5d0c4101785b9a48331f85e
> hadoop/hadoop-0.20.205.0_hadoop-env.sh.patch 1b8f4d06323a3ca3f624d9374066e29be5a18acd
> hadoop/mapred-site.xml.patch c2d6397d8fc1d662afd606e1069ff469c3556e19
> hadoop/mesos-executor ce8dcccf7eaecd35dfa041a78aa9760dc2ce451c
> hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkExecutor.java 9238fe53a718c47ff4bbe2d555d1c072fa8bd6e2
> hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java 754b198a5c7baacc92ce2adc96da15aa97c47ad1
> hadoop/mesos/src/java/org/apache/hadoop/mapred/HadoopFrameworkMessage.java 1e9510873bab962250afbaf81a5c63a6b191e299
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java PRE-CREATION
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 1408f300a43eab043b40e4d313e6632cb21fd7b2
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosTaskTrackerInstrumentation.java 8c517d80e54555e12f906a219b12eb572ed7e3a3
>
> Diff: https://reviews.apache.org/r/8965/diff/
>
>
> Testing
> -------
>
> make hadoop-0.20.2-cdh3u3
> make hadoop-0.20.205.0
>
> Also, verified that the generator executor bundle is self-contained and works as expected for non-local runs.
>
>
> Thanks,
>
> Vinod Kone
>
>