You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Matei Zaharia (JIRA)" <ji...@apache.org> on 2010/10/19 02:50:27 UTC
[jira] Created: (MAPREDUCE-2141) Add an "extra data" field to Task
for use by Mesos
Add an "extra data" field to Task for use by Mesos
--------------------------------------------------
Key: MAPREDUCE-2141
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2141
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Matei Zaharia
Assignee: Matei Zaharia
Priority: Minor
In order to support running Hadoop on the Mesos cluster manager (http://mesos.berkeley.edu/), I'd like to add an extra String field to the Task class to allow extra data (a Mesos task ID) to be associated with each task. This should have no impact on normal operation other than making the serialized form of Task a few bytes longer. In the Mesos support patch for Hadoop, this field is set by a pluggable Hadoop scheduler implementation to allow code on the TaskTracker side to see which Mesos task corresponds to each Hadoop task.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2141) Add an "extra data" field to
Task for use by Mesos
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922420#action_12922420 ]
Matei Zaharia commented on MAPREDUCE-2141:
------------------------------------------
Thanks for the comments, Owen and Arun. We actually plan to submit Mesos to the Apache incubator soon, so maybe it will become Apache software eventually! I'll get back to you with an updated patch.
> Add an "extra data" field to Task for use by Mesos
> --------------------------------------------------
>
> Key: MAPREDUCE-2141
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2141
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Priority: Minor
> Attachments: mapreduce-2141-v1.patch
>
>
> In order to support running Hadoop on the Mesos cluster manager (http://mesos.berkeley.edu/), I'd like to add an extra String field to the Task class to allow extra data (a Mesos task ID) to be associated with each task. This should have no impact on normal operation other than making the serialized form of Task a few bytes longer. In the Mesos support patch for Hadoop, this field is set by a pluggable Hadoop scheduler implementation to allow code on the TaskTracker side to see which Mesos task corresponds to each Hadoop task.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2141) Add an "extra data" field to Task
for use by Mesos
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2141:
-------------------------------------
Status: Open (was: Patch Available)
Matei, the patch itself is small enough that it probably doesn't matter, but I'd still like to call out that this is a first for Hadoop Map-Reduce i.e. explicit support for non-Apache software such as Mesos.
Personally I am happy to commit this, especially given it's innocuous nature - but I'd like to ensure others in the community are fine with this. Tom? Dhruba? (I see you guys are watching this.)
----
Also, it might be worth-while to make this a 'context object' rather than a String.
> Add an "extra data" field to Task for use by Mesos
> --------------------------------------------------
>
> Key: MAPREDUCE-2141
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2141
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Priority: Minor
> Attachments: mapreduce-2141-v1.patch
>
>
> In order to support running Hadoop on the Mesos cluster manager (http://mesos.berkeley.edu/), I'd like to add an extra String field to the Task class to allow extra data (a Mesos task ID) to be associated with each task. This should have no impact on normal operation other than making the serialized form of Task a few bytes longer. In the Mesos support patch for Hadoop, this field is set by a pluggable Hadoop scheduler implementation to allow code on the TaskTracker side to see which Mesos task corresponds to each Hadoop task.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2141) Add an "extra data" field to Task
for use by Mesos
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matei Zaharia updated MAPREDUCE-2141:
-------------------------------------
Attachment: mapreduce-2141-v2.patch
Here's an updated patch that uses BytesWritable instead.
> Add an "extra data" field to Task for use by Mesos
> --------------------------------------------------
>
> Key: MAPREDUCE-2141
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2141
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Priority: Minor
> Attachments: mapreduce-2141-v1.patch, mapreduce-2141-v2.patch
>
>
> In order to support running Hadoop on the Mesos cluster manager (http://mesos.berkeley.edu/), I'd like to add an extra String field to the Task class to allow extra data (a Mesos task ID) to be associated with each task. This should have no impact on normal operation other than making the serialized form of Task a few bytes longer. In the Mesos support patch for Hadoop, this field is set by a pluggable Hadoop scheduler implementation to allow code on the TaskTracker side to see which Mesos task corresponds to each Hadoop task.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2141) Add an "extra data" field to
Task for use by Mesos
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922423#action_12922423 ]
Tom White commented on MAPREDUCE-2141:
--------------------------------------
Arun, I'm happy with this (using BytesWritable) too.
> Add an "extra data" field to Task for use by Mesos
> --------------------------------------------------
>
> Key: MAPREDUCE-2141
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2141
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Priority: Minor
> Attachments: mapreduce-2141-v1.patch
>
>
> In order to support running Hadoop on the Mesos cluster manager (http://mesos.berkeley.edu/), I'd like to add an extra String field to the Task class to allow extra data (a Mesos task ID) to be associated with each task. This should have no impact on normal operation other than making the serialized form of Task a few bytes longer. In the Mesos support patch for Hadoop, this field is set by a pluggable Hadoop scheduler implementation to allow code on the TaskTracker side to see which Mesos task corresponds to each Hadoop task.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2141) Add an "extra data" field to Task
for use by Mesos
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matei Zaharia updated MAPREDUCE-2141:
-------------------------------------
Attachment: mapreduce-2141-v1.patch
Here is a patch for this issue. It gives the new field package level access using get and set methods, which should be enough for our purposes.
> Add an "extra data" field to Task for use by Mesos
> --------------------------------------------------
>
> Key: MAPREDUCE-2141
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2141
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Priority: Minor
> Attachments: mapreduce-2141-v1.patch
>
>
> In order to support running Hadoop on the Mesos cluster manager (http://mesos.berkeley.edu/), I'd like to add an extra String field to the Task class to allow extra data (a Mesos task ID) to be associated with each task. This should have no impact on normal operation other than making the serialized form of Task a few bytes longer. In the Mesos support patch for Hadoop, this field is set by a pluggable Hadoop scheduler implementation to allow code on the TaskTracker side to see which Mesos task corresponds to each Hadoop task.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2141) Add an "extra data" field to
Task for use by Mesos
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930373#action_12930373 ]
Owen O'Malley commented on MAPREDUCE-2141:
------------------------------------------
+1, looks good.
> Add an "extra data" field to Task for use by Mesos
> --------------------------------------------------
>
> Key: MAPREDUCE-2141
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2141
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Priority: Minor
> Attachments: mapreduce-2141-v1.patch, mapreduce-2141-v2.patch
>
>
> In order to support running Hadoop on the Mesos cluster manager (http://mesos.berkeley.edu/), I'd like to add an extra String field to the Task class to allow extra data (a Mesos task ID) to be associated with each task. This should have no impact on normal operation other than making the serialized form of Task a few bytes longer. In the Mesos support patch for Hadoop, this field is set by a pluggable Hadoop scheduler implementation to allow code on the TaskTracker side to see which Mesos task corresponds to each Hadoop task.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2141) Add an "extra data" field to
Task for use by Mesos
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924537#action_12924537 ]
Hadoop QA commented on MAPREDUCE-2141:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12457646/mapreduce-2141-v2.patch
against trunk revision 1026450.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
+1 system tests framework. The patch passed system tests framework compile.
Test results: http://hudson.zones.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/3/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/3/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/3/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/3/console
This message is automatically generated.
> Add an "extra data" field to Task for use by Mesos
> --------------------------------------------------
>
> Key: MAPREDUCE-2141
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2141
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Priority: Minor
> Attachments: mapreduce-2141-v1.patch, mapreduce-2141-v2.patch
>
>
> In order to support running Hadoop on the Mesos cluster manager (http://mesos.berkeley.edu/), I'd like to add an extra String field to the Task class to allow extra data (a Mesos task ID) to be associated with each task. This should have no impact on normal operation other than making the serialized form of Task a few bytes longer. In the Mesos support patch for Hadoop, this field is set by a pluggable Hadoop scheduler implementation to allow code on the TaskTracker side to see which Mesos task corresponds to each Hadoop task.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2141) Add an "extra data" field to
Task for use by Mesos
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924702#action_12924702 ]
Matei Zaharia commented on MAPREDUCE-2141:
------------------------------------------
The test failures seem to be unrelated to the patch. Let me know if there's anything else I need to do for it.
> Add an "extra data" field to Task for use by Mesos
> --------------------------------------------------
>
> Key: MAPREDUCE-2141
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2141
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Priority: Minor
> Attachments: mapreduce-2141-v1.patch, mapreduce-2141-v2.patch
>
>
> In order to support running Hadoop on the Mesos cluster manager (http://mesos.berkeley.edu/), I'd like to add an extra String field to the Task class to allow extra data (a Mesos task ID) to be associated with each task. This should have no impact on normal operation other than making the serialized form of Task a few bytes longer. In the Mesos support patch for Hadoop, this field is set by a pluggable Hadoop scheduler implementation to allow code on the TaskTracker side to see which Mesos task corresponds to each Hadoop task.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2141) Add an "extra data" field to
Task for use by Mesos
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922416#action_12922416 ]
Todd Lipcon commented on MAPREDUCE-2141:
----------------------------------------
+1 for BytesWritable instead of String. Encoding is a pain in the butt.
> Add an "extra data" field to Task for use by Mesos
> --------------------------------------------------
>
> Key: MAPREDUCE-2141
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2141
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Priority: Minor
> Attachments: mapreduce-2141-v1.patch
>
>
> In order to support running Hadoop on the Mesos cluster manager (http://mesos.berkeley.edu/), I'd like to add an extra String field to the Task class to allow extra data (a Mesos task ID) to be associated with each task. This should have no impact on normal operation other than making the serialized form of Task a few bytes longer. In the Mesos support patch for Hadoop, this field is set by a pluggable Hadoop scheduler implementation to allow code on the TaskTracker side to see which Mesos task corresponds to each Hadoop task.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2141) Add an "extra data" field to
Task for use by Mesos
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922409#action_12922409 ]
Owen O'Malley commented on MAPREDUCE-2141:
------------------------------------------
I'd suggest using either BytesWritable or Text rather than String. Other than that it looks good. Some additional JavaDoc that explained that the field was for the scheduler would be helpful too, I think.
> Add an "extra data" field to Task for use by Mesos
> --------------------------------------------------
>
> Key: MAPREDUCE-2141
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2141
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Priority: Minor
> Attachments: mapreduce-2141-v1.patch
>
>
> In order to support running Hadoop on the Mesos cluster manager (http://mesos.berkeley.edu/), I'd like to add an extra String field to the Task class to allow extra data (a Mesos task ID) to be associated with each task. This should have no impact on normal operation other than making the serialized form of Task a few bytes longer. In the Mesos support patch for Hadoop, this field is set by a pluggable Hadoop scheduler implementation to allow code on the TaskTracker side to see which Mesos task corresponds to each Hadoop task.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2141) Add an "extra data" field to Task
for use by Mesos
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matei Zaharia updated MAPREDUCE-2141:
-------------------------------------
Status: Patch Available (was: Open)
> Add an "extra data" field to Task for use by Mesos
> --------------------------------------------------
>
> Key: MAPREDUCE-2141
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2141
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Priority: Minor
> Attachments: mapreduce-2141-v1.patch, mapreduce-2141-v2.patch
>
>
> In order to support running Hadoop on the Mesos cluster manager (http://mesos.berkeley.edu/), I'd like to add an extra String field to the Task class to allow extra data (a Mesos task ID) to be associated with each task. This should have no impact on normal operation other than making the serialized form of Task a few bytes longer. In the Mesos support patch for Hadoop, this field is set by a pluggable Hadoop scheduler implementation to allow code on the TaskTracker side to see which Mesos task corresponds to each Hadoop task.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2141) Add an "extra data" field to
Task for use by Mesos
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922433#action_12922433 ]
dhruba borthakur commented on MAPREDUCE-2141:
---------------------------------------------
+1 on getting this into Apache Hadoop!
> Add an "extra data" field to Task for use by Mesos
> --------------------------------------------------
>
> Key: MAPREDUCE-2141
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2141
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Priority: Minor
> Attachments: mapreduce-2141-v1.patch
>
>
> In order to support running Hadoop on the Mesos cluster manager (http://mesos.berkeley.edu/), I'd like to add an extra String field to the Task class to allow extra data (a Mesos task ID) to be associated with each task. This should have no impact on normal operation other than making the serialized form of Task a few bytes longer. In the Mesos support patch for Hadoop, this field is set by a pluggable Hadoop scheduler implementation to allow code on the TaskTracker side to see which Mesos task corresponds to each Hadoop task.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2141) Add an "extra data" field to Task
for use by Mesos
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matei Zaharia updated MAPREDUCE-2141:
-------------------------------------
Status: Patch Available (was: Open)
> Add an "extra data" field to Task for use by Mesos
> --------------------------------------------------
>
> Key: MAPREDUCE-2141
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2141
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Priority: Minor
> Attachments: mapreduce-2141-v1.patch
>
>
> In order to support running Hadoop on the Mesos cluster manager (http://mesos.berkeley.edu/), I'd like to add an extra String field to the Task class to allow extra data (a Mesos task ID) to be associated with each task. This should have no impact on normal operation other than making the serialized form of Task a few bytes longer. In the Mesos support patch for Hadoop, this field is set by a pluggable Hadoop scheduler implementation to allow code on the TaskTracker side to see which Mesos task corresponds to each Hadoop task.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.