You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Amar Kamat (JIRA)" <ji...@apache.org> on 2008/09/16 15:54:44 UTC
[jira] Created: (HADOOP-4189) HADOOP-3245 is incomplete
HADOOP-3245 is incomplete
-------------------------
Key: HADOOP-4189
URL: https://issues.apache.org/jira/browse/HADOOP-4189
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Reporter: Amar Kamat
Assignee: Amar Kamat
Priority: Blocker
There are 2 issues with HADOOP-3245
- The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
- Same goes for the buffer size.
- InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-4189:
-------------------------------
Status: Patch Available (was: Open)
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.1.patch, HADOOP-4189-v3.patch, HADOOP-4189-v5.1.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635409#action_12635409 ]
Hudson commented on HADOOP-4189:
--------------------------------
Integrated in Hadoop-trunk #618 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/618/])
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.1.patch, HADOOP-4189-v3.patch, HADOOP-4189-v5.1.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-4189:
-------------------------------
Status: Open (was: Patch Available)
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.1.patch, HADOOP-4189-v3.patch, HADOOP-4189-v5.1.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12632518#action_12632518 ]
Arun C Murthy commented on HADOOP-4189:
---------------------------------------
I'm arguing that that the jobhistory code should use the right blocksize and buffersize, I don't see the value in making this configurable.
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.1.patch, HADOOP-4189-v3.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12633898#action_12633898 ]
Owen O'Malley commented on HADOOP-4189:
---------------------------------------
Ok, I think
* we should actually set the default in hadoop-default to the real value instead of 0, -1, or "".
* we should mark it expert
* if it isn't already, I think this should be only configured by the cluster, not the job.
As to the defaults, I guess the question is how many minutes of run are we willing to lose. Let's say 10 minutes, and let's say 1000 job events / minute at 100 bytes each. That would give a ballpark figure of 1mb, which seems reasonable.
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.1.patch, HADOOP-4189-v3.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-4189:
-------------------------------
Status: Open (was: Patch Available)
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-4189:
-------------------------------
Attachment: HADOOP-4189.patch
Straight forward patch.
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Attachments: HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-4189:
-------------------------------
Status: Open (was: Patch Available)
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.1.patch, HADOOP-4189-v3.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12632506#action_12632506 ]
Amar Kamat commented on HADOOP-4189:
------------------------------------
bq. Null config values are not the right way to go. At best it should be "-1"
I am not sure if -1 internally selects the system defaults. If nothing is passed, by default the history code selects filesystem's default values for block size and buffer.
bq. In these specific cases I'm not sure we even need to make them configurable.
With the system defaults, the % recovery is very less as history info is not frequently flushed to the disk. So controlling these parameters, one can control the extent of recovery. Again there is a tradeoff between how much to recover versus the dfs performance w.r.t low block-size/buffer-size.
bq. Do we need to bump up TaskUmbilicalProtocol's version?
I have added the comment in HADOOP-3245 but forgot to bump up the version number. So fixing it in this patch.
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.1.patch, HADOOP-4189-v3.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-4189:
-------------------------------
Status: Patch Available (was: Open)
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-4189:
-------------------------------
Attachment: HADOOP-4189-v1.patch
Attaching a updated patch that solves the issue of stale job history analysis. Pre HADOOP-3245, missing job state meant that the job is running. HADOOP-3245 introduces the RUNNING state for jobs which indicates that the job meta-info is available in the history. {{loadhistory.jsp}} cached the analysis results and refreshed only if the job status was missing (which meant that the job is running). Fixed that to explicitly test the job runstate as RUNNING. Tested the patch with and without restarts. Also checked if the history analysis works after the job is complete. I apologize for not testing this before, my bad.
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-4189:
--------------------------------------------
Hadoop Flags: [Reviewed]
+1 Patch looks good
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-4189:
-------------------------------
Status: Open (was: Patch Available)
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634034#action_12634034 ]
Amar Kamat commented on HADOOP-4189:
------------------------------------
bq. we should actually set the default in hadoop-default to the real value instead of 0, -1, or "".
+1.
bq. we should mark it expert
+1. Will add this to the comment
bq. if it isn't already, I think this should be only configured by the cluster, not the job.
This value is used when the jobtracker starts. So its per cluster and not per job.
bq. As to the defaults ....
On an avg I found ~300-350 characters per line. Which makes 600-700 bytes per line. Hence with 1mb block size we risk ~1750 lines, i.e 875 task updates (as they come in pairs). Which I think should be ok. Thoughts?
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.1.patch, HADOOP-4189-v3.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631866#action_12631866 ]
Hadoop QA commented on HADOOP-4189:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12390243/HADOOP-4189-v2.patch
against trunk revision 696228.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3285/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3285/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3285/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3285/console
This message is automatically generated.
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12632087#action_12632087 ]
Amareshwari Sriramadasu commented on HADOOP-4189:
-------------------------------------------------
In loadhistory.jsp, add a check for job.get(Keys.JOB_STATUS) == RUNNING, instead of check for not a SUCCESS.
Other than that patch looks good.
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-4189:
-------------------------------
Status: Patch Available (was: Open)
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Attachments: HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-4189:
----------------------------------
Affects Version/s: 0.19.0
Hadoop Flags: (was: [Reviewed])
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.1.patch, HADOOP-4189-v3.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-4189:
-------------------------------
Status: Patch Available (was: Open)
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-4189:
-------------------------------
Status: Open (was: Patch Available)
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Attachments: HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-4189:
-------------------------------
Attachment: HADOOP-4189-v2.patch
There was a corner case in the previous patch where in if the history is accessed after the history file is created and before the job status is logged, then the cached result will never be refreshed. Now the check makes sure that the analysis is redone whenever the job is incomplete. Tested this on 20 nodes and the issue to do with stale values after the job is complete is addressed. Its difficult to test this corner case.
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634185#action_12634185 ]
Owen O'Malley commented on HADOOP-4189:
---------------------------------------
{quote}
task updates (as they come in pairs)
{quote}
For restartability, we really should separate the start event from the finish event. The start event should be generated when the task starts, not when it ends. (Assuming that hasn't been changed already.)
{quote}
On an avg I found ~300-350 characters per line. Which makes 600-700 bytes per line
{quote}
Why does it double? Hopefully we are using UTF-8 encoding so it should be 1 byte/char.
If I assume it is actually 300 bytes per an event, I'd bump the default to 3mb.
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.1.patch, HADOOP-4189-v3.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-4189:
-------------------------------
Attachment: HADOOP-4189-v5.1.patch
Attaching a updated patch the sets the default job-history block size to 3MB.
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.1.patch, HADOOP-4189-v3.patch, HADOOP-4189-v5.1.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634372#action_12634372 ]
Amar Kamat commented on HADOOP-4189:
------------------------------------
bq. For restartability, we really should separate the start event from the finish event.
I dont think it really matters. Any task-attempt that has started but not closed is probably useless w.r.t recovery. Hence task-attempt which has a _START_ and _END_ is really what we want and hence we can keep it as it is.
bq. Why does it double? Hopefully we are using UTF-8 encoding so it should be 1 byte/char.
I checked on my local box and its ASCII. Assuming 3mb as the default, we now risk 5.2k tasks which should be ok.
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.1.patch, HADOOP-4189-v3.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley updated HADOOP-4189:
--------------------------------
Fix Version/s: 0.19.0
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-4189:
-------------------------------
Status: Patch Available (was: Open)
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-4189:
--------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Amar!
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.1.patch, HADOOP-4189-v3.patch, HADOOP-4189-v5.1.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634973#action_12634973 ]
Hadoop QA commented on HADOOP-4189:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12390995/HADOOP-4189-v5.1.patch
against trunk revision 699119.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3381/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3381/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3381/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3381/console
This message is automatically generated.
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.1.patch, HADOOP-4189-v3.patch, HADOOP-4189-v5.1.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12632423#action_12632423 ]
Hadoop QA commented on HADOOP-4189:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12390362/HADOOP-4189-v3.1.patch
against trunk revision 696777.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3306/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3306/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3306/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3306/console
This message is automatically generated.
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.1.patch, HADOOP-4189-v3.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-4189:
-------------------------------
Status: Patch Available (was: Open)
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.1.patch, HADOOP-4189-v3.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12632591#action_12632591 ]
Amar Kamat commented on HADOOP-4189:
------------------------------------
Arun,
I was not sure as to what the optimal values are. If we use default values from FileSystem then the amount of data recovered is very low. For long running jobs with less maps, these values should be low so that every update gets logged while for long running jobs with large maps, it is ok to use system defaults as the frequency of updates are high. So instead of using some heuristics and making it hardcoded, I thought its better to keep it (admin)configurable and then come up with some better heuristics or technique to set the defaults. Any thought on what the default values should be considering various types of jobs that can be run?
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.1.patch, HADOOP-4189-v3.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-4189:
-------------------------------
Status: Patch Available (was: Open)
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.1.patch, HADOOP-4189-v3.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-4189:
-------------------------------
Attachment: HADOOP-4189-v3.1.patch
Amareshwari pointed out a major bug, thanks for that. Attaching a new patch with the changes.
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.1.patch, HADOOP-4189-v3.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634753#action_12634753 ]
Amar Kamat commented on HADOOP-4189:
------------------------------------
_test-patch_ and _ant test_ passed on my box.
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.1.patch, HADOOP-4189-v3.patch, HADOOP-4189-v5.1.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635287#action_12635287 ]
Amareshwari Sriramadasu commented on HADOOP-4189:
-------------------------------------------------
+1
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.1.patch, HADOOP-4189-v3.patch, HADOOP-4189-v5.1.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-4189:
----------------------------------
Status: Open (was: Patch Available)
Null config values are not the right way to go. At best it should be "-1" in these cases to let the system pick the default values.
In these specific cases I'm not sure we even need to make them configurable... the system should just pick the right values?
Do we need to bump up TaskUmbilicalProtocol's version? If so, we at least should add a comment describing why the change was necessary.
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.1.patch, HADOOP-4189-v3.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-4189:
-------------------------------
Attachment: HADOOP-4189-v3.patch
Attaching a new patch that incorporates Amareshwari's comments.
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634774#action_12634774 ]
Owen O'Malley commented on HADOOP-4189:
---------------------------------------
I think that we should address putting in the start event immediately, but I don't think that it should hold up this patch.
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.1.patch, HADOOP-4189-v3.patch, HADOOP-4189-v5.1.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4189) HADOOP-3245 is incomplete
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12633647#action_12633647 ]
Hadoop QA commented on HADOOP-4189:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12390362/HADOOP-4189-v3.1.patch
against trunk revision 697306.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3351/console
This message is automatically generated.
> HADOOP-3245 is incomplete
> -------------------------
>
> Key: HADOOP-4189
> URL: https://issues.apache.org/jira/browse/HADOOP-4189
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4189-v1.patch, HADOOP-4189-v2.patch, HADOOP-4189-v3.1.patch, HADOOP-4189-v3.patch, HADOOP-4189.patch
>
>
> There are 2 issues with HADOOP-3245
> - The default block size for the history files in hadoop-default.conf is set to 0. This file will be empty throughout. It should be made null.
> - Same goes for the buffer size.
> - InterTrackerProtocol version needs to be bumped.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.