You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Hemanth Yamijala (JIRA)" <ji...@apache.org> on 2008/09/03 16:25:44 UTC
[jira] Created: (HADOOP-4060) [HOD] Make HOD to roll log files on
the client
[HOD] Make HOD to roll log files on the client
-----------------------------------------------
Key: HADOOP-4060
URL: https://issues.apache.org/jira/browse/HADOOP-4060
Project: Hadoop Core
Issue Type: Bug
Components: contrib/hod
Reporter: Hemanth Yamijala
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
Fix For: 0.18.1, 0.19.0
Currently HOD writes a log file on the client in the cluster directory, named hod-<username>.log. This file is appended to for each run of hod allocation that runs on the same cluster directory. Thus, the file can become quite large - particularly if the job is queued for a long time. If there are problems with the HOD client subsequently, this could result in stale file handles pointing to this large file, and cause disk space to fill up. Another problem is that a large log file is usually unusable for a user.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4060) [HOD] Make HOD to roll log files on
the client
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated HADOOP-4060:
--------------------------------------------
Status: Patch Available (was: Open)
> [HOD] Make HOD to roll log files on the client
> -----------------------------------------------
>
> Key: HADOOP-4060
> URL: https://issues.apache.org/jira/browse/HADOOP-4060
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HOD-LOG-ROTATION.txt
>
>
> Currently HOD writes a log file on the client in the cluster directory, named hod-<username>.log. This file is appended to for each run of hod allocation that runs on the same cluster directory. Thus, the file can become quite large - particularly if the job is queued for a long time. If there are problems with the HOD client subsequently, this could result in stale file handles pointing to this large file, and cause disk space to fill up. Another problem is that a large log file is usually unusable for a user.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4060) [HOD] Make HOD to roll log files on
the client
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629704#action_12629704 ]
Hemanth Yamijala commented on HADOOP-4060:
------------------------------------------
The core test failures are not related to HOD.
> [HOD] Make HOD to roll log files on the client
> -----------------------------------------------
>
> Key: HADOOP-4060
> URL: https://issues.apache.org/jira/browse/HADOOP-4060
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HOD-LOG-ROTATION.1.txt, HOD-LOG-ROTATION.txt
>
>
> Currently HOD writes a log file on the client in the cluster directory, named hod-<username>.log. This file is appended to for each run of hod allocation that runs on the same cluster directory. Thus, the file can become quite large - particularly if the job is queued for a long time. If there are problems with the HOD client subsequently, this could result in stale file handles pointing to this large file, and cause disk space to fill up. Another problem is that a large log file is usually unusable for a user.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4060) [HOD] Make HOD to roll log files on
the client
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated HADOOP-4060:
--------------------------------------------
Status: Patch Available (was: Open)
> [HOD] Make HOD to roll log files on the client
> -----------------------------------------------
>
> Key: HADOOP-4060
> URL: https://issues.apache.org/jira/browse/HADOOP-4060
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HOD-LOG-ROTATION.1.txt, HOD-LOG-ROTATION.txt
>
>
> Currently HOD writes a log file on the client in the cluster directory, named hod-<username>.log. This file is appended to for each run of hod allocation that runs on the same cluster directory. Thus, the file can become quite large - particularly if the job is queued for a long time. If there are problems with the HOD client subsequently, this could result in stale file handles pointing to this large file, and cause disk space to fill up. Another problem is that a large log file is usually unusable for a user.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4060) [HOD] Make HOD to roll log files on
the client
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated HADOOP-4060:
--------------------------------------------
Attachment: HOD-LOG-ROTATION.1.txt
New patch adding a short option -L for --hod.log-rollover-count.
> [HOD] Make HOD to roll log files on the client
> -----------------------------------------------
>
> Key: HADOOP-4060
> URL: https://issues.apache.org/jira/browse/HADOOP-4060
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HOD-LOG-ROTATION.1.txt, HOD-LOG-ROTATION.txt
>
>
> Currently HOD writes a log file on the client in the cluster directory, named hod-<username>.log. This file is appended to for each run of hod allocation that runs on the same cluster directory. Thus, the file can become quite large - particularly if the job is queued for a long time. If there are problems with the HOD client subsequently, this could result in stale file handles pointing to this large file, and cause disk space to fill up. Another problem is that a large log file is usually unusable for a user.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4060) [HOD] Make HOD to roll log files on
the client
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12628615#action_12628615 ]
Hadoop QA commented on HADOOP-4060:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12389551/HOD-LOG-ROTATION.1.txt
against trunk revision 692335.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3185/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3185/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3185/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3185/console
This message is automatically generated.
> [HOD] Make HOD to roll log files on the client
> -----------------------------------------------
>
> Key: HADOOP-4060
> URL: https://issues.apache.org/jira/browse/HADOOP-4060
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HOD-LOG-ROTATION.1.txt, HOD-LOG-ROTATION.txt
>
>
> Currently HOD writes a log file on the client in the cluster directory, named hod-<username>.log. This file is appended to for each run of hod allocation that runs on the same cluster directory. Thus, the file can become quite large - particularly if the job is queued for a long time. If there are problems with the HOD client subsequently, this could result in stale file handles pointing to this large file, and cause disk space to fill up. Another problem is that a large log file is usually unusable for a user.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4060) [HOD] Make HOD to roll log files on
the client
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley updated HADOOP-4060:
--------------------------------
Resolution: Fixed
Fix Version/s: (was: 0.19.0)
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed, Incompatible change])
Status: Resolved (was: Patch Available)
Commit merged to branch-0.18.
> [HOD] Make HOD to roll log files on the client
> -----------------------------------------------
>
> Key: HADOOP-4060
> URL: https://issues.apache.org/jira/browse/HADOOP-4060
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.18.1
>
> Attachments: HOD-LOG-ROTATION.1.txt, HOD-LOG-ROTATION.txt
>
>
> Currently HOD writes a log file on the client in the cluster directory, named hod-<username>.log. This file is appended to for each run of hod allocation that runs on the same cluster directory. Thus, the file can become quite large - particularly if the job is queued for a long time. If there are problems with the HOD client subsequently, this could result in stale file handles pointing to this large file, and cause disk space to fill up. Another problem is that a large log file is usually unusable for a user.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4060) [HOD] Make HOD to roll log files on
the client
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12628427#action_12628427 ]
Hemanth Yamijala commented on HADOOP-4060:
------------------------------------------
One minor comment. Since the new config option is publicly visible in hod --help, it should be given a fixed small option in the help, like the other options. Currently, we are automatically assigning a free character to it, and it is always coming as -g, but this is not required, for e.g. if we add new options.
Otherwise +1 for the patch.
> [HOD] Make HOD to roll log files on the client
> -----------------------------------------------
>
> Key: HADOOP-4060
> URL: https://issues.apache.org/jira/browse/HADOOP-4060
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HOD-LOG-ROTATION.txt
>
>
> Currently HOD writes a log file on the client in the cluster directory, named hod-<username>.log. This file is appended to for each run of hod allocation that runs on the same cluster directory. Thus, the file can become quite large - particularly if the job is queued for a long time. If there are problems with the HOD client subsequently, this could result in stale file handles pointing to this large file, and cause disk space to fill up. Another problem is that a large log file is usually unusable for a user.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4060) [HOD] Make HOD to roll log files on
the client
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod K V updated HADOOP-4060:
------------------------------
Hadoop Flags: [Reviewed] (was: [Reviewed, Incompatible change])
The change is backwards compatible. A value of zero for -L brings back the old behaviour.
> [HOD] Make HOD to roll log files on the client
> -----------------------------------------------
>
> Key: HADOOP-4060
> URL: https://issues.apache.org/jira/browse/HADOOP-4060
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.18.1
>
> Attachments: HOD-LOG-ROTATION.1.txt, HOD-LOG-ROTATION.txt
>
>
> Currently HOD writes a log file on the client in the cluster directory, named hod-<username>.log. This file is appended to for each run of hod allocation that runs on the same cluster directory. Thus, the file can become quite large - particularly if the job is queued for a long time. If there are problems with the HOD client subsequently, this could result in stale file handles pointing to this large file, and cause disk space to fill up. Another problem is that a large log file is usually unusable for a user.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4060) [HOD] Make HOD to roll log files on
the client
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630180#action_12630180 ]
Hudson commented on HADOOP-4060:
--------------------------------
Integrated in Hadoop-trunk #600 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/600/])
> [HOD] Make HOD to roll log files on the client
> -----------------------------------------------
>
> Key: HADOOP-4060
> URL: https://issues.apache.org/jira/browse/HADOOP-4060
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HOD-LOG-ROTATION.1.txt, HOD-LOG-ROTATION.txt
>
>
> Currently HOD writes a log file on the client in the cluster directory, named hod-<username>.log. This file is appended to for each run of hod allocation that runs on the same cluster directory. Thus, the file can become quite large - particularly if the job is queued for a long time. If there are problems with the HOD client subsequently, this could result in stale file handles pointing to this large file, and cause disk space to fill up. Another problem is that a large log file is usually unusable for a user.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4060) [HOD] Make HOD to roll log files on
the client
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-4060:
-------------------------------------
Release Note: HOD client was modified to roll over client logs being written to the cluster directory. A new configuration parameter, hod.log-rollover-count, was introduced to specify how many rollover log files to retain.
Hadoop Flags: [Incompatible change, Reviewed]
> [HOD] Make HOD to roll log files on the client
> -----------------------------------------------
>
> Key: HADOOP-4060
> URL: https://issues.apache.org/jira/browse/HADOOP-4060
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HOD-LOG-ROTATION.1.txt, HOD-LOG-ROTATION.txt
>
>
> Currently HOD writes a log file on the client in the cluster directory, named hod-<username>.log. This file is appended to for each run of hod allocation that runs on the same cluster directory. Thus, the file can become quite large - particularly if the job is queued for a long time. If there are problems with the HOD client subsequently, this could result in stale file handles pointing to this large file, and cause disk space to fill up. Another problem is that a large log file is usually unusable for a user.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4060) [HOD] Make HOD to roll log files on
the client
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated HADOOP-4060:
--------------------------------------------
Status: Open (was: Patch Available)
> [HOD] Make HOD to roll log files on the client
> -----------------------------------------------
>
> Key: HADOOP-4060
> URL: https://issues.apache.org/jira/browse/HADOOP-4060
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HOD-LOG-ROTATION.txt
>
>
> Currently HOD writes a log file on the client in the cluster directory, named hod-<username>.log. This file is appended to for each run of hod allocation that runs on the same cluster directory. Thus, the file can become quite large - particularly if the job is queued for a long time. If there are problems with the HOD client subsequently, this could result in stale file handles pointing to this large file, and cause disk space to fill up. Another problem is that a large log file is usually unusable for a user.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4060) [HOD] Make HOD to roll log files on
the client
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated HADOOP-4060:
--------------------------------------------
Attachment: HOD-LOG-ROTATION.txt
Attaching a patch. This introduces a config parameter hod.log-rollover-count that takes the number of rolled-over logs that HOD client has to keep. It has to be a positive integer, defaults to 5. A zero value disables the roll-over and the log grows indefinitely (current behaviour). Rolled over logs have file-names of the form hod-%user.log.1, hod-%user.log.2 etc.
> [HOD] Make HOD to roll log files on the client
> -----------------------------------------------
>
> Key: HADOOP-4060
> URL: https://issues.apache.org/jira/browse/HADOOP-4060
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HOD-LOG-ROTATION.txt
>
>
> Currently HOD writes a log file on the client in the cluster directory, named hod-<username>.log. This file is appended to for each run of hod allocation that runs on the same cluster directory. Thus, the file can become quite large - particularly if the job is queued for a long time. If there are problems with the HOD client subsequently, this could result in stale file handles pointing to this large file, and cause disk space to fill up. Another problem is that a large log file is usually unusable for a user.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4060) [HOD] Make HOD to roll log files on
the client
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629706#action_12629706 ]
Hemanth Yamijala commented on HADOOP-4060:
------------------------------------------
I just committed this to trunk. Thanks, Vinod.
Can someone with access please commit this to the 0.18 branch also ? The revision number for trunk is 693705. I verified it merges cleanly.
> [HOD] Make HOD to roll log files on the client
> -----------------------------------------------
>
> Key: HADOOP-4060
> URL: https://issues.apache.org/jira/browse/HADOOP-4060
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HOD-LOG-ROTATION.1.txt, HOD-LOG-ROTATION.txt
>
>
> Currently HOD writes a log file on the client in the cluster directory, named hod-<username>.log. This file is appended to for each run of hod allocation that runs on the same cluster directory. Thus, the file can become quite large - particularly if the job is queued for a long time. If there are problems with the HOD client subsequently, this could result in stale file handles pointing to this large file, and cause disk space to fill up. Another problem is that a large log file is usually unusable for a user.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.