You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Hemanth Yamijala (JIRA)" <ji...@apache.org> on 2008/04/24 13:31:23 UTC
[jira] Created: (HADOOP-3304) [HOD] logcondense fails if DFS has
files that are not log files, but match a certain pattern
[HOD] logcondense fails if DFS has files that are not log files, but match a certain pattern
--------------------------------------------------------------------------------------------
Key: HADOOP-3304
URL: https://issues.apache.org/jira/browse/HADOOP-3304
Project: Hadoop Core
Issue Type: Bug
Components: contrib/hod
Affects Versions: 0.16.2
Reporter: Hemanth Yamijala
Assignee: Hemanth Yamijala
Priority: Critical
logcondense works by listing files in dfs and match them against a certain pattern. This pattern is incorrect in the sense that it can potentially match files that are not log files. And this can cause it to fail and not delete files correctly as it should.
It should use a correct regular expression that will only list log files. Also, possibly it should log a stack trace if it happens, but continue to delete files and not stop.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3304) [HOD] logcondense fails if DFS has
files that are not log files, but match a certain pattern
Posted by "Karam Singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karam Singh updated HADOOP-3304:
--------------------------------
When dynamicdfs is used and logcondense fails to delete tasktracker logs if -n options e.g -:
[
python logcondense.py -p ~/hadoop-0.17.0/bin/hadoop -d 7 -c ~/hadoop-conf -l /user
]
logcondense tries to delete 0-tasketracker e.g -:
[
~/hadoop-0.17.0/bin/hadoop --config ~/hadoop-conf dfs -rm /user/user-id/hod-logs/jobid.hostname/0-tasktracker-*
]
where 0-tasktracker-* file are not there because in case of dynamic dfs logs are like 0-datanode-* and 1-tasktracker-*
> [HOD] logcondense fails if DFS has files that are not log files, but match a certain pattern
> --------------------------------------------------------------------------------------------
>
> Key: HADOOP-3304
> URL: https://issues.apache.org/jira/browse/HADOOP-3304
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.16.2
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.16.4
>
> Attachments: 3304.1.patch, 3304.2.patch, 3304.patch
>
>
> logcondense works by listing files in dfs and match them against a certain pattern. This pattern is incorrect in the sense that it can potentially match files that are not log files. And this can cause it to fail and not delete files correctly as it should.
> It should use a correct regular expression that will only list log files. Also, possibly it should log a stack trace if it happens, but continue to delete files and not stop.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3304) [HOD] logcondense fails if DFS has
files that are not log files, but match a certain pattern
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-3304:
-------------------------------------
Attachment: 3304.2.patch
Attaching a new patch file that updates the HOD changes.txt file.
> [HOD] logcondense fails if DFS has files that are not log files, but match a certain pattern
> --------------------------------------------------------------------------------------------
>
> Key: HADOOP-3304
> URL: https://issues.apache.org/jira/browse/HADOOP-3304
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.16.2
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: 3304.1.patch, 3304.2.patch, 3304.patch
>
>
> logcondense works by listing files in dfs and match them against a certain pattern. This pattern is incorrect in the sense that it can potentially match files that are not log files. And this can cause it to fail and not delete files correctly as it should.
> It should use a correct regular expression that will only list log files. Also, possibly it should log a stack trace if it happens, but continue to delete files and not stop.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3304) [HOD] logcondense fails if DFS has
files that are not log files, but match a certain pattern
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-3304:
-------------------------------------
Priority: Blocker (was: Critical)
Fix Version/s: 0.17.0
> [HOD] logcondense fails if DFS has files that are not log files, but match a certain pattern
> --------------------------------------------------------------------------------------------
>
> Key: HADOOP-3304
> URL: https://issues.apache.org/jira/browse/HADOOP-3304
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.16.2
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: 3304.patch
>
>
> logcondense works by listing files in dfs and match them against a certain pattern. This pattern is incorrect in the sense that it can potentially match files that are not log files. And this can cause it to fail and not delete files correctly as it should.
> It should use a correct regular expression that will only list log files. Also, possibly it should log a stack trace if it happens, but continue to delete files and not stop.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3304) [HOD] logcondense fails if DFS has
files that are not log files, but match a certain pattern
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-3304:
-------------------------------------
Fix Version/s: (was: 0.17.0)
0.16.4
I committed this to 0.16.4. Thanks Hemanth.
> [HOD] logcondense fails if DFS has files that are not log files, but match a certain pattern
> --------------------------------------------------------------------------------------------
>
> Key: HADOOP-3304
> URL: https://issues.apache.org/jira/browse/HADOOP-3304
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.16.2
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.16.4
>
> Attachments: 3304.1.patch, 3304.2.patch, 3304.patch
>
>
> logcondense works by listing files in dfs and match them against a certain pattern. This pattern is incorrect in the sense that it can potentially match files that are not log files. And this can cause it to fail and not delete files correctly as it should.
> It should use a correct regular expression that will only list log files. Also, possibly it should log a stack trace if it happens, but continue to delete files and not stop.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3304) [HOD] logcondense fails if DFS has
files that are not log files, but match a certain pattern
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-3304:
-------------------------------------
Status: Patch Available (was: Open)
> [HOD] logcondense fails if DFS has files that are not log files, but match a certain pattern
> --------------------------------------------------------------------------------------------
>
> Key: HADOOP-3304
> URL: https://issues.apache.org/jira/browse/HADOOP-3304
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.16.2
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: 3304.1.patch, 3304.patch
>
>
> logcondense works by listing files in dfs and match them against a certain pattern. This pattern is incorrect in the sense that it can potentially match files that are not log files. And this can cause it to fail and not delete files correctly as it should.
> It should use a correct regular expression that will only list log files. Also, possibly it should log a stack trace if it happens, but continue to delete files and not stop.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3304) [HOD] logcondense fails if DFS has
files that are not log files, but match a certain pattern
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12592395#action_12592395 ]
Hadoop QA commented on HADOOP-3304:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12380922/3304.2.patch
against trunk revision 645773.
@author +1. The patch does not contain any @author tags.
tests included -1. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2327/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2327/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2327/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2327/console
This message is automatically generated.
> [HOD] logcondense fails if DFS has files that are not log files, but match a certain pattern
> --------------------------------------------------------------------------------------------
>
> Key: HADOOP-3304
> URL: https://issues.apache.org/jira/browse/HADOOP-3304
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.16.2
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: 3304.1.patch, 3304.2.patch, 3304.patch
>
>
> logcondense works by listing files in dfs and match them against a certain pattern. This pattern is incorrect in the sense that it can potentially match files that are not log files. And this can cause it to fail and not delete files correctly as it should.
> It should use a correct regular expression that will only list log files. Also, possibly it should log a stack trace if it happens, but continue to delete files and not stop.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3304) [HOD] logcondense fails if DFS has
files that are not log files, but match a certain pattern
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12592142#action_12592142 ]
Hadoop QA commented on HADOOP-3304:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12380854/3304.patch
against trunk revision 645773.
@author +1. The patch does not contain any @author tags.
tests included -1. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2317/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2317/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2317/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2317/console
This message is automatically generated.
> [HOD] logcondense fails if DFS has files that are not log files, but match a certain pattern
> --------------------------------------------------------------------------------------------
>
> Key: HADOOP-3304
> URL: https://issues.apache.org/jira/browse/HADOOP-3304
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.16.2
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Critical
> Attachments: 3304.patch
>
>
> logcondense works by listing files in dfs and match them against a certain pattern. This pattern is incorrect in the sense that it can potentially match files that are not log files. And this can cause it to fail and not delete files correctly as it should.
> It should use a correct regular expression that will only list log files. Also, possibly it should log a stack trace if it happens, but continue to delete files and not stop.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3304) [HOD] logcondense fails if DFS has
files that are not log files, but match a certain pattern
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-3304:
-------------------------------------
Status: Patch Available (was: Open)
> [HOD] logcondense fails if DFS has files that are not log files, but match a certain pattern
> --------------------------------------------------------------------------------------------
>
> Key: HADOOP-3304
> URL: https://issues.apache.org/jira/browse/HADOOP-3304
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.16.2
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: 3304.1.patch, 3304.2.patch, 3304.patch
>
>
> logcondense works by listing files in dfs and match them against a certain pattern. This pattern is incorrect in the sense that it can potentially match files that are not log files. And this can cause it to fail and not delete files correctly as it should.
> It should use a correct regular expression that will only list log files. Also, possibly it should log a stack trace if it happens, but continue to delete files and not stop.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3304) [HOD] logcondense fails if DFS has
files that are not log files, but match a certain pattern
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-3304:
-------------------------------------
Status: Open (was: Patch Available)
> [HOD] logcondense fails if DFS has files that are not log files, but match a certain pattern
> --------------------------------------------------------------------------------------------
>
> Key: HADOOP-3304
> URL: https://issues.apache.org/jira/browse/HADOOP-3304
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.16.2
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: 3304.patch
>
>
> logcondense works by listing files in dfs and match them against a certain pattern. This pattern is incorrect in the sense that it can potentially match files that are not log files. And this can cause it to fail and not delete files correctly as it should.
> It should use a correct regular expression that will only list log files. Also, possibly it should log a stack trace if it happens, but continue to delete files and not stop.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3304) [HOD] logcondense fails if DFS has
files that are not log files, but match a certain pattern
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12592798#action_12592798 ]
Hudson commented on HADOOP-3304:
--------------------------------
Integrated in Hadoop-trunk #473 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/473/])
> [HOD] logcondense fails if DFS has files that are not log files, but match a certain pattern
> --------------------------------------------------------------------------------------------
>
> Key: HADOOP-3304
> URL: https://issues.apache.org/jira/browse/HADOOP-3304
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.16.2
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: 3304.1.patch, 3304.2.patch, 3304.patch
>
>
> logcondense works by listing files in dfs and match them against a certain pattern. This pattern is incorrect in the sense that it can potentially match files that are not log files. And this can cause it to fail and not delete files correctly as it should.
> It should use a correct regular expression that will only list log files. Also, possibly it should log a stack trace if it happens, but continue to delete files and not stop.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3304) [HOD] logcondense fails if DFS has
files that are not log files, but match a certain pattern
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12592293#action_12592293 ]
Vinod Kumar Vavilapalli commented on HADOOP-3304:
-------------------------------------------------
* Tested to check that it now only searches in <prefix>/*/hod-logs for purging.
* We should have documentation for logcondense, describing what/how it does, and its usage; currently we don't have any documentation.
* Hudson's "tests included -1" message: Not writing any test cases.(for now?)
* Found some bugs and good to have enhancements to logcondense. Filing separate issues.
+1 for the fix in general.
> [HOD] logcondense fails if DFS has files that are not log files, but match a certain pattern
> --------------------------------------------------------------------------------------------
>
> Key: HADOOP-3304
> URL: https://issues.apache.org/jira/browse/HADOOP-3304
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.16.2
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: 3304.patch
>
>
> logcondense works by listing files in dfs and match them against a certain pattern. This pattern is incorrect in the sense that it can potentially match files that are not log files. And this can cause it to fail and not delete files correctly as it should.
> It should use a correct regular expression that will only list log files. Also, possibly it should log a stack trace if it happens, but continue to delete files and not stop.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3304) [HOD] logcondense fails if DFS has
files that are not log files, but match a certain pattern
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-3304:
-------------------------------------
Status: Open (was: Patch Available)
> [HOD] logcondense fails if DFS has files that are not log files, but match a certain pattern
> --------------------------------------------------------------------------------------------
>
> Key: HADOOP-3304
> URL: https://issues.apache.org/jira/browse/HADOOP-3304
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.16.2
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: 3304.1.patch, 3304.patch
>
>
> logcondense works by listing files in dfs and match them against a certain pattern. This pattern is incorrect in the sense that it can potentially match files that are not log files. And this can cause it to fail and not delete files correctly as it should.
> It should use a correct regular expression that will only list log files. Also, possibly it should log a stack trace if it happens, but continue to delete files and not stop.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3304) [HOD] logcondense fails if DFS has
files that are not log files, but match a certain pattern
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-3304:
-------------------------------------
Resolution: Fixed
Release Note: Fixed the regular expression used by the logcondense utility to generate the list of log files to purge from HDFS. Also, improved error handling, so a single invalid log file does not fail the purge process.
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Devaraj committed this patch to both trunk and the Hadoop 0.17 branch.
> [HOD] logcondense fails if DFS has files that are not log files, but match a certain pattern
> --------------------------------------------------------------------------------------------
>
> Key: HADOOP-3304
> URL: https://issues.apache.org/jira/browse/HADOOP-3304
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.16.2
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: 3304.1.patch, 3304.2.patch, 3304.patch
>
>
> logcondense works by listing files in dfs and match them against a certain pattern. This pattern is incorrect in the sense that it can potentially match files that are not log files. And this can cause it to fail and not delete files correctly as it should.
> It should use a correct regular expression that will only list log files. Also, possibly it should log a stack trace if it happens, but continue to delete files and not stop.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3304) [HOD] logcondense fails if DFS has
files that are not log files, but match a certain pattern
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-3304:
-------------------------------------
Attachment: 3304.patch
Attached patch fixes the following:
- Moves the shell shebang to the top of the file
- Modifies the default value of the log destination to '/user'
- Modifies the regular expression for looking up files, to be restricted to files under hod-logs only. This will also narrow the scope of search greatly, and should be much faster.
- Adds a try...except block so that a single problematic file won't stop logcondense.py.
> [HOD] logcondense fails if DFS has files that are not log files, but match a certain pattern
> --------------------------------------------------------------------------------------------
>
> Key: HADOOP-3304
> URL: https://issues.apache.org/jira/browse/HADOOP-3304
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.16.2
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Critical
> Attachments: 3304.patch
>
>
> logcondense works by listing files in dfs and match them against a certain pattern. This pattern is incorrect in the sense that it can potentially match files that are not log files. And this can cause it to fail and not delete files correctly as it should.
> It should use a correct regular expression that will only list log files. Also, possibly it should log a stack trace if it happens, but continue to delete files and not stop.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3304) [HOD] logcondense fails if DFS has
files that are not log files, but match a certain pattern
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-3304:
-------------------------------------
Attachment: 3304.1.patch
The attached patch adds documentation as mentioned by Vinod to the HOD admin guide. Some refactoring is required in writing unit tests, but reserving that for another patch as we would like to get this into Hadoop 0.17. ASAP.
> [HOD] logcondense fails if DFS has files that are not log files, but match a certain pattern
> --------------------------------------------------------------------------------------------
>
> Key: HADOOP-3304
> URL: https://issues.apache.org/jira/browse/HADOOP-3304
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.16.2
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: 3304.1.patch, 3304.patch
>
>
> logcondense works by listing files in dfs and match them against a certain pattern. This pattern is incorrect in the sense that it can potentially match files that are not log files. And this can cause it to fail and not delete files correctly as it should.
> It should use a correct regular expression that will only list log files. Also, possibly it should log a stack trace if it happens, but continue to delete files and not stop.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3304) [HOD] logcondense fails if DFS has
files that are not log files, but match a certain pattern
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-3304:
-------------------------------------
Status: Patch Available (was: Open)
> [HOD] logcondense fails if DFS has files that are not log files, but match a certain pattern
> --------------------------------------------------------------------------------------------
>
> Key: HADOOP-3304
> URL: https://issues.apache.org/jira/browse/HADOOP-3304
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.16.2
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Critical
> Attachments: 3304.patch
>
>
> logcondense works by listing files in dfs and match them against a certain pattern. This pattern is incorrect in the sense that it can potentially match files that are not log files. And this can cause it to fail and not delete files correctly as it should.
> It should use a correct regular expression that will only list log files. Also, possibly it should log a stack trace if it happens, but continue to delete files and not stop.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.