You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Chris Douglas (JIRA)" <ji...@apache.org> on 2008/10/16 03:46:46 UTC
[jira] Created: (HADOOP-4425) Edits log takes much longer to load
Edits log takes much longer to load
-----------------------------------
Key: HADOOP-4425
URL: https://issues.apache.org/jira/browse/HADOOP-4425
Project: Hadoop Core
Issue Type: Bug
Components: dfs
Affects Versions: 0.19.0
Reporter: Chris Douglas
Priority: Blocker
Fix For: 0.19.0
The edits log takes significantly longer (+52% to +377% in tests simulating 200k to 20M files) to load in 0.19 than it did in 0.18.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4425) Edits log takes much longer to load
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640272#action_12640272 ]
Konstantin Shvachko commented on HADOOP-4425:
---------------------------------------------
- Implementation of {{read(byte[] b)}} is not required here, since InputStream already has it doing exactly the same.
- Could you please use {{@Override}} for overridden methods.
- Very nice catch btw!
> Edits log takes much longer to load
> -----------------------------------
>
> Key: HADOOP-4425
> URL: https://issues.apache.org/jira/browse/HADOOP-4425
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: 4425-0.patch
>
>
> The edits log takes significantly longer (+52% to +377% in tests simulating 200k to 20M files) to load in 0.19 than it did in 0.18.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4425) Edits log takes much longer to load
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640373#action_12640373 ]
Raghu Angadi commented on HADOOP-4425:
--------------------------------------
> +1 on correcting the typo in this issue.
+1.
> Edits log takes much longer to load
> -----------------------------------
>
> Key: HADOOP-4425
> URL: https://issues.apache.org/jira/browse/HADOOP-4425
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: 4425-0.patch, 4425-1.patch
>
>
> The edits log takes significantly longer (+52% to +377% in tests simulating 200k to 20M files) to load in 0.19 than it did in 0.18.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4425) Edits log takes much longer to load
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-4425:
----------------------------------
Attachment: 4425-1.patch
Updated patch to reflect feedback from Raghu and Konstantin.
BTW- we should probably correct the typo'd Ed*l*itLogInputStream.java, either as part of this checkin or in another issue.
On a related note, Konstantin and I looked at EditLogFileOutputStream, which doesn't override {{write(byte[],int,int)}}, but concluded that this method is not called.
> Edits log takes much longer to load
> -----------------------------------
>
> Key: HADOOP-4425
> URL: https://issues.apache.org/jira/browse/HADOOP-4425
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: 4425-0.patch, 4425-1.patch
>
>
> The edits log takes significantly longer (+52% to +377% in tests simulating 200k to 20M files) to load in 0.19 than it did in 0.18.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4425) Edits log takes much longer to load
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640054#action_12640054 ]
Hadoop QA commented on HADOOP-4425:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12392220/4425-0.patch
against trunk revision 705073.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3471/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3471/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3471/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3471/console
This message is automatically generated.
> Edits log takes much longer to load
> -----------------------------------
>
> Key: HADOOP-4425
> URL: https://issues.apache.org/jira/browse/HADOOP-4425
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: 4425-0.patch
>
>
> The edits log takes significantly longer (+52% to +377% in tests simulating 200k to 20M files) to load in 0.19 than it did in 0.18.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4425) Edits log takes much longer to load
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-4425:
----------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this.
> Edits log takes much longer to load
> -----------------------------------
>
> Key: HADOOP-4425
> URL: https://issues.apache.org/jira/browse/HADOOP-4425
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: 4425-0.patch, 4425-1.patch
>
>
> The edits log takes significantly longer (+52% to +377% in tests simulating 200k to 20M files) to load in 0.19 than it did in 0.18.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4425) Edits log takes much longer to load
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-4425:
----------------------------------
Attachment: 4425-0.patch
FSEditLog.EditLogFileInputStream doesn't overload the read(byte[]) methods in InputStream, which is causing performance to degrade. The patch restores and improves edit log load times.
> Edits log takes much longer to load
> -----------------------------------
>
> Key: HADOOP-4425
> URL: https://issues.apache.org/jira/browse/HADOOP-4425
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Chris Douglas
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: 4425-0.patch
>
>
> The edits log takes significantly longer (+52% to +377% in tests simulating 200k to 20M files) to load in 0.19 than it did in 0.18.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4425) Edits log takes much longer to load
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-4425:
----------------------------------
Assignee: Chris Douglas
Status: Patch Available (was: Open)
> Edits log takes much longer to load
> -----------------------------------
>
> Key: HADOOP-4425
> URL: https://issues.apache.org/jira/browse/HADOOP-4425
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: 4425-0.patch
>
>
> The edits log takes significantly longer (+52% to +377% in tests simulating 200k to 20M files) to load in 0.19 than it did in 0.18.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4425) Edits log takes much longer to load
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-4425:
----------------------------------
Status: Patch Available (was: Open)
> Edits log takes much longer to load
> -----------------------------------
>
> Key: HADOOP-4425
> URL: https://issues.apache.org/jira/browse/HADOOP-4425
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: 4425-0.patch, 4425-1.patch
>
>
> The edits log takes significantly longer (+52% to +377% in tests simulating 200k to 20M files) to load in 0.19 than it did in 0.18.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4425) Edits log takes much longer to load
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640568#action_12640568 ]
Hudson commented on HADOOP-4425:
--------------------------------
Integrated in Hadoop-trunk #636 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/636/])
. Fix EditLogInputStream to overload the bulk read method.
> Edits log takes much longer to load
> -----------------------------------
>
> Key: HADOOP-4425
> URL: https://issues.apache.org/jira/browse/HADOOP-4425
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: 4425-0.patch, 4425-1.patch
>
>
> The edits log takes significantly longer (+52% to +377% in tests simulating 200k to 20M files) to load in 0.19 than it did in 0.18.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4425) Edits log takes much longer to load
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640251#action_12640251 ]
Raghu Angadi commented on HADOOP-4425:
--------------------------------------
+1. Looks good.
To avoid this problem again, should we make 'read(buf, off, len)' abstract in EditLogOutputStream?
> Edits log takes much longer to load
> -----------------------------------
>
> Key: HADOOP-4425
> URL: https://issues.apache.org/jira/browse/HADOOP-4425
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: 4425-0.patch
>
>
> The edits log takes significantly longer (+52% to +377% in tests simulating 200k to 20M files) to load in 0.19 than it did in 0.18.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4425) Edits log takes much longer to load
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640369#action_12640369 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4425:
------------------------------------------------
> BTW- we should probably correct the typo'd Ed*l*itLogInputStream.java, either as part of this checkin or in another issue.
+1 on correcting the typo in this issue.
> Edits log takes much longer to load
> -----------------------------------
>
> Key: HADOOP-4425
> URL: https://issues.apache.org/jira/browse/HADOOP-4425
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: 4425-0.patch, 4425-1.patch
>
>
> The edits log takes significantly longer (+52% to +377% in tests simulating 200k to 20M files) to load in 0.19 than it did in 0.18.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4425) Edits log takes much longer to load
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-4425:
----------------------------------
Status: Open (was: Patch Available)
> Edits log takes much longer to load
> -----------------------------------
>
> Key: HADOOP-4425
> URL: https://issues.apache.org/jira/browse/HADOOP-4425
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: 4425-0.patch, 4425-1.patch
>
>
> The edits log takes significantly longer (+52% to +377% in tests simulating 200k to 20M files) to load in 0.19 than it did in 0.18.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.