You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Konstantin Shvachko (JIRA)" <ji...@apache.org> on 2008/07/09 20:43:31 UTC
[jira] Created: (HADOOP-3732) Block scanner should read block
information during initialization.
Block scanner should read block information during initialization.
------------------------------------------------------------------
Key: HADOOP-3732
URL: https://issues.apache.org/jira/browse/HADOOP-3732
Project: Hadoop Core
Issue Type: Bug
Components: dfs
Affects Versions: 0.17.0
Reporter: Konstantin Shvachko
Assignee: Raghu Angadi
Priority: Blocker
Fix For: 0.18.0
We see a lot of warning messages on each data-node during startup and upgrading from 0.17 to 0.18 saying:
{code}
2008-07-08 22:22:15,711 WARN org.apache.hadoop.dfs.DataNode: Block /grid/3/hadoop/var/hdfs/data/current/blk_3359714082706415785 does not have a metafile!
{code}
The message received twice for each block, because the block information is first read buy the data-node itself and then by the block scanner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3732) Block scanner should read block
information during initialization.
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3732:
---------------------------------
Attachment: HADOOP-3732.patch
> I would use blockMap == null instead of new boolean <initialized>.
thats right. Updated patch uses "throttler == null" since this condition should be changed only at the end of {{init()}}.
> "generate serial version ID" for the servlet
I think Hadoop's policy is not to include them.. this warning can be disabled in Eclipse.
> Block scanner should read block information during initialization.
> ------------------------------------------------------------------
>
> Key: HADOOP-3732
> URL: https://issues.apache.org/jira/browse/HADOOP-3732
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Konstantin Shvachko
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: HADOOP-3732.patch, HADOOP-3732.patch
>
>
> We see a lot of warning messages on each data-node during startup and upgrading from 0.17 to 0.18 saying:
> {code}
> 2008-07-08 22:22:15,711 WARN org.apache.hadoop.dfs.DataNode: Block /grid/3/hadoop/var/hdfs/data/current/blk_3359714082706415785 does not have a metafile!
> {code}
> The message received twice for each block, because the block information is first read buy the data-node itself and then by the block scanner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3732) Block scanner should read block
information during initialization.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12612705#action_12612705 ]
Konstantin Shvachko commented on HADOOP-3732:
---------------------------------------------
I would use blockMap == null instead of new boolean <initialized>.
+1 on the rest.
> Block scanner should read block information during initialization.
> ------------------------------------------------------------------
>
> Key: HADOOP-3732
> URL: https://issues.apache.org/jira/browse/HADOOP-3732
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Konstantin Shvachko
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: HADOOP-3732.patch
>
>
> We see a lot of warning messages on each data-node during startup and upgrading from 0.17 to 0.18 saying:
> {code}
> 2008-07-08 22:22:15,711 WARN org.apache.hadoop.dfs.DataNode: Block /grid/3/hadoop/var/hdfs/data/current/blk_3359714082706415785 does not have a metafile!
> {code}
> The message received twice for each block, because the block information is first read buy the data-node itself and then by the block scanner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3732) Block scanner should read block
information during initialization.
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-3732:
------------------------------------
Release Note: (was: Delay intialization of datanode block verification till the verification thread is started.
)
> Block scanner should read block information during initialization.
> ------------------------------------------------------------------
>
> Key: HADOOP-3732
> URL: https://issues.apache.org/jira/browse/HADOOP-3732
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Konstantin Shvachko
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-3732.patch, HADOOP-3732.patch
>
>
> We see a lot of warning messages on each data-node during startup and upgrading from 0.17 to 0.18 saying:
> {code}
> 2008-07-08 22:22:15,711 WARN org.apache.hadoop.dfs.DataNode: Block /grid/3/hadoop/var/hdfs/data/current/blk_3359714082706415785 does not have a metafile!
> {code}
> The message received twice for each block, because the block information is first read buy the data-node itself and then by the block scanner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3732) Block scanner should read block
information during initialization.
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3732:
---------------------------------
Fix Version/s: (was: 0.18.0)
0.19.0
Status: Patch Available (was: Open)
> Block scanner should read block information during initialization.
> ------------------------------------------------------------------
>
> Key: HADOOP-3732
> URL: https://issues.apache.org/jira/browse/HADOOP-3732
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Konstantin Shvachko
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-3732.patch, HADOOP-3732.patch
>
>
> We see a lot of warning messages on each data-node during startup and upgrading from 0.17 to 0.18 saying:
> {code}
> 2008-07-08 22:22:15,711 WARN org.apache.hadoop.dfs.DataNode: Block /grid/3/hadoop/var/hdfs/data/current/blk_3359714082706415785 does not have a metafile!
> {code}
> The message received twice for each block, because the block information is first read buy the data-node itself and then by the block scanner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3732) Block scanner should read block
information during initialization.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12612231#action_12612231 ]
Konstantin Shvachko commented on HADOOP-3732:
---------------------------------------------
# Right now the DataBlockScanner is initialized during data-node startup, and reads block information from the data-node storage directory at that time. See init() in DataBlockScanner(). A better time to obtain block information would be when the block scanner actually starts scanning and verifying, that is in DataBlockScanner.run(). During regular startup the difference in timing is negligible, but if there is distributed upgrade then it can take a while until the scanner will have a chance to run, besides it may read incorrect information since it will be reading the pre-upgrade block files.
# The message itself is confusing for administrators mostly because it is repeated 80,000 times (2 * #blocks). I am not sure how to deal with this. If we downgrade it to DEBUG level we risk to miss it during regular dn operation. May be we should postpone creating the dn block map until the upgrade is done. It is the same thins as with the bscanner, before the upgrade is finished block information might not be correct from the new software point of view.
> Block scanner should read block information during initialization.
> ------------------------------------------------------------------
>
> Key: HADOOP-3732
> URL: https://issues.apache.org/jira/browse/HADOOP-3732
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Konstantin Shvachko
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.18.0
>
>
> We see a lot of warning messages on each data-node during startup and upgrading from 0.17 to 0.18 saying:
> {code}
> 2008-07-08 22:22:15,711 WARN org.apache.hadoop.dfs.DataNode: Block /grid/3/hadoop/var/hdfs/data/current/blk_3359714082706415785 does not have a metafile!
> {code}
> The message received twice for each block, because the block information is first read buy the data-node itself and then by the block scanner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3732) Block scanner should read block
information during initialization.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12624741#action_12624741 ]
Hudson commented on HADOOP-3732:
--------------------------------
Integrated in Hadoop-trunk #581 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/581/])
> Block scanner should read block information during initialization.
> ------------------------------------------------------------------
>
> Key: HADOOP-3732
> URL: https://issues.apache.org/jira/browse/HADOOP-3732
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Konstantin Shvachko
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-3732.patch, HADOOP-3732.patch
>
>
> We see a lot of warning messages on each data-node during startup and upgrading from 0.17 to 0.18 saying:
> {code}
> 2008-07-08 22:22:15,711 WARN org.apache.hadoop.dfs.DataNode: Block /grid/3/hadoop/var/hdfs/data/current/blk_3359714082706415785 does not have a metafile!
> {code}
> The message received twice for each block, because the block information is first read buy the data-node itself and then by the block scanner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3732) Block scanner should read block
information during initialization.
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3732:
---------------------------------
Resolution: Fixed
Release Note:
Delay intialization of datanode block verification till the verification thread is started.
Status: Resolved (was: Patch Available)
I just committed this.
> Block scanner should read block information during initialization.
> ------------------------------------------------------------------
>
> Key: HADOOP-3732
> URL: https://issues.apache.org/jira/browse/HADOOP-3732
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Konstantin Shvachko
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-3732.patch, HADOOP-3732.patch
>
>
> We see a lot of warning messages on each data-node during startup and upgrading from 0.17 to 0.18 saying:
> {code}
> 2008-07-08 22:22:15,711 WARN org.apache.hadoop.dfs.DataNode: Block /grid/3/hadoop/var/hdfs/data/current/blk_3359714082706415785 does not have a metafile!
> {code}
> The message received twice for each block, because the block information is first read buy the data-node itself and then by the block scanner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3732) Block scanner should read block
information during initialization.
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3732:
---------------------------------
Attachment: HADOOP-3732.patch
Patch for trunk is attached. The fix differs loading block info until the scanner thread is started. If there are any calls to addBlock() or deleteBlock() in between, these are ignored.
> Block scanner should read block information during initialization.
> ------------------------------------------------------------------
>
> Key: HADOOP-3732
> URL: https://issues.apache.org/jira/browse/HADOOP-3732
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Konstantin Shvachko
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: HADOOP-3732.patch
>
>
> We see a lot of warning messages on each data-node during startup and upgrading from 0.17 to 0.18 saying:
> {code}
> 2008-07-08 22:22:15,711 WARN org.apache.hadoop.dfs.DataNode: Block /grid/3/hadoop/var/hdfs/data/current/blk_3359714082706415785 does not have a metafile!
> {code}
> The message received twice for each block, because the block information is first read buy the data-node itself and then by the block scanner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3732) Block scanner should read block
information during initialization.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12612708#action_12612708 ]
Konstantin Shvachko commented on HADOOP-3732:
---------------------------------------------
Since you are in this file could you pls "generate serial version ID" for the servlet in order to get rid of the warning.
Eclipse will do it for you.
> Block scanner should read block information during initialization.
> ------------------------------------------------------------------
>
> Key: HADOOP-3732
> URL: https://issues.apache.org/jira/browse/HADOOP-3732
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Konstantin Shvachko
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: HADOOP-3732.patch
>
>
> We see a lot of warning messages on each data-node during startup and upgrading from 0.17 to 0.18 saying:
> {code}
> 2008-07-08 22:22:15,711 WARN org.apache.hadoop.dfs.DataNode: Block /grid/3/hadoop/var/hdfs/data/current/blk_3359714082706415785 does not have a metafile!
> {code}
> The message received twice for each block, because the block information is first read buy the data-node itself and then by the block scanner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3732) Block scanner should read block
information during initialization.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613780#action_12613780 ]
Hadoop QA commented on HADOOP-3732:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12385818/HADOOP-3732.patch
against trunk revision 677054.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2863/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2863/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2863/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2863/console
This message is automatically generated.
> Block scanner should read block information during initialization.
> ------------------------------------------------------------------
>
> Key: HADOOP-3732
> URL: https://issues.apache.org/jira/browse/HADOOP-3732
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Konstantin Shvachko
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-3732.patch, HADOOP-3732.patch
>
>
> We see a lot of warning messages on each data-node during startup and upgrading from 0.17 to 0.18 saying:
> {code}
> 2008-07-08 22:22:15,711 WARN org.apache.hadoop.dfs.DataNode: Block /grid/3/hadoop/var/hdfs/data/current/blk_3359714082706415785 does not have a metafile!
> {code}
> The message received twice for each block, because the block information is first read buy the data-node itself and then by the block scanner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3732) Block scanner should read block
information during initialization.
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12612269#action_12612269 ]
Raghu Angadi commented on HADOOP-3732:
--------------------------------------
> May be we should postpone creating the dn block map until the upgrade is done. It is the same thins as with the bscanner, before the upgrade is finished block information might not be correct from the new software point of view.
If FSDataset does not scan the directories, datanode won't know what to upgrade. So some where FSDataset should be able to understand prev layout.
For blockScanner I will attach a patch that differs the initialization.
> Block scanner should read block information during initialization.
> ------------------------------------------------------------------
>
> Key: HADOOP-3732
> URL: https://issues.apache.org/jira/browse/HADOOP-3732
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Konstantin Shvachko
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.18.0
>
>
> We see a lot of warning messages on each data-node during startup and upgrading from 0.17 to 0.18 saying:
> {code}
> 2008-07-08 22:22:15,711 WARN org.apache.hadoop.dfs.DataNode: Block /grid/3/hadoop/var/hdfs/data/current/blk_3359714082706415785 does not have a metafile!
> {code}
> The message received twice for each block, because the block information is first read buy the data-node itself and then by the block scanner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.