You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "J.Andreina (JIRA)" <ji...@apache.org> on 2012/05/23 07:08:40 UTC
[jira] [Created] (HDFS-3457) Number of UnderReplicated blocks and
Number of Files in the cluster, displayed in UI and Fsck report is
different
J.Andreina created HDFS-3457:
--------------------------------
Summary: Number of UnderReplicated blocks and Number of Files in the cluster, displayed in UI and Fsck report is different
Key: HDFS-3457
URL: https://issues.apache.org/jira/browse/HDFS-3457
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.0.0
Reporter: J.Andreina
Priority: Minor
Scenario:
=========
Write an HDFS application with the following sequence of operations
1. Create file.
2. Append and sync file.
3. Delete file.
4. Create file.
5. Rename file.
Run the application using 50 threads for 4 hours.
Next Run the same application using 200 threads for the next 4 hours.
Next Run the application using 50 threads for the next 4 hours.
The Number of under-Replicated blocks and Total number of files mentioned in the UI and fsck report differs
Fsck report for the mismatch in Number of under-Replicated blocks :
===================================================================
Status: HEALTHY
Total size: 5670200922 B
Total dirs: 2
Total files: 2015
Total blocks (validated): 977 (avg. block size 5803685 B)
Minimally replicated blocks: 977 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 94 (9.621289 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 1.9037871
Corrupt blocks: 0
Missing replicas: 94 (5.0537634 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Mon Mar 19 11:14:41 IST 2012 in 94 milliseconds
The filesystem under path '/' is HEALTHY
Fsck report for the mismatch in Total number of Files :
=======================================================
Status: HEALTHY
Total size: 19418 B (Total open files size: 42729325 B)
Total dirs: 2
Total files: 4226 (Files currently being written: 15)
Total blocks (validated): 266 (avg. block size 73 B) (Total open file blocks (not validated): 5)
Minimally replicated blocks: 266 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 3
Number of racks: 2
FSCK ended at Wed Apr 04 10:41:44 IST 2012 in 2302 milliseconds
The filesystem under path '/' is HEALTHY
Have attached UI screenshot for both the issues
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3457) Number of UnderReplicated blocks and
Number of Files in the cluster, displayed in UI and Fsck report is
different
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13285088#comment-13285088 ]
Aaron T. Myers commented on HDFS-3457:
--------------------------------------
So shall we resolve this as "invalid" ?
> Number of UnderReplicated blocks and Number of Files in the cluster, displayed in UI and Fsck report is different
> -----------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3457
> URL: https://issues.apache.org/jira/browse/HDFS-3457
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: J.Andreina
> Priority: Minor
> Attachments: Mismatch in Number of Underreplicated blocks UI.jpg, Mismatch in number of files UI.jpg
>
>
> Scenario:
> =========
> Write an HDFS application with the following sequence of operations
> 1. Create file.
> 2. Append and sync file.
> 3. Delete file.
> 4. Create file.
> 5. Rename file.
> Run the application using 50 threads for 4 hours.
> Next Run the same application using 200 threads for the next 4 hours.
> Next Run the application using 50 threads for the next 4 hours.
> The Number of under-Replicated blocks and Total number of files mentioned in the UI and fsck report differs
> Fsck report for the mismatch in Number of under-Replicated blocks :
> ===================================================================
> Status: HEALTHY
> Total size: 5670200922 B
> Total dirs: 2
> Total files: 2015
> Total blocks (validated): 977 (avg. block size 5803685 B)
> Minimally replicated blocks: 977 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 94 (9.621289 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 2
> Average block replication: 1.9037871
> Corrupt blocks: 0
> Missing replicas: 94 (5.0537634 %)
> Number of data-nodes: 3
> Number of racks: 1
> FSCK ended at Mon Mar 19 11:14:41 IST 2012 in 94 milliseconds
> The filesystem under path '/' is HEALTHY
> Fsck report for the mismatch in Total number of Files :
> =======================================================
> Status: HEALTHY
> Total size: 19418 B (Total open files size: 42729325 B)
> Total dirs: 2
> Total files: 4226 (Files currently being written: 15)
> Total blocks (validated): 266 (avg. block size 73 B) (Total open file blocks (not validated): 5)
> Minimally replicated blocks: 266 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 0 (0.0 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 2
> Average block replication: 2.0
> Corrupt blocks: 0
> Missing replicas: 0 (0.0 %)
> Number of data-nodes: 3
> Number of racks: 2
> FSCK ended at Wed Apr 04 10:41:44 IST 2012 in 2302 milliseconds
> The filesystem under path '/' is HEALTHY
> Have attached UI screenshot for both the issues
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3457) Number of UnderReplicated blocks
displayed in UI and Fsck report is different
Posted by "J.Andreina (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13285374#comment-13285374 ]
J.Andreina commented on HDFS-3457:
----------------------------------
Hi Aaron,
Thanks for looking into this defect.
Actually i raised this defect pointing to following two issues
(1) Number of Underreplicated blocks
(2) Number of files
displayed in UI and fsck report is different
But as Ashish has commented : Number of files displayed in UI and fsck report is proper and i too agree with that.
But still the number of underreplicated blocks are different in UI and fsck .Please let me know why this is happening.
> Number of UnderReplicated blocks displayed in UI and Fsck report is different
> -----------------------------------------------------------------------------
>
> Key: HDFS-3457
> URL: https://issues.apache.org/jira/browse/HDFS-3457
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: J.Andreina
> Priority: Minor
> Attachments: Mismatch in Number of Underreplicated blocks UI.jpg, Mismatch in number of files UI.jpg
>
>
> Scenario:
> =========
> Write an HDFS application with the following sequence of operations
> 1. Create file.
> 2. Append and sync file.
> 3. Delete file.
> 4. Create file.
> 5. Rename file.
> Run the application using 50 threads for 4 hours.
> Next Run the same application using 200 threads for the next 4 hours.
> Next Run the application using 50 threads for the next 4 hours.
> The Number of under-Replicated blocks mentioned in the UI and fsck report differs
> Fsck report for the mismatch in Number of under-Replicated blocks :
> ===================================================================
> Status: HEALTHY
> Total size: 5670200922 B
> Total dirs: 2
> Total files: 2015
> Total blocks (validated): 977 (avg. block size 5803685 B)
> Minimally replicated blocks: 977 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 94 (9.621289 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 2
> Average block replication: 1.9037871
> Corrupt blocks: 0
> Missing replicas: 94 (5.0537634 %)
> Number of data-nodes: 3
> Number of racks: 1
> FSCK ended at Mon Mar 19 11:14:41 IST 2012 in 94 milliseconds
> The filesystem under path '/' is HEALTHY
> Have attached UI screenshot for this issue
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3457) Number of UnderReplicated blocks and
Number of Files in the cluster, displayed in UI and Fsck report is
different
Posted by "Ashish Singhi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284319#comment-13284319 ]
Ashish Singhi commented on HDFS-3457:
-------------------------------------
Number of files in fsck and UI report are displayed correctly.
In UI report the total number of files and directories = 4243.
In Fsck report the total number of file and directories = Total dirs + Total files + Files currently being written, i.e ., 2 + 4226 + 15 = 4243.
> Number of UnderReplicated blocks and Number of Files in the cluster, displayed in UI and Fsck report is different
> -----------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3457
> URL: https://issues.apache.org/jira/browse/HDFS-3457
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: J.Andreina
> Priority: Minor
> Attachments: Mismatch in Number of Underreplicated blocks UI.jpg, Mismatch in number of files UI.jpg
>
>
> Scenario:
> =========
> Write an HDFS application with the following sequence of operations
> 1. Create file.
> 2. Append and sync file.
> 3. Delete file.
> 4. Create file.
> 5. Rename file.
> Run the application using 50 threads for 4 hours.
> Next Run the same application using 200 threads for the next 4 hours.
> Next Run the application using 50 threads for the next 4 hours.
> The Number of under-Replicated blocks and Total number of files mentioned in the UI and fsck report differs
> Fsck report for the mismatch in Number of under-Replicated blocks :
> ===================================================================
> Status: HEALTHY
> Total size: 5670200922 B
> Total dirs: 2
> Total files: 2015
> Total blocks (validated): 977 (avg. block size 5803685 B)
> Minimally replicated blocks: 977 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 94 (9.621289 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 2
> Average block replication: 1.9037871
> Corrupt blocks: 0
> Missing replicas: 94 (5.0537634 %)
> Number of data-nodes: 3
> Number of racks: 1
> FSCK ended at Mon Mar 19 11:14:41 IST 2012 in 94 milliseconds
> The filesystem under path '/' is HEALTHY
> Fsck report for the mismatch in Total number of Files :
> =======================================================
> Status: HEALTHY
> Total size: 19418 B (Total open files size: 42729325 B)
> Total dirs: 2
> Total files: 4226 (Files currently being written: 15)
> Total blocks (validated): 266 (avg. block size 73 B) (Total open file blocks (not validated): 5)
> Minimally replicated blocks: 266 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 0 (0.0 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 2
> Average block replication: 2.0
> Corrupt blocks: 0
> Missing replicas: 0 (0.0 %)
> Number of data-nodes: 3
> Number of racks: 2
> FSCK ended at Wed Apr 04 10:41:44 IST 2012 in 2302 milliseconds
> The filesystem under path '/' is HEALTHY
> Have attached UI screenshot for both the issues
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3457) Number of UnderReplicated blocks and
Number of Files in the cluster, displayed in UI and Fsck report is
different
Posted by "J.Andreina (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
J.Andreina updated HDFS-3457:
-----------------------------
Attachment: UI screenshots.docx
Attached the screenshot for UI report
> Number of UnderReplicated blocks and Number of Files in the cluster, displayed in UI and Fsck report is different
> -----------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3457
> URL: https://issues.apache.org/jira/browse/HDFS-3457
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0
> Reporter: J.Andreina
> Priority: Minor
> Attachments: UI screenshots.docx
>
>
> Scenario:
> =========
> Write an HDFS application with the following sequence of operations
> 1. Create file.
> 2. Append and sync file.
> 3. Delete file.
> 4. Create file.
> 5. Rename file.
> Run the application using 50 threads for 4 hours.
> Next Run the same application using 200 threads for the next 4 hours.
> Next Run the application using 50 threads for the next 4 hours.
> The Number of under-Replicated blocks and Total number of files mentioned in the UI and fsck report differs
> Fsck report for the mismatch in Number of under-Replicated blocks :
> ===================================================================
> Status: HEALTHY
> Total size: 5670200922 B
> Total dirs: 2
> Total files: 2015
> Total blocks (validated): 977 (avg. block size 5803685 B)
> Minimally replicated blocks: 977 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 94 (9.621289 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 2
> Average block replication: 1.9037871
> Corrupt blocks: 0
> Missing replicas: 94 (5.0537634 %)
> Number of data-nodes: 3
> Number of racks: 1
> FSCK ended at Mon Mar 19 11:14:41 IST 2012 in 94 milliseconds
> The filesystem under path '/' is HEALTHY
> Fsck report for the mismatch in Total number of Files :
> =======================================================
> Status: HEALTHY
> Total size: 19418 B (Total open files size: 42729325 B)
> Total dirs: 2
> Total files: 4226 (Files currently being written: 15)
> Total blocks (validated): 266 (avg. block size 73 B) (Total open file blocks (not validated): 5)
> Minimally replicated blocks: 266 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 0 (0.0 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 2
> Average block replication: 2.0
> Corrupt blocks: 0
> Missing replicas: 0 (0.0 %)
> Number of data-nodes: 3
> Number of racks: 2
> FSCK ended at Wed Apr 04 10:41:44 IST 2012 in 2302 milliseconds
> The filesystem under path '/' is HEALTHY
> Have attached UI screenshot for both the issues
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3457) Number of UnderReplicated blocks and
Number of Files in the cluster, displayed in UI and Fsck report is
different
Posted by "J.Andreina (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
J.Andreina updated HDFS-3457:
-----------------------------
Attachment: (was: UI screenshots.docx)
> Number of UnderReplicated blocks and Number of Files in the cluster, displayed in UI and Fsck report is different
> -----------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3457
> URL: https://issues.apache.org/jira/browse/HDFS-3457
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0
> Reporter: J.Andreina
> Priority: Minor
>
> Scenario:
> =========
> Write an HDFS application with the following sequence of operations
> 1. Create file.
> 2. Append and sync file.
> 3. Delete file.
> 4. Create file.
> 5. Rename file.
> Run the application using 50 threads for 4 hours.
> Next Run the same application using 200 threads for the next 4 hours.
> Next Run the application using 50 threads for the next 4 hours.
> The Number of under-Replicated blocks and Total number of files mentioned in the UI and fsck report differs
> Fsck report for the mismatch in Number of under-Replicated blocks :
> ===================================================================
> Status: HEALTHY
> Total size: 5670200922 B
> Total dirs: 2
> Total files: 2015
> Total blocks (validated): 977 (avg. block size 5803685 B)
> Minimally replicated blocks: 977 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 94 (9.621289 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 2
> Average block replication: 1.9037871
> Corrupt blocks: 0
> Missing replicas: 94 (5.0537634 %)
> Number of data-nodes: 3
> Number of racks: 1
> FSCK ended at Mon Mar 19 11:14:41 IST 2012 in 94 milliseconds
> The filesystem under path '/' is HEALTHY
> Fsck report for the mismatch in Total number of Files :
> =======================================================
> Status: HEALTHY
> Total size: 19418 B (Total open files size: 42729325 B)
> Total dirs: 2
> Total files: 4226 (Files currently being written: 15)
> Total blocks (validated): 266 (avg. block size 73 B) (Total open file blocks (not validated): 5)
> Minimally replicated blocks: 266 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 0 (0.0 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 2
> Average block replication: 2.0
> Corrupt blocks: 0
> Missing replicas: 0 (0.0 %)
> Number of data-nodes: 3
> Number of racks: 2
> FSCK ended at Wed Apr 04 10:41:44 IST 2012 in 2302 milliseconds
> The filesystem under path '/' is HEALTHY
> Have attached UI screenshot for both the issues
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3457) Number of UnderReplicated blocks and
Number of Files in the cluster, displayed in UI and Fsck report is
different
Posted by "J.Andreina (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284725#comment-13284725 ]
J.Andreina commented on HDFS-3457:
----------------------------------
yes Ashish you are right, UI report displays the count of files+directories. Sorry that was my mistake. Thanks for clarifying.
> Number of UnderReplicated blocks and Number of Files in the cluster, displayed in UI and Fsck report is different
> -----------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3457
> URL: https://issues.apache.org/jira/browse/HDFS-3457
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: J.Andreina
> Priority: Minor
> Attachments: Mismatch in Number of Underreplicated blocks UI.jpg, Mismatch in number of files UI.jpg
>
>
> Scenario:
> =========
> Write an HDFS application with the following sequence of operations
> 1. Create file.
> 2. Append and sync file.
> 3. Delete file.
> 4. Create file.
> 5. Rename file.
> Run the application using 50 threads for 4 hours.
> Next Run the same application using 200 threads for the next 4 hours.
> Next Run the application using 50 threads for the next 4 hours.
> The Number of under-Replicated blocks and Total number of files mentioned in the UI and fsck report differs
> Fsck report for the mismatch in Number of under-Replicated blocks :
> ===================================================================
> Status: HEALTHY
> Total size: 5670200922 B
> Total dirs: 2
> Total files: 2015
> Total blocks (validated): 977 (avg. block size 5803685 B)
> Minimally replicated blocks: 977 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 94 (9.621289 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 2
> Average block replication: 1.9037871
> Corrupt blocks: 0
> Missing replicas: 94 (5.0537634 %)
> Number of data-nodes: 3
> Number of racks: 1
> FSCK ended at Mon Mar 19 11:14:41 IST 2012 in 94 milliseconds
> The filesystem under path '/' is HEALTHY
> Fsck report for the mismatch in Total number of Files :
> =======================================================
> Status: HEALTHY
> Total size: 19418 B (Total open files size: 42729325 B)
> Total dirs: 2
> Total files: 4226 (Files currently being written: 15)
> Total blocks (validated): 266 (avg. block size 73 B) (Total open file blocks (not validated): 5)
> Minimally replicated blocks: 266 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 0 (0.0 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 2
> Average block replication: 2.0
> Corrupt blocks: 0
> Missing replicas: 0 (0.0 %)
> Number of data-nodes: 3
> Number of racks: 2
> FSCK ended at Wed Apr 04 10:41:44 IST 2012 in 2302 milliseconds
> The filesystem under path '/' is HEALTHY
> Have attached UI screenshot for both the issues
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3457) Number of UnderReplicated blocks and
Number of Files in the cluster, displayed in UI and Fsck report is
different
Posted by "J.Andreina (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
J.Andreina updated HDFS-3457:
-----------------------------
Attachment: Mismatch in Number of Underreplicated blocks UI.jpg
Mismatch in number of files UI.jpg
> Number of UnderReplicated blocks and Number of Files in the cluster, displayed in UI and Fsck report is different
> -----------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3457
> URL: https://issues.apache.org/jira/browse/HDFS-3457
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0
> Reporter: J.Andreina
> Priority: Minor
> Attachments: Mismatch in Number of Underreplicated blocks UI.jpg, Mismatch in number of files UI.jpg
>
>
> Scenario:
> =========
> Write an HDFS application with the following sequence of operations
> 1. Create file.
> 2. Append and sync file.
> 3. Delete file.
> 4. Create file.
> 5. Rename file.
> Run the application using 50 threads for 4 hours.
> Next Run the same application using 200 threads for the next 4 hours.
> Next Run the application using 50 threads for the next 4 hours.
> The Number of under-Replicated blocks and Total number of files mentioned in the UI and fsck report differs
> Fsck report for the mismatch in Number of under-Replicated blocks :
> ===================================================================
> Status: HEALTHY
> Total size: 5670200922 B
> Total dirs: 2
> Total files: 2015
> Total blocks (validated): 977 (avg. block size 5803685 B)
> Minimally replicated blocks: 977 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 94 (9.621289 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 2
> Average block replication: 1.9037871
> Corrupt blocks: 0
> Missing replicas: 94 (5.0537634 %)
> Number of data-nodes: 3
> Number of racks: 1
> FSCK ended at Mon Mar 19 11:14:41 IST 2012 in 94 milliseconds
> The filesystem under path '/' is HEALTHY
> Fsck report for the mismatch in Total number of Files :
> =======================================================
> Status: HEALTHY
> Total size: 19418 B (Total open files size: 42729325 B)
> Total dirs: 2
> Total files: 4226 (Files currently being written: 15)
> Total blocks (validated): 266 (avg. block size 73 B) (Total open file blocks (not validated): 5)
> Minimally replicated blocks: 266 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 0 (0.0 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 2
> Average block replication: 2.0
> Corrupt blocks: 0
> Missing replicas: 0 (0.0 %)
> Number of data-nodes: 3
> Number of racks: 2
> FSCK ended at Wed Apr 04 10:41:44 IST 2012 in 2302 milliseconds
> The filesystem under path '/' is HEALTHY
> Have attached UI screenshot for both the issues
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3457) Number of UnderReplicated blocks and
Number of Files in the cluster, displayed in UI and Fsck report is
different
Posted by "J.Andreina (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283112#comment-13283112 ]
J.Andreina commented on HDFS-3457:
----------------------------------
{quote}
any chance you can figure out an easier way to reproduce this?
{quote}
Iam working on that.Once i found out an easier scenario to reproduce this issue ill update it.
> Number of UnderReplicated blocks and Number of Files in the cluster, displayed in UI and Fsck report is different
> -----------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3457
> URL: https://issues.apache.org/jira/browse/HDFS-3457
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: J.Andreina
> Priority: Minor
> Attachments: Mismatch in Number of Underreplicated blocks UI.jpg, Mismatch in number of files UI.jpg
>
>
> Scenario:
> =========
> Write an HDFS application with the following sequence of operations
> 1. Create file.
> 2. Append and sync file.
> 3. Delete file.
> 4. Create file.
> 5. Rename file.
> Run the application using 50 threads for 4 hours.
> Next Run the same application using 200 threads for the next 4 hours.
> Next Run the application using 50 threads for the next 4 hours.
> The Number of under-Replicated blocks and Total number of files mentioned in the UI and fsck report differs
> Fsck report for the mismatch in Number of under-Replicated blocks :
> ===================================================================
> Status: HEALTHY
> Total size: 5670200922 B
> Total dirs: 2
> Total files: 2015
> Total blocks (validated): 977 (avg. block size 5803685 B)
> Minimally replicated blocks: 977 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 94 (9.621289 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 2
> Average block replication: 1.9037871
> Corrupt blocks: 0
> Missing replicas: 94 (5.0537634 %)
> Number of data-nodes: 3
> Number of racks: 1
> FSCK ended at Mon Mar 19 11:14:41 IST 2012 in 94 milliseconds
> The filesystem under path '/' is HEALTHY
> Fsck report for the mismatch in Total number of Files :
> =======================================================
> Status: HEALTHY
> Total size: 19418 B (Total open files size: 42729325 B)
> Total dirs: 2
> Total files: 4226 (Files currently being written: 15)
> Total blocks (validated): 266 (avg. block size 73 B) (Total open file blocks (not validated): 5)
> Minimally replicated blocks: 266 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 0 (0.0 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 2
> Average block replication: 2.0
> Corrupt blocks: 0
> Missing replicas: 0 (0.0 %)
> Number of data-nodes: 3
> Number of racks: 2
> FSCK ended at Wed Apr 04 10:41:44 IST 2012 in 2302 milliseconds
> The filesystem under path '/' is HEALTHY
> Have attached UI screenshot for both the issues
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3457) Number of UnderReplicated blocks and
Number of Files in the cluster, displayed in UI and Fsck report is
different
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281732#comment-13281732 ]
Aaron T. Myers commented on HDFS-3457:
--------------------------------------
Hello, any chance you can figure out an easier way to reproduce this? Running some prescribed workload for a total of 12 hours doesn't will make this tough to verify/fix.
> Number of UnderReplicated blocks and Number of Files in the cluster, displayed in UI and Fsck report is different
> -----------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3457
> URL: https://issues.apache.org/jira/browse/HDFS-3457
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0
> Reporter: J.Andreina
> Priority: Minor
> Attachments: Mismatch in Number of Underreplicated blocks UI.jpg, Mismatch in number of files UI.jpg
>
>
> Scenario:
> =========
> Write an HDFS application with the following sequence of operations
> 1. Create file.
> 2. Append and sync file.
> 3. Delete file.
> 4. Create file.
> 5. Rename file.
> Run the application using 50 threads for 4 hours.
> Next Run the same application using 200 threads for the next 4 hours.
> Next Run the application using 50 threads for the next 4 hours.
> The Number of under-Replicated blocks and Total number of files mentioned in the UI and fsck report differs
> Fsck report for the mismatch in Number of under-Replicated blocks :
> ===================================================================
> Status: HEALTHY
> Total size: 5670200922 B
> Total dirs: 2
> Total files: 2015
> Total blocks (validated): 977 (avg. block size 5803685 B)
> Minimally replicated blocks: 977 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 94 (9.621289 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 2
> Average block replication: 1.9037871
> Corrupt blocks: 0
> Missing replicas: 94 (5.0537634 %)
> Number of data-nodes: 3
> Number of racks: 1
> FSCK ended at Mon Mar 19 11:14:41 IST 2012 in 94 milliseconds
> The filesystem under path '/' is HEALTHY
> Fsck report for the mismatch in Total number of Files :
> =======================================================
> Status: HEALTHY
> Total size: 19418 B (Total open files size: 42729325 B)
> Total dirs: 2
> Total files: 4226 (Files currently being written: 15)
> Total blocks (validated): 266 (avg. block size 73 B) (Total open file blocks (not validated): 5)
> Minimally replicated blocks: 266 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 0 (0.0 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 2
> Average block replication: 2.0
> Corrupt blocks: 0
> Missing replicas: 0 (0.0 %)
> Number of data-nodes: 3
> Number of racks: 2
> FSCK ended at Wed Apr 04 10:41:44 IST 2012 in 2302 milliseconds
> The filesystem under path '/' is HEALTHY
> Have attached UI screenshot for both the issues
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3457) Number of UnderReplicated blocks
displayed in UI and Fsck report is different
Posted by "J.Andreina (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
J.Andreina updated HDFS-3457:
-----------------------------
Description:
Scenario:
=========
Write an HDFS application with the following sequence of operations
1. Create file.
2. Append and sync file.
3. Delete file.
4. Create file.
5. Rename file.
Run the application using 50 threads for 4 hours.
Next Run the same application using 200 threads for the next 4 hours.
Next Run the application using 50 threads for the next 4 hours.
The Number of under-Replicated blocks mentioned in the UI and fsck report differs
Fsck report for the mismatch in Number of under-Replicated blocks :
===================================================================
Status: HEALTHY
Total size: 5670200922 B
Total dirs: 2
Total files: 2015
Total blocks (validated): 977 (avg. block size 5803685 B)
Minimally replicated blocks: 977 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 94 (9.621289 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 1.9037871
Corrupt blocks: 0
Missing replicas: 94 (5.0537634 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Mon Mar 19 11:14:41 IST 2012 in 94 milliseconds
The filesystem under path '/' is HEALTHY
Have attached UI screenshot for this issue
was:
Scenario:
=========
Write an HDFS application with the following sequence of operations
1. Create file.
2. Append and sync file.
3. Delete file.
4. Create file.
5. Rename file.
Run the application using 50 threads for 4 hours.
Next Run the same application using 200 threads for the next 4 hours.
Next Run the application using 50 threads for the next 4 hours.
The Number of under-Replicated blocks and Total number of files mentioned in the UI and fsck report differs
Fsck report for the mismatch in Number of under-Replicated blocks :
===================================================================
Status: HEALTHY
Total size: 5670200922 B
Total dirs: 2
Total files: 2015
Total blocks (validated): 977 (avg. block size 5803685 B)
Minimally replicated blocks: 977 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 94 (9.621289 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 1.9037871
Corrupt blocks: 0
Missing replicas: 94 (5.0537634 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Mon Mar 19 11:14:41 IST 2012 in 94 milliseconds
The filesystem under path '/' is HEALTHY
Fsck report for the mismatch in Total number of Files :
=======================================================
Status: HEALTHY
Total size: 19418 B (Total open files size: 42729325 B)
Total dirs: 2
Total files: 4226 (Files currently being written: 15)
Total blocks (validated): 266 (avg. block size 73 B) (Total open file blocks (not validated): 5)
Minimally replicated blocks: 266 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 3
Number of racks: 2
FSCK ended at Wed Apr 04 10:41:44 IST 2012 in 2302 milliseconds
The filesystem under path '/' is HEALTHY
Have attached UI screenshot for both the issues
Summary: Number of UnderReplicated blocks displayed in UI and Fsck report is different (was: Number of UnderReplicated blocks and Number of Files in the cluster, displayed in UI and Fsck report is different)
> Number of UnderReplicated blocks displayed in UI and Fsck report is different
> -----------------------------------------------------------------------------
>
> Key: HDFS-3457
> URL: https://issues.apache.org/jira/browse/HDFS-3457
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: J.Andreina
> Priority: Minor
> Attachments: Mismatch in Number of Underreplicated blocks UI.jpg, Mismatch in number of files UI.jpg
>
>
> Scenario:
> =========
> Write an HDFS application with the following sequence of operations
> 1. Create file.
> 2. Append and sync file.
> 3. Delete file.
> 4. Create file.
> 5. Rename file.
> Run the application using 50 threads for 4 hours.
> Next Run the same application using 200 threads for the next 4 hours.
> Next Run the application using 50 threads for the next 4 hours.
> The Number of under-Replicated blocks mentioned in the UI and fsck report differs
> Fsck report for the mismatch in Number of under-Replicated blocks :
> ===================================================================
> Status: HEALTHY
> Total size: 5670200922 B
> Total dirs: 2
> Total files: 2015
> Total blocks (validated): 977 (avg. block size 5803685 B)
> Minimally replicated blocks: 977 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 94 (9.621289 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 2
> Average block replication: 1.9037871
> Corrupt blocks: 0
> Missing replicas: 94 (5.0537634 %)
> Number of data-nodes: 3
> Number of racks: 1
> FSCK ended at Mon Mar 19 11:14:41 IST 2012 in 94 milliseconds
> The filesystem under path '/' is HEALTHY
> Have attached UI screenshot for this issue
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira