You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by "Grant Overby (groverby)" <gr...@cisco.com> on 2015/09/15 17:20:19 UTC
BlockMissingException, but file seems fine.
I'm getting the following exception when using Drill. Asked on Drill mailing list a few days back; no one had any ideas.
As near as I can tell, the block is there and retrievable. Any thoughts on what could be causing this exception.
This is the stack trace:
Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1605794487-10.0.1.3-1435700184285:blk_1073829560_90288 file=/warehouse2/completed/events/connection_events/1441747800/1441747805625-1-fe3a49ba-3697-47ff-ab2c-614281346ff6.parquet
at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:880) ~[hadoop-hdfs-2.4.1.jar:na]
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:560) ~[hadoop-hdfs-2.4.1.jar:na]
at org.apache.hadoop.hdfs.DFSInputStream.seekToNewSource(DFSInputStream.java:1443) ~[hadoop-hdfs-2.4.1.jar:na]
at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:766) ~[hadoop-hdfs-2.4.1.jar:na]
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:796) ~[hadoop-hdfs-2.4.1.jar:na]
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:837) ~[hadoop-hdfs-2.4.1.jar:na]
at java.io.DataInputStream.read(DataInputStream.java:149) ~[na:1.7.0_79]
at org.apache.drill.exec.store.dfs.DrillFSDataInputStream$WrappedInputStream.read(DrillFSDataInputStream.java:211) ~[drill-java-exec-1.1.0.jar:1.1.0]
at java.io.DataInputStream.read(DataInputStream.java:149) ~[na:1.7.0_79]
at parquet.org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ~[parquet-format-2.1.1-drill-r1.jar:na]
... 40 common frames omitted
This is a two node cluster, both nodes can retrieve the file.
root@twig03:/opt/drill/log# hdfs dfs -get /warehouse2/completed/events/connection_events/1441747800/1441747805625-1-fe3a49ba-3697-47ff-ab2c-614281346ff6.parquet /tmp/.
root@twig03:/opt/drill/log# ls /tmp/*.parquet
/tmp/1441747805625-1-fe3a49ba-3697-47ff-ab2c-614281346ff6.parquet
root@twig04:~# hdfs dfs -get /warehouse2/completed/events/connection_events/1441747800/1441747805625-1-fe3a49ba-3697-47ff-ab2c-614281346ff6.parquet /tmp/.
root@twig04:~# ls /tmp/*.parquet
/tmp/1441747805625-1-fe3a49ba-3697-47ff-ab2c-614281346ff6.parquet
hdfs reports no missing blocks:
root@twig04:~# hdfs dfsadmin -report
Configured Capacity: 7856899358720 (7.15 TB)
Present Capacity: 7856899358720 (7.15 TB)
DFS Remaining: 4310190084096 (3.92 TB)
DFS Used: 3546709274624 (3.23 TB)
DFS Used%: 45.14%
Under replicated blocks: 23854
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Live datanodes (2):
Name: 10.0.1.4:50010 (twig04.twigs)
Hostname: twig04.twigs
Decommission Status : Normal
Configured Capacity: 3928449679360 (3.57 TB)
DFS Used: 1773355933696 (1.61 TB)
Non DFS Used: 0 (0 B)
DFS Remaining: 2155093745664 (1.96 TB)
DFS Used%: 45.14%
DFS Remaining%: 54.86%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 3
Last contact: Tue Sep 15 11:18:00 EDT 2015
Name: 10.0.1.3:50010 (twig03.twigs)
Hostname: twig03.twigs
Decommission Status : Normal
Configured Capacity: 3928449679360 (3.57 TB)
DFS Used: 1773353340928 (1.61 TB)
Non DFS Used: 0 (0 B)
DFS Remaining: 2155096338432 (1.96 TB)
DFS Used%: 45.14%
DFS Remaining%: 54.86%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Sep 15 11:18:02 EDT 2015
[http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726]
Grant Overby
Software Engineer
Cisco.com<http://www.cisco.com/>
groverby@cisco.com<ma...@cisco.com>
Mobile: 865 724 4910
[http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think before you print.
This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.
Please click here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for Company Registration Information.