You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by vishalsant <Vi...@gmail.com> on 2010/07/21 01:05:35 UTC
Files reducing to zero byte size
I ran a Map Reduce job , and created using Multiple Outputs and reduced to a
bunch of files
I catted them , did a copytoLocal and all the good stuff one does.
I come back after couple of days and those files are 0 bytes.
drwxr-xr-x - hadoop supergroup 0 2010-07-19 14:29
/user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/_logs
-rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
/user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00000
-rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
/user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00001
-rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
/user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00002
-rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
/user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00003
-rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
/user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00004
-rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
/user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00005
-rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
/user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00006
-rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
/user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00007
-rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
/user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00008
-rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
/user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00009
-rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
/user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00010
-rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
/user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00011
-rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
/user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00012
-rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
/user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00013
And this has happened more than once.
I am not sure where to start looking. I run fsck -blocks and it seems to be
OK.
Total size: 2661524498815 B (Total open files size: 60545982 B)
Total dirs: 7273
Total files: 598942 (Files currently being written: 108)
Total blocks (validated): 460457 (avg. block size 5780180 B)
(Total open file blocks (not validated): 108)
Minimally replicated blocks: 460457 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 28 (0.006080915 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.947261
Corrupt blocks: 0
Missing replicas: 56 (0.0041264854 %)
Number of data-nodes: 8
Number of racks: 1
I have checked the name node logs and nothing , apart from
2010-07-15 10:06:02,336 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 3 on 8020, call
complete(/user/hadoop/aggregations/adevent/event/20100714/_temporary/_attempt_201007031521_0340_r_000031_0/hh-r-00031,
DFSClient_attempt_201007031521_0340_r_000031_0) from 10.17.28.210:50225:
error: java.io.IOException: Could not complete write to file
/user/hadoop/aggregations/adevent/event/20100714/_temporary/_attempt_201007031521_0340_r_000031_0/hh-r-00031
by DFSClient_attempt_201007031521_0340_r_000031_0
java.io.IOException: Could not complete write to file
/user/hadoop/aggregations/adevent/event/20100714/_temporary/_attempt_201007031521_0340_r_000031_0/hh-r-00031
by DFSClient_attempt_201007031521_0340_r_000031_0
2010-07-15 10:07:10,947 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=hadoop,hadoop
ip=/10.17.28.218 cmd=open
src=/user/hadoop/aggregations/adevent/event/20100714/hh-r-00031
dst=null perm=null
2010-07-15 10:11:47,609 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=hadoop,hadoop
ip=/10.36.67.130 cmd=open
src=/user/hadoop/aggregations/adevent/event/20100714/hh-r-00031
dst=null perm=null
2010-07-15 11:05:51,833 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Recovering
lease=[Lease. Holder: DFSClient_attempt_201007031521_0340_r_000031_0,
pendingcreates: 3],
src=/user/hadoop/aggregations/adevent/event/20100714/hh-r-00031
2010-07-15 12:05:52,906 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Recovering
lease=[Lease. Holder: DFSClient_attempt_201007031521_0340_r_000031_0,
pendingcreates: 3],
src=/user/hadoop/aggregations/adevent/event/20100714/hh-r-00031
2010-07-15 12:05:54,607 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
commitBlockSynchronization(newblock=blk_4668499616665973908_2986137,
file=/user/hadoop/aggregations/adevent/event/20100714/hh-r-00031,
newgenerationstamp=0, newlength=0, newtargets=[]) successful
As I perceive this .. it did succeed ...
So not sure what is happening.
--
View this message in context: http://old.nabble.com/Files-reducing-to-zero-byte-size-tp29220487p29220487.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: Files reducing to zero byte size
Posted by vishalsant <Vi...@gmail.com>.
I think I got it ..
The reason is that I do do
public void configure(JobConf job) {
mos = new MultipleOutputs(job);
}
for multipe output
but I forgot ( kicking myself very very very very very hard )
/**
* close {@link #mos}
*/
public void close() throws IOException {
if (mos != null) {
mos.close();
}
}
and that causes the exception below and it does not write it.
I do see it for a time and that is pbly b'coz . the namenode feeds it from
the temp directory.
Please correct if I am wrong .
vishalsant wrote:
>
> I ran a Map Reduce job , and created using Multiple Outputs and reduced to
> a bunch of files
> I catted them , did a copytoLocal and all the good stuff one does.
>
> I come back after couple of days and those files are 0 bytes.
>
> drwxr-xr-x - hadoop supergroup 0 2010-07-19 14:29
> /user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/_logs
> -rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
> /user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00000
> -rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
> /user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00001
> -rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
> /user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00002
> -rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
> /user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00003
> -rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
> /user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00004
> -rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
> /user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00005
> -rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
> /user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00006
> -rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
> /user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00007
> -rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
> /user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00008
> -rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
> /user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00009
> -rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
> /user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00010
> -rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
> /user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00011
> -rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
> /user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00012
> -rw-r--r-- 3 hadoop supergroup 0 2010-07-19 14:30
> /user/hadoop/aggregations/adevent/event/2010_07_18/dba4e0ea936911dfbf590050c2490048/dd-r-00013
>
>
> And this has happened more than once.
>
>
> I am not sure where to start looking. I run fsck -blocks and it seems to
> be OK.
>
> Total size: 2661524498815 B (Total open files size: 60545982 B)
> Total dirs: 7273
> Total files: 598942 (Files currently being written: 108)
> Total blocks (validated): 460457 (avg. block size 5780180 B)
> (Total open file blocks (not validated): 108)
> Minimally replicated blocks: 460457 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 28 (0.006080915 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 2
> Average block replication: 2.947261
> Corrupt blocks: 0
> Missing replicas: 56 (0.0041264854 %)
> Number of data-nodes: 8
> Number of racks: 1
>
>
> I have checked the name node logs and nothing , apart from
>
>
>
> 2010-07-15 10:06:02,336 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 3 on 8020, call
> complete(/user/hadoop/aggregations/adevent/event/20100714/_temporary/_attempt_201007031521_0340_r_000031_0/hh-r-00031,
> DFSClient_attempt_201007031521_0340_r_000031_0) from 10.17.28.210:50225:
> error: java.io.IOException: Could not complete write to file
> /user/hadoop/aggregations/adevent/event/20100714/_temporary/_attempt_201007031521_0340_r_000031_0/hh-r-00031
> by DFSClient_attempt_201007031521_0340_r_000031_0
> java.io.IOException: Could not complete write to file
> /user/hadoop/aggregations/adevent/event/20100714/_temporary/_attempt_201007031521_0340_r_000031_0/hh-r-00031
> by DFSClient_attempt_201007031521_0340_r_000031_0
> 2010-07-15 10:07:10,947 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
> ugi=hadoop,hadoop ip=/10.17.28.218 cmd=open
> src=/user/hadoop/aggregations/adevent/event/20100714/hh-r-00031
> dst=null perm=null
> 2010-07-15 10:11:47,609 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
> ugi=hadoop,hadoop ip=/10.36.67.130 cmd=open
> src=/user/hadoop/aggregations/adevent/event/20100714/hh-r-00031
> dst=null perm=null
> 2010-07-15 11:05:51,833 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Recovering
> lease=[Lease. Holder: DFSClient_attempt_201007031521_0340_r_000031_0,
> pendingcreates: 3],
> src=/user/hadoop/aggregations/adevent/event/20100714/hh-r-00031
> 2010-07-15 12:05:52,906 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Recovering
> lease=[Lease. Holder: DFSClient_attempt_201007031521_0340_r_000031_0,
> pendingcreates: 3],
> src=/user/hadoop/aggregations/adevent/event/20100714/hh-r-00031
> 2010-07-15 12:05:54,607 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> commitBlockSynchronization(newblock=blk_4668499616665973908_2986137,
> file=/user/hadoop/aggregations/adevent/event/20100714/hh-r-00031,
> newgenerationstamp=0, newlength=0, newtargets=[]) successful
>
>
> As I perceive this .. it did succeed ...
>
>
> So not sure what is happening.
>
>
>
>
--
View this message in context: http://old.nabble.com/Files-reducing-to-zero-byte-size-tp29220487p29220586.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.