You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Aditya exalter <ad...@gmail.com> on 2016/12/31 18:25:26 UTC
Mismatch in length of source:
Hi All,
A very happy new year to ALL.
I am facing issue while executing distcp between two different clusters,
Caused by: java.io.IOException: Mismatch in length of
source:hdfs://ip1/xxxxxxxxxx/xxxxx and
target:hdfs://nameservice1/xxxxxx/.distcp.tmp.attempt_1483200922993_0056_m_000011_2
I tried using -pb and -skipcrccheck
hadoop distcp -pb -skipcrccheck -update hdfs://ip1/xxxxxxxxxx/xxxxx
hdfs:///xxxxxxxxxxxx/
hadoop distcp -pb hdfs://ip1/xxxxxxxxxx/xxxxx hdfs:///xxxxxxxxxxxx/
hadoop distcp -skipcrccheck -update
hdfs://ip1/xxxxxxxxxx/xxxxx hdfs:///xxxxxxxxxxxx/
but nothing seems to be working .Any solutions please.
Regards,
Aditya.
RE: Mismatch in length of source:
Posted by Brahma Reddy Battula <br...@huawei.com>.
Distcp can be success with snapshot, but open files length can be zero..? see HDFS-11402
AFAIK, if you know the open files you can call recoverlease or wait for hardlimit (let Namenode trigger lease recovery).
i) Get the list of open files
e.g hdfs fsck -openforwrite / -files -blocks -locations | grep -i "OPENFORWRITE:"
ii) call recoverylease on each open files
e.g hdfs debug recoverlease
Note: Service like HBase where RS will keep open WAL files, better stop HBase service which can automatically close the file.
iii) and then go for distcp
Bytheway,HDFS-10480 gives list of open files.
--Brahma Reddy Battula
-----Original Message-----
From: Ulul [mailto:hadoop@ulul.org]
Sent: 02 January 2017 23:05
To: user@hadoop.apache.org
Subject: Re: Mismatch in length of source:
Hi
I can't remember the exact error message but distcp consistently fails when trying to copy open files. Is it your case ?
Workaround it to snapshot prior to copying
Ulul
On 31/12/2016 19:25, Aditya exalter wrote:
> Hi All,
> A very happy new year to ALL.
>
> I am facing issue while executing distcp between two different
> clusters,
>
> Caused by: java.io.IOException: Mismatch in length of
> source:hdfs://ip1/xxxxxxxxxx/xxxxx and
> target:hdfs://nameservice1/xxxxxx/.distcp.tmp.attempt_1483200922993_00
> 56_m_000011_2
>
> I tried using -pb and -skipcrccheck
>
> hadoop distcp -pb -skipcrccheck -update hdfs://ip1/xxxxxxxxxx/xxxxx
> hdfs:///xxxxxxxxxxxx/
>
> hadoop distcp -pb hdfs://ip1/xxxxxxxxxx/xxxxx hdfs:///xxxxxxxxxxxx/
>
> hadoop distcp -skipcrccheck -update
> hdfs://ip1/xxxxxxxxxx/xxxxx hdfs:///xxxxxxxxxxxx/
>
>
> but nothing seems to be working .Any solutions please.
>
>
> Regards,
> Aditya.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: Mismatch in length of source:
Posted by Ulul <ha...@ulul.org>.
Hi
I can't remember the exact error message but distcp consistently fails
when trying to copy open files. Is it your case ?
Workaround it to snapshot prior to copying
Ulul
On 31/12/2016 19:25, Aditya exalter wrote:
> Hi All,
> A very happy new year to ALL.
>
> I am facing issue while executing distcp between two different clusters,
>
> Caused by: java.io.IOException: Mismatch in length of
> source:hdfs://ip1/xxxxxxxxxx/xxxxx and
> target:hdfs://nameservice1/xxxxxx/.distcp.tmp.attempt_1483200922993_0056_m_000011_2
>
> I tried using -pb and -skipcrccheck
>
> hadoop distcp -pb -skipcrccheck -update hdfs://ip1/xxxxxxxxxx/xxxxx
> hdfs:///xxxxxxxxxxxx/
>
> hadoop distcp -pb hdfs://ip1/xxxxxxxxxx/xxxxx hdfs:///xxxxxxxxxxxx/
>
> hadoop distcp -skipcrccheck -update
> hdfs://ip1/xxxxxxxxxx/xxxxx hdfs:///xxxxxxxxxxxx/
>
>
> but nothing seems to be working .Any solutions please.
>
>
> Regards,
> Aditya.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org