You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Brian Jeltema <br...@digitalenvoy.net> on 2014/10/03 17:01:31 UTC
ExportSnapshot webhdfs problems
I posted this on users@hbase, but got not response, so I thought I’d try here:
I’m trying to use ExportSnapshot to copy a snapshot from a Hadoop 1 to a Hadoop 2 cluster using the webhdfs protocol.
I’ve done this successfully before, though there are always mapper failures and retries in the job log. However, I’m not
having success with a rather large table due to an excessive number of failures. The exceptions in the job log are always:
14/09/29 20:28:11 INFO mapred.JobClient: Task Id : attempt_201409241055_0024_m_000005_1, Status : FAILED
org.apache.hadoop.ipc.RemoteException
at org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:290)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:98)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$2.close(WebHdfsFileSystem.java:653)
at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyFile(ExportSnapshot.java:204)
at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:146)
…
…
…
So I presume the real exception is taking place on the target system. However, examining the namenode logs and a handful of
the datanode logs has not revealed any exceptions that correlate with those in the job log. Is there some other log I should be looking at?
I reduced the number of mappers to 6 and the target cluster has 10 datanodes, so it’s hard to believe its a capacity problem.
Thanks
Brian
Re: ExportSnapshot webhdfs problems
Posted by Brian Jeltema <br...@digitalenvoy.net>.
I’ll try hftp next time.
This is an intermittent problem. Mapper retries will typically complete eventually.
I got past the problem by tolerating more mapper failures before killing the job.
On Oct 6, 2014, at 9:24 AM, Artem Ervits <ar...@nyp.org> wrote:
> Does it work with a small table? I prefer to use hftp instead of webhdfs.
>
> From: Brian Jeltema [mailto:brian.jeltema@digitalenvoy.net]
> Sent: Friday, October 03, 2014 11:01 AM
> To: user@hadoop.apache.org <us...@hadoop.apache.org>
> Subject: ExportSnapshot webhdfs problems
>
> I posted this on users@hbase, but got not response, so I thought I’d try here:
>
> I’m trying to use ExportSnapshot to copy a snapshot from a Hadoop 1 to a Hadoop 2 cluster using the webhdfs protocol.
> I’ve done this successfully before, though there are always mapper failures and retries in the job log. However, I’m not
> having success with a rather large table due to an excessive number of failures. The exceptions in the job log are always:
>
> 14/09/29 20:28:11 INFO mapred.JobClient: Task Id : attempt_201409241055_0024_m_000005_1, Status : FAILED
> org.apache.hadoop.ipc.RemoteException
> at org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:290)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:98)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$2.close(WebHdfsFileSystem.java:653)
> at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyFile(ExportSnapshot.java:204)
> at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:146)
> …
> …
> …
>
>
> So I presume the real exception is taking place on the target system. However, examining the namenode logs and a handful of
> the datanode logs has not revealed any exceptions that correlate with those in the job log. Is there some other log I should be looking at?
> I reduced the number of mappers to 6 and the target cluster has 10 datanodes, so it’s hard to believe its a capacity problem.
>
> Thanks
> Brian
>
> This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
Re: ExportSnapshot webhdfs problems
Posted by Brian Jeltema <br...@digitalenvoy.net>.
I’ll try hftp next time.
This is an intermittent problem. Mapper retries will typically complete eventually.
I got past the problem by tolerating more mapper failures before killing the job.
On Oct 6, 2014, at 9:24 AM, Artem Ervits <ar...@nyp.org> wrote:
> Does it work with a small table? I prefer to use hftp instead of webhdfs.
>
> From: Brian Jeltema [mailto:brian.jeltema@digitalenvoy.net]
> Sent: Friday, October 03, 2014 11:01 AM
> To: user@hadoop.apache.org <us...@hadoop.apache.org>
> Subject: ExportSnapshot webhdfs problems
>
> I posted this on users@hbase, but got not response, so I thought I’d try here:
>
> I’m trying to use ExportSnapshot to copy a snapshot from a Hadoop 1 to a Hadoop 2 cluster using the webhdfs protocol.
> I’ve done this successfully before, though there are always mapper failures and retries in the job log. However, I’m not
> having success with a rather large table due to an excessive number of failures. The exceptions in the job log are always:
>
> 14/09/29 20:28:11 INFO mapred.JobClient: Task Id : attempt_201409241055_0024_m_000005_1, Status : FAILED
> org.apache.hadoop.ipc.RemoteException
> at org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:290)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:98)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$2.close(WebHdfsFileSystem.java:653)
> at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyFile(ExportSnapshot.java:204)
> at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:146)
> …
> …
> …
>
>
> So I presume the real exception is taking place on the target system. However, examining the namenode logs and a handful of
> the datanode logs has not revealed any exceptions that correlate with those in the job log. Is there some other log I should be looking at?
> I reduced the number of mappers to 6 and the target cluster has 10 datanodes, so it’s hard to believe its a capacity problem.
>
> Thanks
> Brian
>
> This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
Re: ExportSnapshot webhdfs problems
Posted by Brian Jeltema <br...@digitalenvoy.net>.
I’ll try hftp next time.
This is an intermittent problem. Mapper retries will typically complete eventually.
I got past the problem by tolerating more mapper failures before killing the job.
On Oct 6, 2014, at 9:24 AM, Artem Ervits <ar...@nyp.org> wrote:
> Does it work with a small table? I prefer to use hftp instead of webhdfs.
>
> From: Brian Jeltema [mailto:brian.jeltema@digitalenvoy.net]
> Sent: Friday, October 03, 2014 11:01 AM
> To: user@hadoop.apache.org <us...@hadoop.apache.org>
> Subject: ExportSnapshot webhdfs problems
>
> I posted this on users@hbase, but got not response, so I thought I’d try here:
>
> I’m trying to use ExportSnapshot to copy a snapshot from a Hadoop 1 to a Hadoop 2 cluster using the webhdfs protocol.
> I’ve done this successfully before, though there are always mapper failures and retries in the job log. However, I’m not
> having success with a rather large table due to an excessive number of failures. The exceptions in the job log are always:
>
> 14/09/29 20:28:11 INFO mapred.JobClient: Task Id : attempt_201409241055_0024_m_000005_1, Status : FAILED
> org.apache.hadoop.ipc.RemoteException
> at org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:290)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:98)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$2.close(WebHdfsFileSystem.java:653)
> at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyFile(ExportSnapshot.java:204)
> at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:146)
> …
> …
> …
>
>
> So I presume the real exception is taking place on the target system. However, examining the namenode logs and a handful of
> the datanode logs has not revealed any exceptions that correlate with those in the job log. Is there some other log I should be looking at?
> I reduced the number of mappers to 6 and the target cluster has 10 datanodes, so it’s hard to believe its a capacity problem.
>
> Thanks
> Brian
>
> This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
Re: ExportSnapshot webhdfs problems
Posted by Brian Jeltema <br...@digitalenvoy.net>.
I’ll try hftp next time.
This is an intermittent problem. Mapper retries will typically complete eventually.
I got past the problem by tolerating more mapper failures before killing the job.
On Oct 6, 2014, at 9:24 AM, Artem Ervits <ar...@nyp.org> wrote:
> Does it work with a small table? I prefer to use hftp instead of webhdfs.
>
> From: Brian Jeltema [mailto:brian.jeltema@digitalenvoy.net]
> Sent: Friday, October 03, 2014 11:01 AM
> To: user@hadoop.apache.org <us...@hadoop.apache.org>
> Subject: ExportSnapshot webhdfs problems
>
> I posted this on users@hbase, but got not response, so I thought I’d try here:
>
> I’m trying to use ExportSnapshot to copy a snapshot from a Hadoop 1 to a Hadoop 2 cluster using the webhdfs protocol.
> I’ve done this successfully before, though there are always mapper failures and retries in the job log. However, I’m not
> having success with a rather large table due to an excessive number of failures. The exceptions in the job log are always:
>
> 14/09/29 20:28:11 INFO mapred.JobClient: Task Id : attempt_201409241055_0024_m_000005_1, Status : FAILED
> org.apache.hadoop.ipc.RemoteException
> at org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:290)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:98)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$2.close(WebHdfsFileSystem.java:653)
> at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyFile(ExportSnapshot.java:204)
> at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:146)
> …
> …
> …
>
>
> So I presume the real exception is taking place on the target system. However, examining the namenode logs and a handful of
> the datanode logs has not revealed any exceptions that correlate with those in the job log. Is there some other log I should be looking at?
> I reduced the number of mappers to 6 and the target cluster has 10 datanodes, so it’s hard to believe its a capacity problem.
>
> Thanks
> Brian
>
> This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
Re: ExportSnapshot webhdfs problems
Posted by Artem Ervits <ar...@nyp.org>.
Does it work with a small table? I prefer to use hftp instead of webhdfs.
From: Brian Jeltema [mailto:brian.jeltema@digitalenvoy.net]
Sent: Friday, October 03, 2014 11:01 AM
To: user@hadoop.apache.org <us...@hadoop.apache.org>
Subject: ExportSnapshot webhdfs problems
I posted this on users@hbase, but got not response, so I thought I’d try here:
I’m trying to use ExportSnapshot to copy a snapshot from a Hadoop 1 to a Hadoop 2 cluster using the webhdfs protocol.
I’ve done this successfully before, though there are always mapper failures and retries in the job log. However, I’m not
having success with a rather large table due to an excessive number of failures. The exceptions in the job log are always:
14/09/29 20:28:11 INFO mapred.JobClient: Task Id : attempt_201409241055_0024_m_000005_1, Status : FAILED
org.apache.hadoop.ipc.RemoteException
at org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:290)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:98)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$2.close(WebHdfsFileSystem.java:653)
at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyFile(ExportSnapshot.java:204)
at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:146)
…
…
…
So I presume the real exception is taking place on the target system. However, examining the namenode logs and a handful of
the datanode logs has not revealed any exceptions that correlate with those in the job log. Is there some other log I should be looking at?
I reduced the number of mappers to 6 and the target cluster has 10 datanodes, so it’s hard to believe its a capacity problem.
Thanks
Brian
This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
Re: ExportSnapshot webhdfs problems
Posted by Artem Ervits <ar...@nyp.org>.
Does it work with a small table? I prefer to use hftp instead of webhdfs.
From: Brian Jeltema [mailto:brian.jeltema@digitalenvoy.net]
Sent: Friday, October 03, 2014 11:01 AM
To: user@hadoop.apache.org <us...@hadoop.apache.org>
Subject: ExportSnapshot webhdfs problems
I posted this on users@hbase, but got not response, so I thought I’d try here:
I’m trying to use ExportSnapshot to copy a snapshot from a Hadoop 1 to a Hadoop 2 cluster using the webhdfs protocol.
I’ve done this successfully before, though there are always mapper failures and retries in the job log. However, I’m not
having success with a rather large table due to an excessive number of failures. The exceptions in the job log are always:
14/09/29 20:28:11 INFO mapred.JobClient: Task Id : attempt_201409241055_0024_m_000005_1, Status : FAILED
org.apache.hadoop.ipc.RemoteException
at org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:290)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:98)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$2.close(WebHdfsFileSystem.java:653)
at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyFile(ExportSnapshot.java:204)
at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:146)
…
…
…
So I presume the real exception is taking place on the target system. However, examining the namenode logs and a handful of
the datanode logs has not revealed any exceptions that correlate with those in the job log. Is there some other log I should be looking at?
I reduced the number of mappers to 6 and the target cluster has 10 datanodes, so it’s hard to believe its a capacity problem.
Thanks
Brian
This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
Re: ExportSnapshot webhdfs problems
Posted by Artem Ervits <ar...@nyp.org>.
Does it work with a small table? I prefer to use hftp instead of webhdfs.
From: Brian Jeltema [mailto:brian.jeltema@digitalenvoy.net]
Sent: Friday, October 03, 2014 11:01 AM
To: user@hadoop.apache.org <us...@hadoop.apache.org>
Subject: ExportSnapshot webhdfs problems
I posted this on users@hbase, but got not response, so I thought I’d try here:
I’m trying to use ExportSnapshot to copy a snapshot from a Hadoop 1 to a Hadoop 2 cluster using the webhdfs protocol.
I’ve done this successfully before, though there are always mapper failures and retries in the job log. However, I’m not
having success with a rather large table due to an excessive number of failures. The exceptions in the job log are always:
14/09/29 20:28:11 INFO mapred.JobClient: Task Id : attempt_201409241055_0024_m_000005_1, Status : FAILED
org.apache.hadoop.ipc.RemoteException
at org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:290)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:98)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$2.close(WebHdfsFileSystem.java:653)
at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyFile(ExportSnapshot.java:204)
at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:146)
…
…
…
So I presume the real exception is taking place on the target system. However, examining the namenode logs and a handful of
the datanode logs has not revealed any exceptions that correlate with those in the job log. Is there some other log I should be looking at?
I reduced the number of mappers to 6 and the target cluster has 10 datanodes, so it’s hard to believe its a capacity problem.
Thanks
Brian
This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
Re: ExportSnapshot webhdfs problems
Posted by Artem Ervits <ar...@nyp.org>.
Does it work with a small table? I prefer to use hftp instead of webhdfs.
From: Brian Jeltema [mailto:brian.jeltema@digitalenvoy.net]
Sent: Friday, October 03, 2014 11:01 AM
To: user@hadoop.apache.org <us...@hadoop.apache.org>
Subject: ExportSnapshot webhdfs problems
I posted this on users@hbase, but got not response, so I thought I’d try here:
I’m trying to use ExportSnapshot to copy a snapshot from a Hadoop 1 to a Hadoop 2 cluster using the webhdfs protocol.
I’ve done this successfully before, though there are always mapper failures and retries in the job log. However, I’m not
having success with a rather large table due to an excessive number of failures. The exceptions in the job log are always:
14/09/29 20:28:11 INFO mapred.JobClient: Task Id : attempt_201409241055_0024_m_000005_1, Status : FAILED
org.apache.hadoop.ipc.RemoteException
at org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:290)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:98)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$2.close(WebHdfsFileSystem.java:653)
at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyFile(ExportSnapshot.java:204)
at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:146)
…
…
…
So I presume the real exception is taking place on the target system. However, examining the namenode logs and a handful of
the datanode logs has not revealed any exceptions that correlate with those in the job log. Is there some other log I should be looking at?
I reduced the number of mappers to 6 and the target cluster has 10 datanodes, so it’s hard to believe its a capacity problem.
Thanks
Brian
This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.