You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Alexander Aristov <al...@gmail.com> on 2008/11/24 11:11:27 UTC
s3n exceptions
Hi all
I am testing s3n file system facilities and try to copy from hdfs to S3 in
original format
And I get next errors
08/11/24 05:04:49 INFO mapred.JobClient: Running job: job_200811240437_0004
08/11/24 05:04:50 INFO mapred.JobClient: map 0% reduce 0%
08/11/24 05:05:00 INFO mapred.JobClient: map 44% reduce 0%
08/11/24 05:05:03 INFO mapred.JobClient: map 0% reduce 0%
08/11/24 05:05:03 INFO mapred.JobClient: Task Id :
attempt_200811240437_0004_m_000000_0, Status : FAILED
java.io.IOException: Copied: 0 Skipped: 0 Failed: 7
at
org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:542)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
08/11/24 05:05:15 INFO mapred.JobClient: Task Id :
attempt_200811240437_0004_m_000000_1, Status : FAILED
java.io.IOException: Copied: 0 Skipped: 0 Failed: 7
at
org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:542)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
--
Best Regards
Alexander Aristov
Re: s3n exceptions
Posted by Per Jacobsson <pe...@pjacobsson.com>.
I'm thinking of this bug:
http://issues.apache.org/jira/browse/HADOOP-3361
It was the cause of problems I hit trying to copy from HDFS to S3N with
0.18.1.
/ Per
On Wed, Nov 26, 2008 at 12:57 PM, Alexander Aristov <
alexander.aristov@gmail.com> wrote:
> yes, I am trying 0.18.2
> But according to hadoop wiki it is supported
>
> http://wiki.apache.org/hadoop/AmazonS3
> and
> http://issues.apache.org/jira/browse/HADOOP-930
>
> Alexander
>
>
> 2008/11/26 Per Jacobsson <pe...@pjacobsson.com>
>
> > Are you using 0.18? I know that the copy HDFS to s3n isn't supported
> there
> > yet. I think there's a fix in 0.19.
> > / Per
> >
> > On Mon, Nov 24, 2008 at 2:11 AM, Alexander Aristov <
> > alexander.aristov@gmail.com> wrote:
> >
> > > Hi all
> > >
> > > I am testing s3n file system facilities and try to copy from hdfs to S3
> > in
> > > original format
> > >
> > > And I get next errors
> > >
> > > 08/11/24 05:04:49 INFO mapred.JobClient: Running job:
> > job_200811240437_0004
> > > 08/11/24 05:04:50 INFO mapred.JobClient: map 0% reduce 0%
> > > 08/11/24 05:05:00 INFO mapred.JobClient: map 44% reduce 0%
> > > 08/11/24 05:05:03 INFO mapred.JobClient: map 0% reduce 0%
> > > 08/11/24 05:05:03 INFO mapred.JobClient: Task Id :
> > > attempt_200811240437_0004_m_000000_0, Status : FAILED
> > > java.io.IOException: Copied: 0 Skipped: 0 Failed: 7
> > > at
> > > org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:542)
> > > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227)
> > > at
> > > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
> > >
> > > 08/11/24 05:05:15 INFO mapred.JobClient: Task Id :
> > > attempt_200811240437_0004_m_000000_1, Status : FAILED
> > > java.io.IOException: Copied: 0 Skipped: 0 Failed: 7
> > > at
> > > org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:542)
> > > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227)
> > > at
> > > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
> > >
> > >
> > > --
> > > Best Regards
> > > Alexander Aristov
> > >
> >
>
>
>
> --
> Best Regards
> Alexander Aristov
>
Re: s3n exceptions
Posted by Alexander Aristov <al...@gmail.com>.
yes, I am trying 0.18.2
But according to hadoop wiki it is supported
http://wiki.apache.org/hadoop/AmazonS3
and
http://issues.apache.org/jira/browse/HADOOP-930
Alexander
2008/11/26 Per Jacobsson <pe...@pjacobsson.com>
> Are you using 0.18? I know that the copy HDFS to s3n isn't supported there
> yet. I think there's a fix in 0.19.
> / Per
>
> On Mon, Nov 24, 2008 at 2:11 AM, Alexander Aristov <
> alexander.aristov@gmail.com> wrote:
>
> > Hi all
> >
> > I am testing s3n file system facilities and try to copy from hdfs to S3
> in
> > original format
> >
> > And I get next errors
> >
> > 08/11/24 05:04:49 INFO mapred.JobClient: Running job:
> job_200811240437_0004
> > 08/11/24 05:04:50 INFO mapred.JobClient: map 0% reduce 0%
> > 08/11/24 05:05:00 INFO mapred.JobClient: map 44% reduce 0%
> > 08/11/24 05:05:03 INFO mapred.JobClient: map 0% reduce 0%
> > 08/11/24 05:05:03 INFO mapred.JobClient: Task Id :
> > attempt_200811240437_0004_m_000000_0, Status : FAILED
> > java.io.IOException: Copied: 0 Skipped: 0 Failed: 7
> > at
> > org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:542)
> > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227)
> > at
> > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
> >
> > 08/11/24 05:05:15 INFO mapred.JobClient: Task Id :
> > attempt_200811240437_0004_m_000000_1, Status : FAILED
> > java.io.IOException: Copied: 0 Skipped: 0 Failed: 7
> > at
> > org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:542)
> > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227)
> > at
> > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
> >
> >
> > --
> > Best Regards
> > Alexander Aristov
> >
>
--
Best Regards
Alexander Aristov
Re: s3n exceptions
Posted by Per Jacobsson <pe...@pjacobsson.com>.
Are you using 0.18? I know that the copy HDFS to s3n isn't supported there
yet. I think there's a fix in 0.19.
/ Per
On Mon, Nov 24, 2008 at 2:11 AM, Alexander Aristov <
alexander.aristov@gmail.com> wrote:
> Hi all
>
> I am testing s3n file system facilities and try to copy from hdfs to S3 in
> original format
>
> And I get next errors
>
> 08/11/24 05:04:49 INFO mapred.JobClient: Running job: job_200811240437_0004
> 08/11/24 05:04:50 INFO mapred.JobClient: map 0% reduce 0%
> 08/11/24 05:05:00 INFO mapred.JobClient: map 44% reduce 0%
> 08/11/24 05:05:03 INFO mapred.JobClient: map 0% reduce 0%
> 08/11/24 05:05:03 INFO mapred.JobClient: Task Id :
> attempt_200811240437_0004_m_000000_0, Status : FAILED
> java.io.IOException: Copied: 0 Skipped: 0 Failed: 7
> at
> org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:542)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227)
> at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
>
> 08/11/24 05:05:15 INFO mapred.JobClient: Task Id :
> attempt_200811240437_0004_m_000000_1, Status : FAILED
> java.io.IOException: Copied: 0 Skipped: 0 Failed: 7
> at
> org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:542)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227)
> at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
>
>
> --
> Best Regards
> Alexander Aristov
>