You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Alan Crosswell <al...@crosswell.us> on 2013/06/08 21:26:37 UTC
JsonStorage() fails to write .pig_schema to S3 after correctly
writing alias to json files.
Hello,
I've been having trouble with JsonStorage(). First, since my Python UDF had
an outputSchema that returned floats, I was getting an error in JsonStorage
trying to cast Double to Float. I resolved this by changing my UDF to
return doubles.
Pig-0.11.1, hadoop-1.0.3.
Next, I am able to successfully write json files out to s3 (I was watching
as my Pig job was running and grabbed a sample) but then at what appears to
be the final step of writing .pig_schema, this error is thrown:
grunt> *STORE firsts INTO 's3n://n2ygk/firsthops.json' using JsonStorage();*
*
*
*... *chugs along for a while successfully writing
s3://n2ygk/firsthops.json/part-r-* into the bucket.... and then:
*java.lang.IllegalArgumentException: This file system object (hdfs://
10.253.44.244:9000) does not support access to the request path
's3n://n2ygk/firsthops.json/.pig_schema' You possibly called
FileSystem.get(conf) when you should have called FileSystem.get(uri, conf)
to obtain a file system supporting your path.*
at
org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:384)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:129)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:513)
at
org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:770)
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.isContainer(HDataStorage.java:200)
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:128)
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:144)
at
org.apache.pig.builtin.JsonMetadata.storeSchema(JsonMetadata.java:294)
at
org.apache.pig.builtin.JsonStorage.storeSchema(JsonStorage.java:274)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.storeCleanup(PigOutputCommitter.java:141)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:204)
at
org.apache.hadoop.mapred.Task.runJobCleanupTask(Task.java:1060)
at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:362)
at
org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at
java.security.AccessController.doPrivileged(Native Method)
at
javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
at
org.apache.hadoop.mapred.Child.main(Child.java:249)
Any ideas?
Thanks.
/a
Re: JsonStorage() fails to write .pig_schema to S3 after correctly
writing alias to json files.
Posted by Alan Crosswell <al...@crosswell.us>.
I'm using the builtin JsonStorage() AFAIK (e.g. the one documented in the
Pig manual).
On Sat, Jun 8, 2013 at 7:14 PM, Shahab Yunus <sh...@gmail.com> wrote:
> I don't have a solution but just wondering what lib are you using for
> JsonStorage? I hope it doesn't have any issues in it.
>
> Regards,
> Shahab
>
>
> On Sat, Jun 8, 2013 at 3:26 PM, Alan Crosswell <al...@crosswell.us> wrote:
>
> > Hello,
> >
> > I've been having trouble with JsonStorage(). First, since my Python UDF
> had
> > an outputSchema that returned floats, I was getting an error in
> JsonStorage
> > trying to cast Double to Float. I resolved this by changing my UDF to
> > return doubles.
> >
> > Pig-0.11.1, hadoop-1.0.3.
> >
> > Next, I am able to successfully write json files out to s3 (I was
> watching
> > as my Pig job was running and grabbed a sample) but then at what appears
> to
> > be the final step of writing .pig_schema, this error is thrown:
> >
> > grunt> *STORE firsts INTO 's3n://n2ygk/firsthops.json' using
> > JsonStorage();*
> > *
> > *
> > *... *chugs along for a while successfully writing
> > s3://n2ygk/firsthops.json/part-r-* into the bucket.... and then:
> >
> > *java.lang.IllegalArgumentException: This file system object (hdfs://
> > 10.253.44.244:9000) does not support access to the request path
> > 's3n://n2ygk/firsthops.json/.pig_schema' You possibly called
> > FileSystem.get(conf) when you should have called FileSystem.get(uri,
> conf)
> > to obtain a file system supporting your path.*
> > at
> > org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:384)
> > at
> >
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:129)
> > at
> >
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:513)
> > at
> > org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:770)
> > at
> >
> >
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.isContainer(HDataStorage.java:200)
> > at
> >
> >
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:128)
> > at
> >
> >
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:144)
> > at
> > org.apache.pig.builtin.JsonMetadata.storeSchema(JsonMetadata.java:294)
> > at
> > org.apache.pig.builtin.JsonStorage.storeSchema(JsonStorage.java:274)
> > at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.storeCleanup(PigOutputCommitter.java:141)
> > at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:204)
> > at
> > org.apache.hadoop.mapred.Task.runJobCleanupTask(Task.java:1060)
> > at
> > org.apache.hadoop.mapred.MapTask.run(MapTask.java:362)
> > at
> > org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> > at
> > java.security.AccessController.doPrivileged(Native Method)
> > at
> > javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
> > at
> > org.apache.hadoop.mapred.Child.main(Child.java:249)
> >
> > Any ideas?
> >
> > Thanks.
> > /a
> >
>
Re: JsonStorage() fails to write .pig_schema to S3 after correctly
writing alias to json files.
Posted by Shahab Yunus <sh...@gmail.com>.
I don't have a solution but just wondering what lib are you using for
JsonStorage? I hope it doesn't have any issues in it.
Regards,
Shahab
On Sat, Jun 8, 2013 at 3:26 PM, Alan Crosswell <al...@crosswell.us> wrote:
> Hello,
>
> I've been having trouble with JsonStorage(). First, since my Python UDF had
> an outputSchema that returned floats, I was getting an error in JsonStorage
> trying to cast Double to Float. I resolved this by changing my UDF to
> return doubles.
>
> Pig-0.11.1, hadoop-1.0.3.
>
> Next, I am able to successfully write json files out to s3 (I was watching
> as my Pig job was running and grabbed a sample) but then at what appears to
> be the final step of writing .pig_schema, this error is thrown:
>
> grunt> *STORE firsts INTO 's3n://n2ygk/firsthops.json' using
> JsonStorage();*
> *
> *
> *... *chugs along for a while successfully writing
> s3://n2ygk/firsthops.json/part-r-* into the bucket.... and then:
>
> *java.lang.IllegalArgumentException: This file system object (hdfs://
> 10.253.44.244:9000) does not support access to the request path
> 's3n://n2ygk/firsthops.json/.pig_schema' You possibly called
> FileSystem.get(conf) when you should have called FileSystem.get(uri, conf)
> to obtain a file system supporting your path.*
> at
> org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:384)
> at
>
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:129)
> at
>
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:513)
> at
> org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:770)
> at
>
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.isContainer(HDataStorage.java:200)
> at
>
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:128)
> at
>
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:144)
> at
> org.apache.pig.builtin.JsonMetadata.storeSchema(JsonMetadata.java:294)
> at
> org.apache.pig.builtin.JsonStorage.storeSchema(JsonStorage.java:274)
> at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.storeCleanup(PigOutputCommitter.java:141)
> at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:204)
> at
> org.apache.hadoop.mapred.Task.runJobCleanupTask(Task.java:1060)
> at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:362)
> at
> org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at
> java.security.AccessController.doPrivileged(Native Method)
> at
> javax.security.auth.Subject.doAs(Subject.java:396)
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
> at
> org.apache.hadoop.mapred.Child.main(Child.java:249)
>
> Any ideas?
>
> Thanks.
> /a
>
Re: JsonStorage() fails to write .pig_schema to S3 after correctly writing alias to json files.
Posted by Russell Jurney <ru...@gmail.com>.
Fwiw, I get the same error and can't use the built in JsonStorage.
Sent from my iPhone
On Jun 8, 2013, at 12:26 PM, Alan Crosswell <al...@crosswell.us> wrote:
> Hello,
>
> I've been having trouble with JsonStorage(). First, since my Python UDF had
> an outputSchema that returned floats, I was getting an error in JsonStorage
> trying to cast Double to Float. I resolved this by changing my UDF to
> return doubles.
>
> Pig-0.11.1, hadoop-1.0.3.
>
> Next, I am able to successfully write json files out to s3 (I was watching
> as my Pig job was running and grabbed a sample) but then at what appears to
> be the final step of writing .pig_schema, this error is thrown:
>
> grunt> *STORE firsts INTO 's3n://n2ygk/firsthops.json' using JsonStorage();*
> *
> *
> *... *chugs along for a while successfully writing
> s3://n2ygk/firsthops.json/part-r-* into the bucket.... and then:
>
> *java.lang.IllegalArgumentException: This file system object (hdfs://
> 10.253.44.244:9000) does not support access to the request path
> 's3n://n2ygk/firsthops.json/.pig_schema' You possibly called
> FileSystem.get(conf) when you should have called FileSystem.get(uri, conf)
> to obtain a file system supporting your path.*
> at
> org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:384)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:129)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:513)
> at
> org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:770)
> at
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.isContainer(HDataStorage.java:200)
> at
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:128)
> at
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:144)
> at
> org.apache.pig.builtin.JsonMetadata.storeSchema(JsonMetadata.java:294)
> at
> org.apache.pig.builtin.JsonStorage.storeSchema(JsonStorage.java:274)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.storeCleanup(PigOutputCommitter.java:141)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:204)
> at
> org.apache.hadoop.mapred.Task.runJobCleanupTask(Task.java:1060)
> at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:362)
> at
> org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at
> java.security.AccessController.doPrivileged(Native Method)
> at
> javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
> at
> org.apache.hadoop.mapred.Child.main(Child.java:249)
>
> Any ideas?
>
> Thanks.
> /a