You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Rajnish Garg <ra...@gmail.com> on 2015/06/19 18:32:17 UTC
Can not connect to S3
I have installed Zeppelin, on my aws EC2 machine to connect to my spark
cluster.
Spark Version: Standalone: spark-1.2.1-bin-hadoop1.tgz
I am able to connect to spark cluster but getting following error, when
trying to access the file in S3 in my usecase.
Code:
sc.hadoopConfiguration.set("fs.s3n.awsAccessKeyId", "YOUR_KEY_ID")
sc.hadoopConfiguration.set("fs.s3n.awsSecretAccessKey","YOUR_SEC_KEY")
val file = "s3n://<bucket>/<key>"
val data = sc.textFile(file)
data.count
java.lang.NoSuchMethodError:
org.jets3t.service.impl.rest.httpclient.RestS3Service.<init>(Lorg/jets3t/service/security/AWSCredentials;)V
at
org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.initialize(Jets3tNativeFileSystemStore.java:55)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
at org.apache.hadoop.fs.s3native.$Proxy8.initialize(Unknown Source)
at
org.apache.hadoop.fs.s3native.NativeS3FileSystem.initialize(NativeS3FileSystem.java:234)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1446)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1464)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:263)
I have build the Zeppelin by following command:
mvn clean package -Pspark-1.2.1 -Dhadoop.version=1.0.4 -DskipTests
when I trying to build with hadoop profile "-Phadoop-1.0.4", it is giving
warning that it doesn't exist.
I have also tried -Phadoop-1 mentioned in this spark website. but got the
same error. 1.x to 2.1.x hadoop-1
http://stackoverflow.com/questions/30884662/why-zeppelin-notebook-is-not-able-to-connect-to-s3
Re: Can not connect to S3
Posted by Rajnish Garg <ra...@gmail.com>.
Hi,
I have created an issue "https://issues.apache.org/jira/browse/ZEPPELIN-121"
for this issue.
Regards
Rajnish
On Fri, Jun 19, 2015 at 5:05 PM, moon soo Lee <mo...@apache.org> wrote:
> Hi,
>
> Thanks for share the problem.
> I've tried and got the same error.
> Do you mind create a issue for it?
>
> Thanks,
> moon
>
> On Fri, Jun 19, 2015 at 9:32 AM Rajnish Garg <ra...@gmail.com>
> wrote:
>
>> I have installed Zeppelin, on my aws EC2 machine to connect to my spark
>> cluster.
>>
>> Spark Version: Standalone: spark-1.2.1-bin-hadoop1.tgz
>>
>> I am able to connect to spark cluster but getting following error, when
>> trying to access the file in S3 in my usecase.
>>
>> Code:
>>
>> sc.hadoopConfiguration.set("fs.s3n.awsAccessKeyId", "YOUR_KEY_ID")
>> sc.hadoopConfiguration.set("fs.s3n.awsSecretAccessKey","YOUR_SEC_KEY")
>> val file = "s3n://<bucket>/<key>"
>> val data = sc.textFile(file)
>> data.count
>>
>> java.lang.NoSuchMethodError:
>> org.jets3t.service.impl.rest.httpclient.RestS3Service.<init>(Lorg/jets3t/service/security/AWSCredentials;)V
>> at
>> org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.initialize(Jets3tNativeFileSystemStore.java:55)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
>> at org.apache.hadoop.fs.s3native.$Proxy8.initialize(Unknown Source)
>> at
>> org.apache.hadoop.fs.s3native.NativeS3FileSystem.initialize(NativeS3FileSystem.java:234)
>> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1446)
>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1464)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:263)
>>
>> I have build the Zeppelin by following command:
>>
>> mvn clean package -Pspark-1.2.1 -Dhadoop.version=1.0.4 -DskipTests
>> when I trying to build with hadoop profile "-Phadoop-1.0.4", it is giving
>> warning that it doesn't exist.
>>
>> I have also tried -Phadoop-1 mentioned in this spark website. but got the
>> same error. 1.x to 2.1.x hadoop-1
>>
>>
>> http://stackoverflow.com/questions/30884662/why-zeppelin-notebook-is-not-able-to-connect-to-s3
>>
>
Re: Can not connect to S3
Posted by moon soo Lee <mo...@apache.org>.
Hi,
Thanks for share the problem.
I've tried and got the same error.
Do you mind create a issue for it?
Thanks,
moon
On Fri, Jun 19, 2015 at 9:32 AM Rajnish Garg <ra...@gmail.com>
wrote:
> I have installed Zeppelin, on my aws EC2 machine to connect to my spark
> cluster.
>
> Spark Version: Standalone: spark-1.2.1-bin-hadoop1.tgz
>
> I am able to connect to spark cluster but getting following error, when
> trying to access the file in S3 in my usecase.
>
> Code:
>
> sc.hadoopConfiguration.set("fs.s3n.awsAccessKeyId", "YOUR_KEY_ID")
> sc.hadoopConfiguration.set("fs.s3n.awsSecretAccessKey","YOUR_SEC_KEY")
> val file = "s3n://<bucket>/<key>"
> val data = sc.textFile(file)
> data.count
>
> java.lang.NoSuchMethodError:
> org.jets3t.service.impl.rest.httpclient.RestS3Service.<init>(Lorg/jets3t/service/security/AWSCredentials;)V
> at
> org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.initialize(Jets3tNativeFileSystemStore.java:55)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
> at org.apache.hadoop.fs.s3native.$Proxy8.initialize(Unknown Source)
> at
> org.apache.hadoop.fs.s3native.NativeS3FileSystem.initialize(NativeS3FileSystem.java:234)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1446)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1464)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:263)
>
> I have build the Zeppelin by following command:
>
> mvn clean package -Pspark-1.2.1 -Dhadoop.version=1.0.4 -DskipTests
> when I trying to build with hadoop profile "-Phadoop-1.0.4", it is giving
> warning that it doesn't exist.
>
> I have also tried -Phadoop-1 mentioned in this spark website. but got the
> same error. 1.x to 2.1.x hadoop-1
>
>
> http://stackoverflow.com/questions/30884662/why-zeppelin-notebook-is-not-able-to-connect-to-s3
>