You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by cs user <ac...@gmail.com> on 2016/12/16 21:24:41 UTC
Class org.apache.hadoop.fs.s3a.S3A not found
Hello,
Can someone tell me which version of Hadoop I should be able to find this
class in?
Thanks!
Re: Class org.apache.hadoop.fs.s3a.S3A not found
Posted by Ted Yu <yu...@gmail.com>.
MutableCounterLong is in hadoop-common jar.
Can you check the classpath to see if there is another hadoop-common jar
(other than 3.0-alpha1) ?
Cheers
On Fri, Dec 16, 2016 at 1:49 PM, cs user <ac...@gmail.com> wrote:
> Hello,
>
> Thank you both for getting back to me so quickly,
>
> Can you explain how I can get this to work with the latest version? I'm
> trying to submit a spark job with the following ( With the hadoop 3
> release).
>
> Sys.setenv('SPARKR_SUBMIT_ARGS'='"--packages" "com.amazonaws:aws-java-sdk-
> pom:1.10.34,org.apache.hadoop:hadoop-aws:3.0.0-alpha1" "sparkr-shell"',
>
> But then I get:
>
> 16/12/16 21:47:56 ERROR RBackendHandler: createSparkContext on
> org.apache.spark.api.r.RRDD failed
> Error in invokeJava(isStatic = TRUE, className, methodName, ...) :
> java.lang.IllegalAccessError: tried to access method
> org.apache.hadoop.metrics2.lib.MutableCounterLong.<init>(
> Lorg/apache/hadoop/metrics2/MetricsInfo;J)V from class
> org.apache.hadoop.fs.s3a.S3AInstrumentation
>
>
> How can I set my environment to pull down the hadoop 2.8 release?
>
> Thank you again for your help.
>
>
>
>
>
>
Re: Class org.apache.hadoop.fs.s3a.S3A not found
Posted by cs user <ac...@gmail.com>.
Hello,
Thank you both for getting back to me so quickly,
Can you explain how I can get this to work with the latest version? I'm
trying to submit a spark job with the following ( With the hadoop 3
release).
Sys.setenv('SPARKR_SUBMIT_ARGS'='"--packages"
"com.amazonaws:aws-java-sdk-pom:1.10.34,org.apache.hadoop:hadoop-aws:3.0.0-alpha1"
"sparkr-shell"',
But then I get:
16/12/16 21:47:56 ERROR RBackendHandler: createSparkContext on
org.apache.spark.api.r.RRDD failed
Error in invokeJava(isStatic = TRUE, className, methodName, ...) :
java.lang.IllegalAccessError: tried to access method
org.apache.hadoop.metrics2.lib.MutableCounterLong.<init>(Lorg/apache/hadoop/metrics2/MetricsInfo;J)V
from class org.apache.hadoop.fs.s3a.S3AInstrumentation
How can I set my environment to pull down the hadoop 2.8 release?
Thank you again for your help.
On Fri, Dec 16, 2016 at 9:33 PM, Mingliang Liu <li...@gmail.com> wrote:
> You may need the hadoop-aws*-version*.jar (and aws-java-sdk.jar) in your
> classpath.
>
> For new project, have a look at the latest S3A progress:
> https://github.com/apache/hadoop/blob/release-3.
> 0.0-alpha1-RC0/hadoop-tools/hadoop-aws/src/site/markdown/
> tools/hadoop-aws/index.md
>
> Good luck,
>
> L
>
> On Dec 16, 2016, at 1:24 PM, cs user <ac...@gmail.com> wrote:
>
> Hello,
>
> Can someone tell me which version of Hadoop I should be able to find this
> class in?
>
> Thanks!
>
>
>
Re: Class org.apache.hadoop.fs.s3a.S3A not found
Posted by Mingliang Liu <li...@gmail.com>.
You may need the hadoop-aws-version.jar (and aws-java-sdk.jar) in your classpath.
For new project, have a look at the latest S3A progress: https://github.com/apache/hadoop/blob/release-3.0.0-alpha1-RC0/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md <https://github.com/apache/hadoop/blob/release-3.0.0-alpha1-RC0/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md>
Good luck,
L
> On Dec 16, 2016, at 1:24 PM, cs user <ac...@gmail.com> wrote:
>
> Hello,
>
> Can someone tell me which version of Hadoop I should be able to find this class in?
>
> Thanks!