Jar Dependencies to use S3

[Schmirr Wurst <sc...@gmail.com>]

Hi,

I've just try to access S3 from zeppelin, but it seems I'm missing
some libraries ?

Can somebody tell me what libraries I have to install were ? I'm not
familiar with java...

Best.
SM

Re: Jar Dependencies to use S3

[Schmirr Wurst <sc...@gmail.com>]

Hi,
I could get some steps forward, I recompiled Zeppelin for Hadoop 2.6,
seems that S3 has only been fixed there, but now I have the next
Problem :

java.io.IOException: No FileSystem for scheme: s3n at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)

I've found following workaround, but I'm not able to transpose the
steps for Zeppelin, can anybody help ? :

1. fs.s3n.impl

> We added a property to the core-site.xml file:

You don't need to explicitly set this. It has never been done so in
previous versions.

Take a look at FileSystem#loadFileSystem, which is called from
FileSystem#getFileSystemClass.
Subclasses of FileSystem are loaded automatically if they are
available on a classloader you care.

So you just need to make sure hadoop-aws.jar is on a classpath.

For file system shell, this is done in hadoop-env.sh,
while for a MR job, in mapreduce.application.classpath,
or for YARN, in yarn.application.classpath.

2. mapreduce.application.classpath

> And updated the classpath for mapreduce applications:

Note that it points to a distributed cache on the default HDP 2.2 distribution.

    <property>
        <name>mapreduce.application.classpath</name>
        <value>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure</value>
    </property>
* $PWD/mr-framework/hadoop/share/hadoop/tools/lib/* contains
hadoop-aws.jar(S3NFileSystem)

While on a vanilla hadoop, it looks like standard paths as yours.

    <property>
        <name>mapreduce.application.classpath</name>
        <value>/hadoop-2.6.0/etc/hadoop:/hadoop-2.6.0/share/hadoop/common/lib/*:/hadoop-2.6.0/share/hadoop/common/*:/hadoop-2.6.0/share/hadoop/hdfs:/hadoop-2.6.0/share/hadoop/hdfs/lib/*:/hadoop-2.6.0/share/hadoop/hdfs/*:/hadoop-2.6.0/share/hadoop/yarn/lib/*:/hadoop-2.6.0/share/hadoop/yarn/*:/hadoop-2.6.0/share/hadoop/mapreduce/lib/*:/hadoop-2.6.0/share/hadoop/mapreduce/*:/hadoop-2.6.0/contrib/capacity-scheduler/*.jar:/hadoop-2.6.0/share/hadoop/tools/lib/*</value>
    </property>

Thanks,
Sato

2015-07-16 11:03 GMT+02:00 Schmirr Wurst <sc...@gmail.com>:
> Hi,
>
> I've just try to access S3 from zeppelin, but it seems I'm missing
> some libraries ?
>
> Can somebody tell me what libraries I have to install were ? I'm not
> familiar with java...
>
> Best.
> SM