You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by vinay patil <vi...@gmail.com> on 2016/08/05 09:32:53 UTC
Facing Issues while migrating to S3A
Hi All,
I am facing issues when I changed from S3n to S3A, I am using
hadoop-aws-2.7.2 and aws-java-sdk-1.7.4 as dependencies
Getting this error : java.lang.UnsatisfiedLinkError:
org.apache.hadoop.io.nativeio.NativeIO$Windows.access
Tried to change the fs.s3.buffer.dir path but getting same error.
According to the documentation when I try to provide values in these
properties, it does not work and gives the credential exception:
<property>
<name>fs.s3.awsAccessKeyId</name>
<value></value>
</property>
<property>
<name>fs.s3.awsSecretAccessKey</name>
<value></value>
</property>
When I change these properties to fs.s3a.access.key and fs.s3a.secret.key it
accepts the credentials but gives the above exception.
Can you please guide me the correct way of migrating from S3n to S3A
P.S I have followed all the steps in the documentation.
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Facing-Issues-while-migrating-to-S3A-tp8337.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Facing Issues while migrating to S3A
Posted by "yinhua.dai" <yi...@outlook.com>.
Hi,
I am still have the same problem, googled many ways but still failed.
I have downloaded and added hadoop.dll and winutils.exe to class path.
To verify that is working, I called "System.loadLibrary("haddop")" at the
beginning of my java program and it succeed.
BTW: I run my program in windows 7 64bit.
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Facing Issues while migrating to S3A
Posted by Stefan Richter <s....@data-artisans.com>.
Hi,
yes, it is only an issue on Windows because the system is using native functions and it seems your classpath lacks the Windows specific binary. Those functions are called from Java through JNI and their implementation for Windows is in some dll file that you need to put on your classpath. I think that file is $HADOOP_HOME\bin\hadoop.dll, where $HADOOP_HOME is the location of your Hadoop installation on Windows. Locate this file on your machine and include it to your classpath. This should help.
Best,
Stefan
> Am 06.08.2016 um 08:34 schrieb vinay patil <vi...@gmail.com>:
>
> Hi Stefan,
>
> The migration from S3n to S3A is working fine when I try it on cluster (linux box), the issue is only with the local windows machine.
>
> I am not clear on this ,Can you please elaborate:
> "The code of the native method should be contained in something like a HADOOP_HOME\bin\hadoop.dll"
>
> I want to test it locally so that I don't have to run every-time on cluster
>
> Regards,
> Vinay Patil
>
> On Fri, Aug 5, 2016 at 3:18 PM, Stefan Richter [via Apache Flink User Mailing List archive.] <[hidden email] <x-msg://2/user/SendEmail.jtp?type=node&node=8349&i=0>> wrote:
> Hi,
>
> from the exception, it seems like some native library is missing in your classpath. The code of the native method should be contained in something like a HADOOP_HOME\bin\hadoop.dll.
>
> Best,
> Stefan
>
> > Am 05.08.2016 um 11:32 schrieb vinay patil <[hidden email] <http://user/SendEmail.jtp?type=node&node=8338&i=0>>:
> >
> > Hi All,
> >
> > I am facing issues when I changed from S3n to S3A, I am using
> > hadoop-aws-2.7.2 and aws-java-sdk-1.7.4 as dependencies
> >
> > Getting this error : java.lang.UnsatisfiedLinkError:
> > org.apache.hadoop.io.nativeio.NativeIO$Windows.access
> > Tried to change the fs.s3.buffer.dir path but getting same error.
> >
> > According to the documentation when I try to provide values in these
> > properties, it does not work and gives the credential exception:
> >
> > <property>
> > <name>fs.s3.awsAccessKeyId</name>
> > <value></value>
> > </property>
> >
> > <property>
> > <name>fs.s3.awsSecretAccessKey</name>
> > <value></value>
> > </property>
> >
> > When I change these properties to fs.s3a.access.key and fs.s3a.secret.key it
> > accepts the credentials but gives the above exception.
> >
> > Can you please guide me the correct way of migrating from S3n to S3A
> >
> > P.S I have followed all the steps in the documentation.
> >
> >
> >
> > --
> > View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Facing-Issues-while-migrating-to-S3A-tp8337.html <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Facing-Issues-while-migrating-to-S3A-tp8337.html>
> > Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
>
>
>
> If you reply to this email, your message will be added to the discussion below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Facing-Issues-while-migrating-to-S3A-tp8337p8338.html <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Facing-Issues-while-migrating-to-S3A-tp8337p8338.html>
> To start a new topic under Apache Flink User Mailing List archive., email [hidden email] <x-msg://2/user/SendEmail.jtp?type=node&node=8349&i=1>
> To unsubscribe from Apache Flink User Mailing List archive., click here <applewebdata://A758B8E6-7D53-4B01-9B44-429E4FBD8F27>.
> NAML <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
> View this message in context: Re: Facing Issues while migrating to S3A <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Facing-Issues-while-migrating-to-S3A-tp8337p8349.html>
> Sent from the Apache Flink User Mailing List archive. mailing list archive <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at Nabble.com <http://nabble.com/>.
Re: Facing Issues while migrating to S3A
Posted by vinay patil <vi...@gmail.com>.
Hi Stefan,
The migration from S3n to S3A is working fine when I try it on cluster
(linux box), the issue is only with the local windows machine.
I am not clear on this ,Can you please elaborate:
*"The code of the native method should be contained in something like a
HADOOP_HOME\bin\hadoop.dll"*
I want to test it locally so that I don't have to run every-time on cluster
Regards,
Vinay Patil
On Fri, Aug 5, 2016 at 3:18 PM, Stefan Richter [via Apache Flink User
Mailing List archive.] <ml...@n4.nabble.com> wrote:
> Hi,
>
> from the exception, it seems like some native library is missing in your
> classpath. The code of the native method should be contained in something
> like a HADOOP_HOME\bin\hadoop.dll.
>
> Best,
> Stefan
>
> > Am 05.08.2016 um 11:32 schrieb vinay patil <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=8338&i=0>>:
> >
> > Hi All,
> >
> > I am facing issues when I changed from S3n to S3A, I am using
> > hadoop-aws-2.7.2 and aws-java-sdk-1.7.4 as dependencies
> >
> > Getting this error : java.lang.UnsatisfiedLinkError:
> > org.apache.hadoop.io.nativeio.NativeIO$Windows.access
> > Tried to change the fs.s3.buffer.dir path but getting same error.
> >
> > According to the documentation when I try to provide values in these
> > properties, it does not work and gives the credential exception:
> >
> > <property>
> > <name>fs.s3.awsAccessKeyId</name>
> > <value></value>
> > </property>
> >
> > <property>
> > <name>fs.s3.awsSecretAccessKey</name>
> > <value></value>
> > </property>
> >
> > When I change these properties to fs.s3a.access.key and
> fs.s3a.secret.key it
> > accepts the credentials but gives the above exception.
> >
> > Can you please guide me the correct way of migrating from S3n to S3A
> >
> > P.S I have followed all the steps in the documentation.
> >
> >
> >
> > --
> > View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/Facing-Issues-
> while-migrating-to-S3A-tp8337.html
> > Sent from the Apache Flink User Mailing List archive. mailing list
> archive at Nabble.com.
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/Facing-Issues-while-migrating-to-S3A-tp8337p8338.html
> To start a new topic under Apache Flink User Mailing List archive., email
> ml-node+s2336050n1h83@n4.nabble.com
> To unsubscribe from Apache Flink User Mailing List archive., click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Facing-Issues-while-migrating-to-S3A-tp8337p8349.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Facing Issues while migrating to S3A
Posted by Stefan Richter <s....@data-artisans.com>.
Hi,
from the exception, it seems like some native library is missing in your classpath. The code of the native method should be contained in something like a HADOOP_HOME\bin\hadoop.dll.
Best,
Stefan
> Am 05.08.2016 um 11:32 schrieb vinay patil <vi...@gmail.com>:
>
> Hi All,
>
> I am facing issues when I changed from S3n to S3A, I am using
> hadoop-aws-2.7.2 and aws-java-sdk-1.7.4 as dependencies
>
> Getting this error : java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.io.nativeio.NativeIO$Windows.access
> Tried to change the fs.s3.buffer.dir path but getting same error.
>
> According to the documentation when I try to provide values in these
> properties, it does not work and gives the credential exception:
>
> <property>
> <name>fs.s3.awsAccessKeyId</name>
> <value></value>
> </property>
>
> <property>
> <name>fs.s3.awsSecretAccessKey</name>
> <value></value>
> </property>
>
> When I change these properties to fs.s3a.access.key and fs.s3a.secret.key it
> accepts the credentials but gives the above exception.
>
> Can you please guide me the correct way of migrating from S3n to S3A
>
> P.S I have followed all the steps in the documentation.
>
>
>
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Facing-Issues-while-migrating-to-S3A-tp8337.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.