You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by "Phillips, Caleb" <Ca...@nrel.gov> on 2015/12/22 21:39:32 UTC

fs.s3a.endpoint not working

Hi All,

New to this list. Looking for a bit of help:

I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object store.

This issue was discussed, but left unresolved, in this thread:

https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E

And here, on Cloudera's forums (the second post is mine):

https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoint-ignored-in-hdfs-site-xml/m-p/33694#M1180

I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.

However, when I point the fs.s3a.endpoint configuration directive at my non-AWS S3-Compatable object storage, it appears to still point at (and authenticate against) AWS.

I've checked and double-checked my credentials and configuration using both Python's boto library and the s3cmd tool, both of which connect to this non-AWS data store just fine.

Any help would be much appreciated. Thanks!

--
Caleb Phillips, Ph.D.
Data Scientist | Computational Science Center

National Renewable Energy Laboratory (NREL)
15013 Denver West Parkway | Golden, CO 80401
303-275-4297 | caleb.phillips@nrel.gov

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org

Re: fs.s3a.endpoint not working

Posted by Billy Watson <wi...@gmail.com>.

Thanks for following up!

William Watson
Lead Software Engineer

On Tue, Feb 16, 2016 at 5:08 PM, Phillips, Caleb <Ca...@nrel.gov>
wrote:

> Hi All,
>
> Just wanted to follow on that we got this working with the help of the
> object storage vendor. After running in circles for a bit, the issue seems
> to have been as simple as using the correct FQDN in the endpoint fields and
> disabling SSL. We used the jet3st properties, but it turns out those aren’t
> actually needed with recent Hadoop versions (?).
>
> For anyone who might be having similar issues, here are the relevant
> configuration in core-site.xml for S3A and S3N with Hadoop 2.7.1:
>
> <configuration>
>
> <!-- S3N Connector to Obsidian -->
> <property>
>  <name>fs.s3n.awsAccessKeyId</name>
>  <description>AWS access key ID</description>
>  <value>yourusername</value>
> </property>
>
> <property>
>  <name>fs.s3n.awsSecretAccessKey</name>
>  <description>AWS secret key</description>
>  <value>sweetpassword</value>
> </property>
>
> <property>
>  <name>fs.s3n.endpoint</name>
>  <value>youre.fqdn.here</value>
> </property>
>
> <property>
>  <name>fs.s3n.ssl.enabled</name>
>  <value>false</value>
> </property>
>
> <!-- S3A Connector to Obsidian -->
>
> <property>
>  <name>fs.s3a.access.key</name>
>  <description>AWS access key ID. Omit for Role-based
> authentication.</description>
>  <value>yourusername</value>
> </property>
>
> <property>
>  <name>fs.s3a.secret.key</name>
>  <description>AWS secret key. Omit for Role-based
> authentication.</description>
>  <value>sweetpassword</value>
> </property>
>
> <property>
>  <name>fs.s3a.connection.ssl.enabled</name>
>  <value>false</value>
>  <description>Enables or disables SSL connections to S3.</description>
> </property>
>
> <property>
>  <name>fs.s3a.endpoint</name>
>  <description>AWS S3 endpoint to connect to. An up-to-date list is
>     provided in the AWS Documentation: regions and endpoints. Without this
>     property, the standard region (s3.amazonaws.com) is assumed.
>  </description>
>  <value>your.fqdn.here</value>
> </property>
>
> </configuration>
>
> Also, as mentioned previously in the thread, it’s necessary to add some
> things to your HADOOP_CLASSPATH:
>
> export
> HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/path/to/hadoop-2.7.1/share/hadoop/tools/lib/*
>
> You can test by:
>
> s3cmd mb s3://some-bucket               # <- note that you have to do this
> with s3cmd, not hadoop, at least with our object store
> hadoop fs -ls s3n://some-bucket/
> hadoop fs -ls s3a://some-bucket/
> hadoop distcp /your/favorite/hdfs/data s3a://some-bucket/
>
> HTH,
>
> --
> Caleb Phillips, Ph.D.
> Data Scientist | Computational Science Center
>
> National Renewable Energy Laboratory (NREL)
> 15013 Denver West Parkway | Golden, CO 80401
> 303-275-4297 | caleb.phillips@nrel.gov
>
> From: Billy Watson <williamrwatson@gmail.com<mailto:
> williamrwatson@gmail.com>>
> Date: Tuesday, January 19, 2016 at 8:41 AM
> To: Alexander Pivovarov <apivovarov@gmail.com<mailto:apivovarov@gmail.com
> >>
> Cc: Caleb Phillips <ca...@nrel.gov>>,
> "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>
> Subject: Re: fs.s3a.endpoint not working
>
> Stupid question, I assume you're using a URL that starts with s3a and that
> your custom endpoint supports s3a?
>
> William Watson
> Lead Software Engineer
>
> On Thu, Jan 14, 2016 at 1:57 PM, Alexander Pivovarov <apivovarov@gmail.com
> <ma...@gmail.com>> wrote:
>
> http://www.jets3t.org/toolkit/configuration.html
>
> On Jan 14, 2016 10:56 AM, "Alexander Pivovarov" <apivovarov@gmail.com
> <ma...@gmail.com>> wrote:
>
> Add jets3t.properties file with s3service.s3-endpoint=<endpoint> to
> /etc/hadoop/conf folder
>
> The folder with the file should be in HADOOP_CLASSPATH
>
> JetS3t library which is used by hadoop is looking for this file.
>
> On Dec 22, 2015 12:39 PM, "Phillips, Caleb" <Caleb.Phillips@nrel.gov
> <ma...@nrel.gov>> wrote:
> Hi All,
>
> New to this list. Looking for a bit of help:
>
> I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
> store.
>
> This issue was discussed, but left unresolved, in this thread:
>
>
> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E
>
> And here, on Cloudera's forums (the second post is mine):
>
>
> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoint-ignored-in-hdfs-site-xml/m-p/33694#M1180
>
> I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using Hadoop,
> I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>
> However, when I point the fs.s3a.endpoint configuration directive at my
> non-AWS S3-Compatable object storage, it appears to still point at (and
> authenticate against) AWS.
>
> I've checked and double-checked my credentials and configuration using
> both Python's boto library and the s3cmd tool, both of which connect to
> this non-AWS data store just fine.
>
> Any help would be much appreciated. Thanks!
>
> --
> Caleb Phillips, Ph.D.
> Data Scientist | Computational Science Center
>
> National Renewable Energy Laboratory (NREL)
> 15013 Denver West Parkway | Golden, CO 80401
> 303-275-4297<tel:303-275-4297> | caleb.phillips@nrel.gov<mailto:
> caleb.phillips@nrel.gov>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org<mailto:
> user-unsubscribe@hadoop.apache.org>
> For additional commands, e-mail: user-help@hadoop.apache.org<mailto:
> user-help@hadoop.apache.org>
>
>
>

Re: fs.s3a.endpoint not working

Posted by Billy Watson <wi...@gmail.com>.

Thanks for following up!

William Watson
Lead Software Engineer

On Tue, Feb 16, 2016 at 5:08 PM, Phillips, Caleb <Ca...@nrel.gov>
wrote:

> Hi All,
>
> Just wanted to follow on that we got this working with the help of the
> object storage vendor. After running in circles for a bit, the issue seems
> to have been as simple as using the correct FQDN in the endpoint fields and
> disabling SSL. We used the jet3st properties, but it turns out those aren’t
> actually needed with recent Hadoop versions (?).
>
> For anyone who might be having similar issues, here are the relevant
> configuration in core-site.xml for S3A and S3N with Hadoop 2.7.1:
>
> <configuration>
>
> <!-- S3N Connector to Obsidian -->
> <property>
>  <name>fs.s3n.awsAccessKeyId</name>
>  <description>AWS access key ID</description>
>  <value>yourusername</value>
> </property>
>
> <property>
>  <name>fs.s3n.awsSecretAccessKey</name>
>  <description>AWS secret key</description>
>  <value>sweetpassword</value>
> </property>
>
> <property>
>  <name>fs.s3n.endpoint</name>
>  <value>youre.fqdn.here</value>
> </property>
>
> <property>
>  <name>fs.s3n.ssl.enabled</name>
>  <value>false</value>
> </property>
>
> <!-- S3A Connector to Obsidian -->
>
> <property>
>  <name>fs.s3a.access.key</name>
>  <description>AWS access key ID. Omit for Role-based
> authentication.</description>
>  <value>yourusername</value>
> </property>
>
> <property>
>  <name>fs.s3a.secret.key</name>
>  <description>AWS secret key. Omit for Role-based
> authentication.</description>
>  <value>sweetpassword</value>
> </property>
>
> <property>
>  <name>fs.s3a.connection.ssl.enabled</name>
>  <value>false</value>
>  <description>Enables or disables SSL connections to S3.</description>
> </property>
>
> <property>
>  <name>fs.s3a.endpoint</name>
>  <description>AWS S3 endpoint to connect to. An up-to-date list is
>     provided in the AWS Documentation: regions and endpoints. Without this
>     property, the standard region (s3.amazonaws.com) is assumed.
>  </description>
>  <value>your.fqdn.here</value>
> </property>
>
> </configuration>
>
> Also, as mentioned previously in the thread, it’s necessary to add some
> things to your HADOOP_CLASSPATH:
>
> export
> HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/path/to/hadoop-2.7.1/share/hadoop/tools/lib/*
>
> You can test by:
>
> s3cmd mb s3://some-bucket               # <- note that you have to do this
> with s3cmd, not hadoop, at least with our object store
> hadoop fs -ls s3n://some-bucket/
> hadoop fs -ls s3a://some-bucket/
> hadoop distcp /your/favorite/hdfs/data s3a://some-bucket/
>
> HTH,
>
> --
> Caleb Phillips, Ph.D.
> Data Scientist | Computational Science Center
>
> National Renewable Energy Laboratory (NREL)
> 15013 Denver West Parkway | Golden, CO 80401
> 303-275-4297 | caleb.phillips@nrel.gov
>
> From: Billy Watson <williamrwatson@gmail.com<mailto:
> williamrwatson@gmail.com>>
> Date: Tuesday, January 19, 2016 at 8:41 AM
> To: Alexander Pivovarov <apivovarov@gmail.com<mailto:apivovarov@gmail.com
> >>
> Cc: Caleb Phillips <ca...@nrel.gov>>,
> "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>
> Subject: Re: fs.s3a.endpoint not working
>
> Stupid question, I assume you're using a URL that starts with s3a and that
> your custom endpoint supports s3a?
>
> William Watson
> Lead Software Engineer
>
> On Thu, Jan 14, 2016 at 1:57 PM, Alexander Pivovarov <apivovarov@gmail.com
> <ma...@gmail.com>> wrote:
>
> http://www.jets3t.org/toolkit/configuration.html
>
> On Jan 14, 2016 10:56 AM, "Alexander Pivovarov" <apivovarov@gmail.com
> <ma...@gmail.com>> wrote:
>
> Add jets3t.properties file with s3service.s3-endpoint=<endpoint> to
> /etc/hadoop/conf folder
>
> The folder with the file should be in HADOOP_CLASSPATH
>
> JetS3t library which is used by hadoop is looking for this file.
>
> On Dec 22, 2015 12:39 PM, "Phillips, Caleb" <Caleb.Phillips@nrel.gov
> <ma...@nrel.gov>> wrote:
> Hi All,
>
> New to this list. Looking for a bit of help:
>
> I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
> store.
>
> This issue was discussed, but left unresolved, in this thread:
>
>
> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E
>
> And here, on Cloudera's forums (the second post is mine):
>
>
> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoint-ignored-in-hdfs-site-xml/m-p/33694#M1180
>
> I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using Hadoop,
> I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>
> However, when I point the fs.s3a.endpoint configuration directive at my
> non-AWS S3-Compatable object storage, it appears to still point at (and
> authenticate against) AWS.
>
> I've checked and double-checked my credentials and configuration using
> both Python's boto library and the s3cmd tool, both of which connect to
> this non-AWS data store just fine.
>
> Any help would be much appreciated. Thanks!
>
> --
> Caleb Phillips, Ph.D.
> Data Scientist | Computational Science Center
>
> National Renewable Energy Laboratory (NREL)
> 15013 Denver West Parkway | Golden, CO 80401
> 303-275-4297<tel:303-275-4297> | caleb.phillips@nrel.gov<mailto:
> caleb.phillips@nrel.gov>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org<mailto:
> user-unsubscribe@hadoop.apache.org>
> For additional commands, e-mail: user-help@hadoop.apache.org<mailto:
> user-help@hadoop.apache.org>
>
>
>

Re: fs.s3a.endpoint not working

Posted by Billy Watson <wi...@gmail.com>.

Thanks for following up!

William Watson
Lead Software Engineer

On Tue, Feb 16, 2016 at 5:08 PM, Phillips, Caleb <Ca...@nrel.gov>
wrote:

> Hi All,
>
> Just wanted to follow on that we got this working with the help of the
> object storage vendor. After running in circles for a bit, the issue seems
> to have been as simple as using the correct FQDN in the endpoint fields and
> disabling SSL. We used the jet3st properties, but it turns out those aren’t
> actually needed with recent Hadoop versions (?).
>
> For anyone who might be having similar issues, here are the relevant
> configuration in core-site.xml for S3A and S3N with Hadoop 2.7.1:
>
> <configuration>
>
> <!-- S3N Connector to Obsidian -->
> <property>
>  <name>fs.s3n.awsAccessKeyId</name>
>  <description>AWS access key ID</description>
>  <value>yourusername</value>
> </property>
>
> <property>
>  <name>fs.s3n.awsSecretAccessKey</name>
>  <description>AWS secret key</description>
>  <value>sweetpassword</value>
> </property>
>
> <property>
>  <name>fs.s3n.endpoint</name>
>  <value>youre.fqdn.here</value>
> </property>
>
> <property>
>  <name>fs.s3n.ssl.enabled</name>
>  <value>false</value>
> </property>
>
> <!-- S3A Connector to Obsidian -->
>
> <property>
>  <name>fs.s3a.access.key</name>
>  <description>AWS access key ID. Omit for Role-based
> authentication.</description>
>  <value>yourusername</value>
> </property>
>
> <property>
>  <name>fs.s3a.secret.key</name>
>  <description>AWS secret key. Omit for Role-based
> authentication.</description>
>  <value>sweetpassword</value>
> </property>
>
> <property>
>  <name>fs.s3a.connection.ssl.enabled</name>
>  <value>false</value>
>  <description>Enables or disables SSL connections to S3.</description>
> </property>
>
> <property>
>  <name>fs.s3a.endpoint</name>
>  <description>AWS S3 endpoint to connect to. An up-to-date list is
>     provided in the AWS Documentation: regions and endpoints. Without this
>     property, the standard region (s3.amazonaws.com) is assumed.
>  </description>
>  <value>your.fqdn.here</value>
> </property>
>
> </configuration>
>
> Also, as mentioned previously in the thread, it’s necessary to add some
> things to your HADOOP_CLASSPATH:
>
> export
> HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/path/to/hadoop-2.7.1/share/hadoop/tools/lib/*
>
> You can test by:
>
> s3cmd mb s3://some-bucket               # <- note that you have to do this
> with s3cmd, not hadoop, at least with our object store
> hadoop fs -ls s3n://some-bucket/
> hadoop fs -ls s3a://some-bucket/
> hadoop distcp /your/favorite/hdfs/data s3a://some-bucket/
>
> HTH,
>
> --
> Caleb Phillips, Ph.D.
> Data Scientist | Computational Science Center
>
> National Renewable Energy Laboratory (NREL)
> 15013 Denver West Parkway | Golden, CO 80401
> 303-275-4297 | caleb.phillips@nrel.gov
>
> From: Billy Watson <williamrwatson@gmail.com<mailto:
> williamrwatson@gmail.com>>
> Date: Tuesday, January 19, 2016 at 8:41 AM
> To: Alexander Pivovarov <apivovarov@gmail.com<mailto:apivovarov@gmail.com
> >>
> Cc: Caleb Phillips <ca...@nrel.gov>>,
> "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>
> Subject: Re: fs.s3a.endpoint not working
>
> Stupid question, I assume you're using a URL that starts with s3a and that
> your custom endpoint supports s3a?
>
> William Watson
> Lead Software Engineer
>
> On Thu, Jan 14, 2016 at 1:57 PM, Alexander Pivovarov <apivovarov@gmail.com
> <ma...@gmail.com>> wrote:
>
> http://www.jets3t.org/toolkit/configuration.html
>
> On Jan 14, 2016 10:56 AM, "Alexander Pivovarov" <apivovarov@gmail.com
> <ma...@gmail.com>> wrote:
>
> Add jets3t.properties file with s3service.s3-endpoint=<endpoint> to
> /etc/hadoop/conf folder
>
> The folder with the file should be in HADOOP_CLASSPATH
>
> JetS3t library which is used by hadoop is looking for this file.
>
> On Dec 22, 2015 12:39 PM, "Phillips, Caleb" <Caleb.Phillips@nrel.gov
> <ma...@nrel.gov>> wrote:
> Hi All,
>
> New to this list. Looking for a bit of help:
>
> I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
> store.
>
> This issue was discussed, but left unresolved, in this thread:
>
>
> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E
>
> And here, on Cloudera's forums (the second post is mine):
>
>
> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoint-ignored-in-hdfs-site-xml/m-p/33694#M1180
>
> I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using Hadoop,
> I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>
> However, when I point the fs.s3a.endpoint configuration directive at my
> non-AWS S3-Compatable object storage, it appears to still point at (and
> authenticate against) AWS.
>
> I've checked and double-checked my credentials and configuration using
> both Python's boto library and the s3cmd tool, both of which connect to
> this non-AWS data store just fine.
>
> Any help would be much appreciated. Thanks!
>
> --
> Caleb Phillips, Ph.D.
> Data Scientist | Computational Science Center
>
> National Renewable Energy Laboratory (NREL)
> 15013 Denver West Parkway | Golden, CO 80401
> 303-275-4297<tel:303-275-4297> | caleb.phillips@nrel.gov<mailto:
> caleb.phillips@nrel.gov>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org<mailto:
> user-unsubscribe@hadoop.apache.org>
> For additional commands, e-mail: user-help@hadoop.apache.org<mailto:
> user-help@hadoop.apache.org>
>
>
>

Re: fs.s3a.endpoint not working

Posted by Billy Watson <wi...@gmail.com>.

Thanks for following up!

William Watson
Lead Software Engineer

On Tue, Feb 16, 2016 at 5:08 PM, Phillips, Caleb <Ca...@nrel.gov>
wrote:

> Hi All,
>
> Just wanted to follow on that we got this working with the help of the
> object storage vendor. After running in circles for a bit, the issue seems
> to have been as simple as using the correct FQDN in the endpoint fields and
> disabling SSL. We used the jet3st properties, but it turns out those aren’t
> actually needed with recent Hadoop versions (?).
>
> For anyone who might be having similar issues, here are the relevant
> configuration in core-site.xml for S3A and S3N with Hadoop 2.7.1:
>
> <configuration>
>
> <!-- S3N Connector to Obsidian -->
> <property>
>  <name>fs.s3n.awsAccessKeyId</name>
>  <description>AWS access key ID</description>
>  <value>yourusername</value>
> </property>
>
> <property>
>  <name>fs.s3n.awsSecretAccessKey</name>
>  <description>AWS secret key</description>
>  <value>sweetpassword</value>
> </property>
>
> <property>
>  <name>fs.s3n.endpoint</name>
>  <value>youre.fqdn.here</value>
> </property>
>
> <property>
>  <name>fs.s3n.ssl.enabled</name>
>  <value>false</value>
> </property>
>
> <!-- S3A Connector to Obsidian -->
>
> <property>
>  <name>fs.s3a.access.key</name>
>  <description>AWS access key ID. Omit for Role-based
> authentication.</description>
>  <value>yourusername</value>
> </property>
>
> <property>
>  <name>fs.s3a.secret.key</name>
>  <description>AWS secret key. Omit for Role-based
> authentication.</description>
>  <value>sweetpassword</value>
> </property>
>
> <property>
>  <name>fs.s3a.connection.ssl.enabled</name>
>  <value>false</value>
>  <description>Enables or disables SSL connections to S3.</description>
> </property>
>
> <property>
>  <name>fs.s3a.endpoint</name>
>  <description>AWS S3 endpoint to connect to. An up-to-date list is
>     provided in the AWS Documentation: regions and endpoints. Without this
>     property, the standard region (s3.amazonaws.com) is assumed.
>  </description>
>  <value>your.fqdn.here</value>
> </property>
>
> </configuration>
>
> Also, as mentioned previously in the thread, it’s necessary to add some
> things to your HADOOP_CLASSPATH:
>
> export
> HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/path/to/hadoop-2.7.1/share/hadoop/tools/lib/*
>
> You can test by:
>
> s3cmd mb s3://some-bucket               # <- note that you have to do this
> with s3cmd, not hadoop, at least with our object store
> hadoop fs -ls s3n://some-bucket/
> hadoop fs -ls s3a://some-bucket/
> hadoop distcp /your/favorite/hdfs/data s3a://some-bucket/
>
> HTH,
>
> --
> Caleb Phillips, Ph.D.
> Data Scientist | Computational Science Center
>
> National Renewable Energy Laboratory (NREL)
> 15013 Denver West Parkway | Golden, CO 80401
> 303-275-4297 | caleb.phillips@nrel.gov
>
> From: Billy Watson <williamrwatson@gmail.com<mailto:
> williamrwatson@gmail.com>>
> Date: Tuesday, January 19, 2016 at 8:41 AM
> To: Alexander Pivovarov <apivovarov@gmail.com<mailto:apivovarov@gmail.com
> >>
> Cc: Caleb Phillips <ca...@nrel.gov>>,
> "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>
> Subject: Re: fs.s3a.endpoint not working
>
> Stupid question, I assume you're using a URL that starts with s3a and that
> your custom endpoint supports s3a?
>
> William Watson
> Lead Software Engineer
>
> On Thu, Jan 14, 2016 at 1:57 PM, Alexander Pivovarov <apivovarov@gmail.com
> <ma...@gmail.com>> wrote:
>
> http://www.jets3t.org/toolkit/configuration.html
>
> On Jan 14, 2016 10:56 AM, "Alexander Pivovarov" <apivovarov@gmail.com
> <ma...@gmail.com>> wrote:
>
> Add jets3t.properties file with s3service.s3-endpoint=<endpoint> to
> /etc/hadoop/conf folder
>
> The folder with the file should be in HADOOP_CLASSPATH
>
> JetS3t library which is used by hadoop is looking for this file.
>
> On Dec 22, 2015 12:39 PM, "Phillips, Caleb" <Caleb.Phillips@nrel.gov
> <ma...@nrel.gov>> wrote:
> Hi All,
>
> New to this list. Looking for a bit of help:
>
> I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
> store.
>
> This issue was discussed, but left unresolved, in this thread:
>
>
> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E
>
> And here, on Cloudera's forums (the second post is mine):
>
>
> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoint-ignored-in-hdfs-site-xml/m-p/33694#M1180
>
> I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using Hadoop,
> I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>
> However, when I point the fs.s3a.endpoint configuration directive at my
> non-AWS S3-Compatable object storage, it appears to still point at (and
> authenticate against) AWS.
>
> I've checked and double-checked my credentials and configuration using
> both Python's boto library and the s3cmd tool, both of which connect to
> this non-AWS data store just fine.
>
> Any help would be much appreciated. Thanks!
>
> --
> Caleb Phillips, Ph.D.
> Data Scientist | Computational Science Center
>
> National Renewable Energy Laboratory (NREL)
> 15013 Denver West Parkway | Golden, CO 80401
> 303-275-4297<tel:303-275-4297> | caleb.phillips@nrel.gov<mailto:
> caleb.phillips@nrel.gov>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org<mailto:
> user-unsubscribe@hadoop.apache.org>
> For additional commands, e-mail: user-help@hadoop.apache.org<mailto:
> user-help@hadoop.apache.org>
>
>
>

Re: fs.s3a.endpoint not working

Posted by "Phillips, Caleb" <Ca...@nrel.gov>.

Hi All,

Just wanted to follow on that we got this working with the help of the object storage vendor. After running in circles for a bit, the issue seems to have been as simple as using the correct FQDN in the endpoint fields and disabling SSL. We used the jet3st properties, but it turns out those aren’t actually needed with recent Hadoop versions (?).

For anyone who might be having similar issues, here are the relevant configuration in core-site.xml for S3A and S3N with Hadoop 2.7.1:

<configuration>

<!-- S3N Connector to Obsidian -->
<property>
 <name>fs.s3n.awsAccessKeyId</name>
 <description>AWS access key ID</description>
 <value>yourusername</value>
</property>

<property>
 <name>fs.s3n.awsSecretAccessKey</name>
 <description>AWS secret key</description>
 <value>sweetpassword</value>
</property>

<property>
 <name>fs.s3n.endpoint</name>
 <value>youre.fqdn.here</value>
</property>

<property>
 <name>fs.s3n.ssl.enabled</name>
 <value>false</value>
</property>

<!-- S3A Connector to Obsidian -->

<property>
 <name>fs.s3a.access.key</name>
 <description>AWS access key ID. Omit for Role-based authentication.</description>
 <value>yourusername</value>
</property>

<property>
 <name>fs.s3a.secret.key</name>
 <description>AWS secret key. Omit for Role-based authentication.</description>
 <value>sweetpassword</value>
</property>

<property>
 <name>fs.s3a.connection.ssl.enabled</name>
 <value>false</value>
 <description>Enables or disables SSL connections to S3.</description>
</property>

<property>
 <name>fs.s3a.endpoint</name>
 <description>AWS S3 endpoint to connect to. An up-to-date list is
    provided in the AWS Documentation: regions and endpoints. Without this
    property, the standard region (s3.amazonaws.com) is assumed.
 </description>
 <value>your.fqdn.here</value>
</property>

</configuration>

Also, as mentioned previously in the thread, it’s necessary to add some things to your HADOOP_CLASSPATH:

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/path/to/hadoop-2.7.1/share/hadoop/tools/lib/*

You can test by:

s3cmd mb s3://some-bucket               # <- note that you have to do this with s3cmd, not hadoop, at least with our object store
hadoop fs -ls s3n://some-bucket/
hadoop fs -ls s3a://some-bucket/
hadoop distcp /your/favorite/hdfs/data s3a://some-bucket/

HTH,

--
Caleb Phillips, Ph.D.
Data Scientist | Computational Science Center

National Renewable Energy Laboratory (NREL)
15013 Denver West Parkway | Golden, CO 80401
303-275-4297 | caleb.phillips@nrel.gov

From: Billy Watson <wi...@gmail.com>>
Date: Tuesday, January 19, 2016 at 8:41 AM
To: Alexander Pivovarov <ap...@gmail.com>>
Cc: Caleb Phillips <ca...@nrel.gov>>, "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: fs.s3a.endpoint not working

Stupid question, I assume you're using a URL that starts with s3a and that your custom endpoint supports s3a?

William Watson
Lead Software Engineer

On Thu, Jan 14, 2016 at 1:57 PM, Alexander Pivovarov <ap...@gmail.com>> wrote:

http://www.jets3t.org/toolkit/configuration.html

On Jan 14, 2016 10:56 AM, "Alexander Pivovarov" <ap...@gmail.com>> wrote:

Add jets3t.properties file with s3service.s3-endpoint=<endpoint> to /etc/hadoop/conf folder

The folder with the file should be in HADOOP_CLASSPATH

JetS3t library which is used by hadoop is looking for this file.

On Dec 22, 2015 12:39 PM, "Phillips, Caleb" <Ca...@nrel.gov>> wrote:
Hi All,

New to this list. Looking for a bit of help:

I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object store.

This issue was discussed, but left unresolved, in this thread:

https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E

And here, on Cloudera's forums (the second post is mine):

https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoint-ignored-in-hdfs-site-xml/m-p/33694#M1180

I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.

However, when I point the fs.s3a.endpoint configuration directive at my non-AWS S3-Compatable object storage, it appears to still point at (and authenticate against) AWS.

I've checked and double-checked my credentials and configuration using both Python's boto library and the s3cmd tool, both of which connect to this non-AWS data store just fine.

Any help would be much appreciated. Thanks!

--
Caleb Phillips, Ph.D.
Data Scientist | Computational Science Center

National Renewable Energy Laboratory (NREL)
15013 Denver West Parkway | Golden, CO 80401
303-275-4297<tel:303-275-4297> | caleb.phillips@nrel.gov<ma...@nrel.gov>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org<ma...@hadoop.apache.org>
For additional commands, e-mail: user-help@hadoop.apache.org<ma...@hadoop.apache.org>

Re: fs.s3a.endpoint not working

Posted by "Phillips, Caleb" <Ca...@nrel.gov>.

Hi All,

Just wanted to follow on that we got this working with the help of the object storage vendor. After running in circles for a bit, the issue seems to have been as simple as using the correct FQDN in the endpoint fields and disabling SSL. We used the jet3st properties, but it turns out those aren’t actually needed with recent Hadoop versions (?).

For anyone who might be having similar issues, here are the relevant configuration in core-site.xml for S3A and S3N with Hadoop 2.7.1:

<configuration>

<!-- S3N Connector to Obsidian -->
<property>
 <name>fs.s3n.awsAccessKeyId</name>
 <description>AWS access key ID</description>
 <value>yourusername</value>
</property>

<property>
 <name>fs.s3n.awsSecretAccessKey</name>
 <description>AWS secret key</description>
 <value>sweetpassword</value>
</property>

<property>
 <name>fs.s3n.endpoint</name>
 <value>youre.fqdn.here</value>
</property>

<property>
 <name>fs.s3n.ssl.enabled</name>
 <value>false</value>
</property>

<!-- S3A Connector to Obsidian -->

<property>
 <name>fs.s3a.access.key</name>
 <description>AWS access key ID. Omit for Role-based authentication.</description>
 <value>yourusername</value>
</property>

<property>
 <name>fs.s3a.secret.key</name>
 <description>AWS secret key. Omit for Role-based authentication.</description>
 <value>sweetpassword</value>
</property>

<property>
 <name>fs.s3a.connection.ssl.enabled</name>
 <value>false</value>
 <description>Enables or disables SSL connections to S3.</description>
</property>

<property>
 <name>fs.s3a.endpoint</name>
 <description>AWS S3 endpoint to connect to. An up-to-date list is
    provided in the AWS Documentation: regions and endpoints. Without this
    property, the standard region (s3.amazonaws.com) is assumed.
 </description>
 <value>your.fqdn.here</value>
</property>

</configuration>

Also, as mentioned previously in the thread, it’s necessary to add some things to your HADOOP_CLASSPATH:

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/path/to/hadoop-2.7.1/share/hadoop/tools/lib/*

You can test by:

s3cmd mb s3://some-bucket               # <- note that you have to do this with s3cmd, not hadoop, at least with our object store
hadoop fs -ls s3n://some-bucket/
hadoop fs -ls s3a://some-bucket/
hadoop distcp /your/favorite/hdfs/data s3a://some-bucket/

HTH,

--
Caleb Phillips, Ph.D.
Data Scientist | Computational Science Center

National Renewable Energy Laboratory (NREL)
15013 Denver West Parkway | Golden, CO 80401
303-275-4297 | caleb.phillips@nrel.gov

From: Billy Watson <wi...@gmail.com>>
Date: Tuesday, January 19, 2016 at 8:41 AM
To: Alexander Pivovarov <ap...@gmail.com>>
Cc: Caleb Phillips <ca...@nrel.gov>>, "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: fs.s3a.endpoint not working

Stupid question, I assume you're using a URL that starts with s3a and that your custom endpoint supports s3a?

William Watson
Lead Software Engineer

On Thu, Jan 14, 2016 at 1:57 PM, Alexander Pivovarov <ap...@gmail.com>> wrote:

http://www.jets3t.org/toolkit/configuration.html

On Jan 14, 2016 10:56 AM, "Alexander Pivovarov" <ap...@gmail.com>> wrote:

Add jets3t.properties file with s3service.s3-endpoint=<endpoint> to /etc/hadoop/conf folder

The folder with the file should be in HADOOP_CLASSPATH

JetS3t library which is used by hadoop is looking for this file.

On Dec 22, 2015 12:39 PM, "Phillips, Caleb" <Ca...@nrel.gov>> wrote:
Hi All,

New to this list. Looking for a bit of help:

I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object store.

This issue was discussed, but left unresolved, in this thread:

https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E

And here, on Cloudera's forums (the second post is mine):

https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoint-ignored-in-hdfs-site-xml/m-p/33694#M1180

I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.

However, when I point the fs.s3a.endpoint configuration directive at my non-AWS S3-Compatable object storage, it appears to still point at (and authenticate against) AWS.

I've checked and double-checked my credentials and configuration using both Python's boto library and the s3cmd tool, both of which connect to this non-AWS data store just fine.

Any help would be much appreciated. Thanks!

--
Caleb Phillips, Ph.D.
Data Scientist | Computational Science Center

National Renewable Energy Laboratory (NREL)
15013 Denver West Parkway | Golden, CO 80401
303-275-4297<tel:303-275-4297> | caleb.phillips@nrel.gov<ma...@nrel.gov>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org<ma...@hadoop.apache.org>
For additional commands, e-mail: user-help@hadoop.apache.org<ma...@hadoop.apache.org>

Re: fs.s3a.endpoint not working

Posted by "Phillips, Caleb" <Ca...@nrel.gov>.

Hi All,

Just wanted to follow on that we got this working with the help of the object storage vendor. After running in circles for a bit, the issue seems to have been as simple as using the correct FQDN in the endpoint fields and disabling SSL. We used the jet3st properties, but it turns out those aren’t actually needed with recent Hadoop versions (?).

For anyone who might be having similar issues, here are the relevant configuration in core-site.xml for S3A and S3N with Hadoop 2.7.1:

<configuration>

<!-- S3N Connector to Obsidian -->
<property>
 <name>fs.s3n.awsAccessKeyId</name>
 <description>AWS access key ID</description>
 <value>yourusername</value>
</property>

<property>
 <name>fs.s3n.awsSecretAccessKey</name>
 <description>AWS secret key</description>
 <value>sweetpassword</value>
</property>

<property>
 <name>fs.s3n.endpoint</name>
 <value>youre.fqdn.here</value>
</property>

<property>
 <name>fs.s3n.ssl.enabled</name>
 <value>false</value>
</property>

<!-- S3A Connector to Obsidian -->

<property>
 <name>fs.s3a.access.key</name>
 <description>AWS access key ID. Omit for Role-based authentication.</description>
 <value>yourusername</value>
</property>

<property>
 <name>fs.s3a.secret.key</name>
 <description>AWS secret key. Omit for Role-based authentication.</description>
 <value>sweetpassword</value>
</property>

<property>
 <name>fs.s3a.connection.ssl.enabled</name>
 <value>false</value>
 <description>Enables or disables SSL connections to S3.</description>
</property>

<property>
 <name>fs.s3a.endpoint</name>
 <description>AWS S3 endpoint to connect to. An up-to-date list is
    provided in the AWS Documentation: regions and endpoints. Without this
    property, the standard region (s3.amazonaws.com) is assumed.
 </description>
 <value>your.fqdn.here</value>
</property>

</configuration>

Also, as mentioned previously in the thread, it’s necessary to add some things to your HADOOP_CLASSPATH:

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/path/to/hadoop-2.7.1/share/hadoop/tools/lib/*

You can test by:

s3cmd mb s3://some-bucket               # <- note that you have to do this with s3cmd, not hadoop, at least with our object store
hadoop fs -ls s3n://some-bucket/
hadoop fs -ls s3a://some-bucket/
hadoop distcp /your/favorite/hdfs/data s3a://some-bucket/

HTH,

--
Caleb Phillips, Ph.D.
Data Scientist | Computational Science Center

National Renewable Energy Laboratory (NREL)
15013 Denver West Parkway | Golden, CO 80401
303-275-4297 | caleb.phillips@nrel.gov

From: Billy Watson <wi...@gmail.com>>
Date: Tuesday, January 19, 2016 at 8:41 AM
To: Alexander Pivovarov <ap...@gmail.com>>
Cc: Caleb Phillips <ca...@nrel.gov>>, "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: fs.s3a.endpoint not working

Stupid question, I assume you're using a URL that starts with s3a and that your custom endpoint supports s3a?

William Watson
Lead Software Engineer

On Thu, Jan 14, 2016 at 1:57 PM, Alexander Pivovarov <ap...@gmail.com>> wrote:

http://www.jets3t.org/toolkit/configuration.html

On Jan 14, 2016 10:56 AM, "Alexander Pivovarov" <ap...@gmail.com>> wrote:

Add jets3t.properties file with s3service.s3-endpoint=<endpoint> to /etc/hadoop/conf folder

The folder with the file should be in HADOOP_CLASSPATH

JetS3t library which is used by hadoop is looking for this file.

On Dec 22, 2015 12:39 PM, "Phillips, Caleb" <Ca...@nrel.gov>> wrote:
Hi All,

New to this list. Looking for a bit of help:

I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object store.

This issue was discussed, but left unresolved, in this thread:

https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E

And here, on Cloudera's forums (the second post is mine):

https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoint-ignored-in-hdfs-site-xml/m-p/33694#M1180

I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.

However, when I point the fs.s3a.endpoint configuration directive at my non-AWS S3-Compatable object storage, it appears to still point at (and authenticate against) AWS.

I've checked and double-checked my credentials and configuration using both Python's boto library and the s3cmd tool, both of which connect to this non-AWS data store just fine.

Any help would be much appreciated. Thanks!

--
Caleb Phillips, Ph.D.
Data Scientist | Computational Science Center

National Renewable Energy Laboratory (NREL)
15013 Denver West Parkway | Golden, CO 80401
303-275-4297<tel:303-275-4297> | caleb.phillips@nrel.gov<ma...@nrel.gov>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org<ma...@hadoop.apache.org>
For additional commands, e-mail: user-help@hadoop.apache.org<ma...@hadoop.apache.org>

Re: fs.s3a.endpoint not working

Posted by "Phillips, Caleb" <Ca...@nrel.gov>.

Hi All,

Just wanted to follow on that we got this working with the help of the object storage vendor. After running in circles for a bit, the issue seems to have been as simple as using the correct FQDN in the endpoint fields and disabling SSL. We used the jet3st properties, but it turns out those aren’t actually needed with recent Hadoop versions (?).

For anyone who might be having similar issues, here are the relevant configuration in core-site.xml for S3A and S3N with Hadoop 2.7.1:

<configuration>

<!-- S3N Connector to Obsidian -->
<property>
 <name>fs.s3n.awsAccessKeyId</name>
 <description>AWS access key ID</description>
 <value>yourusername</value>
</property>

<property>
 <name>fs.s3n.awsSecretAccessKey</name>
 <description>AWS secret key</description>
 <value>sweetpassword</value>
</property>

<property>
 <name>fs.s3n.endpoint</name>
 <value>youre.fqdn.here</value>
</property>

<property>
 <name>fs.s3n.ssl.enabled</name>
 <value>false</value>
</property>

<!-- S3A Connector to Obsidian -->

<property>
 <name>fs.s3a.access.key</name>
 <description>AWS access key ID. Omit for Role-based authentication.</description>
 <value>yourusername</value>
</property>

<property>
 <name>fs.s3a.secret.key</name>
 <description>AWS secret key. Omit for Role-based authentication.</description>
 <value>sweetpassword</value>
</property>

<property>
 <name>fs.s3a.connection.ssl.enabled</name>
 <value>false</value>
 <description>Enables or disables SSL connections to S3.</description>
</property>

<property>
 <name>fs.s3a.endpoint</name>
 <description>AWS S3 endpoint to connect to. An up-to-date list is
    provided in the AWS Documentation: regions and endpoints. Without this
    property, the standard region (s3.amazonaws.com) is assumed.
 </description>
 <value>your.fqdn.here</value>
</property>

</configuration>

Also, as mentioned previously in the thread, it’s necessary to add some things to your HADOOP_CLASSPATH:

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/path/to/hadoop-2.7.1/share/hadoop/tools/lib/*

You can test by:

s3cmd mb s3://some-bucket               # <- note that you have to do this with s3cmd, not hadoop, at least with our object store
hadoop fs -ls s3n://some-bucket/
hadoop fs -ls s3a://some-bucket/
hadoop distcp /your/favorite/hdfs/data s3a://some-bucket/

HTH,

--
Caleb Phillips, Ph.D.
Data Scientist | Computational Science Center

National Renewable Energy Laboratory (NREL)
15013 Denver West Parkway | Golden, CO 80401
303-275-4297 | caleb.phillips@nrel.gov

From: Billy Watson <wi...@gmail.com>>
Date: Tuesday, January 19, 2016 at 8:41 AM
To: Alexander Pivovarov <ap...@gmail.com>>
Cc: Caleb Phillips <ca...@nrel.gov>>, "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: fs.s3a.endpoint not working

Stupid question, I assume you're using a URL that starts with s3a and that your custom endpoint supports s3a?

William Watson
Lead Software Engineer

On Thu, Jan 14, 2016 at 1:57 PM, Alexander Pivovarov <ap...@gmail.com>> wrote:

http://www.jets3t.org/toolkit/configuration.html

On Jan 14, 2016 10:56 AM, "Alexander Pivovarov" <ap...@gmail.com>> wrote:

Add jets3t.properties file with s3service.s3-endpoint=<endpoint> to /etc/hadoop/conf folder

The folder with the file should be in HADOOP_CLASSPATH

JetS3t library which is used by hadoop is looking for this file.

On Dec 22, 2015 12:39 PM, "Phillips, Caleb" <Ca...@nrel.gov>> wrote:
Hi All,

New to this list. Looking for a bit of help:

I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object store.

This issue was discussed, but left unresolved, in this thread:

https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E

And here, on Cloudera's forums (the second post is mine):

https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoint-ignored-in-hdfs-site-xml/m-p/33694#M1180

I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.

However, when I point the fs.s3a.endpoint configuration directive at my non-AWS S3-Compatable object storage, it appears to still point at (and authenticate against) AWS.

I've checked and double-checked my credentials and configuration using both Python's boto library and the s3cmd tool, both of which connect to this non-AWS data store just fine.

Any help would be much appreciated. Thanks!

--
Caleb Phillips, Ph.D.
Data Scientist | Computational Science Center

National Renewable Energy Laboratory (NREL)
15013 Denver West Parkway | Golden, CO 80401
303-275-4297<tel:303-275-4297> | caleb.phillips@nrel.gov<ma...@nrel.gov>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org<ma...@hadoop.apache.org>
For additional commands, e-mail: user-help@hadoop.apache.org<ma...@hadoop.apache.org>

Re: fs.s3a.endpoint not working

Posted by Billy Watson <wi...@gmail.com>.

Stupid question, I assume you're using a URL that starts with s3a and that
your custom endpoint supports s3a?

William Watson
Lead Software Engineer

On Thu, Jan 14, 2016 at 1:57 PM, Alexander Pivovarov <ap...@gmail.com>
wrote:

> http://www.jets3t.org/toolkit/configuration.html
> On Jan 14, 2016 10:56 AM, "Alexander Pivovarov" <ap...@gmail.com>
> wrote:
>
>> Add jets3t.properties file with s3service.s3-endpoint=<endpoint> to
>> /etc/hadoop/conf folder
>>
>> The folder with the file should be in HADOOP_CLASSPATH
>>
>> JetS3t library which is used by hadoop is looking for this file.
>> On Dec 22, 2015 12:39 PM, "Phillips, Caleb" <Ca...@nrel.gov>
>> wrote:
>>
>>> Hi All,
>>>
>>> New to this list. Looking for a bit of help:
>>>
>>> I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
>>> store.
>>>
>>> This issue was discussed, but left unresolved, in this thread:
>>>
>>>
>>> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E
>>>
>>> And here, on Cloudera's forums (the second post is mine):
>>>
>>>
>>> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoint-ignored-in-hdfs-site-xml/m-p/33694#M1180
>>>
>>> I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
>>> Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>>>
>>> However, when I point the fs.s3a.endpoint configuration directive at my
>>> non-AWS S3-Compatable object storage, it appears to still point at (and
>>> authenticate against) AWS.
>>>
>>> I've checked and double-checked my credentials and configuration using
>>> both Python's boto library and the s3cmd tool, both of which connect to
>>> this non-AWS data store just fine.
>>>
>>> Any help would be much appreciated. Thanks!
>>>
>>> --
>>> Caleb Phillips, Ph.D.
>>> Data Scientist | Computational Science Center
>>>
>>> National Renewable Energy Laboratory (NREL)
>>> 15013 Denver West Parkway | Golden, CO 80401
>>> 303-275-4297 | caleb.phillips@nrel.gov
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: user-help@hadoop.apache.org
>>>
>>>

Re: fs.s3a.endpoint not working

Posted by Billy Watson <wi...@gmail.com>.

Stupid question, I assume you're using a URL that starts with s3a and that
your custom endpoint supports s3a?

William Watson
Lead Software Engineer

On Thu, Jan 14, 2016 at 1:57 PM, Alexander Pivovarov <ap...@gmail.com>
wrote:

> http://www.jets3t.org/toolkit/configuration.html
> On Jan 14, 2016 10:56 AM, "Alexander Pivovarov" <ap...@gmail.com>
> wrote:
>
>> Add jets3t.properties file with s3service.s3-endpoint=<endpoint> to
>> /etc/hadoop/conf folder
>>
>> The folder with the file should be in HADOOP_CLASSPATH
>>
>> JetS3t library which is used by hadoop is looking for this file.
>> On Dec 22, 2015 12:39 PM, "Phillips, Caleb" <Ca...@nrel.gov>
>> wrote:
>>
>>> Hi All,
>>>
>>> New to this list. Looking for a bit of help:
>>>
>>> I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
>>> store.
>>>
>>> This issue was discussed, but left unresolved, in this thread:
>>>
>>>
>>> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E
>>>
>>> And here, on Cloudera's forums (the second post is mine):
>>>
>>>
>>> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoint-ignored-in-hdfs-site-xml/m-p/33694#M1180
>>>
>>> I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
>>> Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>>>
>>> However, when I point the fs.s3a.endpoint configuration directive at my
>>> non-AWS S3-Compatable object storage, it appears to still point at (and
>>> authenticate against) AWS.
>>>
>>> I've checked and double-checked my credentials and configuration using
>>> both Python's boto library and the s3cmd tool, both of which connect to
>>> this non-AWS data store just fine.
>>>
>>> Any help would be much appreciated. Thanks!
>>>
>>> --
>>> Caleb Phillips, Ph.D.
>>> Data Scientist | Computational Science Center
>>>
>>> National Renewable Energy Laboratory (NREL)
>>> 15013 Denver West Parkway | Golden, CO 80401
>>> 303-275-4297 | caleb.phillips@nrel.gov
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: user-help@hadoop.apache.org
>>>
>>>

Re: fs.s3a.endpoint not working

Posted by Billy Watson <wi...@gmail.com>.

Stupid question, I assume you're using a URL that starts with s3a and that
your custom endpoint supports s3a?

William Watson
Lead Software Engineer

On Thu, Jan 14, 2016 at 1:57 PM, Alexander Pivovarov <ap...@gmail.com>
wrote:

> http://www.jets3t.org/toolkit/configuration.html
> On Jan 14, 2016 10:56 AM, "Alexander Pivovarov" <ap...@gmail.com>
> wrote:
>
>> Add jets3t.properties file with s3service.s3-endpoint=<endpoint> to
>> /etc/hadoop/conf folder
>>
>> The folder with the file should be in HADOOP_CLASSPATH
>>
>> JetS3t library which is used by hadoop is looking for this file.
>> On Dec 22, 2015 12:39 PM, "Phillips, Caleb" <Ca...@nrel.gov>
>> wrote:
>>
>>> Hi All,
>>>
>>> New to this list. Looking for a bit of help:
>>>
>>> I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
>>> store.
>>>
>>> This issue was discussed, but left unresolved, in this thread:
>>>
>>>
>>> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E
>>>
>>> And here, on Cloudera's forums (the second post is mine):
>>>
>>>
>>> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoint-ignored-in-hdfs-site-xml/m-p/33694#M1180
>>>
>>> I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
>>> Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>>>
>>> However, when I point the fs.s3a.endpoint configuration directive at my
>>> non-AWS S3-Compatable object storage, it appears to still point at (and
>>> authenticate against) AWS.
>>>
>>> I've checked and double-checked my credentials and configuration using
>>> both Python's boto library and the s3cmd tool, both of which connect to
>>> this non-AWS data store just fine.
>>>
>>> Any help would be much appreciated. Thanks!
>>>
>>> --
>>> Caleb Phillips, Ph.D.
>>> Data Scientist | Computational Science Center
>>>
>>> National Renewable Energy Laboratory (NREL)
>>> 15013 Denver West Parkway | Golden, CO 80401
>>> 303-275-4297 | caleb.phillips@nrel.gov
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: user-help@hadoop.apache.org
>>>
>>>

Re: fs.s3a.endpoint not working

Posted by Billy Watson <wi...@gmail.com>.

Stupid question, I assume you're using a URL that starts with s3a and that
your custom endpoint supports s3a?

William Watson
Lead Software Engineer

On Thu, Jan 14, 2016 at 1:57 PM, Alexander Pivovarov <ap...@gmail.com>
wrote:

> http://www.jets3t.org/toolkit/configuration.html
> On Jan 14, 2016 10:56 AM, "Alexander Pivovarov" <ap...@gmail.com>
> wrote:
>
>> Add jets3t.properties file with s3service.s3-endpoint=<endpoint> to
>> /etc/hadoop/conf folder
>>
>> The folder with the file should be in HADOOP_CLASSPATH
>>
>> JetS3t library which is used by hadoop is looking for this file.
>> On Dec 22, 2015 12:39 PM, "Phillips, Caleb" <Ca...@nrel.gov>
>> wrote:
>>
>>> Hi All,
>>>
>>> New to this list. Looking for a bit of help:
>>>
>>> I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
>>> store.
>>>
>>> This issue was discussed, but left unresolved, in this thread:
>>>
>>>
>>> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E
>>>
>>> And here, on Cloudera's forums (the second post is mine):
>>>
>>>
>>> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoint-ignored-in-hdfs-site-xml/m-p/33694#M1180
>>>
>>> I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
>>> Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>>>
>>> However, when I point the fs.s3a.endpoint configuration directive at my
>>> non-AWS S3-Compatable object storage, it appears to still point at (and
>>> authenticate against) AWS.
>>>
>>> I've checked and double-checked my credentials and configuration using
>>> both Python's boto library and the s3cmd tool, both of which connect to
>>> this non-AWS data store just fine.
>>>
>>> Any help would be much appreciated. Thanks!
>>>
>>> --
>>> Caleb Phillips, Ph.D.
>>> Data Scientist | Computational Science Center
>>>
>>> National Renewable Energy Laboratory (NREL)
>>> 15013 Denver West Parkway | Golden, CO 80401
>>> 303-275-4297 | caleb.phillips@nrel.gov
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: user-help@hadoop.apache.org
>>>
>>>

Re: fs.s3a.endpoint not working

Posted by Alexander Pivovarov <ap...@gmail.com>.

http://www.jets3t.org/toolkit/configuration.html
On Jan 14, 2016 10:56 AM, "Alexander Pivovarov" <ap...@gmail.com>
wrote:

> Add jets3t.properties file with s3service.s3-endpoint=<endpoint> to
> /etc/hadoop/conf folder
>
> The folder with the file should be in HADOOP_CLASSPATH
>
> JetS3t library which is used by hadoop is looking for this file.
> On Dec 22, 2015 12:39 PM, "Phillips, Caleb" <Ca...@nrel.gov>
> wrote:
>
>> Hi All,
>>
>> New to this list. Looking for a bit of help:
>>
>> I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
>> store.
>>
>> This issue was discussed, but left unresolved, in this thread:
>>
>>
>> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E
>>
>> And here, on Cloudera's forums (the second post is mine):
>>
>>
>> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoint-ignored-in-hdfs-site-xml/m-p/33694#M1180
>>
>> I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
>> Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>>
>> However, when I point the fs.s3a.endpoint configuration directive at my
>> non-AWS S3-Compatable object storage, it appears to still point at (and
>> authenticate against) AWS.
>>
>> I've checked and double-checked my credentials and configuration using
>> both Python's boto library and the s3cmd tool, both of which connect to
>> this non-AWS data store just fine.
>>
>> Any help would be much appreciated. Thanks!
>>
>> --
>> Caleb Phillips, Ph.D.
>> Data Scientist | Computational Science Center
>>
>> National Renewable Energy Laboratory (NREL)
>> 15013 Denver West Parkway | Golden, CO 80401
>> 303-275-4297 | caleb.phillips@nrel.gov
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: user-help@hadoop.apache.org
>>
>>

Re: fs.s3a.endpoint not working

Posted by Alexander Pivovarov <ap...@gmail.com>.

http://www.jets3t.org/toolkit/configuration.html
On Jan 14, 2016 10:56 AM, "Alexander Pivovarov" <ap...@gmail.com>
wrote:

> Add jets3t.properties file with s3service.s3-endpoint=<endpoint> to
> /etc/hadoop/conf folder
>
> The folder with the file should be in HADOOP_CLASSPATH
>
> JetS3t library which is used by hadoop is looking for this file.
> On Dec 22, 2015 12:39 PM, "Phillips, Caleb" <Ca...@nrel.gov>
> wrote:
>
>> Hi All,
>>
>> New to this list. Looking for a bit of help:
>>
>> I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
>> store.
>>
>> This issue was discussed, but left unresolved, in this thread:
>>
>>
>> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E
>>
>> And here, on Cloudera's forums (the second post is mine):
>>
>>
>> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoint-ignored-in-hdfs-site-xml/m-p/33694#M1180
>>
>> I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
>> Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>>
>> However, when I point the fs.s3a.endpoint configuration directive at my
>> non-AWS S3-Compatable object storage, it appears to still point at (and
>> authenticate against) AWS.
>>
>> I've checked and double-checked my credentials and configuration using
>> both Python's boto library and the s3cmd tool, both of which connect to
>> this non-AWS data store just fine.
>>
>> Any help would be much appreciated. Thanks!
>>
>> --
>> Caleb Phillips, Ph.D.
>> Data Scientist | Computational Science Center
>>
>> National Renewable Energy Laboratory (NREL)
>> 15013 Denver West Parkway | Golden, CO 80401
>> 303-275-4297 | caleb.phillips@nrel.gov
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: user-help@hadoop.apache.org
>>
>>

Re: fs.s3a.endpoint not working

Posted by Alexander Pivovarov <ap...@gmail.com>.

http://www.jets3t.org/toolkit/configuration.html
On Jan 14, 2016 10:56 AM, "Alexander Pivovarov" <ap...@gmail.com>
wrote:

> Add jets3t.properties file with s3service.s3-endpoint=<endpoint> to
> /etc/hadoop/conf folder
>
> The folder with the file should be in HADOOP_CLASSPATH
>
> JetS3t library which is used by hadoop is looking for this file.
> On Dec 22, 2015 12:39 PM, "Phillips, Caleb" <Ca...@nrel.gov>
> wrote:
>
>> Hi All,
>>
>> New to this list. Looking for a bit of help:
>>
>> I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
>> store.
>>
>> This issue was discussed, but left unresolved, in this thread:
>>
>>
>> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E
>>
>> And here, on Cloudera's forums (the second post is mine):
>>
>>
>> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoint-ignored-in-hdfs-site-xml/m-p/33694#M1180
>>
>> I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
>> Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>>
>> However, when I point the fs.s3a.endpoint configuration directive at my
>> non-AWS S3-Compatable object storage, it appears to still point at (and
>> authenticate against) AWS.
>>
>> I've checked and double-checked my credentials and configuration using
>> both Python's boto library and the s3cmd tool, both of which connect to
>> this non-AWS data store just fine.
>>
>> Any help would be much appreciated. Thanks!
>>
>> --
>> Caleb Phillips, Ph.D.
>> Data Scientist | Computational Science Center
>>
>> National Renewable Energy Laboratory (NREL)
>> 15013 Denver West Parkway | Golden, CO 80401
>> 303-275-4297 | caleb.phillips@nrel.gov
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: user-help@hadoop.apache.org
>>
>>

Re: fs.s3a.endpoint not working

Posted by Alexander Pivovarov <ap...@gmail.com>.

http://www.jets3t.org/toolkit/configuration.html
On Jan 14, 2016 10:56 AM, "Alexander Pivovarov" <ap...@gmail.com>
wrote:

> Add jets3t.properties file with s3service.s3-endpoint=<endpoint> to
> /etc/hadoop/conf folder
>
> The folder with the file should be in HADOOP_CLASSPATH
>
> JetS3t library which is used by hadoop is looking for this file.
> On Dec 22, 2015 12:39 PM, "Phillips, Caleb" <Ca...@nrel.gov>
> wrote:
>
>> Hi All,
>>
>> New to this list. Looking for a bit of help:
>>
>> I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
>> store.
>>
>> This issue was discussed, but left unresolved, in this thread:
>>
>>
>> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E
>>
>> And here, on Cloudera's forums (the second post is mine):
>>
>>
>> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoint-ignored-in-hdfs-site-xml/m-p/33694#M1180
>>
>> I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
>> Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>>
>> However, when I point the fs.s3a.endpoint configuration directive at my
>> non-AWS S3-Compatable object storage, it appears to still point at (and
>> authenticate against) AWS.
>>
>> I've checked and double-checked my credentials and configuration using
>> both Python's boto library and the s3cmd tool, both of which connect to
>> this non-AWS data store just fine.
>>
>> Any help would be much appreciated. Thanks!
>>
>> --
>> Caleb Phillips, Ph.D.
>> Data Scientist | Computational Science Center
>>
>> National Renewable Energy Laboratory (NREL)
>> 15013 Denver West Parkway | Golden, CO 80401
>> 303-275-4297 | caleb.phillips@nrel.gov
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: user-help@hadoop.apache.org
>>
>>

Re: fs.s3a.endpoint not working

Posted by Alexander Pivovarov <ap...@gmail.com>.

Add jets3t.properties file with s3service.s3-endpoint=<endpoint> to
/etc/hadoop/conf folder

The folder with the file should be in HADOOP_CLASSPATH

JetS3t library which is used by hadoop is looking for this file.
On Dec 22, 2015 12:39 PM, "Phillips, Caleb" <Ca...@nrel.gov> wrote:

> Hi All,
>
> New to this list. Looking for a bit of help:
>
> I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
> store.
>
> This issue was discussed, but left unresolved, in this thread:
>
>
> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E
>
> And here, on Cloudera's forums (the second post is mine):
>
>
> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoint-ignored-in-hdfs-site-xml/m-p/33694#M1180
>
> I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using Hadoop,
> I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>
> However, when I point the fs.s3a.endpoint configuration directive at my
> non-AWS S3-Compatable object storage, it appears to still point at (and
> authenticate against) AWS.
>
> I've checked and double-checked my credentials and configuration using
> both Python's boto library and the s3cmd tool, both of which connect to
> this non-AWS data store just fine.
>
> Any help would be much appreciated. Thanks!
>
> --
> Caleb Phillips, Ph.D.
> Data Scientist | Computational Science Center
>
> National Renewable Energy Laboratory (NREL)
> 15013 Denver West Parkway | Golden, CO 80401
> 303-275-4297 | caleb.phillips@nrel.gov
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>
>

Re: fs.s3a.endpoint not working

Posted by "Phillips, Caleb" <Ca...@nrel.gov>.

Hi Jonathan,


>Im not totally following this thread from the beginning but I might be
>able to help as I have some experience with Amazon EMR (elastic map
>reduce) when working with custom jar files and s3
>Are you using EMR or something internal and offloading strage to s3?

We have an S3-compatable object store made by Scality
(http://www.scality.com/). A so-called ‘ring’. It’s basically a pile of
linux boxes that behaves like our own internal S3 ‘cloud'. It lives in our
data-center.

What I’d like to do is have hadoop connect to that *instead* of the Amazon
AWS S3.

Yet, no matter how I set the fs.s3a.endpoint directive, it still connects
to Amazon’s S3.

Hope that clarifies,

—
Caleb

>---
>Regards,
>Jonathan Aquilina
>Founder
>
>
> 
>On 2016-01-13 23:21, Phillips, Caleb wrote:
>
>Hi Billy (and others),
>
>One of the threads suggested using the core-site.xml. Did you try putting
>your configuration in there?
>
>Yes, I did try that. I've also tried setting it dynamically in e.g.,
>spark. I can verify that it is getting the configuration correctly:
>
>hadoop org.apache.hadoop.conf.Configuration
>
>Still it never connects to our internal S3-compatable store and always
>connects to AWS.
>
>One thing I've noticed is that the AWS stuff is handled by an underlying
>library (I think jets3t in < 2.6 versions, forget what in 2.6+) and when
>I was trying to mess with stuff and spelunking through the hadoop code, I
>kept running into blocks with that library.
>
>I started digging into the code. I found that the custom endpoint was
>introduced with this patch:
>
>https://issues.apache.org/jira/browse/HADOOP-11261
>
>It seems it was integrated in 2.7.0, so just to be sure I downloaded
>2.7.1, but the problem persists.
>
>That code calls this function in the AWS Java SDK:
>
>http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/service
>s/s3/AmazonS3Client.html#setEndpoint(java.lang.String)
>
>However, no matter what configuration I use, it still seems to connect to
>Amazon AWS. Is it possible that the AWS Java SDK cannot work with
>S3-compatable (non-AWS) stores? If so, it would seem there is no way
>currently to connect hadoop to an S3-compatable
> non-AWS store.
>
>If anyone else has any insight, particularly success using hadoop with a
>non-AWS, S3-compatable store, please chime in!
>
>
>William Watson
>Software Engineer
>(904) 705-7056 PCS
>
>On Mon, Jan 11, 2016 at 10:39 AM, Phillips, Caleb
><Ca...@nrel.gov>> wrote:
>Hi All,
>
>Just wanted to send this out again since there was no response
>(admittedly, originally sent in the midst of the US holiday season) and it
>seems to be an issue that continues to come up (see e.g., the email from
>Han Ju on Jan 5).
>
>If anyone has successfully connected Hadoop to a non-AWS S3-compatable
>object store, it'd be very helpful to hear how you made it work. The
>fs.s3a.endpoint configuration directive appears non-functional at our site
>(with Hadoop 2.6.3).
>
>--
>Caleb Phillips, Ph.D.
>Data Scientist | Computational Science Center
>
>National Renewable Energy Laboratory (NREL)
>15013 Denver West Parkway | Golden, CO 80401
>303-275-4297<tel:303-275-4297> |
>caleb.phillips@nrel.gov<ma...@nrel.gov>
>
>
>
>
>
>
>On 12/22/15, 1:39 PM, "Phillips, Caleb"
><Ca...@nrel.gov>> wrote:
>
>
>Hi All,
>
>New to this list. Looking for a bit of help:
>
>I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
>store.
>
>This issue was discussed, but left unresolved, in this thread:
>
>https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_
>Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com<mailto:Au5Es_
>fLUgZMGwkkgA3JyA1ASi3u%2BisJCuYmfnTvNkGuQ@mail.gmail.com>%3E
>
>And here, on Cloudera's forums (the second post is mine):
>
>https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoi
>nt-ignored-in-hdfs-site-xml/m-p/33694#M1180
>
>I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
>Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>
>However, when I point the fs.s3a.endpoint configuration directive at my
>non-AWS S3-Compatable object storage, it appears to still point at (and
>authenticate against) AWS.
>
>I've checked and double-checked my credentials and configuration using
>both Python's boto library and the s3cmd tool, both of which connect to
>this non-AWS data store just fine.
>
>Any help would be much appreciated. Thanks!
>
>--
>Caleb Phillips, Ph.D.
>Data Scientist | Computational Science Center
>
>National Renewable Energy Laboratory (NREL)
>15013 Denver West Parkway | Golden, CO 80401
>303-275-4297 | caleb.phillips@nrel.gov<ma...@nrel.gov>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail:
>user-unsubscribe@hadoop.apache.org<mailto:user-unsubscribe@hadoop.apache.o
>rg>
>For additional commands, e-mail:
>user-help@hadoop.apache.org<ma...@hadoop.apache.org>
>
>
>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: user-help@hadoop.apache.org

Re: fs.s3a.endpoint not working

Posted by "Phillips, Caleb" <Ca...@nrel.gov>.

Hi Jonathan,


>Im not totally following this thread from the beginning but I might be
>able to help as I have some experience with Amazon EMR (elastic map
>reduce) when working with custom jar files and s3
>Are you using EMR or something internal and offloading strage to s3?

We have an S3-compatable object store made by Scality
(http://www.scality.com/). A so-called ‘ring’. It’s basically a pile of
linux boxes that behaves like our own internal S3 ‘cloud'. It lives in our
data-center.

What I’d like to do is have hadoop connect to that *instead* of the Amazon
AWS S3.

Yet, no matter how I set the fs.s3a.endpoint directive, it still connects
to Amazon’s S3.

Hope that clarifies,

—
Caleb

>---
>Regards,
>Jonathan Aquilina
>Founder
>
>
> 
>On 2016-01-13 23:21, Phillips, Caleb wrote:
>
>Hi Billy (and others),
>
>One of the threads suggested using the core-site.xml. Did you try putting
>your configuration in there?
>
>Yes, I did try that. I've also tried setting it dynamically in e.g.,
>spark. I can verify that it is getting the configuration correctly:
>
>hadoop org.apache.hadoop.conf.Configuration
>
>Still it never connects to our internal S3-compatable store and always
>connects to AWS.
>
>One thing I've noticed is that the AWS stuff is handled by an underlying
>library (I think jets3t in < 2.6 versions, forget what in 2.6+) and when
>I was trying to mess with stuff and spelunking through the hadoop code, I
>kept running into blocks with that library.
>
>I started digging into the code. I found that the custom endpoint was
>introduced with this patch:
>
>https://issues.apache.org/jira/browse/HADOOP-11261
>
>It seems it was integrated in 2.7.0, so just to be sure I downloaded
>2.7.1, but the problem persists.
>
>That code calls this function in the AWS Java SDK:
>
>http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/service
>s/s3/AmazonS3Client.html#setEndpoint(java.lang.String)
>
>However, no matter what configuration I use, it still seems to connect to
>Amazon AWS. Is it possible that the AWS Java SDK cannot work with
>S3-compatable (non-AWS) stores? If so, it would seem there is no way
>currently to connect hadoop to an S3-compatable
> non-AWS store.
>
>If anyone else has any insight, particularly success using hadoop with a
>non-AWS, S3-compatable store, please chime in!
>
>
>William Watson
>Software Engineer
>(904) 705-7056 PCS
>
>On Mon, Jan 11, 2016 at 10:39 AM, Phillips, Caleb
><Ca...@nrel.gov>> wrote:
>Hi All,
>
>Just wanted to send this out again since there was no response
>(admittedly, originally sent in the midst of the US holiday season) and it
>seems to be an issue that continues to come up (see e.g., the email from
>Han Ju on Jan 5).
>
>If anyone has successfully connected Hadoop to a non-AWS S3-compatable
>object store, it'd be very helpful to hear how you made it work. The
>fs.s3a.endpoint configuration directive appears non-functional at our site
>(with Hadoop 2.6.3).
>
>--
>Caleb Phillips, Ph.D.
>Data Scientist | Computational Science Center
>
>National Renewable Energy Laboratory (NREL)
>15013 Denver West Parkway | Golden, CO 80401
>303-275-4297<tel:303-275-4297> |
>caleb.phillips@nrel.gov<ma...@nrel.gov>
>
>
>
>
>
>
>On 12/22/15, 1:39 PM, "Phillips, Caleb"
><Ca...@nrel.gov>> wrote:
>
>
>Hi All,
>
>New to this list. Looking for a bit of help:
>
>I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
>store.
>
>This issue was discussed, but left unresolved, in this thread:
>
>https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_
>Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com<mailto:Au5Es_
>fLUgZMGwkkgA3JyA1ASi3u%2BisJCuYmfnTvNkGuQ@mail.gmail.com>%3E
>
>And here, on Cloudera's forums (the second post is mine):
>
>https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoi
>nt-ignored-in-hdfs-site-xml/m-p/33694#M1180
>
>I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
>Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>
>However, when I point the fs.s3a.endpoint configuration directive at my
>non-AWS S3-Compatable object storage, it appears to still point at (and
>authenticate against) AWS.
>
>I've checked and double-checked my credentials and configuration using
>both Python's boto library and the s3cmd tool, both of which connect to
>this non-AWS data store just fine.
>
>Any help would be much appreciated. Thanks!
>
>--
>Caleb Phillips, Ph.D.
>Data Scientist | Computational Science Center
>
>National Renewable Energy Laboratory (NREL)
>15013 Denver West Parkway | Golden, CO 80401
>303-275-4297 | caleb.phillips@nrel.gov<ma...@nrel.gov>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail:
>user-unsubscribe@hadoop.apache.org<mailto:user-unsubscribe@hadoop.apache.o
>rg>
>For additional commands, e-mail:
>user-help@hadoop.apache.org<ma...@hadoop.apache.org>
>
>
>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: user-help@hadoop.apache.org

Re: fs.s3a.endpoint not working

Posted by "Phillips, Caleb" <Ca...@nrel.gov>.

Hi Jonathan,


>Im not totally following this thread from the beginning but I might be
>able to help as I have some experience with Amazon EMR (elastic map
>reduce) when working with custom jar files and s3
>Are you using EMR or something internal and offloading strage to s3?

We have an S3-compatable object store made by Scality
(http://www.scality.com/). A so-called ‘ring’. It’s basically a pile of
linux boxes that behaves like our own internal S3 ‘cloud'. It lives in our
data-center.

What I’d like to do is have hadoop connect to that *instead* of the Amazon
AWS S3.

Yet, no matter how I set the fs.s3a.endpoint directive, it still connects
to Amazon’s S3.

Hope that clarifies,

—
Caleb

>---
>Regards,
>Jonathan Aquilina
>Founder
>
>
> 
>On 2016-01-13 23:21, Phillips, Caleb wrote:
>
>Hi Billy (and others),
>
>One of the threads suggested using the core-site.xml. Did you try putting
>your configuration in there?
>
>Yes, I did try that. I've also tried setting it dynamically in e.g.,
>spark. I can verify that it is getting the configuration correctly:
>
>hadoop org.apache.hadoop.conf.Configuration
>
>Still it never connects to our internal S3-compatable store and always
>connects to AWS.
>
>One thing I've noticed is that the AWS stuff is handled by an underlying
>library (I think jets3t in < 2.6 versions, forget what in 2.6+) and when
>I was trying to mess with stuff and spelunking through the hadoop code, I
>kept running into blocks with that library.
>
>I started digging into the code. I found that the custom endpoint was
>introduced with this patch:
>
>https://issues.apache.org/jira/browse/HADOOP-11261
>
>It seems it was integrated in 2.7.0, so just to be sure I downloaded
>2.7.1, but the problem persists.
>
>That code calls this function in the AWS Java SDK:
>
>http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/service
>s/s3/AmazonS3Client.html#setEndpoint(java.lang.String)
>
>However, no matter what configuration I use, it still seems to connect to
>Amazon AWS. Is it possible that the AWS Java SDK cannot work with
>S3-compatable (non-AWS) stores? If so, it would seem there is no way
>currently to connect hadoop to an S3-compatable
> non-AWS store.
>
>If anyone else has any insight, particularly success using hadoop with a
>non-AWS, S3-compatable store, please chime in!
>
>
>William Watson
>Software Engineer
>(904) 705-7056 PCS
>
>On Mon, Jan 11, 2016 at 10:39 AM, Phillips, Caleb
><Ca...@nrel.gov>> wrote:
>Hi All,
>
>Just wanted to send this out again since there was no response
>(admittedly, originally sent in the midst of the US holiday season) and it
>seems to be an issue that continues to come up (see e.g., the email from
>Han Ju on Jan 5).
>
>If anyone has successfully connected Hadoop to a non-AWS S3-compatable
>object store, it'd be very helpful to hear how you made it work. The
>fs.s3a.endpoint configuration directive appears non-functional at our site
>(with Hadoop 2.6.3).
>
>--
>Caleb Phillips, Ph.D.
>Data Scientist | Computational Science Center
>
>National Renewable Energy Laboratory (NREL)
>15013 Denver West Parkway | Golden, CO 80401
>303-275-4297<tel:303-275-4297> |
>caleb.phillips@nrel.gov<ma...@nrel.gov>
>
>
>
>
>
>
>On 12/22/15, 1:39 PM, "Phillips, Caleb"
><Ca...@nrel.gov>> wrote:
>
>
>Hi All,
>
>New to this list. Looking for a bit of help:
>
>I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
>store.
>
>This issue was discussed, but left unresolved, in this thread:
>
>https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_
>Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com<mailto:Au5Es_
>fLUgZMGwkkgA3JyA1ASi3u%2BisJCuYmfnTvNkGuQ@mail.gmail.com>%3E
>
>And here, on Cloudera's forums (the second post is mine):
>
>https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoi
>nt-ignored-in-hdfs-site-xml/m-p/33694#M1180
>
>I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
>Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>
>However, when I point the fs.s3a.endpoint configuration directive at my
>non-AWS S3-Compatable object storage, it appears to still point at (and
>authenticate against) AWS.
>
>I've checked and double-checked my credentials and configuration using
>both Python's boto library and the s3cmd tool, both of which connect to
>this non-AWS data store just fine.
>
>Any help would be much appreciated. Thanks!
>
>--
>Caleb Phillips, Ph.D.
>Data Scientist | Computational Science Center
>
>National Renewable Energy Laboratory (NREL)
>15013 Denver West Parkway | Golden, CO 80401
>303-275-4297 | caleb.phillips@nrel.gov<ma...@nrel.gov>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail:
>user-unsubscribe@hadoop.apache.org<mailto:user-unsubscribe@hadoop.apache.o
>rg>
>For additional commands, e-mail:
>user-help@hadoop.apache.org<ma...@hadoop.apache.org>
>
>
>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: user-help@hadoop.apache.org

Re: fs.s3a.endpoint not working

Posted by "Phillips, Caleb" <Ca...@nrel.gov>.

Hi Jonathan,


>Im not totally following this thread from the beginning but I might be
>able to help as I have some experience with Amazon EMR (elastic map
>reduce) when working with custom jar files and s3
>Are you using EMR or something internal and offloading strage to s3?

We have an S3-compatable object store made by Scality
(http://www.scality.com/). A so-called ‘ring’. It’s basically a pile of
linux boxes that behaves like our own internal S3 ‘cloud'. It lives in our
data-center.

What I’d like to do is have hadoop connect to that *instead* of the Amazon
AWS S3.

Yet, no matter how I set the fs.s3a.endpoint directive, it still connects
to Amazon’s S3.

Hope that clarifies,

—
Caleb

>---
>Regards,
>Jonathan Aquilina
>Founder
>
>
> 
>On 2016-01-13 23:21, Phillips, Caleb wrote:
>
>Hi Billy (and others),
>
>One of the threads suggested using the core-site.xml. Did you try putting
>your configuration in there?
>
>Yes, I did try that. I've also tried setting it dynamically in e.g.,
>spark. I can verify that it is getting the configuration correctly:
>
>hadoop org.apache.hadoop.conf.Configuration
>
>Still it never connects to our internal S3-compatable store and always
>connects to AWS.
>
>One thing I've noticed is that the AWS stuff is handled by an underlying
>library (I think jets3t in < 2.6 versions, forget what in 2.6+) and when
>I was trying to mess with stuff and spelunking through the hadoop code, I
>kept running into blocks with that library.
>
>I started digging into the code. I found that the custom endpoint was
>introduced with this patch:
>
>https://issues.apache.org/jira/browse/HADOOP-11261
>
>It seems it was integrated in 2.7.0, so just to be sure I downloaded
>2.7.1, but the problem persists.
>
>That code calls this function in the AWS Java SDK:
>
>http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/service
>s/s3/AmazonS3Client.html#setEndpoint(java.lang.String)
>
>However, no matter what configuration I use, it still seems to connect to
>Amazon AWS. Is it possible that the AWS Java SDK cannot work with
>S3-compatable (non-AWS) stores? If so, it would seem there is no way
>currently to connect hadoop to an S3-compatable
> non-AWS store.
>
>If anyone else has any insight, particularly success using hadoop with a
>non-AWS, S3-compatable store, please chime in!
>
>
>William Watson
>Software Engineer
>(904) 705-7056 PCS
>
>On Mon, Jan 11, 2016 at 10:39 AM, Phillips, Caleb
><Ca...@nrel.gov>> wrote:
>Hi All,
>
>Just wanted to send this out again since there was no response
>(admittedly, originally sent in the midst of the US holiday season) and it
>seems to be an issue that continues to come up (see e.g., the email from
>Han Ju on Jan 5).
>
>If anyone has successfully connected Hadoop to a non-AWS S3-compatable
>object store, it'd be very helpful to hear how you made it work. The
>fs.s3a.endpoint configuration directive appears non-functional at our site
>(with Hadoop 2.6.3).
>
>--
>Caleb Phillips, Ph.D.
>Data Scientist | Computational Science Center
>
>National Renewable Energy Laboratory (NREL)
>15013 Denver West Parkway | Golden, CO 80401
>303-275-4297<tel:303-275-4297> |
>caleb.phillips@nrel.gov<ma...@nrel.gov>
>
>
>
>
>
>
>On 12/22/15, 1:39 PM, "Phillips, Caleb"
><Ca...@nrel.gov>> wrote:
>
>
>Hi All,
>
>New to this list. Looking for a bit of help:
>
>I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
>store.
>
>This issue was discussed, but left unresolved, in this thread:
>
>https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_
>Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com<mailto:Au5Es_
>fLUgZMGwkkgA3JyA1ASi3u%2BisJCuYmfnTvNkGuQ@mail.gmail.com>%3E
>
>And here, on Cloudera's forums (the second post is mine):
>
>https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoi
>nt-ignored-in-hdfs-site-xml/m-p/33694#M1180
>
>I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
>Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>
>However, when I point the fs.s3a.endpoint configuration directive at my
>non-AWS S3-Compatable object storage, it appears to still point at (and
>authenticate against) AWS.
>
>I've checked and double-checked my credentials and configuration using
>both Python's boto library and the s3cmd tool, both of which connect to
>this non-AWS data store just fine.
>
>Any help would be much appreciated. Thanks!
>
>--
>Caleb Phillips, Ph.D.
>Data Scientist | Computational Science Center
>
>National Renewable Energy Laboratory (NREL)
>15013 Denver West Parkway | Golden, CO 80401
>303-275-4297 | caleb.phillips@nrel.gov<ma...@nrel.gov>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail:
>user-unsubscribe@hadoop.apache.org<mailto:user-unsubscribe@hadoop.apache.o
>rg>
>For additional commands, e-mail:
>user-help@hadoop.apache.org<ma...@hadoop.apache.org>
>
>
>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: user-help@hadoop.apache.org

Re: fs.s3a.endpoint not working

Posted by Jonathan Aquilina <ja...@eagleeyet.net>.

Im not totally following this thread from the beginning but I might be
able to help as I have some experience with Amazon EMR (elastic map
reduce) when working with custom jar files and s3 

Are you using EMR or something internal and offloading strage to s3?

---
Regards,
Jonathan Aquilina
Founder 

On 2016-01-13 23:21, Phillips, Caleb wrote:

> Hi Billy (and others),
> 
> One of the threads suggested using the core-site.xml. Did you try putting your configuration in there?
> 
> Yes, I did try that. I've also tried setting it dynamically in e.g., spark. I can verify that it is getting the configuration correctly:
> 
> hadoop org.apache.hadoop.conf.Configuration
> 
> Still it never connects to our internal S3-compatable store and always connects to AWS.
> 
> One thing I've noticed is that the AWS stuff is handled by an underlying library (I think jets3t in < 2.6 versions, forget what in 2.6+) and when I was trying to mess with stuff and spelunking through the hadoop code, I kept running into blocks with that library.
> 
> I started digging into the code. I found that the custom endpoint was introduced with this patch:
> 
> https://issues.apache.org/jira/browse/HADOOP-11261
> 
> It seems it was integrated in 2.7.0, so just to be sure I downloaded 2.7.1, but the problem persists.
> 
> That code calls this function in the AWS Java SDK:
> 
> http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html#setEndpoint(java.lang.String)
> 
> However, no matter what configuration I use, it still seems to connect to Amazon AWS. Is it possible that the AWS Java SDK cannot work with S3-compatable (non-AWS) stores? If so, it would seem there is no way currently to connect hadoop to an S3-compatable non-AWS store.
> 
> If anyone else has any insight, particularly success using hadoop with a non-AWS, S3-compatable store, please chime in!
> 
> William Watson
> Software Engineer
> (904) 705-7056 PCS
> 
> On Mon, Jan 11, 2016 at 10:39 AM, Phillips, Caleb <Ca...@nrel.gov>> wrote:
> Hi All,
> 
> Just wanted to send this out again since there was no response
> (admittedly, originally sent in the midst of the US holiday season) and it
> seems to be an issue that continues to come up (see e.g., the email from
> Han Ju on Jan 5).
> 
> If anyone has successfully connected Hadoop to a non-AWS S3-compatable
> object store, it'd be very helpful to hear how you made it work. The
> fs.s3a.endpoint configuration directive appears non-functional at our site
> (with Hadoop 2.6.3).
> 
> --
> Caleb Phillips, Ph.D.
> Data Scientist | Computational Science Center
> 
> National Renewable Energy Laboratory (NREL)
> 15013 Denver West Parkway | Golden, CO 80401
> 303-275-4297<tel:303-275-4297> | caleb.phillips@nrel.gov<ma...@nrel.gov>
> 
> On 12/22/15, 1:39 PM, "Phillips, Caleb" <Ca...@nrel.gov>> wrote:
> 
>> Hi All,
>> 
>> New to this list. Looking for a bit of help:
>> 
>> I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
>> store.
>> 
>> This issue was discussed, but left unresolved, in this thread:
>> 
>> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_
>> Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com<ma...@mail.gmail.com>%3E
>> 
>> And here, on Cloudera's forums (the second post is mine):
>> 
>> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoi
>> nt-ignored-in-hdfs-site-xml/m-p/33694#M1180
>> 
>> I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
>> Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>> 
>> However, when I point the fs.s3a.endpoint configuration directive at my
>> non-AWS S3-Compatable object storage, it appears to still point at (and
>> authenticate against) AWS.
>> 
>> I've checked and double-checked my credentials and configuration using
>> both Python's boto library and the s3cmd tool, both of which connect to
>> this non-AWS data store just fine.
>> 
>> Any help would be much appreciated. Thanks!
>> 
>> --
>> Caleb Phillips, Ph.D.
>> Data Scientist | Computational Science Center
>> 
>> National Renewable Energy Laboratory (NREL)
>> 15013 Denver West Parkway | Golden, CO 80401
>> 303-275-4297 | caleb.phillips@nrel.gov<ma...@nrel.gov>
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org<ma...@hadoop.apache.org>
>> For additional commands, e-mail: user-help@hadoop.apache.org<ma...@hadoop.apache.org>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org

Re: fs.s3a.endpoint not working

Posted by Jonathan Aquilina <ja...@eagleeyet.net>.

Im not totally following this thread from the beginning but I might be
able to help as I have some experience with Amazon EMR (elastic map
reduce) when working with custom jar files and s3 

Are you using EMR or something internal and offloading strage to s3?

---
Regards,
Jonathan Aquilina
Founder 

On 2016-01-13 23:21, Phillips, Caleb wrote:

> Hi Billy (and others),
> 
> One of the threads suggested using the core-site.xml. Did you try putting your configuration in there?
> 
> Yes, I did try that. I've also tried setting it dynamically in e.g., spark. I can verify that it is getting the configuration correctly:
> 
> hadoop org.apache.hadoop.conf.Configuration
> 
> Still it never connects to our internal S3-compatable store and always connects to AWS.
> 
> One thing I've noticed is that the AWS stuff is handled by an underlying library (I think jets3t in < 2.6 versions, forget what in 2.6+) and when I was trying to mess with stuff and spelunking through the hadoop code, I kept running into blocks with that library.
> 
> I started digging into the code. I found that the custom endpoint was introduced with this patch:
> 
> https://issues.apache.org/jira/browse/HADOOP-11261
> 
> It seems it was integrated in 2.7.0, so just to be sure I downloaded 2.7.1, but the problem persists.
> 
> That code calls this function in the AWS Java SDK:
> 
> http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html#setEndpoint(java.lang.String)
> 
> However, no matter what configuration I use, it still seems to connect to Amazon AWS. Is it possible that the AWS Java SDK cannot work with S3-compatable (non-AWS) stores? If so, it would seem there is no way currently to connect hadoop to an S3-compatable non-AWS store.
> 
> If anyone else has any insight, particularly success using hadoop with a non-AWS, S3-compatable store, please chime in!
> 
> William Watson
> Software Engineer
> (904) 705-7056 PCS
> 
> On Mon, Jan 11, 2016 at 10:39 AM, Phillips, Caleb <Ca...@nrel.gov>> wrote:
> Hi All,
> 
> Just wanted to send this out again since there was no response
> (admittedly, originally sent in the midst of the US holiday season) and it
> seems to be an issue that continues to come up (see e.g., the email from
> Han Ju on Jan 5).
> 
> If anyone has successfully connected Hadoop to a non-AWS S3-compatable
> object store, it'd be very helpful to hear how you made it work. The
> fs.s3a.endpoint configuration directive appears non-functional at our site
> (with Hadoop 2.6.3).
> 
> --
> Caleb Phillips, Ph.D.
> Data Scientist | Computational Science Center
> 
> National Renewable Energy Laboratory (NREL)
> 15013 Denver West Parkway | Golden, CO 80401
> 303-275-4297<tel:303-275-4297> | caleb.phillips@nrel.gov<ma...@nrel.gov>
> 
> On 12/22/15, 1:39 PM, "Phillips, Caleb" <Ca...@nrel.gov>> wrote:
> 
>> Hi All,
>> 
>> New to this list. Looking for a bit of help:
>> 
>> I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
>> store.
>> 
>> This issue was discussed, but left unresolved, in this thread:
>> 
>> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_
>> Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com<ma...@mail.gmail.com>%3E
>> 
>> And here, on Cloudera's forums (the second post is mine):
>> 
>> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoi
>> nt-ignored-in-hdfs-site-xml/m-p/33694#M1180
>> 
>> I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
>> Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>> 
>> However, when I point the fs.s3a.endpoint configuration directive at my
>> non-AWS S3-Compatable object storage, it appears to still point at (and
>> authenticate against) AWS.
>> 
>> I've checked and double-checked my credentials and configuration using
>> both Python's boto library and the s3cmd tool, both of which connect to
>> this non-AWS data store just fine.
>> 
>> Any help would be much appreciated. Thanks!
>> 
>> --
>> Caleb Phillips, Ph.D.
>> Data Scientist | Computational Science Center
>> 
>> National Renewable Energy Laboratory (NREL)
>> 15013 Denver West Parkway | Golden, CO 80401
>> 303-275-4297 | caleb.phillips@nrel.gov<ma...@nrel.gov>
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org<ma...@hadoop.apache.org>
>> For additional commands, e-mail: user-help@hadoop.apache.org<ma...@hadoop.apache.org>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org

Re: fs.s3a.endpoint not working

Posted by Jonathan Aquilina <ja...@eagleeyet.net>.

Im not totally following this thread from the beginning but I might be
able to help as I have some experience with Amazon EMR (elastic map
reduce) when working with custom jar files and s3 

Are you using EMR or something internal and offloading strage to s3?

---
Regards,
Jonathan Aquilina
Founder 

On 2016-01-13 23:21, Phillips, Caleb wrote:

> Hi Billy (and others),
> 
> One of the threads suggested using the core-site.xml. Did you try putting your configuration in there?
> 
> Yes, I did try that. I've also tried setting it dynamically in e.g., spark. I can verify that it is getting the configuration correctly:
> 
> hadoop org.apache.hadoop.conf.Configuration
> 
> Still it never connects to our internal S3-compatable store and always connects to AWS.
> 
> One thing I've noticed is that the AWS stuff is handled by an underlying library (I think jets3t in < 2.6 versions, forget what in 2.6+) and when I was trying to mess with stuff and spelunking through the hadoop code, I kept running into blocks with that library.
> 
> I started digging into the code. I found that the custom endpoint was introduced with this patch:
> 
> https://issues.apache.org/jira/browse/HADOOP-11261
> 
> It seems it was integrated in 2.7.0, so just to be sure I downloaded 2.7.1, but the problem persists.
> 
> That code calls this function in the AWS Java SDK:
> 
> http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html#setEndpoint(java.lang.String)
> 
> However, no matter what configuration I use, it still seems to connect to Amazon AWS. Is it possible that the AWS Java SDK cannot work with S3-compatable (non-AWS) stores? If so, it would seem there is no way currently to connect hadoop to an S3-compatable non-AWS store.
> 
> If anyone else has any insight, particularly success using hadoop with a non-AWS, S3-compatable store, please chime in!
> 
> William Watson
> Software Engineer
> (904) 705-7056 PCS
> 
> On Mon, Jan 11, 2016 at 10:39 AM, Phillips, Caleb <Ca...@nrel.gov>> wrote:
> Hi All,
> 
> Just wanted to send this out again since there was no response
> (admittedly, originally sent in the midst of the US holiday season) and it
> seems to be an issue that continues to come up (see e.g., the email from
> Han Ju on Jan 5).
> 
> If anyone has successfully connected Hadoop to a non-AWS S3-compatable
> object store, it'd be very helpful to hear how you made it work. The
> fs.s3a.endpoint configuration directive appears non-functional at our site
> (with Hadoop 2.6.3).
> 
> --
> Caleb Phillips, Ph.D.
> Data Scientist | Computational Science Center
> 
> National Renewable Energy Laboratory (NREL)
> 15013 Denver West Parkway | Golden, CO 80401
> 303-275-4297<tel:303-275-4297> | caleb.phillips@nrel.gov<ma...@nrel.gov>
> 
> On 12/22/15, 1:39 PM, "Phillips, Caleb" <Ca...@nrel.gov>> wrote:
> 
>> Hi All,
>> 
>> New to this list. Looking for a bit of help:
>> 
>> I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
>> store.
>> 
>> This issue was discussed, but left unresolved, in this thread:
>> 
>> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_
>> Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com<ma...@mail.gmail.com>%3E
>> 
>> And here, on Cloudera's forums (the second post is mine):
>> 
>> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoi
>> nt-ignored-in-hdfs-site-xml/m-p/33694#M1180
>> 
>> I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
>> Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>> 
>> However, when I point the fs.s3a.endpoint configuration directive at my
>> non-AWS S3-Compatable object storage, it appears to still point at (and
>> authenticate against) AWS.
>> 
>> I've checked and double-checked my credentials and configuration using
>> both Python's boto library and the s3cmd tool, both of which connect to
>> this non-AWS data store just fine.
>> 
>> Any help would be much appreciated. Thanks!
>> 
>> --
>> Caleb Phillips, Ph.D.
>> Data Scientist | Computational Science Center
>> 
>> National Renewable Energy Laboratory (NREL)
>> 15013 Denver West Parkway | Golden, CO 80401
>> 303-275-4297 | caleb.phillips@nrel.gov<ma...@nrel.gov>
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org<ma...@hadoop.apache.org>
>> For additional commands, e-mail: user-help@hadoop.apache.org<ma...@hadoop.apache.org>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org

Re: fs.s3a.endpoint not working

Posted by Jonathan Aquilina <ja...@eagleeyet.net>.

Im not totally following this thread from the beginning but I might be
able to help as I have some experience with Amazon EMR (elastic map
reduce) when working with custom jar files and s3 

Are you using EMR or something internal and offloading strage to s3?

---
Regards,
Jonathan Aquilina
Founder 

On 2016-01-13 23:21, Phillips, Caleb wrote:

> Hi Billy (and others),
> 
> One of the threads suggested using the core-site.xml. Did you try putting your configuration in there?
> 
> Yes, I did try that. I've also tried setting it dynamically in e.g., spark. I can verify that it is getting the configuration correctly:
> 
> hadoop org.apache.hadoop.conf.Configuration
> 
> Still it never connects to our internal S3-compatable store and always connects to AWS.
> 
> One thing I've noticed is that the AWS stuff is handled by an underlying library (I think jets3t in < 2.6 versions, forget what in 2.6+) and when I was trying to mess with stuff and spelunking through the hadoop code, I kept running into blocks with that library.
> 
> I started digging into the code. I found that the custom endpoint was introduced with this patch:
> 
> https://issues.apache.org/jira/browse/HADOOP-11261
> 
> It seems it was integrated in 2.7.0, so just to be sure I downloaded 2.7.1, but the problem persists.
> 
> That code calls this function in the AWS Java SDK:
> 
> http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html#setEndpoint(java.lang.String)
> 
> However, no matter what configuration I use, it still seems to connect to Amazon AWS. Is it possible that the AWS Java SDK cannot work with S3-compatable (non-AWS) stores? If so, it would seem there is no way currently to connect hadoop to an S3-compatable non-AWS store.
> 
> If anyone else has any insight, particularly success using hadoop with a non-AWS, S3-compatable store, please chime in!
> 
> William Watson
> Software Engineer
> (904) 705-7056 PCS
> 
> On Mon, Jan 11, 2016 at 10:39 AM, Phillips, Caleb <Ca...@nrel.gov>> wrote:
> Hi All,
> 
> Just wanted to send this out again since there was no response
> (admittedly, originally sent in the midst of the US holiday season) and it
> seems to be an issue that continues to come up (see e.g., the email from
> Han Ju on Jan 5).
> 
> If anyone has successfully connected Hadoop to a non-AWS S3-compatable
> object store, it'd be very helpful to hear how you made it work. The
> fs.s3a.endpoint configuration directive appears non-functional at our site
> (with Hadoop 2.6.3).
> 
> --
> Caleb Phillips, Ph.D.
> Data Scientist | Computational Science Center
> 
> National Renewable Energy Laboratory (NREL)
> 15013 Denver West Parkway | Golden, CO 80401
> 303-275-4297<tel:303-275-4297> | caleb.phillips@nrel.gov<ma...@nrel.gov>
> 
> On 12/22/15, 1:39 PM, "Phillips, Caleb" <Ca...@nrel.gov>> wrote:
> 
>> Hi All,
>> 
>> New to this list. Looking for a bit of help:
>> 
>> I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
>> store.
>> 
>> This issue was discussed, but left unresolved, in this thread:
>> 
>> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_
>> Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com<ma...@mail.gmail.com>%3E
>> 
>> And here, on Cloudera's forums (the second post is mine):
>> 
>> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoi
>> nt-ignored-in-hdfs-site-xml/m-p/33694#M1180
>> 
>> I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
>> Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>> 
>> However, when I point the fs.s3a.endpoint configuration directive at my
>> non-AWS S3-Compatable object storage, it appears to still point at (and
>> authenticate against) AWS.
>> 
>> I've checked and double-checked my credentials and configuration using
>> both Python's boto library and the s3cmd tool, both of which connect to
>> this non-AWS data store just fine.
>> 
>> Any help would be much appreciated. Thanks!
>> 
>> --
>> Caleb Phillips, Ph.D.
>> Data Scientist | Computational Science Center
>> 
>> National Renewable Energy Laboratory (NREL)
>> 15013 Denver West Parkway | Golden, CO 80401
>> 303-275-4297 | caleb.phillips@nrel.gov<ma...@nrel.gov>
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org<ma...@hadoop.apache.org>
>> For additional commands, e-mail: user-help@hadoop.apache.org<ma...@hadoop.apache.org>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org

Re: fs.s3a.endpoint not working

Posted by "Phillips, Caleb" <Ca...@nrel.gov>.

Hi Billy (and others),

One of the threads suggested using the core-site.xml. Did you try putting your configuration in there?

Yes, I did try that. I’ve also tried setting it dynamically in e.g., spark. I can verify that it is getting the configuration correctly:

hadoop org.apache.hadoop.conf.Configuration

Still it never connects to our internal S3-compatable store and always connects to AWS.

One thing I've noticed is that the AWS stuff is handled by an underlying library (I think jets3t in < 2.6 versions, forget what in 2.6+) and when I was trying to mess with stuff and spelunking through the hadoop code, I kept running into blocks with that library.

I started digging into the code. I found that the custom endpoint was introduced with this patch:

https://issues.apache.org/jira/browse/HADOOP-11261

It seems it was integrated in 2.7.0, so just to be sure I downloaded 2.7.1, but the problem persists.

That code calls this function in the AWS Java SDK:

http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html#setEndpoint(java.lang.String)

However, no matter what configuration I use, it still seems to connect to Amazon AWS. Is it possible that the AWS Java SDK cannot work with S3-compatable (non-AWS) stores? If so, it would seem there is no way currently to connect hadoop to an S3-compatable non-AWS store.

If anyone else has any insight, particularly success using hadoop with a non-AWS, S3-compatable store, please chime in!

William Watson
Software Engineer
(904) 705-7056 PCS

On Mon, Jan 11, 2016 at 10:39 AM, Phillips, Caleb <Ca...@nrel.gov>> wrote:
Hi All,

Just wanted to send this out again since there was no response
(admittedly, originally sent in the midst of the US holiday season) and it
seems to be an issue that continues to come up (see e.g., the email from
Han Ju on Jan 5).

If anyone has successfully connected Hadoop to a non-AWS S3-compatable
object store, it’d be very helpful to hear how you made it work. The
fs.s3a.endpoint configuration directive appears non-functional at our site
(with Hadoop 2.6.3).

--
Caleb Phillips, Ph.D.
Data Scientist | Computational Science Center

National Renewable Energy Laboratory (NREL)
15013 Denver West Parkway | Golden, CO 80401
303-275-4297<tel:303-275-4297> | caleb.phillips@nrel.gov<ma...@nrel.gov>

On 12/22/15, 1:39 PM, "Phillips, Caleb" <Ca...@nrel.gov>> wrote:

>Hi All,
>
>New to this list. Looking for a bit of help:
>
>I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
>store.
>
>This issue was discussed, but left unresolved, in this thread:
>
>https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_
>Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com<ma...@mail.gmail.com>%3E
>
>And here, on Cloudera's forums (the second post is mine):
>
>https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoi
>nt-ignored-in-hdfs-site-xml/m-p/33694#M1180
>
>I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
>Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>
>However, when I point the fs.s3a.endpoint configuration directive at my
>non-AWS S3-Compatable object storage, it appears to still point at (and
>authenticate against) AWS.
>
>I've checked and double-checked my credentials and configuration using
>both Python's boto library and the s3cmd tool, both of which connect to
>this non-AWS data store just fine.
>
>Any help would be much appreciated. Thanks!
>
>--
>Caleb Phillips, Ph.D.
>Data Scientist | Computational Science Center
>
>National Renewable Energy Laboratory (NREL)
>15013 Denver West Parkway | Golden, CO 80401
>303-275-4297 | caleb.phillips@nrel.gov<ma...@nrel.gov>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org<ma...@hadoop.apache.org>
>For additional commands, e-mail: user-help@hadoop.apache.org<ma...@hadoop.apache.org>
>

Re: fs.s3a.endpoint not working

Posted by "Phillips, Caleb" <Ca...@nrel.gov>.

Hi Billy (and others),

One of the threads suggested using the core-site.xml. Did you try putting your configuration in there?

Yes, I did try that. I’ve also tried setting it dynamically in e.g., spark. I can verify that it is getting the configuration correctly:

hadoop org.apache.hadoop.conf.Configuration

Still it never connects to our internal S3-compatable store and always connects to AWS.

One thing I've noticed is that the AWS stuff is handled by an underlying library (I think jets3t in < 2.6 versions, forget what in 2.6+) and when I was trying to mess with stuff and spelunking through the hadoop code, I kept running into blocks with that library.

I started digging into the code. I found that the custom endpoint was introduced with this patch:

https://issues.apache.org/jira/browse/HADOOP-11261

It seems it was integrated in 2.7.0, so just to be sure I downloaded 2.7.1, but the problem persists.

That code calls this function in the AWS Java SDK:

http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html#setEndpoint(java.lang.String)

However, no matter what configuration I use, it still seems to connect to Amazon AWS. Is it possible that the AWS Java SDK cannot work with S3-compatable (non-AWS) stores? If so, it would seem there is no way currently to connect hadoop to an S3-compatable non-AWS store.

If anyone else has any insight, particularly success using hadoop with a non-AWS, S3-compatable store, please chime in!

William Watson
Software Engineer
(904) 705-7056 PCS

On Mon, Jan 11, 2016 at 10:39 AM, Phillips, Caleb <Ca...@nrel.gov>> wrote:
Hi All,

Just wanted to send this out again since there was no response
(admittedly, originally sent in the midst of the US holiday season) and it
seems to be an issue that continues to come up (see e.g., the email from
Han Ju on Jan 5).

If anyone has successfully connected Hadoop to a non-AWS S3-compatable
object store, it’d be very helpful to hear how you made it work. The
fs.s3a.endpoint configuration directive appears non-functional at our site
(with Hadoop 2.6.3).

--
Caleb Phillips, Ph.D.
Data Scientist | Computational Science Center

National Renewable Energy Laboratory (NREL)
15013 Denver West Parkway | Golden, CO 80401
303-275-4297<tel:303-275-4297> | caleb.phillips@nrel.gov<ma...@nrel.gov>

On 12/22/15, 1:39 PM, "Phillips, Caleb" <Ca...@nrel.gov>> wrote:

>Hi All,
>
>New to this list. Looking for a bit of help:
>
>I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
>store.
>
>This issue was discussed, but left unresolved, in this thread:
>
>https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_
>Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com<ma...@mail.gmail.com>%3E
>
>And here, on Cloudera's forums (the second post is mine):
>
>https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoi
>nt-ignored-in-hdfs-site-xml/m-p/33694#M1180
>
>I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
>Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>
>However, when I point the fs.s3a.endpoint configuration directive at my
>non-AWS S3-Compatable object storage, it appears to still point at (and
>authenticate against) AWS.
>
>I've checked and double-checked my credentials and configuration using
>both Python's boto library and the s3cmd tool, both of which connect to
>this non-AWS data store just fine.
>
>Any help would be much appreciated. Thanks!
>
>--
>Caleb Phillips, Ph.D.
>Data Scientist | Computational Science Center
>
>National Renewable Energy Laboratory (NREL)
>15013 Denver West Parkway | Golden, CO 80401
>303-275-4297 | caleb.phillips@nrel.gov<ma...@nrel.gov>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org<ma...@hadoop.apache.org>
>For additional commands, e-mail: user-help@hadoop.apache.org<ma...@hadoop.apache.org>
>

Re: fs.s3a.endpoint not working

Posted by "Phillips, Caleb" <Ca...@nrel.gov>.

Hi Billy (and others),

One of the threads suggested using the core-site.xml. Did you try putting your configuration in there?

Yes, I did try that. I’ve also tried setting it dynamically in e.g., spark. I can verify that it is getting the configuration correctly:

hadoop org.apache.hadoop.conf.Configuration

Still it never connects to our internal S3-compatable store and always connects to AWS.

One thing I've noticed is that the AWS stuff is handled by an underlying library (I think jets3t in < 2.6 versions, forget what in 2.6+) and when I was trying to mess with stuff and spelunking through the hadoop code, I kept running into blocks with that library.

I started digging into the code. I found that the custom endpoint was introduced with this patch:

https://issues.apache.org/jira/browse/HADOOP-11261

It seems it was integrated in 2.7.0, so just to be sure I downloaded 2.7.1, but the problem persists.

That code calls this function in the AWS Java SDK:

http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html#setEndpoint(java.lang.String)

However, no matter what configuration I use, it still seems to connect to Amazon AWS. Is it possible that the AWS Java SDK cannot work with S3-compatable (non-AWS) stores? If so, it would seem there is no way currently to connect hadoop to an S3-compatable non-AWS store.

If anyone else has any insight, particularly success using hadoop with a non-AWS, S3-compatable store, please chime in!

William Watson
Software Engineer
(904) 705-7056 PCS

On Mon, Jan 11, 2016 at 10:39 AM, Phillips, Caleb <Ca...@nrel.gov>> wrote:
Hi All,

Just wanted to send this out again since there was no response
(admittedly, originally sent in the midst of the US holiday season) and it
seems to be an issue that continues to come up (see e.g., the email from
Han Ju on Jan 5).

If anyone has successfully connected Hadoop to a non-AWS S3-compatable
object store, it’d be very helpful to hear how you made it work. The
fs.s3a.endpoint configuration directive appears non-functional at our site
(with Hadoop 2.6.3).

--
Caleb Phillips, Ph.D.
Data Scientist | Computational Science Center

National Renewable Energy Laboratory (NREL)
15013 Denver West Parkway | Golden, CO 80401
303-275-4297<tel:303-275-4297> | caleb.phillips@nrel.gov<ma...@nrel.gov>

On 12/22/15, 1:39 PM, "Phillips, Caleb" <Ca...@nrel.gov>> wrote:

>Hi All,
>
>New to this list. Looking for a bit of help:
>
>I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
>store.
>
>This issue was discussed, but left unresolved, in this thread:
>
>https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_
>Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com<ma...@mail.gmail.com>%3E
>
>And here, on Cloudera's forums (the second post is mine):
>
>https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoi
>nt-ignored-in-hdfs-site-xml/m-p/33694#M1180
>
>I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
>Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>
>However, when I point the fs.s3a.endpoint configuration directive at my
>non-AWS S3-Compatable object storage, it appears to still point at (and
>authenticate against) AWS.
>
>I've checked and double-checked my credentials and configuration using
>both Python's boto library and the s3cmd tool, both of which connect to
>this non-AWS data store just fine.
>
>Any help would be much appreciated. Thanks!
>
>--
>Caleb Phillips, Ph.D.
>Data Scientist | Computational Science Center
>
>National Renewable Energy Laboratory (NREL)
>15013 Denver West Parkway | Golden, CO 80401
>303-275-4297 | caleb.phillips@nrel.gov<ma...@nrel.gov>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org<ma...@hadoop.apache.org>
>For additional commands, e-mail: user-help@hadoop.apache.org<ma...@hadoop.apache.org>
>

Re: fs.s3a.endpoint not working

Posted by "Phillips, Caleb" <Ca...@nrel.gov>.

Hi Billy (and others),

One of the threads suggested using the core-site.xml. Did you try putting your configuration in there?

Yes, I did try that. I’ve also tried setting it dynamically in e.g., spark. I can verify that it is getting the configuration correctly:

hadoop org.apache.hadoop.conf.Configuration

Still it never connects to our internal S3-compatable store and always connects to AWS.

One thing I've noticed is that the AWS stuff is handled by an underlying library (I think jets3t in < 2.6 versions, forget what in 2.6+) and when I was trying to mess with stuff and spelunking through the hadoop code, I kept running into blocks with that library.

I started digging into the code. I found that the custom endpoint was introduced with this patch:

https://issues.apache.org/jira/browse/HADOOP-11261

It seems it was integrated in 2.7.0, so just to be sure I downloaded 2.7.1, but the problem persists.

That code calls this function in the AWS Java SDK:

http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html#setEndpoint(java.lang.String)

However, no matter what configuration I use, it still seems to connect to Amazon AWS. Is it possible that the AWS Java SDK cannot work with S3-compatable (non-AWS) stores? If so, it would seem there is no way currently to connect hadoop to an S3-compatable non-AWS store.

If anyone else has any insight, particularly success using hadoop with a non-AWS, S3-compatable store, please chime in!

William Watson
Software Engineer
(904) 705-7056 PCS

On Mon, Jan 11, 2016 at 10:39 AM, Phillips, Caleb <Ca...@nrel.gov>> wrote:
Hi All,

Just wanted to send this out again since there was no response
(admittedly, originally sent in the midst of the US holiday season) and it
seems to be an issue that continues to come up (see e.g., the email from
Han Ju on Jan 5).

If anyone has successfully connected Hadoop to a non-AWS S3-compatable
object store, it’d be very helpful to hear how you made it work. The
fs.s3a.endpoint configuration directive appears non-functional at our site
(with Hadoop 2.6.3).

--
Caleb Phillips, Ph.D.
Data Scientist | Computational Science Center

National Renewable Energy Laboratory (NREL)
15013 Denver West Parkway | Golden, CO 80401
303-275-4297<tel:303-275-4297> | caleb.phillips@nrel.gov<ma...@nrel.gov>

On 12/22/15, 1:39 PM, "Phillips, Caleb" <Ca...@nrel.gov>> wrote:

>Hi All,
>
>New to this list. Looking for a bit of help:
>
>I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
>store.
>
>This issue was discussed, but left unresolved, in this thread:
>
>https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_
>Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com<ma...@mail.gmail.com>%3E
>
>And here, on Cloudera's forums (the second post is mine):
>
>https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoi
>nt-ignored-in-hdfs-site-xml/m-p/33694#M1180
>
>I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
>Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>
>However, when I point the fs.s3a.endpoint configuration directive at my
>non-AWS S3-Compatable object storage, it appears to still point at (and
>authenticate against) AWS.
>
>I've checked and double-checked my credentials and configuration using
>both Python's boto library and the s3cmd tool, both of which connect to
>this non-AWS data store just fine.
>
>Any help would be much appreciated. Thanks!
>
>--
>Caleb Phillips, Ph.D.
>Data Scientist | Computational Science Center
>
>National Renewable Energy Laboratory (NREL)
>15013 Denver West Parkway | Golden, CO 80401
>303-275-4297 | caleb.phillips@nrel.gov<ma...@nrel.gov>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org<ma...@hadoop.apache.org>
>For additional commands, e-mail: user-help@hadoop.apache.org<ma...@hadoop.apache.org>
>

Re: fs.s3a.endpoint not working

Posted by Billy Watson <wi...@gmail.com>.

One of the threads suggested using the core-site.xml. Did you try putting
your configuration in there?

One thing I've noticed is that the AWS stuff is handled by an underlying
library (I think jets3t in < 2.6 versions, forget what in 2.6+) and when I
was trying to mess with stuff and spelunking through the hadoop code, I
kept running into blocks with that library.

William Watson
Software Engineer
(904) 705-7056 PCS

On Mon, Jan 11, 2016 at 10:39 AM, Phillips, Caleb <Ca...@nrel.gov>
wrote:

> Hi All,
>
> Just wanted to send this out again since there was no response
> (admittedly, originally sent in the midst of the US holiday season) and it
> seems to be an issue that continues to come up (see e.g., the email from
> Han Ju on Jan 5).
>
> If anyone has successfully connected Hadoop to a non-AWS S3-compatable
> object store, it’d be very helpful to hear how you made it work. The
> fs.s3a.endpoint configuration directive appears non-functional at our site
> (with Hadoop 2.6.3).
>
> --
> Caleb Phillips, Ph.D.
> Data Scientist | Computational Science Center
>
> National Renewable Energy Laboratory (NREL)
> 15013 Denver West Parkway | Golden, CO 80401
> 303-275-4297 | caleb.phillips@nrel.gov
>
>
>
>
>
>
> On 12/22/15, 1:39 PM, "Phillips, Caleb" <Ca...@nrel.gov> wrote:
>
> >Hi All,
> >
> >New to this list. Looking for a bit of help:
> >
> >I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
> >store.
> >
> >This issue was discussed, but left unresolved, in this thread:
> >
> >
> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_
> >Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E
> >
> >And here, on Cloudera's forums (the second post is mine):
> >
> >
> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoi
> >nt-ignored-in-hdfs-site-xml/m-p/33694#M1180
> >
> >I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
> >Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
> >
> >However, when I point the fs.s3a.endpoint configuration directive at my
> >non-AWS S3-Compatable object storage, it appears to still point at (and
> >authenticate against) AWS.
> >
> >I've checked and double-checked my credentials and configuration using
> >both Python's boto library and the s3cmd tool, both of which connect to
> >this non-AWS data store just fine.
> >
> >Any help would be much appreciated. Thanks!
> >
> >--
> >Caleb Phillips, Ph.D.
> >Data Scientist | Computational Science Center
> >
> >National Renewable Energy Laboratory (NREL)
> >15013 Denver West Parkway | Golden, CO 80401
> >303-275-4297 | caleb.phillips@nrel.gov
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> >For additional commands, e-mail: user-help@hadoop.apache.org
> >
>
>

Re: fs.s3a.endpoint not working

Posted by Billy Watson <wi...@gmail.com>.

One of the threads suggested using the core-site.xml. Did you try putting
your configuration in there?

One thing I've noticed is that the AWS stuff is handled by an underlying
library (I think jets3t in < 2.6 versions, forget what in 2.6+) and when I
was trying to mess with stuff and spelunking through the hadoop code, I
kept running into blocks with that library.

William Watson
Software Engineer
(904) 705-7056 PCS

On Mon, Jan 11, 2016 at 10:39 AM, Phillips, Caleb <Ca...@nrel.gov>
wrote:

> Hi All,
>
> Just wanted to send this out again since there was no response
> (admittedly, originally sent in the midst of the US holiday season) and it
> seems to be an issue that continues to come up (see e.g., the email from
> Han Ju on Jan 5).
>
> If anyone has successfully connected Hadoop to a non-AWS S3-compatable
> object store, it’d be very helpful to hear how you made it work. The
> fs.s3a.endpoint configuration directive appears non-functional at our site
> (with Hadoop 2.6.3).
>
> --
> Caleb Phillips, Ph.D.
> Data Scientist | Computational Science Center
>
> National Renewable Energy Laboratory (NREL)
> 15013 Denver West Parkway | Golden, CO 80401
> 303-275-4297 | caleb.phillips@nrel.gov
>
>
>
>
>
>
> On 12/22/15, 1:39 PM, "Phillips, Caleb" <Ca...@nrel.gov> wrote:
>
> >Hi All,
> >
> >New to this list. Looking for a bit of help:
> >
> >I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
> >store.
> >
> >This issue was discussed, but left unresolved, in this thread:
> >
> >
> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_
> >Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E
> >
> >And here, on Cloudera's forums (the second post is mine):
> >
> >
> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoi
> >nt-ignored-in-hdfs-site-xml/m-p/33694#M1180
> >
> >I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
> >Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
> >
> >However, when I point the fs.s3a.endpoint configuration directive at my
> >non-AWS S3-Compatable object storage, it appears to still point at (and
> >authenticate against) AWS.
> >
> >I've checked and double-checked my credentials and configuration using
> >both Python's boto library and the s3cmd tool, both of which connect to
> >this non-AWS data store just fine.
> >
> >Any help would be much appreciated. Thanks!
> >
> >--
> >Caleb Phillips, Ph.D.
> >Data Scientist | Computational Science Center
> >
> >National Renewable Energy Laboratory (NREL)
> >15013 Denver West Parkway | Golden, CO 80401
> >303-275-4297 | caleb.phillips@nrel.gov
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> >For additional commands, e-mail: user-help@hadoop.apache.org
> >
>
>

Re: fs.s3a.endpoint not working

Posted by Billy Watson <wi...@gmail.com>.

One of the threads suggested using the core-site.xml. Did you try putting
your configuration in there?

One thing I've noticed is that the AWS stuff is handled by an underlying
library (I think jets3t in < 2.6 versions, forget what in 2.6+) and when I
was trying to mess with stuff and spelunking through the hadoop code, I
kept running into blocks with that library.

William Watson
Software Engineer
(904) 705-7056 PCS

On Mon, Jan 11, 2016 at 10:39 AM, Phillips, Caleb <Ca...@nrel.gov>
wrote:

> Hi All,
>
> Just wanted to send this out again since there was no response
> (admittedly, originally sent in the midst of the US holiday season) and it
> seems to be an issue that continues to come up (see e.g., the email from
> Han Ju on Jan 5).
>
> If anyone has successfully connected Hadoop to a non-AWS S3-compatable
> object store, it’d be very helpful to hear how you made it work. The
> fs.s3a.endpoint configuration directive appears non-functional at our site
> (with Hadoop 2.6.3).
>
> --
> Caleb Phillips, Ph.D.
> Data Scientist | Computational Science Center
>
> National Renewable Energy Laboratory (NREL)
> 15013 Denver West Parkway | Golden, CO 80401
> 303-275-4297 | caleb.phillips@nrel.gov
>
>
>
>
>
>
> On 12/22/15, 1:39 PM, "Phillips, Caleb" <Ca...@nrel.gov> wrote:
>
> >Hi All,
> >
> >New to this list. Looking for a bit of help:
> >
> >I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
> >store.
> >
> >This issue was discussed, but left unresolved, in this thread:
> >
> >
> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_
> >Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E
> >
> >And here, on Cloudera's forums (the second post is mine):
> >
> >
> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoi
> >nt-ignored-in-hdfs-site-xml/m-p/33694#M1180
> >
> >I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
> >Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
> >
> >However, when I point the fs.s3a.endpoint configuration directive at my
> >non-AWS S3-Compatable object storage, it appears to still point at (and
> >authenticate against) AWS.
> >
> >I've checked and double-checked my credentials and configuration using
> >both Python's boto library and the s3cmd tool, both of which connect to
> >this non-AWS data store just fine.
> >
> >Any help would be much appreciated. Thanks!
> >
> >--
> >Caleb Phillips, Ph.D.
> >Data Scientist | Computational Science Center
> >
> >National Renewable Energy Laboratory (NREL)
> >15013 Denver West Parkway | Golden, CO 80401
> >303-275-4297 | caleb.phillips@nrel.gov
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> >For additional commands, e-mail: user-help@hadoop.apache.org
> >
>
>

Re: fs.s3a.endpoint not working

Posted by Billy Watson <wi...@gmail.com>.

One of the threads suggested using the core-site.xml. Did you try putting
your configuration in there?

One thing I've noticed is that the AWS stuff is handled by an underlying
library (I think jets3t in < 2.6 versions, forget what in 2.6+) and when I
was trying to mess with stuff and spelunking through the hadoop code, I
kept running into blocks with that library.

William Watson
Software Engineer
(904) 705-7056 PCS

On Mon, Jan 11, 2016 at 10:39 AM, Phillips, Caleb <Ca...@nrel.gov>
wrote:

> Hi All,
>
> Just wanted to send this out again since there was no response
> (admittedly, originally sent in the midst of the US holiday season) and it
> seems to be an issue that continues to come up (see e.g., the email from
> Han Ju on Jan 5).
>
> If anyone has successfully connected Hadoop to a non-AWS S3-compatable
> object store, it’d be very helpful to hear how you made it work. The
> fs.s3a.endpoint configuration directive appears non-functional at our site
> (with Hadoop 2.6.3).
>
> --
> Caleb Phillips, Ph.D.
> Data Scientist | Computational Science Center
>
> National Renewable Energy Laboratory (NREL)
> 15013 Denver West Parkway | Golden, CO 80401
> 303-275-4297 | caleb.phillips@nrel.gov
>
>
>
>
>
>
> On 12/22/15, 1:39 PM, "Phillips, Caleb" <Ca...@nrel.gov> wrote:
>
> >Hi All,
> >
> >New to this list. Looking for a bit of help:
> >
> >I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
> >store.
> >
> >This issue was discussed, but left unresolved, in this thread:
> >
> >
> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_
> >Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E
> >
> >And here, on Cloudera's forums (the second post is mine):
> >
> >
> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoi
> >nt-ignored-in-hdfs-site-xml/m-p/33694#M1180
> >
> >I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
> >Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
> >
> >However, when I point the fs.s3a.endpoint configuration directive at my
> >non-AWS S3-Compatable object storage, it appears to still point at (and
> >authenticate against) AWS.
> >
> >I've checked and double-checked my credentials and configuration using
> >both Python's boto library and the s3cmd tool, both of which connect to
> >this non-AWS data store just fine.
> >
> >Any help would be much appreciated. Thanks!
> >
> >--
> >Caleb Phillips, Ph.D.
> >Data Scientist | Computational Science Center
> >
> >National Renewable Energy Laboratory (NREL)
> >15013 Denver West Parkway | Golden, CO 80401
> >303-275-4297 | caleb.phillips@nrel.gov
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> >For additional commands, e-mail: user-help@hadoop.apache.org
> >
>
>

Re: fs.s3a.endpoint not working

Posted by "Phillips, Caleb" <Ca...@nrel.gov>.

Hi All,

Just wanted to send this out again since there was no response
(admittedly, originally sent in the midst of the US holiday season) and it
seems to be an issue that continues to come up (see e.g., the email from
Han Ju on Jan 5).

If anyone has successfully connected Hadoop to a non-AWS S3-compatable
object store, it’d be very helpful to hear how you made it work. The
fs.s3a.endpoint configuration directive appears non-functional at our site
(with Hadoop 2.6.3).

--
Caleb Phillips, Ph.D.
Data Scientist | Computational Science Center

National Renewable Energy Laboratory (NREL)
15013 Denver West Parkway | Golden, CO 80401
303-275-4297 | caleb.phillips@nrel.gov






On 12/22/15, 1:39 PM, "Phillips, Caleb" <Ca...@nrel.gov> wrote:

>Hi All,
>
>New to this list. Looking for a bit of help:
>
>I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
>store.
>
>This issue was discussed, but left unresolved, in this thread:
>
>https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_
>Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E
>
>And here, on Cloudera's forums (the second post is mine):
>
>https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoi
>nt-ignored-in-hdfs-site-xml/m-p/33694#M1180
>
>I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
>Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>
>However, when I point the fs.s3a.endpoint configuration directive at my
>non-AWS S3-Compatable object storage, it appears to still point at (and
>authenticate against) AWS.
>
>I've checked and double-checked my credentials and configuration using
>both Python's boto library and the s3cmd tool, both of which connect to
>this non-AWS data store just fine.
>
>Any help would be much appreciated. Thanks!
>
>--
>Caleb Phillips, Ph.D.
>Data Scientist | Computational Science Center
>
>National Renewable Energy Laboratory (NREL)
>15013 Denver West Parkway | Golden, CO 80401
>303-275-4297 | caleb.phillips@nrel.gov
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: user-help@hadoop.apache.org
>

Re: fs.s3a.endpoint not working

Posted by Alexander Pivovarov <ap...@gmail.com>.

Add jets3t.properties file with s3service.s3-endpoint=<endpoint> to
/etc/hadoop/conf folder

The folder with the file should be in HADOOP_CLASSPATH

JetS3t library which is used by hadoop is looking for this file.
On Dec 22, 2015 12:39 PM, "Phillips, Caleb" <Ca...@nrel.gov> wrote:

> Hi All,
>
> New to this list. Looking for a bit of help:
>
> I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
> store.
>
> This issue was discussed, but left unresolved, in this thread:
>
>
> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E
>
> And here, on Cloudera's forums (the second post is mine):
>
>
> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoint-ignored-in-hdfs-site-xml/m-p/33694#M1180
>
> I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using Hadoop,
> I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>
> However, when I point the fs.s3a.endpoint configuration directive at my
> non-AWS S3-Compatable object storage, it appears to still point at (and
> authenticate against) AWS.
>
> I've checked and double-checked my credentials and configuration using
> both Python's boto library and the s3cmd tool, both of which connect to
> this non-AWS data store just fine.
>
> Any help would be much appreciated. Thanks!
>
> --
> Caleb Phillips, Ph.D.
> Data Scientist | Computational Science Center
>
> National Renewable Energy Laboratory (NREL)
> 15013 Denver West Parkway | Golden, CO 80401
> 303-275-4297 | caleb.phillips@nrel.gov
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>
>

Re: fs.s3a.endpoint not working

Posted by "Phillips, Caleb" <Ca...@nrel.gov>.

Hi All,

Just wanted to send this out again since there was no response
(admittedly, originally sent in the midst of the US holiday season) and it
seems to be an issue that continues to come up (see e.g., the email from
Han Ju on Jan 5).

If anyone has successfully connected Hadoop to a non-AWS S3-compatable
object store, it’d be very helpful to hear how you made it work. The
fs.s3a.endpoint configuration directive appears non-functional at our site
(with Hadoop 2.6.3).

--
Caleb Phillips, Ph.D.
Data Scientist | Computational Science Center

National Renewable Energy Laboratory (NREL)
15013 Denver West Parkway | Golden, CO 80401
303-275-4297 | caleb.phillips@nrel.gov






On 12/22/15, 1:39 PM, "Phillips, Caleb" <Ca...@nrel.gov> wrote:

>Hi All,
>
>New to this list. Looking for a bit of help:
>
>I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
>store.
>
>This issue was discussed, but left unresolved, in this thread:
>
>https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_
>Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E
>
>And here, on Cloudera's forums (the second post is mine):
>
>https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoi
>nt-ignored-in-hdfs-site-xml/m-p/33694#M1180
>
>I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
>Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>
>However, when I point the fs.s3a.endpoint configuration directive at my
>non-AWS S3-Compatable object storage, it appears to still point at (and
>authenticate against) AWS.
>
>I've checked and double-checked my credentials and configuration using
>both Python's boto library and the s3cmd tool, both of which connect to
>this non-AWS data store just fine.
>
>Any help would be much appreciated. Thanks!
>
>--
>Caleb Phillips, Ph.D.
>Data Scientist | Computational Science Center
>
>National Renewable Energy Laboratory (NREL)
>15013 Denver West Parkway | Golden, CO 80401
>303-275-4297 | caleb.phillips@nrel.gov
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: user-help@hadoop.apache.org
>

Re: fs.s3a.endpoint not working

Posted by Alexander Pivovarov <ap...@gmail.com>.

Add jets3t.properties file with s3service.s3-endpoint=<endpoint> to
/etc/hadoop/conf folder

The folder with the file should be in HADOOP_CLASSPATH

JetS3t library which is used by hadoop is looking for this file.
On Dec 22, 2015 12:39 PM, "Phillips, Caleb" <Ca...@nrel.gov> wrote:

> Hi All,
>
> New to this list. Looking for a bit of help:
>
> I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
> store.
>
> This issue was discussed, but left unresolved, in this thread:
>
>
> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E
>
> And here, on Cloudera's forums (the second post is mine):
>
>
> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoint-ignored-in-hdfs-site-xml/m-p/33694#M1180
>
> I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using Hadoop,
> I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>
> However, when I point the fs.s3a.endpoint configuration directive at my
> non-AWS S3-Compatable object storage, it appears to still point at (and
> authenticate against) AWS.
>
> I've checked and double-checked my credentials and configuration using
> both Python's boto library and the s3cmd tool, both of which connect to
> this non-AWS data store just fine.
>
> Any help would be much appreciated. Thanks!
>
> --
> Caleb Phillips, Ph.D.
> Data Scientist | Computational Science Center
>
> National Renewable Energy Laboratory (NREL)
> 15013 Denver West Parkway | Golden, CO 80401
> 303-275-4297 | caleb.phillips@nrel.gov
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>
>

Re: fs.s3a.endpoint not working

Posted by "Phillips, Caleb" <Ca...@nrel.gov>.

Hi All,

Just wanted to send this out again since there was no response
(admittedly, originally sent in the midst of the US holiday season) and it
seems to be an issue that continues to come up (see e.g., the email from
Han Ju on Jan 5).

If anyone has successfully connected Hadoop to a non-AWS S3-compatable
object store, it’d be very helpful to hear how you made it work. The
fs.s3a.endpoint configuration directive appears non-functional at our site
(with Hadoop 2.6.3).

--
Caleb Phillips, Ph.D.
Data Scientist | Computational Science Center

National Renewable Energy Laboratory (NREL)
15013 Denver West Parkway | Golden, CO 80401
303-275-4297 | caleb.phillips@nrel.gov






On 12/22/15, 1:39 PM, "Phillips, Caleb" <Ca...@nrel.gov> wrote:

>Hi All,
>
>New to this list. Looking for a bit of help:
>
>I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
>store.
>
>This issue was discussed, but left unresolved, in this thread:
>
>https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_
>Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E
>
>And here, on Cloudera's forums (the second post is mine):
>
>https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoi
>nt-ignored-in-hdfs-site-xml/m-p/33694#M1180
>
>I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
>Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>
>However, when I point the fs.s3a.endpoint configuration directive at my
>non-AWS S3-Compatable object storage, it appears to still point at (and
>authenticate against) AWS.
>
>I've checked and double-checked my credentials and configuration using
>both Python's boto library and the s3cmd tool, both of which connect to
>this non-AWS data store just fine.
>
>Any help would be much appreciated. Thanks!
>
>--
>Caleb Phillips, Ph.D.
>Data Scientist | Computational Science Center
>
>National Renewable Energy Laboratory (NREL)
>15013 Denver West Parkway | Golden, CO 80401
>303-275-4297 | caleb.phillips@nrel.gov
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: user-help@hadoop.apache.org
>

Re: fs.s3a.endpoint not working

Posted by Alexander Pivovarov <ap...@gmail.com>.

Add jets3t.properties file with s3service.s3-endpoint=<endpoint> to
/etc/hadoop/conf folder

The folder with the file should be in HADOOP_CLASSPATH

JetS3t library which is used by hadoop is looking for this file.
On Dec 22, 2015 12:39 PM, "Phillips, Caleb" <Ca...@nrel.gov> wrote:

> Hi All,
>
> New to this list. Looking for a bit of help:
>
> I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
> store.
>
> This issue was discussed, but left unresolved, in this thread:
>
>
> https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E
>
> And here, on Cloudera's forums (the second post is mine):
>
>
> https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoint-ignored-in-hdfs-site-xml/m-p/33694#M1180
>
> I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using Hadoop,
> I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>
> However, when I point the fs.s3a.endpoint configuration directive at my
> non-AWS S3-Compatable object storage, it appears to still point at (and
> authenticate against) AWS.
>
> I've checked and double-checked my credentials and configuration using
> both Python's boto library and the s3cmd tool, both of which connect to
> this non-AWS data store just fine.
>
> Any help would be much appreciated. Thanks!
>
> --
> Caleb Phillips, Ph.D.
> Data Scientist | Computational Science Center
>
> National Renewable Energy Laboratory (NREL)
> 15013 Denver West Parkway | Golden, CO 80401
> 303-275-4297 | caleb.phillips@nrel.gov
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>
>

Re: fs.s3a.endpoint not working

Posted by "Phillips, Caleb" <Ca...@nrel.gov>.

Hi All,

Just wanted to send this out again since there was no response
(admittedly, originally sent in the midst of the US holiday season) and it
seems to be an issue that continues to come up (see e.g., the email from
Han Ju on Jan 5).

If anyone has successfully connected Hadoop to a non-AWS S3-compatable
object store, it’d be very helpful to hear how you made it work. The
fs.s3a.endpoint configuration directive appears non-functional at our site
(with Hadoop 2.6.3).

--
Caleb Phillips, Ph.D.
Data Scientist | Computational Science Center

National Renewable Energy Laboratory (NREL)
15013 Denver West Parkway | Golden, CO 80401
303-275-4297 | caleb.phillips@nrel.gov






On 12/22/15, 1:39 PM, "Phillips, Caleb" <Ca...@nrel.gov> wrote:

>Hi All,
>
>New to this list. Looking for a bit of help:
>
>I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object
>store.
>
>This issue was discussed, but left unresolved, in this thread:
>
>https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3CCA+0W_
>Au5Es_fLUgZMGwkkgA3JyA1ASi3u+isJCuYmfnTvNkGuQ@mail.gmail.com%3E
>
>And here, on Cloudera's forums (the second post is mine):
>
>https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoi
>nt-ignored-in-hdfs-site-xml/m-p/33694#M1180
>
>I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using
>Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files.
>
>However, when I point the fs.s3a.endpoint configuration directive at my
>non-AWS S3-Compatable object storage, it appears to still point at (and
>authenticate against) AWS.
>
>I've checked and double-checked my credentials and configuration using
>both Python's boto library and the s3cmd tool, both of which connect to
>this non-AWS data store just fine.
>
>Any help would be much appreciated. Thanks!
>
>--
>Caleb Phillips, Ph.D.
>Data Scientist | Computational Science Center
>
>National Renewable Energy Laboratory (NREL)
>15013 Denver West Parkway | Golden, CO 80401
>303-275-4297 | caleb.phillips@nrel.gov
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: user-help@hadoop.apache.org
>