You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Han JU <ju...@gmail.com> on 2016/01/05 18:35:36 UTC

Custom s3 endpoint for s3n/s3a

Hello,

For test purpose we need to configure a custom s3 endpoint for s3n/s3a.
More precisely we've need to test that parquet writes correctly the content
to s3.
We've setup a s3rver, so the endpoint should be `http://s3rver:8000`. I've
tried different method but no luck so far.

Things I've tried:
  - use url like `s3n://s3rver:8000/mybucket/file`
  - set jets3t.properties (also set them in Configuration)
  - tried a `AmazonS3Client` with `setEndpoint("s3rver:8000")`

Any suggestions?

Thanks!
-- 
*JU Han*

Software Engineer @ Teads.tv

+33 0619608888

Re: Custom s3 endpoint for s3n/s3a

Posted by Anu Engineer <ae...@hortonworks.com>.
I have never tried it myself, so I have no idea if it works. But this JIRA (https://issues.apache.org/jira/browse/HADOOP-11261) seems to indicate that you might want to setup “fs.s3a.endpoint”.

Thanks
Anu

From: Han JU <ju...@gmail.com>>
Date: Tuesday, January 5, 2016 at 9:35 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Custom s3 endpoint for s3n/s3a

Hello,

For test purpose we need to configure a custom s3 endpoint for s3n/s3a. More precisely we've need to test that parquet writes correctly the content to s3.
We've setup a s3rver, so the endpoint should be `http://s3rver:8000`. I've tried different method but no luck so far.

Things I've tried:
  - use url like `s3n://s3rver:8000/mybucket/file`
  - set jets3t.properties (also set them in Configuration)
  - tried a `AmazonS3Client` with `setEndpoint("s3rver:8000")`

Any suggestions?

Thanks!
--
JU Han

Software Engineer @ Teads.tv

+33 0619608888

Re: Custom s3 endpoint for s3n/s3a

Posted by Anu Engineer <ae...@hortonworks.com>.
I have never tried it myself, so I have no idea if it works. But this JIRA (https://issues.apache.org/jira/browse/HADOOP-11261) seems to indicate that you might want to setup “fs.s3a.endpoint”.

Thanks
Anu

From: Han JU <ju...@gmail.com>>
Date: Tuesday, January 5, 2016 at 9:35 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Custom s3 endpoint for s3n/s3a

Hello,

For test purpose we need to configure a custom s3 endpoint for s3n/s3a. More precisely we've need to test that parquet writes correctly the content to s3.
We've setup a s3rver, so the endpoint should be `http://s3rver:8000`. I've tried different method but no luck so far.

Things I've tried:
  - use url like `s3n://s3rver:8000/mybucket/file`
  - set jets3t.properties (also set them in Configuration)
  - tried a `AmazonS3Client` with `setEndpoint("s3rver:8000")`

Any suggestions?

Thanks!
--
JU Han

Software Engineer @ Teads.tv

+33 0619608888

Re: Custom s3 endpoint for s3n/s3a

Posted by Anu Engineer <ae...@hortonworks.com>.
I have never tried it myself, so I have no idea if it works. But this JIRA (https://issues.apache.org/jira/browse/HADOOP-11261) seems to indicate that you might want to setup “fs.s3a.endpoint”.

Thanks
Anu

From: Han JU <ju...@gmail.com>>
Date: Tuesday, January 5, 2016 at 9:35 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Custom s3 endpoint for s3n/s3a

Hello,

For test purpose we need to configure a custom s3 endpoint for s3n/s3a. More precisely we've need to test that parquet writes correctly the content to s3.
We've setup a s3rver, so the endpoint should be `http://s3rver:8000`. I've tried different method but no luck so far.

Things I've tried:
  - use url like `s3n://s3rver:8000/mybucket/file`
  - set jets3t.properties (also set them in Configuration)
  - tried a `AmazonS3Client` with `setEndpoint("s3rver:8000")`

Any suggestions?

Thanks!
--
JU Han

Software Engineer @ Teads.tv

+33 0619608888

Re: Custom s3 endpoint for s3n/s3a

Posted by Anu Engineer <ae...@hortonworks.com>.
I have never tried it myself, so I have no idea if it works. But this JIRA (https://issues.apache.org/jira/browse/HADOOP-11261) seems to indicate that you might want to setup “fs.s3a.endpoint”.

Thanks
Anu

From: Han JU <ju...@gmail.com>>
Date: Tuesday, January 5, 2016 at 9:35 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Custom s3 endpoint for s3n/s3a

Hello,

For test purpose we need to configure a custom s3 endpoint for s3n/s3a. More precisely we've need to test that parquet writes correctly the content to s3.
We've setup a s3rver, so the endpoint should be `http://s3rver:8000`. I've tried different method but no luck so far.

Things I've tried:
  - use url like `s3n://s3rver:8000/mybucket/file`
  - set jets3t.properties (also set them in Configuration)
  - tried a `AmazonS3Client` with `setEndpoint("s3rver:8000")`

Any suggestions?

Thanks!
--
JU Han

Software Engineer @ Teads.tv

+33 0619608888