Webhdfs and S3

[Joseph Henry <Jo...@sas.com>]

Hey,

I am not sure if this is the correct mailing list for this question, but I will start here.

Our client application needs to support accessing S3 buckets from hdfs. We can do this with the Java API using the s3a:// scheme, but also need a way to access the same files in S3 via the HDFS REST API.

Is there a way to access the data stored in S3 via WEBHDFS?

Thanks,
Joseph Henry.

RE: Webhdfs and S3

[Joseph Henry <Jo...@sas.com>]

That is an interesting idea, but I was kind of hoping for something like a query parameter of “&filesystem=s3a” or something like that, where I can choose the filesystem like I can in the Java API.

From: Wei-Chiu Chuang <we...@cloudera.com>
Sent: Wednesday, May 22, 2019 9:50 AM
To: Joseph Henry <Jo...@sas.com>
Cc: user@hadoop.apache.org
Subject: Re: Webhdfs and S3


EXTERNAL
You can start 2 httpfs servers (or even more), and let one set fs.defaultFS to s3a://, and the other set to hdfs.
Will that work for you? Or is this not what you need?

On Wed, May 22, 2019 at 3:40 PM Joseph Henry <Jo...@sas.com>> wrote:
I thought about that, but we need to be able to access storage in native hdfs as well as S3 in the same cluster. If we change fs.defaultFS then I would not be able to access the HDFS storage.

From: Wei-Chiu Chuang <we...@cloudera.com>>
Sent: Wednesday, May 22, 2019 9:36 AM
To: Joseph Henry <Jo...@sas.com>>
Cc: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Webhdfs and S3


EXTERNAL
I've never tried, but it seems possible to start a Httpfs server with fs.defaultFS = s3a://your-bucket
Httpfs server speaks WebHDFS protocol so your webhdfs client can use webhdfs. And then for each webhdfs request, httpfs server translates that into the corresponding FileSystem API call. If the fs.defaultFS is the s3a:// URI, it may be able to talk to s3.

On Wed, May 22, 2019 at 3:29 PM Joseph Henry <Jo...@sas.com>> wrote:
Hey,

I am not sure if this is the correct mailing list for this question, but I will start here.

Our client application needs to support accessing S3 buckets from hdfs. We can do this with the Java API using the s3a:// scheme, but also need a way to access the same files in S3 via the HDFS REST API.

Is there a way to access the data stored in S3 via WEBHDFS?

Thanks,
Joseph Henry.

Re: Webhdfs and S3

[Wei-Chiu Chuang <we...@cloudera.com.INVALID>]

You can start 2 httpfs servers (or even more), and let one set fs.defaultFS
to s3a://, and the other set to hdfs.
Will that work for you? Or is this not what you need?

On Wed, May 22, 2019 at 3:40 PM Joseph Henry <Jo...@sas.com> wrote:

> I thought about that, but we need to be able to access storage in native
> hdfs as well as S3 in the same cluster. If we change fs.defaultFS then I
> would not be able to access the HDFS storage.
>
>
>
> *From:* Wei-Chiu Chuang <we...@cloudera.com>
> *Sent:* Wednesday, May 22, 2019 9:36 AM
> *To:* Joseph Henry <Jo...@sas.com>
> *Cc:* user@hadoop.apache.org
> *Subject:* Re: Webhdfs and S3
>
>
>
> *EXTERNAL*
>
> I've never tried, but it seems possible to start a Httpfs server with
> fs.defaultFS = s3a://your-bucket
>
> Httpfs server speaks WebHDFS protocol so your webhdfs client can use
> webhdfs. And then for each webhdfs request, httpfs server translates that
> into the corresponding FileSystem API call. If the fs.defaultFS is the
> s3a:// URI, it may be able to talk to s3.
>
>
>
> On Wed, May 22, 2019 at 3:29 PM Joseph Henry <Jo...@sas.com> wrote:
>
> Hey,
>
>
>
> I am not sure if this is the correct mailing list for this question, but I
> will start here.
>
>
>
> Our client application needs to support accessing S3 buckets from hdfs. We
> can do this with the Java API using the s3a:// scheme, but also need a way
> to access the same files in S3 via the HDFS REST API.
>
>
>
> Is there a way to access the data stored in S3 via WEBHDFS?
>
>
>
> Thanks,
>
> Joseph Henry.
>
>
>
>

RE: Webhdfs and S3

[Joseph Henry <Jo...@sas.com>]

I thought about that, but we need to be able to access storage in native hdfs as well as S3 in the same cluster. If we change fs.defaultFS then I would not be able to access the HDFS storage.

From: Wei-Chiu Chuang <we...@cloudera.com>
Sent: Wednesday, May 22, 2019 9:36 AM
To: Joseph Henry <Jo...@sas.com>
Cc: user@hadoop.apache.org
Subject: Re: Webhdfs and S3


EXTERNAL
I've never tried, but it seems possible to start a Httpfs server with fs.defaultFS = s3a://your-bucket
Httpfs server speaks WebHDFS protocol so your webhdfs client can use webhdfs. And then for each webhdfs request, httpfs server translates that into the corresponding FileSystem API call. If the fs.defaultFS is the s3a:// URI, it may be able to talk to s3.

On Wed, May 22, 2019 at 3:29 PM Joseph Henry <Jo...@sas.com>> wrote:
Hey,

I am not sure if this is the correct mailing list for this question, but I will start here.

Our client application needs to support accessing S3 buckets from hdfs. We can do this with the Java API using the s3a:// scheme, but also need a way to access the same files in S3 via the HDFS REST API.

Is there a way to access the data stored in S3 via WEBHDFS?

Thanks,
Joseph Henry.

Re: Webhdfs and S3

[Wei-Chiu Chuang <we...@cloudera.com.INVALID>]

I've never tried, but it seems possible to start a Httpfs server with
fs.defaultFS = s3a://your-bucket
Httpfs server speaks WebHDFS protocol so your webhdfs client can use
webhdfs. And then for each webhdfs request, httpfs server translates that
into the corresponding FileSystem API call. If the fs.defaultFS is the
s3a:// URI, it may be able to talk to s3.

On Wed, May 22, 2019 at 3:29 PM Joseph Henry <Jo...@sas.com> wrote:

> Hey,
>
>
>
> I am not sure if this is the correct mailing list for this question, but I
> will start here.
>
>
>
> Our client application needs to support accessing S3 buckets from hdfs. We
> can do this with the Java API using the s3a:// scheme, but also need a way
> to access the same files in S3 via the HDFS REST API.
>
>
>
> Is there a way to access the data stored in S3 via WEBHDFS?
>
>
>
> Thanks,
>
> Joseph Henry.
>
>
>