You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Aaron Carey <ac...@ilm.com> on 2016/02/26 17:17:49 UTC

Downloading s3 uris

I'm attempting to fetch files from s3 uris in mesos, but we're not using hdfs in our cluster... however I believe I need the client installed.

Is it possible to just have the client running without a full hdfs setup?

I haven't been able to find much information in the docs, could someone point me in the right direction?

Thanks!

Aaron

RE: Downloading s3 uris

Posted by Aaron Carey <ac...@ilm.com>.
Ah I think I've solved my own problem:

We are using the dind mesos slave container and hadn't mounted the host's /var/lib/mesos folder within the container so it wasn't showing up in the other containers.

Oops!

Thanks for your help,

Aaron

________________________________
From: Aaron Carey [acarey@ilm.com]
Sent: 29 February 2016 08:53
To: user@mesos.apache.org
Subject: RE: Downloading s3 uris

Sorry apparently the inline image didn't work:

http://i.imgur.com/x1cPXvW.png

________________________________
From: Aaron Carey [acarey@ilm.com]
Sent: 29 February 2016 08:50
To: user@mesos.apache.org
Subject: RE: Downloading s3 uris

To illustrate:

[X]

/var/lib/mesos/slaves/20160212-131720-1510021036-5050-1-S0/frameworks/20160212-131720-1510021036-5050-1-0000/executors/test.3d01cbd4-dcb4-11e5-868c-02420a4d969a/runs/25d29966-0515-4aa6-8d99-63a377aa68e8$ ls -alh
total 8.0K
drwxr-xr-x 2 root root 4.0K Feb 26 18:10 .
drwxr-xr-x 3 root root 4.0K Feb 26 18:10 ..




________________________________
From: Aaron Carey [acarey@ilm.com]
Sent: 29 February 2016 08:45
To: user@mesos.apache.org
Subject: RE: Downloading s3 uris

Yeah, I've managed to find the sandbox itself on disk, but it's empty, even though the file shows up in the web UI...

My task is a docker container and it doesn't show up in the container either

Any ideas?

Thanks!
Aaron

________________________________
From: Joseph Wu [joseph@mesosphere.io]
Sent: 26 February 2016 18:27
To: user@mesos.apache.org
Subject: Re: Downloading s3 uris

The sandbox directory structure is a bit deep...  See the "Where is the sandbox?" section here: http://mesos.apache.org/documentation/latest/sandbox/


On Fri, Feb 26, 2016 at 10:15 AM, Aaron Carey <ac...@ilm.com>> wrote:
A second question for you all..

I'm testing http uri downloads, and all the logs say that the file has downloaded (it even shows up in the mesos UI in the sandbox) but I can't find the file on disk anywhere. It doesn't appear in the docker container I'm running either (shouldn't it be in /mnt/mesos/sandbox?)

Am I missing something here?

Thanks for your help,

Aaron


________________________________
From: Radoslaw Gruchalski [radek@gruchalski.com<ma...@gruchalski.com>]
Sent: 26 February 2016 17:41

To: user@mesos.apache.org<ma...@mesos.apache.org>; user@mesos.apache.org<ma...@mesos.apache.org>
Subject: Re: Downloading s3 uris

Just keep in mind that every execution of such command starts a jvm and is, generally, heavyweight. Use WebHDFS if you can.

Sent from Outlook Mobile<https://aka.ms/qtex0l>




On Fri, Feb 26, 2016 at 9:13 AM -0800, "Shuai Lin" <li...@gmail.com>> wrote:

If you don't want to configure hadoop on your mesos slaves, the only workaround I see is to write a "hadoop" script and put it in your PATH. It need to support the following usage patterns:

- hadoop version
- hadoop fs -copyToLocal s3n://path /target/directory/

On Sat, Feb 27, 2016 at 12:31 AM, Aaron Carey <ac...@ilm.com>> wrote:
I was trying to avoid generating urls for everything as this will complicate things a lot.

Is there a straight forward way to get the fetcher to do it directly?

________________________________
From: haosdent [haosdent@gmail.com<ma...@gmail.com>]
Sent: 26 February 2016 16:27
To: user
Subject: Re: Downloading s3 uris

I think still could pass AWSAccessKeyId if it is private? http://www.bucketexplorer.com/documentation/amazon-s3--how-to-generate-url-for-amazon-s3-files.html

On Sat, Feb 27, 2016 at 12:25 AM, Abhishek Amralkar <Ab...@talentica.com>> wrote:
In that case do we need to keep bucket/files public?

-Abhishek

From: Zhitao Li <zh...@uber.com>>
Reply-To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Date: Friday, 26 February 2016 at 8:23 AM
To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Subject: Re: Downloading s3 uris

Haven't directly used s3 download, but I think a workaround (if you don't care ACL about the files) is to use http<http://stackoverflow.com/questions/18239567/how-can-i-download-a-file-from-an-s3-bucket-with-wget> url instead.
On Feb 26, 2016, at 8:17 AM, Aaron Carey <ac...@ilm.com>> wrote:

I'm attempting to fetch files from s3 uris in mesos, but we're not using hdfs in our cluster... however I believe I need the client installed.

Is it possible to just have the client running without a full hdfs setup?

I haven't been able to find much information in the docs, could someone point me in the right direction?

Thanks!

Aaron




--
Best Regards,
Haosdent Huang



RE: Downloading s3 uris

Posted by Aaron Carey <ac...@ilm.com>.
Sorry apparently the inline image didn't work:

http://i.imgur.com/x1cPXvW.png

________________________________
From: Aaron Carey [acarey@ilm.com]
Sent: 29 February 2016 08:50
To: user@mesos.apache.org
Subject: RE: Downloading s3 uris

To illustrate:

[X]

/var/lib/mesos/slaves/20160212-131720-1510021036-5050-1-S0/frameworks/20160212-131720-1510021036-5050-1-0000/executors/test.3d01cbd4-dcb4-11e5-868c-02420a4d969a/runs/25d29966-0515-4aa6-8d99-63a377aa68e8$ ls -alh
total 8.0K
drwxr-xr-x 2 root root 4.0K Feb 26 18:10 .
drwxr-xr-x 3 root root 4.0K Feb 26 18:10 ..




________________________________
From: Aaron Carey [acarey@ilm.com]
Sent: 29 February 2016 08:45
To: user@mesos.apache.org
Subject: RE: Downloading s3 uris

Yeah, I've managed to find the sandbox itself on disk, but it's empty, even though the file shows up in the web UI...

My task is a docker container and it doesn't show up in the container either

Any ideas?

Thanks!
Aaron

________________________________
From: Joseph Wu [joseph@mesosphere.io]
Sent: 26 February 2016 18:27
To: user@mesos.apache.org
Subject: Re: Downloading s3 uris

The sandbox directory structure is a bit deep...  See the "Where is the sandbox?" section here: http://mesos.apache.org/documentation/latest/sandbox/


On Fri, Feb 26, 2016 at 10:15 AM, Aaron Carey <ac...@ilm.com>> wrote:
A second question for you all..

I'm testing http uri downloads, and all the logs say that the file has downloaded (it even shows up in the mesos UI in the sandbox) but I can't find the file on disk anywhere. It doesn't appear in the docker container I'm running either (shouldn't it be in /mnt/mesos/sandbox?)

Am I missing something here?

Thanks for your help,

Aaron


________________________________
From: Radoslaw Gruchalski [radek@gruchalski.com<ma...@gruchalski.com>]
Sent: 26 February 2016 17:41

To: user@mesos.apache.org<ma...@mesos.apache.org>; user@mesos.apache.org<ma...@mesos.apache.org>
Subject: Re: Downloading s3 uris

Just keep in mind that every execution of such command starts a jvm and is, generally, heavyweight. Use WebHDFS if you can.

Sent from Outlook Mobile<https://aka.ms/qtex0l>




On Fri, Feb 26, 2016 at 9:13 AM -0800, "Shuai Lin" <li...@gmail.com>> wrote:

If you don't want to configure hadoop on your mesos slaves, the only workaround I see is to write a "hadoop" script and put it in your PATH. It need to support the following usage patterns:

- hadoop version
- hadoop fs -copyToLocal s3n://path /target/directory/

On Sat, Feb 27, 2016 at 12:31 AM, Aaron Carey <ac...@ilm.com>> wrote:
I was trying to avoid generating urls for everything as this will complicate things a lot.

Is there a straight forward way to get the fetcher to do it directly?

________________________________
From: haosdent [haosdent@gmail.com<ma...@gmail.com>]
Sent: 26 February 2016 16:27
To: user
Subject: Re: Downloading s3 uris

I think still could pass AWSAccessKeyId if it is private? http://www.bucketexplorer.com/documentation/amazon-s3--how-to-generate-url-for-amazon-s3-files.html

On Sat, Feb 27, 2016 at 12:25 AM, Abhishek Amralkar <Ab...@talentica.com>> wrote:
In that case do we need to keep bucket/files public?

-Abhishek

From: Zhitao Li <zh...@uber.com>>
Reply-To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Date: Friday, 26 February 2016 at 8:23 AM
To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Subject: Re: Downloading s3 uris

Haven't directly used s3 download, but I think a workaround (if you don't care ACL about the files) is to use http<http://stackoverflow.com/questions/18239567/how-can-i-download-a-file-from-an-s3-bucket-with-wget> url instead.
On Feb 26, 2016, at 8:17 AM, Aaron Carey <ac...@ilm.com>> wrote:

I'm attempting to fetch files from s3 uris in mesos, but we're not using hdfs in our cluster... however I believe I need the client installed.

Is it possible to just have the client running without a full hdfs setup?

I haven't been able to find much information in the docs, could someone point me in the right direction?

Thanks!

Aaron




--
Best Regards,
Haosdent Huang



RE: Downloading s3 uris

Posted by Aaron Carey <ac...@ilm.com>.
To illustrate:

[X]

/var/lib/mesos/slaves/20160212-131720-1510021036-5050-1-S0/frameworks/20160212-131720-1510021036-5050-1-0000/executors/test.3d01cbd4-dcb4-11e5-868c-02420a4d969a/runs/25d29966-0515-4aa6-8d99-63a377aa68e8$ ls -alh
total 8.0K
drwxr-xr-x 2 root root 4.0K Feb 26 18:10 .
drwxr-xr-x 3 root root 4.0K Feb 26 18:10 ..




________________________________
From: Aaron Carey [acarey@ilm.com]
Sent: 29 February 2016 08:45
To: user@mesos.apache.org
Subject: RE: Downloading s3 uris

Yeah, I've managed to find the sandbox itself on disk, but it's empty, even though the file shows up in the web UI...

My task is a docker container and it doesn't show up in the container either

Any ideas?

Thanks!
Aaron

________________________________
From: Joseph Wu [joseph@mesosphere.io]
Sent: 26 February 2016 18:27
To: user@mesos.apache.org
Subject: Re: Downloading s3 uris

The sandbox directory structure is a bit deep...  See the "Where is the sandbox?" section here: http://mesos.apache.org/documentation/latest/sandbox/


On Fri, Feb 26, 2016 at 10:15 AM, Aaron Carey <ac...@ilm.com>> wrote:
A second question for you all..

I'm testing http uri downloads, and all the logs say that the file has downloaded (it even shows up in the mesos UI in the sandbox) but I can't find the file on disk anywhere. It doesn't appear in the docker container I'm running either (shouldn't it be in /mnt/mesos/sandbox?)

Am I missing something here?

Thanks for your help,

Aaron


________________________________
From: Radoslaw Gruchalski [radek@gruchalski.com<ma...@gruchalski.com>]
Sent: 26 February 2016 17:41

To: user@mesos.apache.org<ma...@mesos.apache.org>; user@mesos.apache.org<ma...@mesos.apache.org>
Subject: Re: Downloading s3 uris

Just keep in mind that every execution of such command starts a jvm and is, generally, heavyweight. Use WebHDFS if you can.

Sent from Outlook Mobile<https://aka.ms/qtex0l>




On Fri, Feb 26, 2016 at 9:13 AM -0800, "Shuai Lin" <li...@gmail.com>> wrote:

If you don't want to configure hadoop on your mesos slaves, the only workaround I see is to write a "hadoop" script and put it in your PATH. It need to support the following usage patterns:

- hadoop version
- hadoop fs -copyToLocal s3n://path /target/directory/

On Sat, Feb 27, 2016 at 12:31 AM, Aaron Carey <ac...@ilm.com>> wrote:
I was trying to avoid generating urls for everything as this will complicate things a lot.

Is there a straight forward way to get the fetcher to do it directly?

________________________________
From: haosdent [haosdent@gmail.com<ma...@gmail.com>]
Sent: 26 February 2016 16:27
To: user
Subject: Re: Downloading s3 uris

I think still could pass AWSAccessKeyId if it is private? http://www.bucketexplorer.com/documentation/amazon-s3--how-to-generate-url-for-amazon-s3-files.html

On Sat, Feb 27, 2016 at 12:25 AM, Abhishek Amralkar <Ab...@talentica.com>> wrote:
In that case do we need to keep bucket/files public?

-Abhishek

From: Zhitao Li <zh...@uber.com>>
Reply-To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Date: Friday, 26 February 2016 at 8:23 AM
To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Subject: Re: Downloading s3 uris

Haven't directly used s3 download, but I think a workaround (if you don't care ACL about the files) is to use http<http://stackoverflow.com/questions/18239567/how-can-i-download-a-file-from-an-s3-bucket-with-wget> url instead.
On Feb 26, 2016, at 8:17 AM, Aaron Carey <ac...@ilm.com>> wrote:

I'm attempting to fetch files from s3 uris in mesos, but we're not using hdfs in our cluster... however I believe I need the client installed.

Is it possible to just have the client running without a full hdfs setup?

I haven't been able to find much information in the docs, could someone point me in the right direction?

Thanks!

Aaron




--
Best Regards,
Haosdent Huang



RE: Downloading s3 uris

Posted by Aaron Carey <ac...@ilm.com>.
Yeah, I've managed to find the sandbox itself on disk, but it's empty, even though the file shows up in the web UI...

My task is a docker container and it doesn't show up in the container either

Any ideas?

Thanks!
Aaron

________________________________
From: Joseph Wu [joseph@mesosphere.io]
Sent: 26 February 2016 18:27
To: user@mesos.apache.org
Subject: Re: Downloading s3 uris

The sandbox directory structure is a bit deep...  See the "Where is the sandbox?" section here: http://mesos.apache.org/documentation/latest/sandbox/


On Fri, Feb 26, 2016 at 10:15 AM, Aaron Carey <ac...@ilm.com>> wrote:
A second question for you all..

I'm testing http uri downloads, and all the logs say that the file has downloaded (it even shows up in the mesos UI in the sandbox) but I can't find the file on disk anywhere. It doesn't appear in the docker container I'm running either (shouldn't it be in /mnt/mesos/sandbox?)

Am I missing something here?

Thanks for your help,

Aaron


________________________________
From: Radoslaw Gruchalski [radek@gruchalski.com<ma...@gruchalski.com>]
Sent: 26 February 2016 17:41

To: user@mesos.apache.org<ma...@mesos.apache.org>; user@mesos.apache.org<ma...@mesos.apache.org>
Subject: Re: Downloading s3 uris

Just keep in mind that every execution of such command starts a jvm and is, generally, heavyweight. Use WebHDFS if you can.

Sent from Outlook Mobile<https://aka.ms/qtex0l>




On Fri, Feb 26, 2016 at 9:13 AM -0800, "Shuai Lin" <li...@gmail.com>> wrote:

If you don't want to configure hadoop on your mesos slaves, the only workaround I see is to write a "hadoop" script and put it in your PATH. It need to support the following usage patterns:

- hadoop version
- hadoop fs -copyToLocal s3n://path /target/directory/

On Sat, Feb 27, 2016 at 12:31 AM, Aaron Carey <ac...@ilm.com>> wrote:
I was trying to avoid generating urls for everything as this will complicate things a lot.

Is there a straight forward way to get the fetcher to do it directly?

________________________________
From: haosdent [haosdent@gmail.com<ma...@gmail.com>]
Sent: 26 February 2016 16:27
To: user
Subject: Re: Downloading s3 uris

I think still could pass AWSAccessKeyId if it is private? http://www.bucketexplorer.com/documentation/amazon-s3--how-to-generate-url-for-amazon-s3-files.html

On Sat, Feb 27, 2016 at 12:25 AM, Abhishek Amralkar <Ab...@talentica.com>> wrote:
In that case do we need to keep bucket/files public?

-Abhishek

From: Zhitao Li <zh...@uber.com>>
Reply-To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Date: Friday, 26 February 2016 at 8:23 AM
To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Subject: Re: Downloading s3 uris

Haven't directly used s3 download, but I think a workaround (if you don't care ACL about the files) is to use http<http://stackoverflow.com/questions/18239567/how-can-i-download-a-file-from-an-s3-bucket-with-wget> url instead.
On Feb 26, 2016, at 8:17 AM, Aaron Carey <ac...@ilm.com>> wrote:

I'm attempting to fetch files from s3 uris in mesos, but we're not using hdfs in our cluster... however I believe I need the client installed.

Is it possible to just have the client running without a full hdfs setup?

I haven't been able to find much information in the docs, could someone point me in the right direction?

Thanks!

Aaron




--
Best Regards,
Haosdent Huang



Re: Downloading s3 uris

Posted by Joseph Wu <jo...@mesosphere.io>.
The sandbox directory structure is a bit deep...  See the "Where is the
sandbox?" section here:
http://mesos.apache.org/documentation/latest/sandbox/


On Fri, Feb 26, 2016 at 10:15 AM, Aaron Carey <ac...@ilm.com> wrote:

> A second question for you all..
>
> I'm testing http uri downloads, and all the logs say that the file has
> downloaded (it even shows up in the mesos UI in the sandbox) but I can't
> find the file on disk anywhere. It doesn't appear in the docker container
> I'm running either (shouldn't it be in /mnt/mesos/sandbox?)
>
> Am I missing something here?
>
> Thanks for your help,
>
> Aaron
>
>
> ------------------------------
> *From:* Radoslaw Gruchalski [radek@gruchalski.com]
> *Sent:* 26 February 2016 17:41
>
> *To:* user@mesos.apache.org; user@mesos.apache.org
> *Subject:* Re: Downloading s3 uris
>
> Just keep in mind that every execution of such command starts a jvm and
> is, generally, heavyweight. Use WebHDFS if you can.
>
> Sent from Outlook Mobile <https://aka.ms/qtex0l>
>
>
>
>
> On Fri, Feb 26, 2016 at 9:13 AM -0800, "Shuai Lin" <linshuai2012@gmail.com
> > wrote:
>
> If you don't want to configure hadoop on your mesos slaves, the only
>> workaround I see is to write a "hadoop" script and put it in your PATH. It
>> need to support the following usage patterns:
>>
>> - hadoop version
>> - hadoop fs -copyToLocal s3n://path /target/directory/
>>
>> On Sat, Feb 27, 2016 at 12:31 AM, Aaron Carey <ac...@ilm.com> wrote:
>>
>>> I was trying to avoid generating urls for everything as this will
>>> complicate things a lot.
>>>
>>> Is there a straight forward way to get the fetcher to do it directly?
>>>
>>> ------------------------------
>>> *From:* haosdent [haosdent@gmail.com]
>>> *Sent:* 26 February 2016 16:27
>>> *To:* user
>>> *Subject:* Re: Downloading s3 uris
>>>
>>> I think still could pass AWSAccessKeyId if it is private?
>>> http://www.bucketexplorer.com/documentation/amazon-s3--how-to-generate-url-for-amazon-s3-files.html
>>>
>>> On Sat, Feb 27, 2016 at 12:25 AM, Abhishek Amralkar <
>>> Abhishek.Amralkar@talentica.com> wrote:
>>>
>>>> In that case do we need to keep bucket/files public?
>>>>
>>>> -Abhishek
>>>>
>>>> From: Zhitao Li <zh...@uber.com>
>>>> Reply-To: "user@mesos.apache.org" <us...@mesos.apache.org>
>>>> Date: Friday, 26 February 2016 at 8:23 AM
>>>> To: "user@mesos.apache.org" <us...@mesos.apache.org>
>>>> Subject: Re: Downloading s3 uris
>>>>
>>>> Haven't directly used s3 download, but I think a workaround (if you
>>>> don't care ACL about the files) is to use http
>>>> <http://stackoverflow.com/questions/18239567/how-can-i-download-a-file-from-an-s3-bucket-with-wget> url
>>>> instead.
>>>>
>>>> On Feb 26, 2016, at 8:17 AM, Aaron Carey <ac...@ilm.com> wrote:
>>>>
>>>> I'm attempting to fetch files from s3 uris in mesos, but we're not
>>>> using hdfs in our cluster... however I believe I need the client installed.
>>>>
>>>> Is it possible to just have the client running without a full hdfs
>>>> setup?
>>>>
>>>> I haven't been able to find much information in the docs, could someone
>>>> point me in the right direction?
>>>>
>>>> Thanks!
>>>>
>>>> Aaron
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Haosdent Huang
>>>
>>
>>

RE: Downloading s3 uris

Posted by Aaron Carey <ac...@ilm.com>.
A second question for you all..

I'm testing http uri downloads, and all the logs say that the file has downloaded (it even shows up in the mesos UI in the sandbox) but I can't find the file on disk anywhere. It doesn't appear in the docker container I'm running either (shouldn't it be in /mnt/mesos/sandbox?)

Am I missing something here?

Thanks for your help,

Aaron


________________________________
From: Radoslaw Gruchalski [radek@gruchalski.com]
Sent: 26 February 2016 17:41
To: user@mesos.apache.org; user@mesos.apache.org
Subject: Re: Downloading s3 uris

Just keep in mind that every execution of such command starts a jvm and is, generally, heavyweight. Use WebHDFS if you can.

Sent from Outlook Mobile<https://aka.ms/qtex0l>




On Fri, Feb 26, 2016 at 9:13 AM -0800, "Shuai Lin" <li...@gmail.com>> wrote:

If you don't want to configure hadoop on your mesos slaves, the only workaround I see is to write a "hadoop" script and put it in your PATH. It need to support the following usage patterns:

- hadoop version
- hadoop fs -copyToLocal s3n://path /target/directory/

On Sat, Feb 27, 2016 at 12:31 AM, Aaron Carey <ac...@ilm.com>> wrote:
I was trying to avoid generating urls for everything as this will complicate things a lot.

Is there a straight forward way to get the fetcher to do it directly?

________________________________
From: haosdent [haosdent@gmail.com<ma...@gmail.com>]
Sent: 26 February 2016 16:27
To: user
Subject: Re: Downloading s3 uris

I think still could pass AWSAccessKeyId if it is private? http://www.bucketexplorer.com/documentation/amazon-s3--how-to-generate-url-for-amazon-s3-files.html

On Sat, Feb 27, 2016 at 12:25 AM, Abhishek Amralkar <Ab...@talentica.com>> wrote:
In that case do we need to keep bucket/files public?

-Abhishek

From: Zhitao Li <zh...@uber.com>>
Reply-To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Date: Friday, 26 February 2016 at 8:23 AM
To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Subject: Re: Downloading s3 uris

Haven't directly used s3 download, but I think a workaround (if you don't care ACL about the files) is to use http<http://stackoverflow.com/questions/18239567/how-can-i-download-a-file-from-an-s3-bucket-with-wget> url instead.
On Feb 26, 2016, at 8:17 AM, Aaron Carey <ac...@ilm.com>> wrote:

I'm attempting to fetch files from s3 uris in mesos, but we're not using hdfs in our cluster... however I believe I need the client installed.

Is it possible to just have the client running without a full hdfs setup?

I haven't been able to find much information in the docs, could someone point me in the right direction?

Thanks!

Aaron




--
Best Regards,
Haosdent Huang


Re: Downloading s3 uris

Posted by Radoslaw Gruchalski <ra...@gruchalski.com>.
Just keep in mind that every execution of such command starts a jvm and is, generally, heavyweight. Use WebHDFS if you can.

Sent from Outlook Mobile




On Fri, Feb 26, 2016 at 9:13 AM -0800, "Shuai Lin" <li...@gmail.com> wrote:










If you don't want to configure hadoop on your mesos slaves, the only workaround I see is to write a "hadoop" script and put it in your PATH. It need to support the following usage patterns:
- hadoop version- hadoop fs -copyToLocal s3n://path /target/directory/
On Sat, Feb 27, 2016 at 12:31 AM, Aaron Carey <ac...@ilm.com> wrote:






I was trying to avoid generating urls for everything as this will complicate things a lot.



Is there a straight forward way to get the fetcher to do it directly?





From: haosdent [haosdent@gmail.com]

Sent: 26 February 2016 16:27

To: user

Subject: Re: Downloading s3 uris






I think still could pass AWSAccessKeyId if it is private? http://www.bucketexplorer.com/documentation/amazon-s3--how-to-generate-url-for-amazon-s3-files.html


On Sat, Feb 27, 2016 at 12:25 AM, Abhishek Amralkar 
<Ab...@talentica.com> wrote:



In that case do we need to keep bucket/files public?



-Abhishek





From: Zhitao Li <zh...@uber.com>

Reply-To: "user@mesos.apache.org" <us...@mesos.apache.org>

Date: Friday, 26 February 2016 at 8:23 AM

To: "user@mesos.apache.org" <us...@mesos.apache.org>

Subject: Re: Downloading s3 uris








Haven't directly used s3 download, but I think a workaround (if you don't care ACL about the files) is to use
 http url instead.



On Feb 26, 2016, at 8:17 AM, Aaron Carey <ac...@ilm.com> wrote:




I'm attempting to fetch files from s3 uris in mesos, but we're not using hdfs in our cluster... however I believe I need the client installed.



Is it possible to just have the client running without a full hdfs setup?



I haven't been able to find much information in the docs, could someone point me in the right direction?



Thanks!



Aaron



















-- 

Best Regards,

Haosdent Huang















Re: Downloading s3 uris

Posted by Shuai Lin <li...@gmail.com>.
If you don't want to configure hadoop on your mesos slaves, the only
workaround I see is to write a "hadoop" script and put it in your PATH. It
need to support the following usage patterns:

- hadoop version
- hadoop fs -copyToLocal s3n://path /target/directory/

On Sat, Feb 27, 2016 at 12:31 AM, Aaron Carey <ac...@ilm.com> wrote:

> I was trying to avoid generating urls for everything as this will
> complicate things a lot.
>
> Is there a straight forward way to get the fetcher to do it directly?
>
> ------------------------------
> *From:* haosdent [haosdent@gmail.com]
> *Sent:* 26 February 2016 16:27
> *To:* user
> *Subject:* Re: Downloading s3 uris
>
> I think still could pass AWSAccessKeyId if it is private?
> http://www.bucketexplorer.com/documentation/amazon-s3--how-to-generate-url-for-amazon-s3-files.html
>
> On Sat, Feb 27, 2016 at 12:25 AM, Abhishek Amralkar <
> Abhishek.Amralkar@talentica.com> wrote:
>
>> In that case do we need to keep bucket/files public?
>>
>> -Abhishek
>>
>> From: Zhitao Li <zh...@uber.com>
>> Reply-To: "user@mesos.apache.org" <us...@mesos.apache.org>
>> Date: Friday, 26 February 2016 at 8:23 AM
>> To: "user@mesos.apache.org" <us...@mesos.apache.org>
>> Subject: Re: Downloading s3 uris
>>
>> Haven't directly used s3 download, but I think a workaround (if you don't
>> care ACL about the files) is to use http
>> <http://stackoverflow.com/questions/18239567/how-can-i-download-a-file-from-an-s3-bucket-with-wget> url
>> instead.
>>
>> On Feb 26, 2016, at 8:17 AM, Aaron Carey <ac...@ilm.com> wrote:
>>
>> I'm attempting to fetch files from s3 uris in mesos, but we're not using
>> hdfs in our cluster... however I believe I need the client installed.
>>
>> Is it possible to just have the client running without a full hdfs setup?
>>
>> I haven't been able to find much information in the docs, could someone
>> point me in the right direction?
>>
>> Thanks!
>>
>> Aaron
>>
>>
>>
>
>
> --
> Best Regards,
> Haosdent Huang
>

RE: Downloading s3 uris

Posted by Aaron Carey <ac...@ilm.com>.
I know this is a stupid question...

but how do I just install the client without all the rest of the stuff?

________________________________
From: haosdent [haosdent@gmail.com]
Sent: 26 February 2016 16:50
To: user
Subject: Re: Downloading s3 uris

So far have to install a hdfs client if you want to use "s3n://xxx". :-(

On Sat, Feb 27, 2016 at 12:39 AM, Abhishek Amralkar <Ab...@talentica.com>> wrote:
Agreed with @aaron, it will be too much manual work to generate S3 url everytime.

Thanks
Abhishek

From: Aaron Carey <ac...@ilm.com>>
Reply-To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Date: Friday, 26 February 2016 at 8:31 AM
To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Subject: RE: Downloading s3 uris

I was trying to avoid generating urls for everything as this will complicate things a lot.

Is there a straight forward way to get the fetcher to do it directly?

________________________________
From: haosdent [haosdent@gmail.com<ma...@gmail.com>]
Sent: 26 February 2016 16:27
To: user
Subject: Re: Downloading s3 uris

I think still could pass AWSAccessKeyId if it is private? http://www.bucketexplorer.com/documentation/amazon-s3--how-to-generate-url-for-amazon-s3-files.html

On Sat, Feb 27, 2016 at 12:25 AM, Abhishek Amralkar <Ab...@talentica.com>> wrote:
In that case do we need to keep bucket/files public?

-Abhishek

From: Zhitao Li <zh...@uber.com>>
Reply-To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Date: Friday, 26 February 2016 at 8:23 AM
To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Subject: Re: Downloading s3 uris

Haven't directly used s3 download, but I think a workaround (if you don't care ACL about the files) is to use http<http://stackoverflow.com/questions/18239567/how-can-i-download-a-file-from-an-s3-bucket-with-wget> url instead.
On Feb 26, 2016, at 8:17 AM, Aaron Carey <ac...@ilm.com>> wrote:

I'm attempting to fetch files from s3 uris in mesos, but we're not using hdfs in our cluster... however I believe I need the client installed.

Is it possible to just have the client running without a full hdfs setup?

I haven't been able to find much information in the docs, could someone point me in the right direction?

Thanks!

Aaron




--
Best Regards,
Haosdent Huang



--
Best Regards,
Haosdent Huang

Re: Downloading s3 uris

Posted by haosdent <ha...@gmail.com>.
So far have to install a hdfs client if you want to use "s3n://xxx". :-(

On Sat, Feb 27, 2016 at 12:39 AM, Abhishek Amralkar <
Abhishek.Amralkar@talentica.com> wrote:

> Agreed with @aaron, it will be too much manual work to generate S3 url
> everytime.
>
> Thanks
> Abhishek
>
> From: Aaron Carey <ac...@ilm.com>
> Reply-To: "user@mesos.apache.org" <us...@mesos.apache.org>
> Date: Friday, 26 February 2016 at 8:31 AM
> To: "user@mesos.apache.org" <us...@mesos.apache.org>
> Subject: RE: Downloading s3 uris
>
> I was trying to avoid generating urls for everything as this will
> complicate things a lot.
>
> Is there a straight forward way to get the fetcher to do it directly?
>
> ------------------------------
> *From:* haosdent [haosdent@gmail.com]
> *Sent:* 26 February 2016 16:27
> *To:* user
> *Subject:* Re: Downloading s3 uris
>
> I think still could pass AWSAccessKeyId if it is private?
> http://www.bucketexplorer.com/documentation/amazon-s3--how-to-generate-url-for-amazon-s3-files.html
>
> On Sat, Feb 27, 2016 at 12:25 AM, Abhishek Amralkar <
> Abhishek.Amralkar@talentica.com> wrote:
>
>> In that case do we need to keep bucket/files public?
>>
>> -Abhishek
>>
>> From: Zhitao Li <zh...@uber.com>
>> Reply-To: "user@mesos.apache.org" <us...@mesos.apache.org>
>> Date: Friday, 26 February 2016 at 8:23 AM
>> To: "user@mesos.apache.org" <us...@mesos.apache.org>
>> Subject: Re: Downloading s3 uris
>>
>> Haven't directly used s3 download, but I think a workaround (if you don't
>> care ACL about the files) is to use http
>> <http://stackoverflow.com/questions/18239567/how-can-i-download-a-file-from-an-s3-bucket-with-wget> url
>> instead.
>>
>> On Feb 26, 2016, at 8:17 AM, Aaron Carey <ac...@ilm.com> wrote:
>>
>> I'm attempting to fetch files from s3 uris in mesos, but we're not using
>> hdfs in our cluster... however I believe I need the client installed.
>>
>> Is it possible to just have the client running without a full hdfs setup?
>>
>> I haven't been able to find much information in the docs, could someone
>> point me in the right direction?
>>
>> Thanks!
>>
>> Aaron
>>
>>
>>
>
>
> --
> Best Regards,
> Haosdent Huang
>



-- 
Best Regards,
Haosdent Huang

RE: Downloading s3 uris

Posted by Aaron Carey <ac...@ilm.com>.
I was trying to avoid generating urls for everything as this will complicate things a lot.

Is there a straight forward way to get the fetcher to do it directly?

________________________________
From: haosdent [haosdent@gmail.com]
Sent: 26 February 2016 16:27
To: user
Subject: Re: Downloading s3 uris

I think still could pass AWSAccessKeyId if it is private? http://www.bucketexplorer.com/documentation/amazon-s3--how-to-generate-url-for-amazon-s3-files.html

On Sat, Feb 27, 2016 at 12:25 AM, Abhishek Amralkar <Ab...@talentica.com>> wrote:
In that case do we need to keep bucket/files public?

-Abhishek

From: Zhitao Li <zh...@uber.com>>
Reply-To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Date: Friday, 26 February 2016 at 8:23 AM
To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Subject: Re: Downloading s3 uris

Haven't directly used s3 download, but I think a workaround (if you don't care ACL about the files) is to use http<http://stackoverflow.com/questions/18239567/how-can-i-download-a-file-from-an-s3-bucket-with-wget> url instead.
On Feb 26, 2016, at 8:17 AM, Aaron Carey <ac...@ilm.com>> wrote:

I'm attempting to fetch files from s3 uris in mesos, but we're not using hdfs in our cluster... however I believe I need the client installed.

Is it possible to just have the client running without a full hdfs setup?

I haven't been able to find much information in the docs, could someone point me in the right direction?

Thanks!

Aaron




--
Best Regards,
Haosdent Huang

Re: Downloading s3 uris

Posted by haosdent <ha...@gmail.com>.
I think still could pass AWSAccessKeyId if it is private?
http://www.bucketexplorer.com/documentation/amazon-s3--how-to-generate-url-for-amazon-s3-files.html

On Sat, Feb 27, 2016 at 12:25 AM, Abhishek Amralkar <
Abhishek.Amralkar@talentica.com> wrote:

> In that case do we need to keep bucket/files public?
>
> -Abhishek
>
> From: Zhitao Li <zh...@uber.com>
> Reply-To: "user@mesos.apache.org" <us...@mesos.apache.org>
> Date: Friday, 26 February 2016 at 8:23 AM
> To: "user@mesos.apache.org" <us...@mesos.apache.org>
> Subject: Re: Downloading s3 uris
>
> Haven't directly used s3 download, but I think a workaround (if you don't
> care ACL about the files) is to use http
> <http://stackoverflow.com/questions/18239567/how-can-i-download-a-file-from-an-s3-bucket-with-wget> url
> instead.
>
> On Feb 26, 2016, at 8:17 AM, Aaron Carey <ac...@ilm.com> wrote:
>
> I'm attempting to fetch files from s3 uris in mesos, but we're not using
> hdfs in our cluster... however I believe I need the client installed.
>
> Is it possible to just have the client running without a full hdfs setup?
>
> I haven't been able to find much information in the docs, could someone
> point me in the right direction?
>
> Thanks!
>
> Aaron
>
>
>


-- 
Best Regards,
Haosdent Huang

Re: Downloading s3 uris

Posted by Zhitao Li <zh...@uber.com>.
Haven't directly used s3 download, but I think a workaround (if you don't care ACL about the files) is to use http <http://stackoverflow.com/questions/18239567/how-can-i-download-a-file-from-an-s3-bucket-with-wget> url instead.
> On Feb 26, 2016, at 8:17 AM, Aaron Carey <ac...@ilm.com> wrote:
> 
> I'm attempting to fetch files from s3 uris in mesos, but we're not using hdfs in our cluster... however I believe I need the client installed.
> 
> Is it possible to just have the client running without a full hdfs setup?
> 
> I haven't been able to find much information in the docs, could someone point me in the right direction?
> 
> Thanks!
> 
> Aaron