You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Hafiz Mujadid <ha...@gmail.com> on 2015/03/17 14:52:42 UTC
Downloading data from url
Hi experts!
Is there any api in spark to download data from url? I want to download data
from url in a spark application. I want to get downloading on all nodes
instead of a single node.
Thanks
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Downloading-data-from-url-tp22102.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Downloading data from url
Posted by Ted Yu <yu...@gmail.com>.
The methods I mentioned are private.
But I hope they give you some idea how downloading from url works.
Cheers
On Tue, Mar 17, 2015 at 7:01 AM, Ted Yu <yu...@gmail.com> wrote:
> Please take a look at downloadFile() method
> in ./core/src/main/scala/org/apache/spark/util/Utils.scala
>
> You can find usage in doFetchFile().
>
> FYI
>
> On Tue, Mar 17, 2015 at 6:52 AM, Hafiz Mujadid <ha...@gmail.com>
> wrote:
>
>> Hi experts!
>>
>>
>> Is there any api in spark to download data from url? I want to download
>> data
>> from url in a spark application. I want to get downloading on all nodes
>> instead of a single node.
>>
>> Thanks
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Downloading-data-from-url-tp22102.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: Downloading data from url
Posted by Ted Yu <yu...@gmail.com>.
Please take a look at downloadFile() method
in ./core/src/main/scala/org/apache/spark/util/Utils.scala
You can find usage in doFetchFile().
FYI
On Tue, Mar 17, 2015 at 6:52 AM, Hafiz Mujadid <ha...@gmail.com>
wrote:
> Hi experts!
>
>
> Is there any api in spark to download data from url? I want to download
> data
> from url in a spark application. I want to get downloading on all nodes
> instead of a single node.
>
> Thanks
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Downloading-data-from-url-tp22102.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>