You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Georg Heiler <ge...@gmail.com> on 2020/08/30 22:06:05 UTC
pip/conda distribution headless mode
Hi,
I want to use pyspark as distributed via conda in headless mode.
It looks like the hadoop binaries are bundles (= pip distributes a default
version)
https://stackoverflow.com/questions/63661404/bootstrap-spark-itself-on-yarn.
I want to ask if it would be possible to A) distribute the headless version
(=without hadoop) instead or B) distribute the headless version
additionally for pip & conda-forge distribution channels.
Best,
Georg
Re: pip/conda distribution headless mode
Posted by Hyukjin Kwon <gu...@gmail.com>.
I am going to take a look if nobody is interested in it.
2020년 8월 31일 (월) 오후 1:48, Georg Heiler <ge...@gmail.com>님이 작성:
> Many thanks.
>
> Best,
> Georg
>
> Am Mo., 31. Aug. 2020 um 01:12 Uhr schrieb Xiao Li <lixiao@databricks.com
> >:
>
>> Hi, Georg,
>>
>> This is being tracked by
>> https://issues.apache.org/jira/browse/SPARK-32017 You can leave comments
>> in the JIRA.
>>
>> Thanks,
>>
>> Xiao
>>
>> On Sun, Aug 30, 2020 at 3:06 PM Georg Heiler <ge...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I want to use pyspark as distributed via conda in headless mode.
>>> It looks like the hadoop binaries are bundles (= pip distributes a
>>> default version)
>>> https://stackoverflow.com/questions/63661404/bootstrap-spark-itself-on-yarn
>>> .
>>>
>>> I want to ask if it would be possible to A) distribute the headless
>>> version (=without hadoop) instead or B) distribute the headless version
>>> additionally for pip & conda-forge distribution channels.
>>>
>>> Best,
>>> Georg
>>>
>>
>>
>> --
>> <https://databricks.com/sparkaisummit/north-america>
>>
>
Re: pip/conda distribution headless mode
Posted by Georg Heiler <ge...@gmail.com>.
Many thanks.
Best,
Georg
Am Mo., 31. Aug. 2020 um 01:12 Uhr schrieb Xiao Li <li...@databricks.com>:
> Hi, Georg,
>
> This is being tracked by https://issues.apache.org/jira/browse/SPARK-32017 You
> can leave comments in the JIRA.
>
> Thanks,
>
> Xiao
>
> On Sun, Aug 30, 2020 at 3:06 PM Georg Heiler <ge...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I want to use pyspark as distributed via conda in headless mode.
>> It looks like the hadoop binaries are bundles (= pip distributes a
>> default version)
>> https://stackoverflow.com/questions/63661404/bootstrap-spark-itself-on-yarn
>> .
>>
>> I want to ask if it would be possible to A) distribute the headless
>> version (=without hadoop) instead or B) distribute the headless version
>> additionally for pip & conda-forge distribution channels.
>>
>> Best,
>> Georg
>>
>
>
> --
> <https://databricks.com/sparkaisummit/north-america>
>
Re: pip/conda distribution headless mode
Posted by Xiao Li <li...@databricks.com>.
Hi, Georg,
This is being tracked by https://issues.apache.org/jira/browse/SPARK-32017 You
can leave comments in the JIRA.
Thanks,
Xiao
On Sun, Aug 30, 2020 at 3:06 PM Georg Heiler <ge...@gmail.com>
wrote:
> Hi,
>
> I want to use pyspark as distributed via conda in headless mode.
> It looks like the hadoop binaries are bundles (= pip distributes a default
> version)
> https://stackoverflow.com/questions/63661404/bootstrap-spark-itself-on-yarn
> .
>
> I want to ask if it would be possible to A) distribute the headless
> version (=without hadoop) instead or B) distribute the headless version
> additionally for pip & conda-forge distribution channels.
>
> Best,
> Georg
>
--
<https://databricks.com/sparkaisummit/north-america>