You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@griffin.apache.org by William Guo <gu...@apache.org> on 2018/04/30 14:47:26 UTC

[DISCUSS] Support CLI launch for spark jobs

hi all,

In some cluster environment, administrator doesn't install livy in their
cluster.

How can we support these users, should we support it by launching spark
jobs by CLI?


What are your comments for this issue?


Thanks,
William

Re: [DISCUSS] Support CLI launch for spark jobs

Posted by Lionel Liu <li...@apache.org>.

I agree that users should be blocked if they don't install livy, livy is
easy for spark job submitting, but we also need some other approach like
submitting in shell way or using APIs of yarn.

However, there're more spikes to adapt to different spark clusters, like
yarn or mesos, which is not the problem we want to solve in data quality
domain.

In my opinion, livy focuses on connecting to spark cluster and works well,
it could be the default solution.
We can also provide a simple way to submit spark jobs, for the basic
functions only, to support the specific environment, that'll be enough. If
we want to get job state, retry the job if failed, it will cost more
efforts.

Thanks,
Lionel

On Mon, Apr 30, 2018 at 10:47 PM, William Guo <gu...@apache.org> wrote:

> hi all,
>
> In some cluster environment, administrator doesn't install livy in their
> cluster.
>
> How can we support these users, should we support it by launching spark
> jobs by CLI?
>
>
> What are your comments for this issue?
>
>
> Thanks,
> William
>

Re: [DISCUSS] Support CLI launch for spark jobs

Posted by Enrico D'Urso <a-...@hotels.com>.

Hi William,

I would use a very light webserver listening for SparkJob submitting request, then I would just run a sub-process running:
spark-submit –master yarn ….

The above process should be enabled only and only if Livy is not available.

Thanks,

Enrico

On 4/30/18, 3:48 PM, "William Guo" <gu...@apache.org> wrote:

    hi all,
    
    In some cluster environment, administrator doesn't install livy in their
    cluster.
    
    How can we support these users, should we support it by launching spark
    jobs by CLI?
    
    
    What are your comments for this issue?
    
    
    Thanks,
    William

Re: [DISCUSS] Support CLI launch for spark jobs

Posted by Lionel Liu <li...@apache.org>.

I agree that users should be blocked if they don't install livy, livy is
easy for spark job submitting, but we also need some other approach like
submitting in shell way or using APIs of yarn.

However, there're more spikes to adapt to different spark clusters, like
yarn or mesos, which is not the problem we want to solve in data quality
domain.

In my opinion, livy focuses on connecting to spark cluster and works well,
it could be the default solution.
We can also provide a simple way to submit spark jobs, for the basic
functions only, to support the specific environment, that'll be enough. If
we want to get job state, retry the job if failed, it will cost more
efforts.

Thanks,
Lionel

On Mon, Apr 30, 2018 at 10:47 PM, William Guo <gu...@apache.org> wrote:

> hi all,
>
> In some cluster environment, administrator doesn't install livy in their
> cluster.
>
> How can we support these users, should we support it by launching spark
> jobs by CLI?
>
>
> What are your comments for this issue?
>
>
> Thanks,
> William
>