Multiple queries to single Spark interpreter

[Sourav Mazumder <so...@gmail.com>]

Hi,

I'm sending multiple queries/tasks to a single Zeppelin instance from
different browsers through various notebooks.

Now I understand all the queries/tasks will share the same Spark
Interpreter process. In that case will the Spark Interpreter process will
process those queries sequentially or will execute them in multiple threads
?

Regards,
Sourav

Re: Multiple queries to single Spark interpreter

[Zhong Wang <wa...@gmail.com>]

I don't think it should work on non-SQL jobs. Because all of the paragraphs
share the same interpreter process, what do you expect the interpreter to
do if you run multiple paragraphs with shared variables?

If you really want to run jobs concurrently, I think you may try these
solutions:
--
1. do not block on long-running spark jobs: try using threads and callbacks
2. try creating several spark interpreters. I believe they shouldn't block
on each other.

Best,
Zhong



On Thu, Feb 11, 2016 at 7:03 AM, Sourav Mazumder <
sourav.mazumder00@gmail.com> wrote:

> Will that work for non SQL jobs too ?
>
> Say I have opened two browser windows pointing to same zeppelin servers.
> I'm working  on separate notebook in each of them. I run different jobs
> (say MLLib job) on two different RDDs from each each of them in parallel. I
> do see a queue getting built up.
>
> Is there any other parameter which can help me in tuning this ?
>
> Regards,
> Sourav
>
> On Wed, Feb 10, 2016 at 5:54 PM, Zhong Wang <wa...@gmail.com>
> wrote:
>
>> I think this configuration of spark interpreter allows you to run queries
>> concurrently:
>>
>> zeppelin.spark.concurrentSQL=true
>>
>> Best,
>> Zhong
>>
>> On Wed, Feb 10, 2016 at 4:25 PM, Sourav Mazumder <
>> sourav.mazumder00@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I'm sending multiple queries/tasks to a single Zeppelin instance from
>>> different browsers through various notebooks.
>>>
>>> Now I understand all the queries/tasks will share the same Spark
>>> Interpreter process. In that case will the Spark Interpreter process will
>>> process those queries sequentially or will execute them in multiple threads
>>> ?
>>>
>>> Regards,
>>> Sourav
>>>
>>
>>
>

Re: Multiple queries to single Spark interpreter

[Sourav Mazumder <so...@gmail.com>]

Will that work for non SQL jobs too ?

Say I have opened two browser windows pointing to same zeppelin servers.
I'm working  on separate notebook in each of them. I run different jobs
(say MLLib job) on two different RDDs from each each of them in parallel. I
do see a queue getting built up.

Is there any other parameter which can help me in tuning this ?

Regards,
Sourav

On Wed, Feb 10, 2016 at 5:54 PM, Zhong Wang <wa...@gmail.com> wrote:

> I think this configuration of spark interpreter allows you to run queries
> concurrently:
>
> zeppelin.spark.concurrentSQL=true
>
> Best,
> Zhong
>
> On Wed, Feb 10, 2016 at 4:25 PM, Sourav Mazumder <
> sourav.mazumder00@gmail.com> wrote:
>
>> Hi,
>>
>> I'm sending multiple queries/tasks to a single Zeppelin instance from
>> different browsers through various notebooks.
>>
>> Now I understand all the queries/tasks will share the same Spark
>> Interpreter process. In that case will the Spark Interpreter process will
>> process those queries sequentially or will execute them in multiple threads
>> ?
>>
>> Regards,
>> Sourav
>>
>
>

Re: Multiple queries to single Spark interpreter

[Zhong Wang <wa...@gmail.com>]

I think this configuration of spark interpreter allows you to run queries
concurrently:

zeppelin.spark.concurrentSQL=true

Best,
Zhong

On Wed, Feb 10, 2016 at 4:25 PM, Sourav Mazumder <
sourav.mazumder00@gmail.com> wrote:

> Hi,
>
> I'm sending multiple queries/tasks to a single Zeppelin instance from
> different browsers through various notebooks.
>
> Now I understand all the queries/tasks will share the same Spark
> Interpreter process. In that case will the Spark Interpreter process will
> process those queries sequentially or will execute them in multiple threads
> ?
>
> Regards,
> Sourav
>