Max limit for Jobs submission to Compute task

[Gaurav Bajaj <ga...@gmail.com>]

Hello,

I was going through various documentation and example of Compute Task to
find out if there is any limit to number Jobs which can be submitted to the
Compute task?
My use case is, I have 200,000 Jobs which I want to submit and do the
aggregation (reduce) on the result of the Job execution.

In other words, can the ArrayList in below example be of size 200000
objects.
Will Ignite do some kind of internal queuing to limit number of jobs which
can run in parallel?
I understand there is feature of Continuous Mapping, should that be used in
such scenarios?

// Execute closure on all cluster nodes.
    Collection<Integer> res = ignite.compute().apply(
    new IgniteClosure<String, Integer>() {
                   @Override public Integer apply(String word) {
                        System.out.println();
                        System.out.println(">>> Printing '" + word + "' on
this node from ignite job.");

                        // Return number of letters in the word.
                        return word.length();
                    }
                },

                // Job parameters. Ignite will create as many jobs as there
are parameters.
                Arrays.asList("Count characters using closure".split(" "))
            );

Thanks,
Gaurav

Re: Max limit for Jobs submission to Compute task

[vkulichenko <va...@gmail.com>]

Gaurav,

Thread pool and collision SPI are separate from each other. First one is
just a set of threads used to run user's code (jobs, closures, event
listeners, etc). While the latter is specific to Compute Grid and allows to
regulate how jobs are executed (limits, ordering, etc).

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Max-limit-for-Jobs-submission-to-Compute-task-tp9993p10041.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Max limit for Jobs submission to Compute task

[Gaurav Bajaj <ga...@gmail.com>]

Hello Val,

Thanks for the reply. I checked the documentation and it seems that by
default jobs are executed in random order which is not an issue for my use
case. I believe like other collision spis (FIFO and Priority ordering)
default thread pool size would be :
No. of cores * 2.  and rest would be queues internally by Ignite?

I will test this over coming days and update my findings.

Thanks,
Gaurav


On Tue, Jan 10, 2017 at 8:44 PM, vkulichenko <va...@gmail.com>
wrote:

> Hi Gaurav,
>
> There is no limitation and this approach should work. The number of jobs
> executed in parallel will be limited by number of threads on public pools
> and by collision SPI [1].
>
> [1] https://apacheignite.readme.io/docs/job-scheduling
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Max-limit-for-Jobs-submission-to-
> Compute-task-tp9993p10002.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Max limit for Jobs submission to Compute task

[vkulichenko <va...@gmail.com>]

Hi Gaurav,

There is no limitation and this approach should work. The number of jobs
executed in parallel will be limited by number of threads on public pools
and by collision SPI [1].

[1] https://apacheignite.readme.io/docs/job-scheduling

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Max-limit-for-Jobs-submission-to-Compute-task-tp9993p10002.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.