You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ofbiz.apache.org by Adrian Crum <ad...@sandglass-software.com> on 2012/08/12 13:36:02 UTC
Job Manager Part 2
On 8/5/2012 1:21 PM, Adrian Crum wrote:
> On 8/5/2012 11:02 AM, Adrian Crum wrote:
>> I just committed a bunch of changes to the Job Manager group of
>> classes. The changes help simplify the code and hopefully make the
>> Job Manager more robust. On the other hand, I might have broken
>> something. ;) I will monitor the mailing list for problems.
>>
>> I believe the JobPoller settings in serviceengine.xml (the
>> <thread-pool> element) should be changed. I think min-threads should
>> be set to "2" and max-threads should be set to "5". Creating lots of
>> threads can hurt throughput because the JVM spends more time managing
>> them. I would be interested in hearing what others think.
>
> Thinking about this more, there are some other things that need to be
> fixed:
>
> 1. The JobPoller uses an unbounded queue. In a busy server, there is
> the potential the queue will grow in size until it causes an
> out-of-memory condition.
> 2. There is no accommodation for when a job cannot be added to the
> queue - it is just lost. We could add a dequeue method to the Job
> interface that will allow implementations to recover or reschedule the
> job when it can't be added to the queue.
> 3. There is a JobPoller instance per delegator, and each instance
> contains the number of threads configured in serviceengine.xml. With
> the current max-threads setting of 15, a multi-tenant installation
> with 100 tenants will create up to 1500 threads. (!!!) A smarter
> strategy might be to have a single JobPoller instance that services
> multiple JobManagers.
I fixed #1 and #2. I am considering working on #3, but I want some
feedback first.
A JobPoller instance is created for each delegator. So, in a
multi-tenant or multi-delegator scenario, multiple JobPollers will be
created - which means one job queue per delegator and (threads per
queue) threads per delegator. In a multi-server installation, things
are multiplied: (# of servers * # of delegators) job queues.
Fortunately, in that scenario we can disable the JobPoller on all but
one server.
So, we are left with the potential problem of too many queues/threads
being created on a multi-delegator or multi-tenant server. So, I think
there should be one JobPoller instance that services all delegators. At
each polling interval, the JobPoller gets a list of jobs from each
delegator (JobManager) - creating a list of lists. Then the JobPoller
creates a queue candidate list from the list of lists - using a
round-robin approach so each delegator gets an equal opportunity to
queue jobs. The JobPoller queues the candidate list, and any candidates
that don't fit in the queue are rescheduled. With this approach the
JobPoller can service any number of delegators without saturating the
server.
What do you think?
-Adrian
Re: Job Manager Part 2
Posted by Jacques Le Roux <ja...@les7arts.com>.
This sounds indeed like a good, simple, and easy to maintain solution to this problem
+
Jacques
From: "Hans Bakker" <ma...@antwebsystems.com>
> Hi Adrian,
>
> thanks for the excellent work you did on mini language and also here for
> the background jobs.
>
> Your proposal sounds like the way to go so +1.
>
> Regards,
> Hans
>
> On 08/12/2012 06:36 PM, Adrian Crum wrote:
>> On 8/5/2012 1:21 PM, Adrian Crum wrote:
>>> On 8/5/2012 11:02 AM, Adrian Crum wrote:
>>>> I just committed a bunch of changes to the Job Manager group of
>>>> classes. The changes help simplify the code and hopefully make the
>>>> Job Manager more robust. On the other hand, I might have broken
>>>> something. ;) I will monitor the mailing list for problems.
>>>>
>>>> I believe the JobPoller settings in serviceengine.xml (the
>>>> <thread-pool> element) should be changed. I think min-threads should
>>>> be set to "2" and max-threads should be set to "5". Creating lots of
>>>> threads can hurt throughput because the JVM spends more time
>>>> managing them. I would be interested in hearing what others think.
>>>
>>> Thinking about this more, there are some other things that need to be
>>> fixed:
>>>
>>> 1. The JobPoller uses an unbounded queue. In a busy server, there is
>>> the potential the queue will grow in size until it causes an
>>> out-of-memory condition.
>>> 2. There is no accommodation for when a job cannot be added to the
>>> queue - it is just lost. We could add a dequeue method to the Job
>>> interface that will allow implementations to recover or reschedule
>>> the job when it can't be added to the queue.
>>> 3. There is a JobPoller instance per delegator, and each instance
>>> contains the number of threads configured in serviceengine.xml. With
>>> the current max-threads setting of 15, a multi-tenant installation
>>> with 100 tenants will create up to 1500 threads. (!!!) A smarter
>>> strategy might be to have a single JobPoller instance that services
>>> multiple JobManagers.
>>
>> I fixed #1 and #2. I am considering working on #3, but I want some
>> feedback first.
>>
>> A JobPoller instance is created for each delegator. So, in a
>> multi-tenant or multi-delegator scenario, multiple JobPollers will be
>> created - which means one job queue per delegator and (threads per
>> queue) threads per delegator. In a multi-server installation, things
>> are multiplied: (# of servers * # of delegators) job queues.
>> Fortunately, in that scenario we can disable the JobPoller on all but
>> one server.
>>
>> So, we are left with the potential problem of too many queues/threads
>> being created on a multi-delegator or multi-tenant server. So, I think
>> there should be one JobPoller instance that services all delegators.
>> At each polling interval, the JobPoller gets a list of jobs from each
>> delegator (JobManager) - creating a list of lists. Then the JobPoller
>> creates a queue candidate list from the list of lists - using a
>> round-robin approach so each delegator gets an equal opportunity to
>> queue jobs. The JobPoller queues the candidate list, and any
>> candidates that don't fit in the queue are rescheduled. With this
>> approach the JobPoller can service any number of delegators without
>> saturating the server.
>>
>> What do you think?
>>
>> -Adrian
>>
>
Re: Job Manager Part 2
Posted by Hans Bakker <ma...@antwebsystems.com>.
Hi Adrian,
thanks for the excellent work you did on mini language and also here for
the background jobs.
Your proposal sounds like the way to go so +1.
Regards,
Hans
On 08/12/2012 06:36 PM, Adrian Crum wrote:
> On 8/5/2012 1:21 PM, Adrian Crum wrote:
>> On 8/5/2012 11:02 AM, Adrian Crum wrote:
>>> I just committed a bunch of changes to the Job Manager group of
>>> classes. The changes help simplify the code and hopefully make the
>>> Job Manager more robust. On the other hand, I might have broken
>>> something. ;) I will monitor the mailing list for problems.
>>>
>>> I believe the JobPoller settings in serviceengine.xml (the
>>> <thread-pool> element) should be changed. I think min-threads should
>>> be set to "2" and max-threads should be set to "5". Creating lots of
>>> threads can hurt throughput because the JVM spends more time
>>> managing them. I would be interested in hearing what others think.
>>
>> Thinking about this more, there are some other things that need to be
>> fixed:
>>
>> 1. The JobPoller uses an unbounded queue. In a busy server, there is
>> the potential the queue will grow in size until it causes an
>> out-of-memory condition.
>> 2. There is no accommodation for when a job cannot be added to the
>> queue - it is just lost. We could add a dequeue method to the Job
>> interface that will allow implementations to recover or reschedule
>> the job when it can't be added to the queue.
>> 3. There is a JobPoller instance per delegator, and each instance
>> contains the number of threads configured in serviceengine.xml. With
>> the current max-threads setting of 15, a multi-tenant installation
>> with 100 tenants will create up to 1500 threads. (!!!) A smarter
>> strategy might be to have a single JobPoller instance that services
>> multiple JobManagers.
>
> I fixed #1 and #2. I am considering working on #3, but I want some
> feedback first.
>
> A JobPoller instance is created for each delegator. So, in a
> multi-tenant or multi-delegator scenario, multiple JobPollers will be
> created - which means one job queue per delegator and (threads per
> queue) threads per delegator. In a multi-server installation, things
> are multiplied: (# of servers * # of delegators) job queues.
> Fortunately, in that scenario we can disable the JobPoller on all but
> one server.
>
> So, we are left with the potential problem of too many queues/threads
> being created on a multi-delegator or multi-tenant server. So, I think
> there should be one JobPoller instance that services all delegators.
> At each polling interval, the JobPoller gets a list of jobs from each
> delegator (JobManager) - creating a list of lists. Then the JobPoller
> creates a queue candidate list from the list of lists - using a
> round-robin approach so each delegator gets an equal opportunity to
> queue jobs. The JobPoller queues the candidate list, and any
> candidates that don't fit in the queue are rescheduled. With this
> approach the JobPoller can service any number of delegators without
> saturating the server.
>
> What do you think?
>
> -Adrian
>
Re: Job Manager Part 2
Posted by Adrian Crum <ad...@sandglass-software.com>.
I just updated the schema with documentation that should help everyone
understand how to set up the Job Manager/Job Poller. It seems to me it
can accommodate the scenario you described.
-Adrian
On 8/13/2012 2:42 AM, Brett Palmer wrote:
> *Adrian,
>
> I think the single JobPoller is a good idea and reduces the chance of too
> many JobPoller’s running on a machine.
>
> We often setup multiple delegators to communicate with different databases.
> For example, our data warehouse is hosted on a separate server. These
> databases usually have a full ofbiz schema on them (jobsandbox table, etc).
>
> Here is how our data warehouse process works:
>
> The application has several ofbiz servers talking to a primary database.
> These servers contain all the user information for our application. When
> a person logs into the application they are redirected to a secondary ofbiz
> server that is used for running the application under heavy loads. The
> data is captured on the secondary server.
>
> A data warehouse process is scheduled to run every 5 mins on these
> secondary servers. The secondary servers have a delegator that talks to
> its local database and a delegator to talk to the data warehouse.
>
> With the new job poller changes would the poller pick up jobs from the data
> warehouse database since it has a delegator that points to that instance?
>
> For this example, we would need to make sure the job poller on the
> secondary server only serviced jobs from its local database (default
> delegator) and not the our configured olap delegator.
>
> Let me know if this will be possible with the new job poller or if you have
> any questions from my scenario. I realize this scenario is not the typical
> ofbiz e-commerce type of setup everyone is use to, but we use ofbiz to
> create lots of different types applications and have found it very flexible
> for creating just about any type of ERP application.
>
>
> Thanks for your work on the job poller.
>
>
> Brett *
>
> On Sun, Aug 12, 2012 at 5:36 AM, Adrian Crum <
> adrian.crum@sandglass-software.com> wrote:
>
>> On 8/5/2012 1:21 PM, Adrian Crum wrote:
>>
>>> On 8/5/2012 11:02 AM, Adrian Crum wrote:
>>>
>>>> I just committed a bunch of changes to the Job Manager group of classes.
>>>> The changes help simplify the code and hopefully make the Job Manager more
>>>> robust. On the other hand, I might have broken something. ;) I will monitor
>>>> the mailing list for problems.
>>>>
>>>> I believe the JobPoller settings in serviceengine.xml (the <thread-pool>
>>>> element) should be changed. I think min-threads should be set to "2" and
>>>> max-threads should be set to "5". Creating lots of threads can hurt
>>>> throughput because the JVM spends more time managing them. I would be
>>>> interested in hearing what others think.
>>>>
>>> Thinking about this more, there are some other things that need to be
>>> fixed:
>>>
>>> 1. The JobPoller uses an unbounded queue. In a busy server, there is the
>>> potential the queue will grow in size until it causes an out-of-memory
>>> condition.
>>> 2. There is no accommodation for when a job cannot be added to the queue
>>> - it is just lost. We could add a dequeue method to the Job interface that
>>> will allow implementations to recover or reschedule the job when it can't
>>> be added to the queue.
>>> 3. There is a JobPoller instance per delegator, and each instance
>>> contains the number of threads configured in serviceengine.xml. With the
>>> current max-threads setting of 15, a multi-tenant installation with 100
>>> tenants will create up to 1500 threads. (!!!) A smarter strategy might be
>>> to have a single JobPoller instance that services multiple JobManagers.
>>>
>> I fixed #1 and #2. I am considering working on #3, but I want some
>> feedback first.
>>
>> A JobPoller instance is created for each delegator. So, in a multi-tenant
>> or multi-delegator scenario, multiple JobPollers will be created - which
>> means one job queue per delegator and (threads per queue) threads per
>> delegator. In a multi-server installation, things are multiplied: (# of
>> servers * # of delegators) job queues. Fortunately, in that scenario we can
>> disable the JobPoller on all but one server.
>>
>> So, we are left with the potential problem of too many queues/threads
>> being created on a multi-delegator or multi-tenant server. So, I think
>> there should be one JobPoller instance that services all delegators. At
>> each polling interval, the JobPoller gets a list of jobs from each
>> delegator (JobManager) - creating a list of lists. Then the JobPoller
>> creates a queue candidate list from the list of lists - using a round-robin
>> approach so each delegator gets an equal opportunity to queue jobs. The
>> JobPoller queues the candidate list, and any candidates that don't fit in
>> the queue are rescheduled. With this approach the JobPoller can service any
>> number of delegators without saturating the server.
>>
>> What do you think?
>>
>> -Adrian
>>
>>
Re: Job Manager Part 2
Posted by Adrian Crum <ad...@sandglass-software.com>.
This change was made in rev 1374596.
The main thing to watch out for is the queue size. The default is 100,
but that might be too small in a multi-delegator or multi-tenant
installation. Follow the queue size advice in service-config.xsd.
-Adrian
On 8/13/2012 2:42 AM, Brett Palmer wrote:
> *Adrian,
>
> I think the single JobPoller is a good idea and reduces the chance of too
> many JobPoller’s running on a machine.
>
> We often setup multiple delegators to communicate with different databases.
> For example, our data warehouse is hosted on a separate server. These
> databases usually have a full ofbiz schema on them (jobsandbox table, etc).
>
> Here is how our data warehouse process works:
>
> The application has several ofbiz servers talking to a primary database.
> These servers contain all the user information for our application. When
> a person logs into the application they are redirected to a secondary ofbiz
> server that is used for running the application under heavy loads. The
> data is captured on the secondary server.
>
> A data warehouse process is scheduled to run every 5 mins on these
> secondary servers. The secondary servers have a delegator that talks to
> its local database and a delegator to talk to the data warehouse.
>
> With the new job poller changes would the poller pick up jobs from the data
> warehouse database since it has a delegator that points to that instance?
>
> For this example, we would need to make sure the job poller on the
> secondary server only serviced jobs from its local database (default
> delegator) and not the our configured olap delegator.
>
> Let me know if this will be possible with the new job poller or if you have
> any questions from my scenario. I realize this scenario is not the typical
> ofbiz e-commerce type of setup everyone is use to, but we use ofbiz to
> create lots of different types applications and have found it very flexible
> for creating just about any type of ERP application.
>
>
> Thanks for your work on the job poller.
>
>
> Brett *
>
> On Sun, Aug 12, 2012 at 5:36 AM, Adrian Crum <
> adrian.crum@sandglass-software.com> wrote:
>
>> On 8/5/2012 1:21 PM, Adrian Crum wrote:
>>
>>> On 8/5/2012 11:02 AM, Adrian Crum wrote:
>>>
>>>> I just committed a bunch of changes to the Job Manager group of classes.
>>>> The changes help simplify the code and hopefully make the Job Manager more
>>>> robust. On the other hand, I might have broken something. ;) I will monitor
>>>> the mailing list for problems.
>>>>
>>>> I believe the JobPoller settings in serviceengine.xml (the <thread-pool>
>>>> element) should be changed. I think min-threads should be set to "2" and
>>>> max-threads should be set to "5". Creating lots of threads can hurt
>>>> throughput because the JVM spends more time managing them. I would be
>>>> interested in hearing what others think.
>>>>
>>> Thinking about this more, there are some other things that need to be
>>> fixed:
>>>
>>> 1. The JobPoller uses an unbounded queue. In a busy server, there is the
>>> potential the queue will grow in size until it causes an out-of-memory
>>> condition.
>>> 2. There is no accommodation for when a job cannot be added to the queue
>>> - it is just lost. We could add a dequeue method to the Job interface that
>>> will allow implementations to recover or reschedule the job when it can't
>>> be added to the queue.
>>> 3. There is a JobPoller instance per delegator, and each instance
>>> contains the number of threads configured in serviceengine.xml. With the
>>> current max-threads setting of 15, a multi-tenant installation with 100
>>> tenants will create up to 1500 threads. (!!!) A smarter strategy might be
>>> to have a single JobPoller instance that services multiple JobManagers.
>>>
>> I fixed #1 and #2. I am considering working on #3, but I want some
>> feedback first.
>>
>> A JobPoller instance is created for each delegator. So, in a multi-tenant
>> or multi-delegator scenario, multiple JobPollers will be created - which
>> means one job queue per delegator and (threads per queue) threads per
>> delegator. In a multi-server installation, things are multiplied: (# of
>> servers * # of delegators) job queues. Fortunately, in that scenario we can
>> disable the JobPoller on all but one server.
>>
>> So, we are left with the potential problem of too many queues/threads
>> being created on a multi-delegator or multi-tenant server. So, I think
>> there should be one JobPoller instance that services all delegators. At
>> each polling interval, the JobPoller gets a list of jobs from each
>> delegator (JobManager) - creating a list of lists. Then the JobPoller
>> creates a queue candidate list from the list of lists - using a round-robin
>> approach so each delegator gets an equal opportunity to queue jobs. The
>> JobPoller queues the candidate list, and any candidates that don't fit in
>> the queue are rescheduled. With this approach the JobPoller can service any
>> number of delegators without saturating the server.
>>
>> What do you think?
>>
>> -Adrian
>>
>>
Re: Job Manager Part 2
Posted by Brett Palmer <br...@gmail.com>.
*Adrian,
I think the single JobPoller is a good idea and reduces the chance of too
many JobPoller’s running on a machine.
We often setup multiple delegators to communicate with different databases.
For example, our data warehouse is hosted on a separate server. These
databases usually have a full ofbiz schema on them (jobsandbox table, etc).
Here is how our data warehouse process works:
The application has several ofbiz servers talking to a primary database.
These servers contain all the user information for our application. When
a person logs into the application they are redirected to a secondary ofbiz
server that is used for running the application under heavy loads. The
data is captured on the secondary server.
A data warehouse process is scheduled to run every 5 mins on these
secondary servers. The secondary servers have a delegator that talks to
its local database and a delegator to talk to the data warehouse.
With the new job poller changes would the poller pick up jobs from the data
warehouse database since it has a delegator that points to that instance?
For this example, we would need to make sure the job poller on the
secondary server only serviced jobs from its local database (default
delegator) and not the our configured olap delegator.
Let me know if this will be possible with the new job poller or if you have
any questions from my scenario. I realize this scenario is not the typical
ofbiz e-commerce type of setup everyone is use to, but we use ofbiz to
create lots of different types applications and have found it very flexible
for creating just about any type of ERP application.
Thanks for your work on the job poller.
Brett *
On Sun, Aug 12, 2012 at 5:36 AM, Adrian Crum <
adrian.crum@sandglass-software.com> wrote:
> On 8/5/2012 1:21 PM, Adrian Crum wrote:
>
>> On 8/5/2012 11:02 AM, Adrian Crum wrote:
>>
>>> I just committed a bunch of changes to the Job Manager group of classes.
>>> The changes help simplify the code and hopefully make the Job Manager more
>>> robust. On the other hand, I might have broken something. ;) I will monitor
>>> the mailing list for problems.
>>>
>>> I believe the JobPoller settings in serviceengine.xml (the <thread-pool>
>>> element) should be changed. I think min-threads should be set to "2" and
>>> max-threads should be set to "5". Creating lots of threads can hurt
>>> throughput because the JVM spends more time managing them. I would be
>>> interested in hearing what others think.
>>>
>>
>> Thinking about this more, there are some other things that need to be
>> fixed:
>>
>> 1. The JobPoller uses an unbounded queue. In a busy server, there is the
>> potential the queue will grow in size until it causes an out-of-memory
>> condition.
>> 2. There is no accommodation for when a job cannot be added to the queue
>> - it is just lost. We could add a dequeue method to the Job interface that
>> will allow implementations to recover or reschedule the job when it can't
>> be added to the queue.
>> 3. There is a JobPoller instance per delegator, and each instance
>> contains the number of threads configured in serviceengine.xml. With the
>> current max-threads setting of 15, a multi-tenant installation with 100
>> tenants will create up to 1500 threads. (!!!) A smarter strategy might be
>> to have a single JobPoller instance that services multiple JobManagers.
>>
>
> I fixed #1 and #2. I am considering working on #3, but I want some
> feedback first.
>
> A JobPoller instance is created for each delegator. So, in a multi-tenant
> or multi-delegator scenario, multiple JobPollers will be created - which
> means one job queue per delegator and (threads per queue) threads per
> delegator. In a multi-server installation, things are multiplied: (# of
> servers * # of delegators) job queues. Fortunately, in that scenario we can
> disable the JobPoller on all but one server.
>
> So, we are left with the potential problem of too many queues/threads
> being created on a multi-delegator or multi-tenant server. So, I think
> there should be one JobPoller instance that services all delegators. At
> each polling interval, the JobPoller gets a list of jobs from each
> delegator (JobManager) - creating a list of lists. Then the JobPoller
> creates a queue candidate list from the list of lists - using a round-robin
> approach so each delegator gets an equal opportunity to queue jobs. The
> JobPoller queues the candidate list, and any candidates that don't fit in
> the queue are rescheduled. With this approach the JobPoller can service any
> number of delegators without saturating the server.
>
> What do you think?
>
> -Adrian
>
>