You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ofbiz.apache.org by Skip <sk...@thedevers.org> on 2013/06/15 02:21:46 UTC
findListIteratorByCondition in 12.04
I have used findListIteratorByCondition successfully in the past to process
large files with no transaction in place.
By putting a TransactionUtil.begin/end around the
findListIteratorByCondition and a second TransactionUtil.begin/end around
the actual writing of the data.
This all worked just fine. Now, with 12.04 I get a (ERROR: portal "C_xxx"
does not exist) after processing about 500 or so records (not sure of the
actual number) on the iterator.next() call.
If I remove the begin/end around the findListIteratorByCondition, everything
runs fine. The only problem is that I get an exception report (no actual
exception) from GenericDelegator that sez (ERROR: Cannot do a find that
returns an EntityListIterator with no transaction in place.)
This is not a thrown exception, only one that is displayed in the log.
So, I have two questions.
1. Why is it dangerous enough to not have a transaction in place to print
this exception in the log but not throw an actual exception.
2. Given my need above (where I might spend an hour processing huge amounts
of data), what is the recommended way to use findListIteratorByCondition and
only have a transaction around the actual commits?
Thanks in Advance
Skip
Re: findListIteratorByCondition in 12.04
Posted by Adrian Crum <ad...@sandglass-software.com>.
Add a flag (or indicator) column to the table being read and set it to
true when each record is processed. The find condition includes an
expression to check for the flag being false.
-Adrian
On 6/19/2013 1:50 AM, Skip wrote:
> Adrian
>
> Thanks for the response. In my case, I do not think that data consistancy
> is an issue because a single thread is reading the incoming data.
>
> The worst of these is a price change operation where a wholesaler sends this
> customer a list of all products and their current selling price (as well as
> other info).
>
> This application scans through this list and does lots of stuff to it like
> change price, add products if they do not exist, expire ones that do and are
> no longer supplied, etc.
>
>
> The only way I can see to do this "breaking up the set into smaller chunks
> would be to"
>
> Transaction.begin
> iter = findListIteratorByCondition(..)
> List thisBatch = iter.getPartialList(..)
> iter.close()
> Transaction end
> Rinse and repeat
>
> Each time, you are reading over from the beginning of the file. With
> typically 1.4 million records to process for one of these price change
> operations, I would think that this would add a considerable time to the
> total process. It currently takes 7 hours to run on a spiffy 32 gig
> machine.
>
> Is there an alternate way to avoid reading from the beginning of the file
> for each set?
>
> Skip
>
>
> -----Original Message-----
> From: Adrian Crum [mailto:adrian.crum@sandglass-software.com]
> Sent: Saturday, June 15, 2013 1:09 AM
> To: user@ofbiz.apache.org
> Subject: Re: findListIteratorByCondition in 12.04
>
>
> Transactions are recommended for queries so you can ensure consistent data.
>
> Put your large processing jobs inside a loop that processes smaller sets
> of data - like 100 records or so.
>
> -Adrian
>
> On 6/15/2013 1:21 AM, Skip wrote:
>> I have used findListIteratorByCondition successfully in the past to
> process
>> large files with no transaction in place.
>>
>> By putting a TransactionUtil.begin/end around the
>> findListIteratorByCondition and a second TransactionUtil.begin/end around
>> the actual writing of the data.
>>
>> This all worked just fine. Now, with 12.04 I get a (ERROR: portal "C_xxx"
>> does not exist) after processing about 500 or so records (not sure of the
>> actual number) on the iterator.next() call.
>>
>> If I remove the begin/end around the findListIteratorByCondition,
> everything
>> runs fine. The only problem is that I get an exception report (no actual
>> exception) from GenericDelegator that sez (ERROR: Cannot do a find that
>> returns an EntityListIterator with no transaction in place.)
>>
>> This is not a thrown exception, only one that is displayed in the log.
>>
>> So, I have two questions.
>>
>> 1. Why is it dangerous enough to not have a transaction in place to print
>> this exception in the log but not throw an actual exception.
>>
>> 2. Given my need above (where I might spend an hour processing huge
> amounts
>> of data), what is the recommended way to use findListIteratorByCondition
> and
>> only have a transaction around the actual commits?
>>
>> Thanks in Advance
>>
>> Skip
>>
>>
>
RE: findListIteratorByCondition in 12.04
Posted by Skip <sk...@thedevers.org>.
Adrian
Thanks for the response. In my case, I do not think that data consistancy
is an issue because a single thread is reading the incoming data.
The worst of these is a price change operation where a wholesaler sends this
customer a list of all products and their current selling price (as well as
other info).
This application scans through this list and does lots of stuff to it like
change price, add products if they do not exist, expire ones that do and are
no longer supplied, etc.
The only way I can see to do this "breaking up the set into smaller chunks
would be to"
Transaction.begin
iter = findListIteratorByCondition(..)
List thisBatch = iter.getPartialList(..)
iter.close()
Transaction end
Rinse and repeat
Each time, you are reading over from the beginning of the file. With
typically 1.4 million records to process for one of these price change
operations, I would think that this would add a considerable time to the
total process. It currently takes 7 hours to run on a spiffy 32 gig
machine.
Is there an alternate way to avoid reading from the beginning of the file
for each set?
Skip
-----Original Message-----
From: Adrian Crum [mailto:adrian.crum@sandglass-software.com]
Sent: Saturday, June 15, 2013 1:09 AM
To: user@ofbiz.apache.org
Subject: Re: findListIteratorByCondition in 12.04
Transactions are recommended for queries so you can ensure consistent data.
Put your large processing jobs inside a loop that processes smaller sets
of data - like 100 records or so.
-Adrian
On 6/15/2013 1:21 AM, Skip wrote:
> I have used findListIteratorByCondition successfully in the past to
process
> large files with no transaction in place.
>
> By putting a TransactionUtil.begin/end around the
> findListIteratorByCondition and a second TransactionUtil.begin/end around
> the actual writing of the data.
>
> This all worked just fine. Now, with 12.04 I get a (ERROR: portal "C_xxx"
> does not exist) after processing about 500 or so records (not sure of the
> actual number) on the iterator.next() call.
>
> If I remove the begin/end around the findListIteratorByCondition,
everything
> runs fine. The only problem is that I get an exception report (no actual
> exception) from GenericDelegator that sez (ERROR: Cannot do a find that
> returns an EntityListIterator with no transaction in place.)
>
> This is not a thrown exception, only one that is displayed in the log.
>
> So, I have two questions.
>
> 1. Why is it dangerous enough to not have a transaction in place to print
> this exception in the log but not throw an actual exception.
>
> 2. Given my need above (where I might spend an hour processing huge
amounts
> of data), what is the recommended way to use findListIteratorByCondition
and
> only have a transaction around the actual commits?
>
> Thanks in Advance
>
> Skip
>
>
Re: findListIteratorByCondition in 12.04
Posted by Adrian Crum <ad...@sandglass-software.com>.
Transactions are recommended for queries so you can ensure consistent data.
Put your large processing jobs inside a loop that processes smaller sets
of data - like 100 records or so.
-Adrian
On 6/15/2013 1:21 AM, Skip wrote:
> I have used findListIteratorByCondition successfully in the past to process
> large files with no transaction in place.
>
> By putting a TransactionUtil.begin/end around the
> findListIteratorByCondition and a second TransactionUtil.begin/end around
> the actual writing of the data.
>
> This all worked just fine. Now, with 12.04 I get a (ERROR: portal "C_xxx"
> does not exist) after processing about 500 or so records (not sure of the
> actual number) on the iterator.next() call.
>
> If I remove the begin/end around the findListIteratorByCondition, everything
> runs fine. The only problem is that I get an exception report (no actual
> exception) from GenericDelegator that sez (ERROR: Cannot do a find that
> returns an EntityListIterator with no transaction in place.)
>
> This is not a thrown exception, only one that is displayed in the log.
>
> So, I have two questions.
>
> 1. Why is it dangerous enough to not have a transaction in place to print
> this exception in the log but not throw an actual exception.
>
> 2. Given my need above (where I might spend an hour processing huge amounts
> of data), what is the recommended way to use findListIteratorByCondition and
> only have a transaction around the actual commits?
>
> Thanks in Advance
>
> Skip
>
>