You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cayenne.apache.org by Hugi Thordarson <hu...@karlmenn.is> on 2015/06/18 23:30:52 UTC
Cayenne and threading
Hi all.
Coming from EOF I’ve been mentally trained to believe anything threaded will bite you in the ass in a bad way. Using Cayenne, I’m now writing threaded code and loving it, but I just wanted to double check with the community; Is it really OK for me to do stuff like I’m doing in the following code? It’s a somewhat extreme example since I'd never actually commit for each iteration, just looking for some good practices, advice or an “ARE YOU CRAZY?” comment :).
public void main( String[] argv ) {
ObjectContext oc = // get my ObjectContext
ExecutorService executorService = Executors.newFixedThreadPool( Runtime.getRuntime().availableProcessors() );
List<Receipt> receipts = oc.select( new SelectQuery<>( Receipt.class ) );
for( Receipt receipt : receipts ) {
executorService.execute( () -> {
receipt.setCreationDate( new Date() );
oc.commitChanges();
} );
}
executorService.shutdown();
}
Cheers,
- hugi
// Hugi Thordarson
// http://www.loftfar.is/ <http://www.loftfar.is/>
// s. 895-6688
Re: Cayenne and threading
Posted by do...@xsinet.co.za.
I've just done something like this, along these lines:
private class ProcessImages extends Task<Void>
{
private int imageCount = 0;
private int listSize;
@Override
protected Void call() throws Exception
{
String sql = ....
SQLTemplate imgQry = new SQLTemplate( ImageMeta.class, sql );
imgQry.setCacheStrategy( QueryCacheStrategy.NO_CACHE );
imgQry.setFetchingDataRows( true );
List<DataRow> imgList = getNewContext().performQuery( imgQry );
listSize = imgList.size();
imgList.parallelStream().forEach( this::processImage );
return null;
}
private void processImage( DataRow imgID )
{
updateProgress( ++imageCount, listSize );
ImageBlob imgBlob = Cayenne.objectForPK( getNewContext(),
ImageBlob.class, imgID.get( "MED_IM_ID" ) );
// do stuff to imgBlob
imgBlob.getObjectContext().commitChanges();
}
}
regards
Jurgen
-----Original Message-----
From: Hugi Thordarson
Sent: Friday, June 19, 2015 12:32 AM
To: user@cayenne.apache.org
Subject: Re: Cayenne and threading
“Contexts are cheap”. Word’s I’m living by from now on, thanks!
- hugi
> On 18. jún. 2015, at 22:29, John Huss <jo...@gmail.com> wrote:
>
> Contexts are cheap. You can create a new one on each iteration (or bind
> one
> to each thread ahead of time and reuse them) and localize the object and
> save. Or do the write as plain sql.
> On Thu, Jun 18, 2015 at 5:17 PM Hugi Thordarson <hu...@karlmenn.is> wrote:
>
>> Thanks for the reply John, I suspected it couldn’t really be *this* good
>> :).
>>
>> But if there are any takers, I’d love to hear how experienced folks would
>> approach multithreaded DB writes for tasks such as this using Cayenne.
>>
>> To make things more clear; what’s happening in the lambda in my actual
>> project is that I’m performing OCR on an image linked to each Receipt
>> object and storing the resulting text. So, basically; a long running
>> task,
>> performing an operation on fetched objects and updating them.
>>
>> When using EOF I’ve been bypassing DB writes entirely during the
>> multithreaded OCR stage by writing OCR results to disk. Once OCR is
>> complete for all receipts, I read the text files back form disk and
>> update
>> the DB.
>>
>> Optimally, I’d like to be able to update the objects (and save to the DB)
>> as I’m iterating through the receipt objects. Any ideas?
>>
>> Cheers,
>> - hugi
>>
>>
>>
>>> On 18. jún. 2015, at 21:48, John Huss <jo...@gmail.com> wrote:
>>>
>>> No, I don't think that's a good idea. The same context shouldn't be
>> shared
>>> among threads. If you're doing pure SQL operations (like SQLTemplate)
>> and
>>> not touching DataObjects you may be ok, but otherwise not.
>>>
>>> But yes, Executors are great, especially with lambdas.
>>>
>>> John
>>>
>>> On Thu, Jun 18, 2015 at 4:31 PM Hugi Thordarson <hu...@karlmenn.is>
>> wrote:
>>>
>>>> Hi all.
>>>>
>>>> Coming from EOF I’ve been mentally trained to believe anything threaded
>>>> will bite you in the ass in a bad way. Using Cayenne, I’m now writing
>>>> threaded code and loving it, but I just wanted to double check with the
>>>> community; Is it really OK for me to do stuff like I’m doing in the
>>>> following code? It’s a somewhat extreme example since I'd never
>>>> actually
>>>> commit for each iteration, just looking for some good practices, advice
>> or
>>>> an “ARE YOU CRAZY?” comment :).
>>>>
>>>> public void main( String[] argv ) {
>>>> ObjectContext oc = // get my ObjectContext
>>>>
>>>> ExecutorService executorService = Executors.newFixedThreadPool(
>>>> Runtime.getRuntime().availableProcessors() );
>>>>
>>>> List<Receipt> receipts = oc.select( new SelectQuery<>(
>>>> Receipt.class ) );
>>>>
>>>> for( Receipt receipt : receipts ) {
>>>> executorService.execute( () -> {
>>>> receipt.setCreationDate( new Date() );
>>>> oc.commitChanges();
>>>> } );
>>>> }
>>>>
>>>> executorService.shutdown();
>>>> }
>>>>
>>>> Cheers,
>>>> - hugi
>>>>
>>>> // Hugi Thordarson
>>>> // http://www.loftfar.is/ <http://www.loftfar.is/>
>>>> // s. 895-6688
>>>>
>>>>
>>
>>
Re: Cayenne and threading
Posted by Hugi Thordarson <hu...@karlmenn.is>.
“Contexts are cheap”. Word’s I’m living by from now on, thanks!
- hugi
> On 18. jún. 2015, at 22:29, John Huss <jo...@gmail.com> wrote:
>
> Contexts are cheap. You can create a new one on each iteration (or bind one
> to each thread ahead of time and reuse them) and localize the object and
> save. Or do the write as plain sql.
> On Thu, Jun 18, 2015 at 5:17 PM Hugi Thordarson <hu...@karlmenn.is> wrote:
>
>> Thanks for the reply John, I suspected it couldn’t really be *this* good
>> :).
>>
>> But if there are any takers, I’d love to hear how experienced folks would
>> approach multithreaded DB writes for tasks such as this using Cayenne.
>>
>> To make things more clear; what’s happening in the lambda in my actual
>> project is that I’m performing OCR on an image linked to each Receipt
>> object and storing the resulting text. So, basically; a long running task,
>> performing an operation on fetched objects and updating them.
>>
>> When using EOF I’ve been bypassing DB writes entirely during the
>> multithreaded OCR stage by writing OCR results to disk. Once OCR is
>> complete for all receipts, I read the text files back form disk and update
>> the DB.
>>
>> Optimally, I’d like to be able to update the objects (and save to the DB)
>> as I’m iterating through the receipt objects. Any ideas?
>>
>> Cheers,
>> - hugi
>>
>>
>>
>>> On 18. jún. 2015, at 21:48, John Huss <jo...@gmail.com> wrote:
>>>
>>> No, I don't think that's a good idea. The same context shouldn't be
>> shared
>>> among threads. If you're doing pure SQL operations (like SQLTemplate)
>> and
>>> not touching DataObjects you may be ok, but otherwise not.
>>>
>>> But yes, Executors are great, especially with lambdas.
>>>
>>> John
>>>
>>> On Thu, Jun 18, 2015 at 4:31 PM Hugi Thordarson <hu...@karlmenn.is>
>> wrote:
>>>
>>>> Hi all.
>>>>
>>>> Coming from EOF I’ve been mentally trained to believe anything threaded
>>>> will bite you in the ass in a bad way. Using Cayenne, I’m now writing
>>>> threaded code and loving it, but I just wanted to double check with the
>>>> community; Is it really OK for me to do stuff like I’m doing in the
>>>> following code? It’s a somewhat extreme example since I'd never actually
>>>> commit for each iteration, just looking for some good practices, advice
>> or
>>>> an “ARE YOU CRAZY?” comment :).
>>>>
>>>> public void main( String[] argv ) {
>>>> ObjectContext oc = // get my ObjectContext
>>>>
>>>> ExecutorService executorService = Executors.newFixedThreadPool(
>>>> Runtime.getRuntime().availableProcessors() );
>>>>
>>>> List<Receipt> receipts = oc.select( new SelectQuery<>(
>>>> Receipt.class ) );
>>>>
>>>> for( Receipt receipt : receipts ) {
>>>> executorService.execute( () -> {
>>>> receipt.setCreationDate( new Date() );
>>>> oc.commitChanges();
>>>> } );
>>>> }
>>>>
>>>> executorService.shutdown();
>>>> }
>>>>
>>>> Cheers,
>>>> - hugi
>>>>
>>>> // Hugi Thordarson
>>>> // http://www.loftfar.is/ <http://www.loftfar.is/>
>>>> // s. 895-6688
>>>>
>>>>
>>
>>
Re: Cayenne and threading
Posted by John Huss <jo...@gmail.com>.
Contexts are cheap. You can create a new one on each iteration (or bind one
to each thread ahead of time and reuse them) and localize the object and
save. Or do the write as plain sql.
On Thu, Jun 18, 2015 at 5:17 PM Hugi Thordarson <hu...@karlmenn.is> wrote:
> Thanks for the reply John, I suspected it couldn’t really be *this* good
> :).
>
> But if there are any takers, I’d love to hear how experienced folks would
> approach multithreaded DB writes for tasks such as this using Cayenne.
>
> To make things more clear; what’s happening in the lambda in my actual
> project is that I’m performing OCR on an image linked to each Receipt
> object and storing the resulting text. So, basically; a long running task,
> performing an operation on fetched objects and updating them.
>
> When using EOF I’ve been bypassing DB writes entirely during the
> multithreaded OCR stage by writing OCR results to disk. Once OCR is
> complete for all receipts, I read the text files back form disk and update
> the DB.
>
> Optimally, I’d like to be able to update the objects (and save to the DB)
> as I’m iterating through the receipt objects. Any ideas?
>
> Cheers,
> - hugi
>
>
>
> > On 18. jún. 2015, at 21:48, John Huss <jo...@gmail.com> wrote:
> >
> > No, I don't think that's a good idea. The same context shouldn't be
> shared
> > among threads. If you're doing pure SQL operations (like SQLTemplate)
> and
> > not touching DataObjects you may be ok, but otherwise not.
> >
> > But yes, Executors are great, especially with lambdas.
> >
> > John
> >
> > On Thu, Jun 18, 2015 at 4:31 PM Hugi Thordarson <hu...@karlmenn.is>
> wrote:
> >
> >> Hi all.
> >>
> >> Coming from EOF I’ve been mentally trained to believe anything threaded
> >> will bite you in the ass in a bad way. Using Cayenne, I’m now writing
> >> threaded code and loving it, but I just wanted to double check with the
> >> community; Is it really OK for me to do stuff like I’m doing in the
> >> following code? It’s a somewhat extreme example since I'd never actually
> >> commit for each iteration, just looking for some good practices, advice
> or
> >> an “ARE YOU CRAZY?” comment :).
> >>
> >> public void main( String[] argv ) {
> >> ObjectContext oc = // get my ObjectContext
> >>
> >> ExecutorService executorService = Executors.newFixedThreadPool(
> >> Runtime.getRuntime().availableProcessors() );
> >>
> >> List<Receipt> receipts = oc.select( new SelectQuery<>(
> >> Receipt.class ) );
> >>
> >> for( Receipt receipt : receipts ) {
> >> executorService.execute( () -> {
> >> receipt.setCreationDate( new Date() );
> >> oc.commitChanges();
> >> } );
> >> }
> >>
> >> executorService.shutdown();
> >> }
> >>
> >> Cheers,
> >> - hugi
> >>
> >> // Hugi Thordarson
> >> // http://www.loftfar.is/ <http://www.loftfar.is/>
> >> // s. 895-6688
> >>
> >>
>
>
Re: Cayenne and threading
Posted by Hugi Thordarson <hu...@karlmenn.is>.
Thanks for the reply John, I suspected it couldn’t really be *this* good :).
But if there are any takers, I’d love to hear how experienced folks would approach multithreaded DB writes for tasks such as this using Cayenne.
To make things more clear; what’s happening in the lambda in my actual project is that I’m performing OCR on an image linked to each Receipt object and storing the resulting text. So, basically; a long running task, performing an operation on fetched objects and updating them.
When using EOF I’ve been bypassing DB writes entirely during the multithreaded OCR stage by writing OCR results to disk. Once OCR is complete for all receipts, I read the text files back form disk and update the DB.
Optimally, I’d like to be able to update the objects (and save to the DB) as I’m iterating through the receipt objects. Any ideas?
Cheers,
- hugi
> On 18. jún. 2015, at 21:48, John Huss <jo...@gmail.com> wrote:
>
> No, I don't think that's a good idea. The same context shouldn't be shared
> among threads. If you're doing pure SQL operations (like SQLTemplate) and
> not touching DataObjects you may be ok, but otherwise not.
>
> But yes, Executors are great, especially with lambdas.
>
> John
>
> On Thu, Jun 18, 2015 at 4:31 PM Hugi Thordarson <hu...@karlmenn.is> wrote:
>
>> Hi all.
>>
>> Coming from EOF I’ve been mentally trained to believe anything threaded
>> will bite you in the ass in a bad way. Using Cayenne, I’m now writing
>> threaded code and loving it, but I just wanted to double check with the
>> community; Is it really OK for me to do stuff like I’m doing in the
>> following code? It’s a somewhat extreme example since I'd never actually
>> commit for each iteration, just looking for some good practices, advice or
>> an “ARE YOU CRAZY?” comment :).
>>
>> public void main( String[] argv ) {
>> ObjectContext oc = // get my ObjectContext
>>
>> ExecutorService executorService = Executors.newFixedThreadPool(
>> Runtime.getRuntime().availableProcessors() );
>>
>> List<Receipt> receipts = oc.select( new SelectQuery<>(
>> Receipt.class ) );
>>
>> for( Receipt receipt : receipts ) {
>> executorService.execute( () -> {
>> receipt.setCreationDate( new Date() );
>> oc.commitChanges();
>> } );
>> }
>>
>> executorService.shutdown();
>> }
>>
>> Cheers,
>> - hugi
>>
>> // Hugi Thordarson
>> // http://www.loftfar.is/ <http://www.loftfar.is/>
>> // s. 895-6688
>>
>>
Re: Cayenne and threading
Posted by John Huss <jo...@gmail.com>.
No, I don't think that's a good idea. The same context shouldn't be shared
among threads. If you're doing pure SQL operations (like SQLTemplate) and
not touching DataObjects you may be ok, but otherwise not.
But yes, Executors are great, especially with lambdas.
John
On Thu, Jun 18, 2015 at 4:31 PM Hugi Thordarson <hu...@karlmenn.is> wrote:
> Hi all.
>
> Coming from EOF I’ve been mentally trained to believe anything threaded
> will bite you in the ass in a bad way. Using Cayenne, I’m now writing
> threaded code and loving it, but I just wanted to double check with the
> community; Is it really OK for me to do stuff like I’m doing in the
> following code? It’s a somewhat extreme example since I'd never actually
> commit for each iteration, just looking for some good practices, advice or
> an “ARE YOU CRAZY?” comment :).
>
> public void main( String[] argv ) {
> ObjectContext oc = // get my ObjectContext
>
> ExecutorService executorService = Executors.newFixedThreadPool(
> Runtime.getRuntime().availableProcessors() );
>
> List<Receipt> receipts = oc.select( new SelectQuery<>(
> Receipt.class ) );
>
> for( Receipt receipt : receipts ) {
> executorService.execute( () -> {
> receipt.setCreationDate( new Date() );
> oc.commitChanges();
> } );
> }
>
> executorService.shutdown();
> }
>
> Cheers,
> - hugi
>
> // Hugi Thordarson
> // http://www.loftfar.is/ <http://www.loftfar.is/>
> // s. 895-6688
>
>