You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cayenne.apache.org by Mark Wardle <ma...@wardle.org> on 2016/10/15 19:29:00 UTC
Batch iteration
I’m using ResultBatchIterator like this:
Consumer<T> forEach = ...
try (ResultBatchIterator<T> iterator = query.batchIterator(context, batchSize)) {
for(List<T> batch : iterator) {
for (T c : batch) {
forEach.accept(c);
}
}
}
but I also want to do some processing, inserting rows, editing objects etc. I see lots of SQL in my logs, but the transaction doesn’t get committed until the batch iterator completes, whether I use context.commitChanges() or context.performGenericQuery() on some custom SQL created using SQLTemplate.
Is there a way of committing the transaction earlier, or should I use a peer object context? Is there an easy of getting a new peer editing context given a single object context or do I need to inject a ServerRuntime here?
Thank you,
Mark
Re: Batch iteration
Posted by Mark Wardle <ma...@wardle.org>.
Thanks Andrus.
Mark
> On 17 Oct 2016, at 06:28, Andrus Adamchik <an...@objectstyle.org> wrote:
>
> Hi Mark,
>
> Scenario B is now the default after a recent change per [1]. We decoupled the transaction handling open iterator (and all its connections) from any transactions started by context commits within the iterator. Here is a related mailing list thread [2].
>
> Andrus
>
> [1] https://issues.apache.org/jira/browse/CAY-2111
> [2] https://lists.apache.org/thread.html/f1ccb1ac0f0aa974ffe32217fe4fcfe4b3d9a11ef2c8513f2f0e8436@%3Cdev.cayenne.apache.org%3E
>
>> On Oct 16, 2016, at 11:45 PM, Mark Wardle <ma...@wardle.org> wrote:
>>
>> Thanks John. Interestingly, I did some testing and peer contexts aren’t committed until the batch iterator JDBC ResultSet is closed at the end of iteration either. This is fine but for a long running task, I thought that there was a problem (but there wasn’t). I think the only workarounds are a) to not care [which I think is probably fine or b) do other processing in another connection.
>>
>> Mark
>>
>>> On 15 Oct 2016, at 22:34, John Huss <jo...@gmail.com> wrote:
>>>
>>> I would use a peer context.
>>>
>>> You can create a peer context like this if you runtime has been bound to
>>> the current thread using
>>> CayenneRuntime.bindThreadInjector(runtime.getInjector())
>>>
>>> public static ObjectContext newObjectContext() {
>>> ObjectContextFactory factory = CayenneRuntime.getThreadInjector
>>> ().getInstance(ObjectContextFactory.class);
>>> return (factory != null) ? factory.createContext() : null;
>>> }
>>>
>>>
>>>
>>> On Sat, Oct 15, 2016 at 2:37 PM Mark Wardle <ma...@wardle.org> wrote:
>>>
>>>> Sorry… meant to say using Cayenne M4.0.M3…
>>>>
>>>> Mark
>>>>
>>>>> On 15 Oct 2016, at 20:29, Mark Wardle <ma...@wardle.org> wrote:
>>>>>
>>>>> I’m using ResultBatchIterator like this:
>>>>>
>>>>> Consumer<T> forEach = ...
>>>>> try (ResultBatchIterator<T> iterator = query.batchIterator(context,
>>>> batchSize)) {
>>>>> for(List<T> batch : iterator) {
>>>>> for (T c : batch) {
>>>>> forEach.accept(c);
>>>>> }
>>>>> }
>>>>> }
>>>>>
>>>>> but I also want to do some processing, inserting rows, editing objects
>>>> etc. I see lots of SQL in my logs, but the transaction doesn’t get
>>>> committed until the batch iterator completes, whether I use
>>>> context.commitChanges() or context.performGenericQuery() on some custom SQL
>>>> created using SQLTemplate.
>>>>>
>>>>> Is there a way of committing the transaction earlier, or should I use a
>>>> peer object context? Is there an easy of getting a new peer editing context
>>>> given a single object context or do I need to inject a ServerRuntime here?
>>>>>
>>>>> Thank you,
>>>>>
>>>>> Mark
>>>>>
>>>>>
>>>>
>>>>
>>
>
Re: Batch iteration
Posted by Andrus Adamchik <an...@objectstyle.org>.
Hi Mark,
Scenario B is now the default after a recent change per [1]. We decoupled the transaction handling open iterator (and all its connections) from any transactions started by context commits within the iterator. Here is a related mailing list thread [2].
Andrus
[1] https://issues.apache.org/jira/browse/CAY-2111
[2] https://lists.apache.org/thread.html/f1ccb1ac0f0aa974ffe32217fe4fcfe4b3d9a11ef2c8513f2f0e8436@%3Cdev.cayenne.apache.org%3E
> On Oct 16, 2016, at 11:45 PM, Mark Wardle <ma...@wardle.org> wrote:
>
> Thanks John. Interestingly, I did some testing and peer contexts aren’t committed until the batch iterator JDBC ResultSet is closed at the end of iteration either. This is fine but for a long running task, I thought that there was a problem (but there wasn’t). I think the only workarounds are a) to not care [which I think is probably fine or b) do other processing in another connection.
>
> Mark
>
>> On 15 Oct 2016, at 22:34, John Huss <jo...@gmail.com> wrote:
>>
>> I would use a peer context.
>>
>> You can create a peer context like this if you runtime has been bound to
>> the current thread using
>> CayenneRuntime.bindThreadInjector(runtime.getInjector())
>>
>> public static ObjectContext newObjectContext() {
>> ObjectContextFactory factory = CayenneRuntime.getThreadInjector
>> ().getInstance(ObjectContextFactory.class);
>> return (factory != null) ? factory.createContext() : null;
>> }
>>
>>
>>
>> On Sat, Oct 15, 2016 at 2:37 PM Mark Wardle <ma...@wardle.org> wrote:
>>
>>> Sorry… meant to say using Cayenne M4.0.M3…
>>>
>>> Mark
>>>
>>>> On 15 Oct 2016, at 20:29, Mark Wardle <ma...@wardle.org> wrote:
>>>>
>>>> I’m using ResultBatchIterator like this:
>>>>
>>>> Consumer<T> forEach = ...
>>>> try (ResultBatchIterator<T> iterator = query.batchIterator(context,
>>> batchSize)) {
>>>> for(List<T> batch : iterator) {
>>>> for (T c : batch) {
>>>> forEach.accept(c);
>>>> }
>>>> }
>>>> }
>>>>
>>>> but I also want to do some processing, inserting rows, editing objects
>>> etc. I see lots of SQL in my logs, but the transaction doesn’t get
>>> committed until the batch iterator completes, whether I use
>>> context.commitChanges() or context.performGenericQuery() on some custom SQL
>>> created using SQLTemplate.
>>>>
>>>> Is there a way of committing the transaction earlier, or should I use a
>>> peer object context? Is there an easy of getting a new peer editing context
>>> given a single object context or do I need to inject a ServerRuntime here?
>>>>
>>>> Thank you,
>>>>
>>>> Mark
>>>>
>>>>
>>>
>>>
>
Re: Batch iteration
Posted by Mark Wardle <ma...@wardle.org>.
Thanks John. Interestingly, I did some testing and peer contexts aren’t committed until the batch iterator JDBC ResultSet is closed at the end of iteration either. This is fine but for a long running task, I thought that there was a problem (but there wasn’t). I think the only workarounds are a) to not care [which I think is probably fine or b) do other processing in another connection.
Mark
> On 15 Oct 2016, at 22:34, John Huss <jo...@gmail.com> wrote:
>
> I would use a peer context.
>
> You can create a peer context like this if you runtime has been bound to
> the current thread using
> CayenneRuntime.bindThreadInjector(runtime.getInjector())
>
> public static ObjectContext newObjectContext() {
> ObjectContextFactory factory = CayenneRuntime.getThreadInjector
> ().getInstance(ObjectContextFactory.class);
> return (factory != null) ? factory.createContext() : null;
> }
>
>
>
> On Sat, Oct 15, 2016 at 2:37 PM Mark Wardle <ma...@wardle.org> wrote:
>
>> Sorry… meant to say using Cayenne M4.0.M3…
>>
>> Mark
>>
>>> On 15 Oct 2016, at 20:29, Mark Wardle <ma...@wardle.org> wrote:
>>>
>>> I’m using ResultBatchIterator like this:
>>>
>>> Consumer<T> forEach = ...
>>> try (ResultBatchIterator<T> iterator = query.batchIterator(context,
>> batchSize)) {
>>> for(List<T> batch : iterator) {
>>> for (T c : batch) {
>>> forEach.accept(c);
>>> }
>>> }
>>> }
>>>
>>> but I also want to do some processing, inserting rows, editing objects
>> etc. I see lots of SQL in my logs, but the transaction doesn’t get
>> committed until the batch iterator completes, whether I use
>> context.commitChanges() or context.performGenericQuery() on some custom SQL
>> created using SQLTemplate.
>>>
>>> Is there a way of committing the transaction earlier, or should I use a
>> peer object context? Is there an easy of getting a new peer editing context
>> given a single object context or do I need to inject a ServerRuntime here?
>>>
>>> Thank you,
>>>
>>> Mark
>>>
>>>
>>
>>
Re: Batch iteration
Posted by John Huss <jo...@gmail.com>.
I would use a peer context.
You can create a peer context like this if you runtime has been bound to
the current thread using
CayenneRuntime.bindThreadInjector(runtime.getInjector())
public static ObjectContext newObjectContext() {
ObjectContextFactory factory = CayenneRuntime.getThreadInjector
().getInstance(ObjectContextFactory.class);
return (factory != null) ? factory.createContext() : null;
}
On Sat, Oct 15, 2016 at 2:37 PM Mark Wardle <ma...@wardle.org> wrote:
> Sorry… meant to say using Cayenne M4.0.M3…
>
> Mark
>
> > On 15 Oct 2016, at 20:29, Mark Wardle <ma...@wardle.org> wrote:
> >
> > I’m using ResultBatchIterator like this:
> >
> > Consumer<T> forEach = ...
> > try (ResultBatchIterator<T> iterator = query.batchIterator(context,
> batchSize)) {
> > for(List<T> batch : iterator) {
> > for (T c : batch) {
> > forEach.accept(c);
> > }
> > }
> > }
> >
> > but I also want to do some processing, inserting rows, editing objects
> etc. I see lots of SQL in my logs, but the transaction doesn’t get
> committed until the batch iterator completes, whether I use
> context.commitChanges() or context.performGenericQuery() on some custom SQL
> created using SQLTemplate.
> >
> > Is there a way of committing the transaction earlier, or should I use a
> peer object context? Is there an easy of getting a new peer editing context
> given a single object context or do I need to inject a ServerRuntime here?
> >
> > Thank you,
> >
> > Mark
> >
> >
>
>
Re: Batch iteration
Posted by Mark Wardle <ma...@wardle.org>.
Sorry… meant to say using Cayenne M4.0.M3…
Mark
> On 15 Oct 2016, at 20:29, Mark Wardle <ma...@wardle.org> wrote:
>
> I’m using ResultBatchIterator like this:
>
> Consumer<T> forEach = ...
> try (ResultBatchIterator<T> iterator = query.batchIterator(context, batchSize)) {
> for(List<T> batch : iterator) {
> for (T c : batch) {
> forEach.accept(c);
> }
> }
> }
>
> but I also want to do some processing, inserting rows, editing objects etc. I see lots of SQL in my logs, but the transaction doesn’t get committed until the batch iterator completes, whether I use context.commitChanges() or context.performGenericQuery() on some custom SQL created using SQLTemplate.
>
> Is there a way of committing the transaction earlier, or should I use a peer object context? Is there an easy of getting a new peer editing context given a single object context or do I need to inject a ServerRuntime here?
>
> Thank you,
>
> Mark
>
>