You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cayenne.apache.org by Ken Anderson <ke...@amphorainc.com> on 2018/03/28 11:56:22 UTC
Staged saves pattern ?
All,
We have a process that reads in a file and, for each line, creates or edits objects in the object graph. We only want to commit to the database once at the end.
We have a finite set of lines, so memory is not an issue. We need to save only once because saving will actually fire triggers that will start doing other things to the database, which will then lead to optimistic lock exceptions for us if we have data that overlaps (which we do).
Please don’t suggest we change how the trigger pattern works – it’s a big system and we don’t have control over it.
So, what we’ve toyed with is using a parent/child context arrangement, where each line is processed in a child, and assuming everything goes OK, we commit only to the parent. This works well as long as we don’t need to access the same objects multiple times, but unfortunately, we do. We can reach into the parent context’s unsaved objects, but those objects do not have any relationships since they were built in the child context. This makes things painful.
In EOF, I might consider using a single context and undo, but it doesn’t seem like Cayenne has this kind of functionality.
Thoughts? Suggestions? In EOF, I had once written a layer that intercepted all queries and tried to find the correct object in unsaved objects, but I don’t have nearly enough experience with Cayenne to do that.
Thanks!
Ken
Confidentiality Notice: This e-mail and accompanying documents contain confidential information intended for a specific individual and purpose. This e-mailed information is private and protected by law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, or distribution, or the taking of any action based on the contents of this information, is strictly prohibited.
Re: Staged saves pattern ?
Posted by Tony Giaccone <an...@nytimes.com>.
How about creating a local in memory database. Do all your file reads
and object creation to the in memory database. You could do the save
changes as many times as you like locally in memory. Then when you're
done, figure out what's new, what's been updated and create a new
connection to your remote db, and do the work once there in one transaction.
Could get a bit messy when it comes to updating objects that exist
already, you'd have to fetch them from the remote db and instantiate
them in the local db, but this way you have all your changes in a place
where the "side-effect" of doing the commits won't trigger the larger
process.
having a modified date, and a create date on your objects would go a
long way to making sure you knew what objects had changed. You might
still have to figure out which properties need to be updated. The
difficulty of this solution directly ties to the number of attributes in
your objects and the number of objects you're modifying.
What's the order of magnitude of the data are you parsing?
Tony Giaccone
On 3/28/18 7:56 AM, Ken Anderson wrote:
> All,
>
> We have a process that reads in a file and, for each line, creates or edits objects in the object graph. We only want to commit to the database once at the end.
>
> We have a finite set of lines, so memory is not an issue. We need to save only once because saving will actually fire triggers that will start doing other things to the database, which will then lead to optimistic lock exceptions for us if we have data that overlaps (which we do).
>
> Please don’t suggest we change how the trigger pattern works – it’s a big system and we don’t have control over it.
>
> So, what we’ve toyed with is using a parent/child context arrangement, where each line is processed in a child, and assuming everything goes OK, we commit only to the parent. This works well as long as we don’t need to access the same objects multiple times, but unfortunately, we do. We can reach into the parent context’s unsaved objects, but those objects do not have any relationships since they were built in the child context. This makes things painful.
>
> In EOF, I might consider using a single context and undo, but it doesn’t seem like Cayenne has this kind of functionality.
>
> Thoughts? Suggestions? In EOF, I had once written a layer that intercepted all queries and tried to find the correct object in unsaved objects, but I don’t have nearly enough experience with Cayenne to do that.
>
> Thanks!
> Ken
>
> Confidentiality Notice: This e-mail and accompanying documents contain confidential information intended for a specific individual and purpose. This e-mailed information is private and protected by law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, or distribution, or the taking of any action based on the contents of this information, is strictly prohibited.
Re: Staged saves pattern ?
Posted by John Huss <jo...@gmail.com>.
Why can't you just wrap the whole thing in a database transaction and
forget the parent child contexts?
On Sun, Apr 1, 2018 at 6:35 AM Ken Anderson <ke...@anderhome.com> wrote:
> In your scenario, what if line 4 requires the objects I saved to the
> parent from line 1? When I query the database, those objects won’t exist,
> so I will create new ones, which will eventually clash with the objects I
> created from line 1.
>
> I’m looking for a simple way where all queries, including firing faults,
> have a way to check for the existence in the parent context before deciding
> to create new.
>
> One option is to keep a hash map of some sort, but that’s a decent amount
> of work for the 8 or so entities that I might work with, and gets even more
> complicated when I start to deal with the chance a fault will be fired.
>
> > On Mar 29, 2018, at 6:39 PM, Mike Kienenberger <mk...@gmail.com>
> wrote:
> >
> > Is there a reason why using a single parent context and a single child
> > context would not work?
> >
> > You said earlier that you couldn't find the objects in the child after
> > a save or a rollback because the parent returns them as hollow when
> > you use localObjects.
> >
> > But if you are doing all of your work in a single child context, why
> > do you need to use localObject?
> >
> > Something like
> >
> > - Create parent context
> > - Create child context
> > - process line 1
> > - save to parent
> > - process line 2
> > - save to parent
> > - process line 3
> > - rollback local changes (should roll back to state after we saved line
> 2)
> > - process line 4
> > - save to parent
> >
> >
> >
> >
> > On Thu, Mar 29, 2018 at 6:21 PM, Ken Anderson <ke...@anderhome.com> wrote:
> >> I’m pretty sure you still have to call localObjects(), even if it’s a
> child context.
> >>
> >> In any case, a hash map doesn’t really work for me, since I have a
> significant graph of objects, and relationships could fire at different
> levels.
> >>
> >> Thanks for your thoughts!
> >>
> >>> On Mar 29, 2018, at 6:07 PM, Mike Kienenberger <mk...@gmail.com>
> wrote:
> >>>
> >>> Like I said, I haven't used nested contexts, but I'm assuming that
> >>> data in the parent context is available to the child contexts, but
> >>> data in sibling contexts is not available to the child context.
> >>>
> >>> As for not nesting contexts too deep, I'm guessing that's a
> >>> performance issue rather than a technical limitation. This process
> >>> doesn't sound like it's an interactive time-sensitive operation, and
> >>> you've already said that memory isn't an issue (and data contexts are
> >>> not memory intensive in any case).
> >>>
> >>> Years ago, before Cayenne had child contexts, I handled situations
> >>> like yours by using a single context and a hashmap.
> >>> As I created new objects, I stored them in the hash map by the line
> >>> identifier, then pulled them back out later when I had a line that
> >>> referred to a previous line.
> >>>
> >>>
> >>> On Thu, Mar 29, 2018 at 5:50 PM, Ken Anderson
> >>> <ke...@amphorainc.com> wrote:
> >>>> Mike,
> >>>>
> >>>> The docs say to not go too deep with nested contexts, so I don't
> think that's a viable solution. I'm also not clear what benefit you think
> this would have over the 2 level plan.
> >>>>
> >>>> Ken
> >>>>
> >>>> On 3/29/18, 5:39 PM, "Mike Kienenberger" <mk...@gmail.com> wrote:
> >>>>
> >>>> Instead of having each line as a child context under a parent
> context,
> >>>>
> >>>> parent
> >>>> - child1 (line 1)
> >>>> - child2 (line 2)
> >>>> - child3 (line 3)
> >>>>
> >>>> could you have each line processed as a child context of the
> previous line?
> >>>>
> >>>> - child1 (line 1)
> >>>> -- child2 (line 2)
> >>>> --- child3 (line 3)
> >>>> ---- child4 (line 4)
> >>>>
> >>>> If you want to "undo" the current line,
> >>>> child4.rollbackChangesLocally(); and start on the next line.
> >>>>
> >>>> Note that I have not done anything with child contexts, but this
> would
> >>>> be how I'd try to solve it.
> >>>>
> >>>>
> >>>> On Wed, Mar 28, 2018 at 9:29 PM, Ken Anderson <ke...@anderhome.com>
> wrote:
> >>>>> Hugi,
> >>>>>
> >>>>> That’s correct - it’s not like we’re just importing a bunch of
> records. Each row in the file could affect the same set of objects.
> >>>>>
> >>>>> So, we did the child context, but obviously if we created an object
> in a prior child and then saved it to the parent, we won’t be able to
> easily find it in the next child context. If you get a localObject in your
> new child context, it is “hollow”, so not connected to all the other
> objects floating around in the parent. We also can’t fire relationships,
> because those relationships will go to the database instead of the parent
> context.
> >>>>>
> >>>>> Ken
> >>>>>
> >>>>>> On Mar 28, 2018, at 7:11 PM, Hugi Thordarson <hu...@karlmenn.is>
> wrote:
> >>>>>>
> >>>>>>> That's exactly what we want to do - save once at the end.
> However, we have 2 problems:
> >>>>>>>
> >>>>>>> 1. How do we find the objects that we already created but haven't
> saved yet
> >>>>>>
> >>>>>> You can go through yur ObjectContext's newObjects() and filter that
> to your liking—pretty much the same as you'd do with EOF.
> >>>>>>
> >>>>>>> 2. How do we roll back each line if there's an error? Not a DB
> error, but the logic gets so far, and then determines that there's no way
> to continue so we must skip this line.
> >>>>>>
> >>>>>> As you tried yourself, I'd use a child context and commit to the
> parent once you're sure everything is in place. Can you explain further
> what was problematic with that (that you need to "access the same objects
> multiple times")? Do you mean that each row of the file is in some way
> looking at data from other rows?
> >>>>>>
> >>>>>> Cheers,
> >>>>>> - hugi
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>> Ken
> >>>>>>>
> >>>>>>> On 3/28/18, 6:07 PM, "John Huss" <jo...@gmail.com> wrote:
> >>>>>>>
> >>>>>>> Well, you could just save once at the end. Why do you need to save
> >>>>>>> multiple times during the processing? Validation exceptions and
> Optimistic
> >>>>>>> Locking errors could be handled in the save with some custom logic
> and a
> >>>>>>> retry.
> >>>>>>>
> >>>>>>> Or if this isn't a super long process you can use a database
> transaction to
> >>>>>>> allow saving multiple times without actually having that data be
> visible
> >>>>>>> outside of the transaction.
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Mar 28, 2018 at 6:56 AM Ken Anderson <
> ken.anderson@amphorainc.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> All,
> >>>>>>>>
> >>>>>>>> We have a process that reads in a file and, for each line,
> creates or
> >>>>>>>> edits objects in the object graph. We only want to commit to the
> database
> >>>>>>>> once at the end.
> >>>>>>>>
> >>>>>>>> We have a finite set of lines, so memory is not an issue. We
> need to save
> >>>>>>>> only once because saving will actually fire triggers that will
> start doing
> >>>>>>>> other things to the database, which will then lead to optimistic
> lock
> >>>>>>>> exceptions for us if we have data that overlaps (which we do).
> >>>>>>>>
> >>>>>>>> Please don’t suggest we change how the trigger pattern works –
> it’s a big
> >>>>>>>> system and we don’t have control over it.
> >>>>>>>>
> >>>>>>>> So, what we’ve toyed with is using a parent/child context
> arrangement,
> >>>>>>>> where each line is processed in a child, and assuming everything
> goes OK,
> >>>>>>>> we commit only to the parent. This works well as long as we
> don’t need to
> >>>>>>>> access the same objects multiple times, but unfortunately, we
> do. We can
> >>>>>>>> reach into the parent context’s unsaved objects, but those
> objects do not
> >>>>>>>> have any relationships since they were built in the child
> context. This
> >>>>>>>> makes things painful.
> >>>>>>>>
> >>>>>>>> In EOF, I might consider using a single context and undo, but it
> doesn’t
> >>>>>>>> seem like Cayenne has this kind of functionality.
> >>>>>>>>
> >>>>>>>> Thoughts? Suggestions? In EOF, I had once written a layer that
> >>>>>>>> intercepted all queries and tried to find the correct object in
> unsaved
> >>>>>>>> objects, but I don’t have nearly enough experience with Cayenne
> to do that.
> >>>>>>>>
> >>>>>>>> Thanks!
> >>>>>>>> Ken
> >>>>>>>>
> >>>>>>>> Confidentiality Notice: This e-mail and accompanying documents
> contain
> >>>>>>>> confidential information intended for a specific individual and
> purpose.
> >>>>>>>> This e-mailed information is private and protected by law. If you
> are not
> >>>>>>>> the intended recipient, you are hereby notified that any
> disclosure,
> >>>>>>>> copying, or distribution, or the taking of any action based on
> the contents
> >>>>>>>> of this information, is strictly prohibited.
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Confidentiality Notice: This e-mail and accompanying documents
> contain confidential information intended for a specific individual and
> purpose. This e-mailed information is private and protected by law. If you
> are not the intended recipient, you are hereby notified that any
> disclosure, copying, or distribution, or the taking of any action based on
> the contents of this information, is strictly prohibited.
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>> Confidentiality Notice: This e-mail and accompanying documents
> contain confidential information intended for a specific individual and
> purpose. This e-mailed information is private and protected by law. If you
> are not the intended recipient, you are hereby notified that any
> disclosure, copying, or distribution, or the taking of any action based on
> the contents of this information, is strictly prohibited.
> >>
>
>
Re: Staged saves pattern ?
Posted by Mike Kienenberger <mk...@gmail.com>.
I suspect it's not possible to answer your questions without a better
idea of the specifics.
If you are currently using localObject() and end up with a hollow
object then you must already have an object available from a context.
If you were only using a single child context, then that same object
would already be in your single child context and you wouldn't need to
use localObject().
Now it's sounding like what you want is a way to have queries resolved
but only in the existing parent object. I would suspect that could
be possible, but I don't know the specifics of how it would be set up.
Probably by using a custom subclass of ObjectContext. You could also
just check which objects are currently stored in the objects
As someone mentioned before, you can check the newObjects() or
modifiedObjects() collection in a context to find an existing loaded
entity before creating a new entity.
On Sun, Apr 1, 2018 at 7:35 AM, Ken Anderson <ke...@anderhome.com> wrote:
> In your scenario, what if line 4 requires the objects I saved to the parent from line 1? When I query the database, those objects won’t exist, so I will create new ones, which will eventually clash with the objects I created from line 1.
>
> I’m looking for a simple way where all queries, including firing faults, have a way to check for the existence in the parent context before deciding to create new.
>
> One option is to keep a hash map of some sort, but that’s a decent amount of work for the 8 or so entities that I might work with, and gets even more complicated when I start to deal with the chance a fault will be fired.
>
>> On Mar 29, 2018, at 6:39 PM, Mike Kienenberger <mk...@gmail.com> wrote:
>>
>> Is there a reason why using a single parent context and a single child
>> context would not work?
>>
>> You said earlier that you couldn't find the objects in the child after
>> a save or a rollback because the parent returns them as hollow when
>> you use localObjects.
>>
>> But if you are doing all of your work in a single child context, why
>> do you need to use localObject?
>>
>> Something like
>>
>> - Create parent context
>> - Create child context
>> - process line 1
>> - save to parent
>> - process line 2
>> - save to parent
>> - process line 3
>> - rollback local changes (should roll back to state after we saved line 2)
>> - process line 4
>> - save to parent
>>
>>
>>
>>
>> On Thu, Mar 29, 2018 at 6:21 PM, Ken Anderson <ke...@anderhome.com> wrote:
>>> I’m pretty sure you still have to call localObjects(), even if it’s a child context.
>>>
>>> In any case, a hash map doesn’t really work for me, since I have a significant graph of objects, and relationships could fire at different levels.
>>>
>>> Thanks for your thoughts!
>>>
>>>> On Mar 29, 2018, at 6:07 PM, Mike Kienenberger <mk...@gmail.com> wrote:
>>>>
>>>> Like I said, I haven't used nested contexts, but I'm assuming that
>>>> data in the parent context is available to the child contexts, but
>>>> data in sibling contexts is not available to the child context.
>>>>
>>>> As for not nesting contexts too deep, I'm guessing that's a
>>>> performance issue rather than a technical limitation. This process
>>>> doesn't sound like it's an interactive time-sensitive operation, and
>>>> you've already said that memory isn't an issue (and data contexts are
>>>> not memory intensive in any case).
>>>>
>>>> Years ago, before Cayenne had child contexts, I handled situations
>>>> like yours by using a single context and a hashmap.
>>>> As I created new objects, I stored them in the hash map by the line
>>>> identifier, then pulled them back out later when I had a line that
>>>> referred to a previous line.
>>>>
>>>>
>>>> On Thu, Mar 29, 2018 at 5:50 PM, Ken Anderson
>>>> <ke...@amphorainc.com> wrote:
>>>>> Mike,
>>>>>
>>>>> The docs say to not go too deep with nested contexts, so I don't think that's a viable solution. I'm also not clear what benefit you think this would have over the 2 level plan.
>>>>>
>>>>> Ken
>>>>>
>>>>> On 3/29/18, 5:39 PM, "Mike Kienenberger" <mk...@gmail.com> wrote:
>>>>>
>>>>> Instead of having each line as a child context under a parent context,
>>>>>
>>>>> parent
>>>>> - child1 (line 1)
>>>>> - child2 (line 2)
>>>>> - child3 (line 3)
>>>>>
>>>>> could you have each line processed as a child context of the previous line?
>>>>>
>>>>> - child1 (line 1)
>>>>> -- child2 (line 2)
>>>>> --- child3 (line 3)
>>>>> ---- child4 (line 4)
>>>>>
>>>>> If you want to "undo" the current line,
>>>>> child4.rollbackChangesLocally(); and start on the next line.
>>>>>
>>>>> Note that I have not done anything with child contexts, but this would
>>>>> be how I'd try to solve it.
>>>>>
>>>>>
>>>>> On Wed, Mar 28, 2018 at 9:29 PM, Ken Anderson <ke...@anderhome.com> wrote:
>>>>>> Hugi,
>>>>>>
>>>>>> That’s correct - it’s not like we’re just importing a bunch of records. Each row in the file could affect the same set of objects.
>>>>>>
>>>>>> So, we did the child context, but obviously if we created an object in a prior child and then saved it to the parent, we won’t be able to easily find it in the next child context. If you get a localObject in your new child context, it is “hollow”, so not connected to all the other objects floating around in the parent. We also can’t fire relationships, because those relationships will go to the database instead of the parent context.
>>>>>>
>>>>>> Ken
>>>>>>
>>>>>>> On Mar 28, 2018, at 7:11 PM, Hugi Thordarson <hu...@karlmenn.is> wrote:
>>>>>>>
>>>>>>>> That's exactly what we want to do - save once at the end. However, we have 2 problems:
>>>>>>>>
>>>>>>>> 1. How do we find the objects that we already created but haven't saved yet
>>>>>>>
>>>>>>> You can go through yur ObjectContext's newObjects() and filter that to your liking—pretty much the same as you'd do with EOF.
>>>>>>>
>>>>>>>> 2. How do we roll back each line if there's an error? Not a DB error, but the logic gets so far, and then determines that there's no way to continue so we must skip this line.
>>>>>>>
>>>>>>> As you tried yourself, I'd use a child context and commit to the parent once you're sure everything is in place. Can you explain further what was problematic with that (that you need to "access the same objects multiple times")? Do you mean that each row of the file is in some way looking at data from other rows?
>>>>>>>
>>>>>>> Cheers,
>>>>>>> - hugi
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Ken
>>>>>>>>
>>>>>>>> On 3/28/18, 6:07 PM, "John Huss" <jo...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Well, you could just save once at the end. Why do you need to save
>>>>>>>> multiple times during the processing? Validation exceptions and Optimistic
>>>>>>>> Locking errors could be handled in the save with some custom logic and a
>>>>>>>> retry.
>>>>>>>>
>>>>>>>> Or if this isn't a super long process you can use a database transaction to
>>>>>>>> allow saving multiple times without actually having that data be visible
>>>>>>>> outside of the transaction.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Mar 28, 2018 at 6:56 AM Ken Anderson <ke...@amphorainc.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> All,
>>>>>>>>>
>>>>>>>>> We have a process that reads in a file and, for each line, creates or
>>>>>>>>> edits objects in the object graph. We only want to commit to the database
>>>>>>>>> once at the end.
>>>>>>>>>
>>>>>>>>> We have a finite set of lines, so memory is not an issue. We need to save
>>>>>>>>> only once because saving will actually fire triggers that will start doing
>>>>>>>>> other things to the database, which will then lead to optimistic lock
>>>>>>>>> exceptions for us if we have data that overlaps (which we do).
>>>>>>>>>
>>>>>>>>> Please don’t suggest we change how the trigger pattern works – it’s a big
>>>>>>>>> system and we don’t have control over it.
>>>>>>>>>
>>>>>>>>> So, what we’ve toyed with is using a parent/child context arrangement,
>>>>>>>>> where each line is processed in a child, and assuming everything goes OK,
>>>>>>>>> we commit only to the parent. This works well as long as we don’t need to
>>>>>>>>> access the same objects multiple times, but unfortunately, we do. We can
>>>>>>>>> reach into the parent context’s unsaved objects, but those objects do not
>>>>>>>>> have any relationships since they were built in the child context. This
>>>>>>>>> makes things painful.
>>>>>>>>>
>>>>>>>>> In EOF, I might consider using a single context and undo, but it doesn’t
>>>>>>>>> seem like Cayenne has this kind of functionality.
>>>>>>>>>
>>>>>>>>> Thoughts? Suggestions? In EOF, I had once written a layer that
>>>>>>>>> intercepted all queries and tried to find the correct object in unsaved
>>>>>>>>> objects, but I don’t have nearly enough experience with Cayenne to do that.
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>> Ken
>>>>>>>>>
>>>>>>>>> Confidentiality Notice: This e-mail and accompanying documents contain
>>>>>>>>> confidential information intended for a specific individual and purpose.
>>>>>>>>> This e-mailed information is private and protected by law. If you are not
>>>>>>>>> the intended recipient, you are hereby notified that any disclosure,
>>>>>>>>> copying, or distribution, or the taking of any action based on the contents
>>>>>>>>> of this information, is strictly prohibited.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Confidentiality Notice: This e-mail and accompanying documents contain confidential information intended for a specific individual and purpose. This e-mailed information is private and protected by law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, or distribution, or the taking of any action based on the contents of this information, is strictly prohibited.
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> Confidentiality Notice: This e-mail and accompanying documents contain confidential information intended for a specific individual and purpose. This e-mailed information is private and protected by law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, or distribution, or the taking of any action based on the contents of this information, is strictly prohibited.
>>>
>
Re: Staged saves pattern ?
Posted by Ken Anderson <ke...@anderhome.com>.
In your scenario, what if line 4 requires the objects I saved to the parent from line 1? When I query the database, those objects won’t exist, so I will create new ones, which will eventually clash with the objects I created from line 1.
I’m looking for a simple way where all queries, including firing faults, have a way to check for the existence in the parent context before deciding to create new.
One option is to keep a hash map of some sort, but that’s a decent amount of work for the 8 or so entities that I might work with, and gets even more complicated when I start to deal with the chance a fault will be fired.
> On Mar 29, 2018, at 6:39 PM, Mike Kienenberger <mk...@gmail.com> wrote:
>
> Is there a reason why using a single parent context and a single child
> context would not work?
>
> You said earlier that you couldn't find the objects in the child after
> a save or a rollback because the parent returns them as hollow when
> you use localObjects.
>
> But if you are doing all of your work in a single child context, why
> do you need to use localObject?
>
> Something like
>
> - Create parent context
> - Create child context
> - process line 1
> - save to parent
> - process line 2
> - save to parent
> - process line 3
> - rollback local changes (should roll back to state after we saved line 2)
> - process line 4
> - save to parent
>
>
>
>
> On Thu, Mar 29, 2018 at 6:21 PM, Ken Anderson <ke...@anderhome.com> wrote:
>> I’m pretty sure you still have to call localObjects(), even if it’s a child context.
>>
>> In any case, a hash map doesn’t really work for me, since I have a significant graph of objects, and relationships could fire at different levels.
>>
>> Thanks for your thoughts!
>>
>>> On Mar 29, 2018, at 6:07 PM, Mike Kienenberger <mk...@gmail.com> wrote:
>>>
>>> Like I said, I haven't used nested contexts, but I'm assuming that
>>> data in the parent context is available to the child contexts, but
>>> data in sibling contexts is not available to the child context.
>>>
>>> As for not nesting contexts too deep, I'm guessing that's a
>>> performance issue rather than a technical limitation. This process
>>> doesn't sound like it's an interactive time-sensitive operation, and
>>> you've already said that memory isn't an issue (and data contexts are
>>> not memory intensive in any case).
>>>
>>> Years ago, before Cayenne had child contexts, I handled situations
>>> like yours by using a single context and a hashmap.
>>> As I created new objects, I stored them in the hash map by the line
>>> identifier, then pulled them back out later when I had a line that
>>> referred to a previous line.
>>>
>>>
>>> On Thu, Mar 29, 2018 at 5:50 PM, Ken Anderson
>>> <ke...@amphorainc.com> wrote:
>>>> Mike,
>>>>
>>>> The docs say to not go too deep with nested contexts, so I don't think that's a viable solution. I'm also not clear what benefit you think this would have over the 2 level plan.
>>>>
>>>> Ken
>>>>
>>>> On 3/29/18, 5:39 PM, "Mike Kienenberger" <mk...@gmail.com> wrote:
>>>>
>>>> Instead of having each line as a child context under a parent context,
>>>>
>>>> parent
>>>> - child1 (line 1)
>>>> - child2 (line 2)
>>>> - child3 (line 3)
>>>>
>>>> could you have each line processed as a child context of the previous line?
>>>>
>>>> - child1 (line 1)
>>>> -- child2 (line 2)
>>>> --- child3 (line 3)
>>>> ---- child4 (line 4)
>>>>
>>>> If you want to "undo" the current line,
>>>> child4.rollbackChangesLocally(); and start on the next line.
>>>>
>>>> Note that I have not done anything with child contexts, but this would
>>>> be how I'd try to solve it.
>>>>
>>>>
>>>> On Wed, Mar 28, 2018 at 9:29 PM, Ken Anderson <ke...@anderhome.com> wrote:
>>>>> Hugi,
>>>>>
>>>>> That’s correct - it’s not like we’re just importing a bunch of records. Each row in the file could affect the same set of objects.
>>>>>
>>>>> So, we did the child context, but obviously if we created an object in a prior child and then saved it to the parent, we won’t be able to easily find it in the next child context. If you get a localObject in your new child context, it is “hollow”, so not connected to all the other objects floating around in the parent. We also can’t fire relationships, because those relationships will go to the database instead of the parent context.
>>>>>
>>>>> Ken
>>>>>
>>>>>> On Mar 28, 2018, at 7:11 PM, Hugi Thordarson <hu...@karlmenn.is> wrote:
>>>>>>
>>>>>>> That's exactly what we want to do - save once at the end. However, we have 2 problems:
>>>>>>>
>>>>>>> 1. How do we find the objects that we already created but haven't saved yet
>>>>>>
>>>>>> You can go through yur ObjectContext's newObjects() and filter that to your liking—pretty much the same as you'd do with EOF.
>>>>>>
>>>>>>> 2. How do we roll back each line if there's an error? Not a DB error, but the logic gets so far, and then determines that there's no way to continue so we must skip this line.
>>>>>>
>>>>>> As you tried yourself, I'd use a child context and commit to the parent once you're sure everything is in place. Can you explain further what was problematic with that (that you need to "access the same objects multiple times")? Do you mean that each row of the file is in some way looking at data from other rows?
>>>>>>
>>>>>> Cheers,
>>>>>> - hugi
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Ken
>>>>>>>
>>>>>>> On 3/28/18, 6:07 PM, "John Huss" <jo...@gmail.com> wrote:
>>>>>>>
>>>>>>> Well, you could just save once at the end. Why do you need to save
>>>>>>> multiple times during the processing? Validation exceptions and Optimistic
>>>>>>> Locking errors could be handled in the save with some custom logic and a
>>>>>>> retry.
>>>>>>>
>>>>>>> Or if this isn't a super long process you can use a database transaction to
>>>>>>> allow saving multiple times without actually having that data be visible
>>>>>>> outside of the transaction.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Mar 28, 2018 at 6:56 AM Ken Anderson <ke...@amphorainc.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> All,
>>>>>>>>
>>>>>>>> We have a process that reads in a file and, for each line, creates or
>>>>>>>> edits objects in the object graph. We only want to commit to the database
>>>>>>>> once at the end.
>>>>>>>>
>>>>>>>> We have a finite set of lines, so memory is not an issue. We need to save
>>>>>>>> only once because saving will actually fire triggers that will start doing
>>>>>>>> other things to the database, which will then lead to optimistic lock
>>>>>>>> exceptions for us if we have data that overlaps (which we do).
>>>>>>>>
>>>>>>>> Please don’t suggest we change how the trigger pattern works – it’s a big
>>>>>>>> system and we don’t have control over it.
>>>>>>>>
>>>>>>>> So, what we’ve toyed with is using a parent/child context arrangement,
>>>>>>>> where each line is processed in a child, and assuming everything goes OK,
>>>>>>>> we commit only to the parent. This works well as long as we don’t need to
>>>>>>>> access the same objects multiple times, but unfortunately, we do. We can
>>>>>>>> reach into the parent context’s unsaved objects, but those objects do not
>>>>>>>> have any relationships since they were built in the child context. This
>>>>>>>> makes things painful.
>>>>>>>>
>>>>>>>> In EOF, I might consider using a single context and undo, but it doesn’t
>>>>>>>> seem like Cayenne has this kind of functionality.
>>>>>>>>
>>>>>>>> Thoughts? Suggestions? In EOF, I had once written a layer that
>>>>>>>> intercepted all queries and tried to find the correct object in unsaved
>>>>>>>> objects, but I don’t have nearly enough experience with Cayenne to do that.
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>> Ken
>>>>>>>>
>>>>>>>> Confidentiality Notice: This e-mail and accompanying documents contain
>>>>>>>> confidential information intended for a specific individual and purpose.
>>>>>>>> This e-mailed information is private and protected by law. If you are not
>>>>>>>> the intended recipient, you are hereby notified that any disclosure,
>>>>>>>> copying, or distribution, or the taking of any action based on the contents
>>>>>>>> of this information, is strictly prohibited.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Confidentiality Notice: This e-mail and accompanying documents contain confidential information intended for a specific individual and purpose. This e-mailed information is private and protected by law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, or distribution, or the taking of any action based on the contents of this information, is strictly prohibited.
>>>>>>
>>>>>
>>>>
>>>>
>>>> Confidentiality Notice: This e-mail and accompanying documents contain confidential information intended for a specific individual and purpose. This e-mailed information is private and protected by law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, or distribution, or the taking of any action based on the contents of this information, is strictly prohibited.
>>
Re: Staged saves pattern ?
Posted by Mike Kienenberger <mk...@gmail.com>.
Is there a reason why using a single parent context and a single child
context would not work?
You said earlier that you couldn't find the objects in the child after
a save or a rollback because the parent returns them as hollow when
you use localObjects.
But if you are doing all of your work in a single child context, why
do you need to use localObject?
Something like
- Create parent context
- Create child context
- process line 1
- save to parent
- process line 2
- save to parent
- process line 3
- rollback local changes (should roll back to state after we saved line 2)
- process line 4
- save to parent
On Thu, Mar 29, 2018 at 6:21 PM, Ken Anderson <ke...@anderhome.com> wrote:
> I’m pretty sure you still have to call localObjects(), even if it’s a child context.
>
> In any case, a hash map doesn’t really work for me, since I have a significant graph of objects, and relationships could fire at different levels.
>
> Thanks for your thoughts!
>
>> On Mar 29, 2018, at 6:07 PM, Mike Kienenberger <mk...@gmail.com> wrote:
>>
>> Like I said, I haven't used nested contexts, but I'm assuming that
>> data in the parent context is available to the child contexts, but
>> data in sibling contexts is not available to the child context.
>>
>> As for not nesting contexts too deep, I'm guessing that's a
>> performance issue rather than a technical limitation. This process
>> doesn't sound like it's an interactive time-sensitive operation, and
>> you've already said that memory isn't an issue (and data contexts are
>> not memory intensive in any case).
>>
>> Years ago, before Cayenne had child contexts, I handled situations
>> like yours by using a single context and a hashmap.
>> As I created new objects, I stored them in the hash map by the line
>> identifier, then pulled them back out later when I had a line that
>> referred to a previous line.
>>
>>
>> On Thu, Mar 29, 2018 at 5:50 PM, Ken Anderson
>> <ke...@amphorainc.com> wrote:
>>> Mike,
>>>
>>> The docs say to not go too deep with nested contexts, so I don't think that's a viable solution. I'm also not clear what benefit you think this would have over the 2 level plan.
>>>
>>> Ken
>>>
>>> On 3/29/18, 5:39 PM, "Mike Kienenberger" <mk...@gmail.com> wrote:
>>>
>>> Instead of having each line as a child context under a parent context,
>>>
>>> parent
>>> - child1 (line 1)
>>> - child2 (line 2)
>>> - child3 (line 3)
>>>
>>> could you have each line processed as a child context of the previous line?
>>>
>>> - child1 (line 1)
>>> -- child2 (line 2)
>>> --- child3 (line 3)
>>> ---- child4 (line 4)
>>>
>>> If you want to "undo" the current line,
>>> child4.rollbackChangesLocally(); and start on the next line.
>>>
>>> Note that I have not done anything with child contexts, but this would
>>> be how I'd try to solve it.
>>>
>>>
>>> On Wed, Mar 28, 2018 at 9:29 PM, Ken Anderson <ke...@anderhome.com> wrote:
>>>> Hugi,
>>>>
>>>> That’s correct - it’s not like we’re just importing a bunch of records. Each row in the file could affect the same set of objects.
>>>>
>>>> So, we did the child context, but obviously if we created an object in a prior child and then saved it to the parent, we won’t be able to easily find it in the next child context. If you get a localObject in your new child context, it is “hollow”, so not connected to all the other objects floating around in the parent. We also can’t fire relationships, because those relationships will go to the database instead of the parent context.
>>>>
>>>> Ken
>>>>
>>>>> On Mar 28, 2018, at 7:11 PM, Hugi Thordarson <hu...@karlmenn.is> wrote:
>>>>>
>>>>>> That's exactly what we want to do - save once at the end. However, we have 2 problems:
>>>>>>
>>>>>> 1. How do we find the objects that we already created but haven't saved yet
>>>>>
>>>>> You can go through yur ObjectContext's newObjects() and filter that to your liking—pretty much the same as you'd do with EOF.
>>>>>
>>>>>> 2. How do we roll back each line if there's an error? Not a DB error, but the logic gets so far, and then determines that there's no way to continue so we must skip this line.
>>>>>
>>>>> As you tried yourself, I'd use a child context and commit to the parent once you're sure everything is in place. Can you explain further what was problematic with that (that you need to "access the same objects multiple times")? Do you mean that each row of the file is in some way looking at data from other rows?
>>>>>
>>>>> Cheers,
>>>>> - hugi
>>>>>
>>>>>
>>>>>>
>>>>>> Ken
>>>>>>
>>>>>> On 3/28/18, 6:07 PM, "John Huss" <jo...@gmail.com> wrote:
>>>>>>
>>>>>> Well, you could just save once at the end. Why do you need to save
>>>>>> multiple times during the processing? Validation exceptions and Optimistic
>>>>>> Locking errors could be handled in the save with some custom logic and a
>>>>>> retry.
>>>>>>
>>>>>> Or if this isn't a super long process you can use a database transaction to
>>>>>> allow saving multiple times without actually having that data be visible
>>>>>> outside of the transaction.
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 28, 2018 at 6:56 AM Ken Anderson <ke...@amphorainc.com>
>>>>>> wrote:
>>>>>>
>>>>>>> All,
>>>>>>>
>>>>>>> We have a process that reads in a file and, for each line, creates or
>>>>>>> edits objects in the object graph. We only want to commit to the database
>>>>>>> once at the end.
>>>>>>>
>>>>>>> We have a finite set of lines, so memory is not an issue. We need to save
>>>>>>> only once because saving will actually fire triggers that will start doing
>>>>>>> other things to the database, which will then lead to optimistic lock
>>>>>>> exceptions for us if we have data that overlaps (which we do).
>>>>>>>
>>>>>>> Please don’t suggest we change how the trigger pattern works – it’s a big
>>>>>>> system and we don’t have control over it.
>>>>>>>
>>>>>>> So, what we’ve toyed with is using a parent/child context arrangement,
>>>>>>> where each line is processed in a child, and assuming everything goes OK,
>>>>>>> we commit only to the parent. This works well as long as we don’t need to
>>>>>>> access the same objects multiple times, but unfortunately, we do. We can
>>>>>>> reach into the parent context’s unsaved objects, but those objects do not
>>>>>>> have any relationships since they were built in the child context. This
>>>>>>> makes things painful.
>>>>>>>
>>>>>>> In EOF, I might consider using a single context and undo, but it doesn’t
>>>>>>> seem like Cayenne has this kind of functionality.
>>>>>>>
>>>>>>> Thoughts? Suggestions? In EOF, I had once written a layer that
>>>>>>> intercepted all queries and tried to find the correct object in unsaved
>>>>>>> objects, but I don’t have nearly enough experience with Cayenne to do that.
>>>>>>>
>>>>>>> Thanks!
>>>>>>> Ken
>>>>>>>
>>>>>>> Confidentiality Notice: This e-mail and accompanying documents contain
>>>>>>> confidential information intended for a specific individual and purpose.
>>>>>>> This e-mailed information is private and protected by law. If you are not
>>>>>>> the intended recipient, you are hereby notified that any disclosure,
>>>>>>> copying, or distribution, or the taking of any action based on the contents
>>>>>>> of this information, is strictly prohibited.
>>>>>>>
>>>>>>
>>>>>>
>>>>>> Confidentiality Notice: This e-mail and accompanying documents contain confidential information intended for a specific individual and purpose. This e-mailed information is private and protected by law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, or distribution, or the taking of any action based on the contents of this information, is strictly prohibited.
>>>>>
>>>>
>>>
>>>
>>> Confidentiality Notice: This e-mail and accompanying documents contain confidential information intended for a specific individual and purpose. This e-mailed information is private and protected by law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, or distribution, or the taking of any action based on the contents of this information, is strictly prohibited.
>
Re: Staged saves pattern ?
Posted by Ken Anderson <ke...@anderhome.com>.
I’m pretty sure you still have to call localObjects(), even if it’s a child context.
In any case, a hash map doesn’t really work for me, since I have a significant graph of objects, and relationships could fire at different levels.
Thanks for your thoughts!
> On Mar 29, 2018, at 6:07 PM, Mike Kienenberger <mk...@gmail.com> wrote:
>
> Like I said, I haven't used nested contexts, but I'm assuming that
> data in the parent context is available to the child contexts, but
> data in sibling contexts is not available to the child context.
>
> As for not nesting contexts too deep, I'm guessing that's a
> performance issue rather than a technical limitation. This process
> doesn't sound like it's an interactive time-sensitive operation, and
> you've already said that memory isn't an issue (and data contexts are
> not memory intensive in any case).
>
> Years ago, before Cayenne had child contexts, I handled situations
> like yours by using a single context and a hashmap.
> As I created new objects, I stored them in the hash map by the line
> identifier, then pulled them back out later when I had a line that
> referred to a previous line.
>
>
> On Thu, Mar 29, 2018 at 5:50 PM, Ken Anderson
> <ke...@amphorainc.com> wrote:
>> Mike,
>>
>> The docs say to not go too deep with nested contexts, so I don't think that's a viable solution. I'm also not clear what benefit you think this would have over the 2 level plan.
>>
>> Ken
>>
>> On 3/29/18, 5:39 PM, "Mike Kienenberger" <mk...@gmail.com> wrote:
>>
>> Instead of having each line as a child context under a parent context,
>>
>> parent
>> - child1 (line 1)
>> - child2 (line 2)
>> - child3 (line 3)
>>
>> could you have each line processed as a child context of the previous line?
>>
>> - child1 (line 1)
>> -- child2 (line 2)
>> --- child3 (line 3)
>> ---- child4 (line 4)
>>
>> If you want to "undo" the current line,
>> child4.rollbackChangesLocally(); and start on the next line.
>>
>> Note that I have not done anything with child contexts, but this would
>> be how I'd try to solve it.
>>
>>
>> On Wed, Mar 28, 2018 at 9:29 PM, Ken Anderson <ke...@anderhome.com> wrote:
>>> Hugi,
>>>
>>> That’s correct - it’s not like we’re just importing a bunch of records. Each row in the file could affect the same set of objects.
>>>
>>> So, we did the child context, but obviously if we created an object in a prior child and then saved it to the parent, we won’t be able to easily find it in the next child context. If you get a localObject in your new child context, it is “hollow”, so not connected to all the other objects floating around in the parent. We also can’t fire relationships, because those relationships will go to the database instead of the parent context.
>>>
>>> Ken
>>>
>>>> On Mar 28, 2018, at 7:11 PM, Hugi Thordarson <hu...@karlmenn.is> wrote:
>>>>
>>>>> That's exactly what we want to do - save once at the end. However, we have 2 problems:
>>>>>
>>>>> 1. How do we find the objects that we already created but haven't saved yet
>>>>
>>>> You can go through yur ObjectContext's newObjects() and filter that to your liking—pretty much the same as you'd do with EOF.
>>>>
>>>>> 2. How do we roll back each line if there's an error? Not a DB error, but the logic gets so far, and then determines that there's no way to continue so we must skip this line.
>>>>
>>>> As you tried yourself, I'd use a child context and commit to the parent once you're sure everything is in place. Can you explain further what was problematic with that (that you need to "access the same objects multiple times")? Do you mean that each row of the file is in some way looking at data from other rows?
>>>>
>>>> Cheers,
>>>> - hugi
>>>>
>>>>
>>>>>
>>>>> Ken
>>>>>
>>>>> On 3/28/18, 6:07 PM, "John Huss" <jo...@gmail.com> wrote:
>>>>>
>>>>> Well, you could just save once at the end. Why do you need to save
>>>>> multiple times during the processing? Validation exceptions and Optimistic
>>>>> Locking errors could be handled in the save with some custom logic and a
>>>>> retry.
>>>>>
>>>>> Or if this isn't a super long process you can use a database transaction to
>>>>> allow saving multiple times without actually having that data be visible
>>>>> outside of the transaction.
>>>>>
>>>>>
>>>>> On Wed, Mar 28, 2018 at 6:56 AM Ken Anderson <ke...@amphorainc.com>
>>>>> wrote:
>>>>>
>>>>>> All,
>>>>>>
>>>>>> We have a process that reads in a file and, for each line, creates or
>>>>>> edits objects in the object graph. We only want to commit to the database
>>>>>> once at the end.
>>>>>>
>>>>>> We have a finite set of lines, so memory is not an issue. We need to save
>>>>>> only once because saving will actually fire triggers that will start doing
>>>>>> other things to the database, which will then lead to optimistic lock
>>>>>> exceptions for us if we have data that overlaps (which we do).
>>>>>>
>>>>>> Please don’t suggest we change how the trigger pattern works – it’s a big
>>>>>> system and we don’t have control over it.
>>>>>>
>>>>>> So, what we’ve toyed with is using a parent/child context arrangement,
>>>>>> where each line is processed in a child, and assuming everything goes OK,
>>>>>> we commit only to the parent. This works well as long as we don’t need to
>>>>>> access the same objects multiple times, but unfortunately, we do. We can
>>>>>> reach into the parent context’s unsaved objects, but those objects do not
>>>>>> have any relationships since they were built in the child context. This
>>>>>> makes things painful.
>>>>>>
>>>>>> In EOF, I might consider using a single context and undo, but it doesn’t
>>>>>> seem like Cayenne has this kind of functionality.
>>>>>>
>>>>>> Thoughts? Suggestions? In EOF, I had once written a layer that
>>>>>> intercepted all queries and tried to find the correct object in unsaved
>>>>>> objects, but I don’t have nearly enough experience with Cayenne to do that.
>>>>>>
>>>>>> Thanks!
>>>>>> Ken
>>>>>>
>>>>>> Confidentiality Notice: This e-mail and accompanying documents contain
>>>>>> confidential information intended for a specific individual and purpose.
>>>>>> This e-mailed information is private and protected by law. If you are not
>>>>>> the intended recipient, you are hereby notified that any disclosure,
>>>>>> copying, or distribution, or the taking of any action based on the contents
>>>>>> of this information, is strictly prohibited.
>>>>>>
>>>>>
>>>>>
>>>>> Confidentiality Notice: This e-mail and accompanying documents contain confidential information intended for a specific individual and purpose. This e-mailed information is private and protected by law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, or distribution, or the taking of any action based on the contents of this information, is strictly prohibited.
>>>>
>>>
>>
>>
>> Confidentiality Notice: This e-mail and accompanying documents contain confidential information intended for a specific individual and purpose. This e-mailed information is private and protected by law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, or distribution, or the taking of any action based on the contents of this information, is strictly prohibited.
Re: Staged saves pattern ?
Posted by Mike Kienenberger <mk...@gmail.com>.
Like I said, I haven't used nested contexts, but I'm assuming that
data in the parent context is available to the child contexts, but
data in sibling contexts is not available to the child context.
As for not nesting contexts too deep, I'm guessing that's a
performance issue rather than a technical limitation. This process
doesn't sound like it's an interactive time-sensitive operation, and
you've already said that memory isn't an issue (and data contexts are
not memory intensive in any case).
Years ago, before Cayenne had child contexts, I handled situations
like yours by using a single context and a hashmap.
As I created new objects, I stored them in the hash map by the line
identifier, then pulled them back out later when I had a line that
referred to a previous line.
On Thu, Mar 29, 2018 at 5:50 PM, Ken Anderson
<ke...@amphorainc.com> wrote:
> Mike,
>
> The docs say to not go too deep with nested contexts, so I don't think that's a viable solution. I'm also not clear what benefit you think this would have over the 2 level plan.
>
> Ken
>
> On 3/29/18, 5:39 PM, "Mike Kienenberger" <mk...@gmail.com> wrote:
>
> Instead of having each line as a child context under a parent context,
>
> parent
> - child1 (line 1)
> - child2 (line 2)
> - child3 (line 3)
>
> could you have each line processed as a child context of the previous line?
>
> - child1 (line 1)
> -- child2 (line 2)
> --- child3 (line 3)
> ---- child4 (line 4)
>
> If you want to "undo" the current line,
> child4.rollbackChangesLocally(); and start on the next line.
>
> Note that I have not done anything with child contexts, but this would
> be how I'd try to solve it.
>
>
> On Wed, Mar 28, 2018 at 9:29 PM, Ken Anderson <ke...@anderhome.com> wrote:
> > Hugi,
> >
> > That’s correct - it’s not like we’re just importing a bunch of records. Each row in the file could affect the same set of objects.
> >
> > So, we did the child context, but obviously if we created an object in a prior child and then saved it to the parent, we won’t be able to easily find it in the next child context. If you get a localObject in your new child context, it is “hollow”, so not connected to all the other objects floating around in the parent. We also can’t fire relationships, because those relationships will go to the database instead of the parent context.
> >
> > Ken
> >
> >> On Mar 28, 2018, at 7:11 PM, Hugi Thordarson <hu...@karlmenn.is> wrote:
> >>
> >>> That's exactly what we want to do - save once at the end. However, we have 2 problems:
> >>>
> >>> 1. How do we find the objects that we already created but haven't saved yet
> >>
> >> You can go through yur ObjectContext's newObjects() and filter that to your liking—pretty much the same as you'd do with EOF.
> >>
> >>> 2. How do we roll back each line if there's an error? Not a DB error, but the logic gets so far, and then determines that there's no way to continue so we must skip this line.
> >>
> >> As you tried yourself, I'd use a child context and commit to the parent once you're sure everything is in place. Can you explain further what was problematic with that (that you need to "access the same objects multiple times")? Do you mean that each row of the file is in some way looking at data from other rows?
> >>
> >> Cheers,
> >> - hugi
> >>
> >>
> >>>
> >>> Ken
> >>>
> >>> On 3/28/18, 6:07 PM, "John Huss" <jo...@gmail.com> wrote:
> >>>
> >>> Well, you could just save once at the end. Why do you need to save
> >>> multiple times during the processing? Validation exceptions and Optimistic
> >>> Locking errors could be handled in the save with some custom logic and a
> >>> retry.
> >>>
> >>> Or if this isn't a super long process you can use a database transaction to
> >>> allow saving multiple times without actually having that data be visible
> >>> outside of the transaction.
> >>>
> >>>
> >>> On Wed, Mar 28, 2018 at 6:56 AM Ken Anderson <ke...@amphorainc.com>
> >>> wrote:
> >>>
> >>>> All,
> >>>>
> >>>> We have a process that reads in a file and, for each line, creates or
> >>>> edits objects in the object graph. We only want to commit to the database
> >>>> once at the end.
> >>>>
> >>>> We have a finite set of lines, so memory is not an issue. We need to save
> >>>> only once because saving will actually fire triggers that will start doing
> >>>> other things to the database, which will then lead to optimistic lock
> >>>> exceptions for us if we have data that overlaps (which we do).
> >>>>
> >>>> Please don’t suggest we change how the trigger pattern works – it’s a big
> >>>> system and we don’t have control over it.
> >>>>
> >>>> So, what we’ve toyed with is using a parent/child context arrangement,
> >>>> where each line is processed in a child, and assuming everything goes OK,
> >>>> we commit only to the parent. This works well as long as we don’t need to
> >>>> access the same objects multiple times, but unfortunately, we do. We can
> >>>> reach into the parent context’s unsaved objects, but those objects do not
> >>>> have any relationships since they were built in the child context. This
> >>>> makes things painful.
> >>>>
> >>>> In EOF, I might consider using a single context and undo, but it doesn’t
> >>>> seem like Cayenne has this kind of functionality.
> >>>>
> >>>> Thoughts? Suggestions? In EOF, I had once written a layer that
> >>>> intercepted all queries and tried to find the correct object in unsaved
> >>>> objects, but I don’t have nearly enough experience with Cayenne to do that.
> >>>>
> >>>> Thanks!
> >>>> Ken
> >>>>
> >>>> Confidentiality Notice: This e-mail and accompanying documents contain
> >>>> confidential information intended for a specific individual and purpose.
> >>>> This e-mailed information is private and protected by law. If you are not
> >>>> the intended recipient, you are hereby notified that any disclosure,
> >>>> copying, or distribution, or the taking of any action based on the contents
> >>>> of this information, is strictly prohibited.
> >>>>
> >>>
> >>>
> >>> Confidentiality Notice: This e-mail and accompanying documents contain confidential information intended for a specific individual and purpose. This e-mailed information is private and protected by law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, or distribution, or the taking of any action based on the contents of this information, is strictly prohibited.
> >>
> >
>
>
> Confidentiality Notice: This e-mail and accompanying documents contain confidential information intended for a specific individual and purpose. This e-mailed information is private and protected by law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, or distribution, or the taking of any action based on the contents of this information, is strictly prohibited.
Re: Staged saves pattern ?
Posted by Ken Anderson <ke...@amphorainc.com>.
Mike,
The docs say to not go too deep with nested contexts, so I don't think that's a viable solution. I'm also not clear what benefit you think this would have over the 2 level plan.
Ken
On 3/29/18, 5:39 PM, "Mike Kienenberger" <mk...@gmail.com> wrote:
Instead of having each line as a child context under a parent context,
parent
- child1 (line 1)
- child2 (line 2)
- child3 (line 3)
could you have each line processed as a child context of the previous line?
- child1 (line 1)
-- child2 (line 2)
--- child3 (line 3)
---- child4 (line 4)
If you want to "undo" the current line,
child4.rollbackChangesLocally(); and start on the next line.
Note that I have not done anything with child contexts, but this would
be how I'd try to solve it.
On Wed, Mar 28, 2018 at 9:29 PM, Ken Anderson <ke...@anderhome.com> wrote:
> Hugi,
>
> That’s correct - it’s not like we’re just importing a bunch of records. Each row in the file could affect the same set of objects.
>
> So, we did the child context, but obviously if we created an object in a prior child and then saved it to the parent, we won’t be able to easily find it in the next child context. If you get a localObject in your new child context, it is “hollow”, so not connected to all the other objects floating around in the parent. We also can’t fire relationships, because those relationships will go to the database instead of the parent context.
>
> Ken
>
>> On Mar 28, 2018, at 7:11 PM, Hugi Thordarson <hu...@karlmenn.is> wrote:
>>
>>> That's exactly what we want to do - save once at the end. However, we have 2 problems:
>>>
>>> 1. How do we find the objects that we already created but haven't saved yet
>>
>> You can go through yur ObjectContext's newObjects() and filter that to your liking—pretty much the same as you'd do with EOF.
>>
>>> 2. How do we roll back each line if there's an error? Not a DB error, but the logic gets so far, and then determines that there's no way to continue so we must skip this line.
>>
>> As you tried yourself, I'd use a child context and commit to the parent once you're sure everything is in place. Can you explain further what was problematic with that (that you need to "access the same objects multiple times")? Do you mean that each row of the file is in some way looking at data from other rows?
>>
>> Cheers,
>> - hugi
>>
>>
>>>
>>> Ken
>>>
>>> On 3/28/18, 6:07 PM, "John Huss" <jo...@gmail.com> wrote:
>>>
>>> Well, you could just save once at the end. Why do you need to save
>>> multiple times during the processing? Validation exceptions and Optimistic
>>> Locking errors could be handled in the save with some custom logic and a
>>> retry.
>>>
>>> Or if this isn't a super long process you can use a database transaction to
>>> allow saving multiple times without actually having that data be visible
>>> outside of the transaction.
>>>
>>>
>>> On Wed, Mar 28, 2018 at 6:56 AM Ken Anderson <ke...@amphorainc.com>
>>> wrote:
>>>
>>>> All,
>>>>
>>>> We have a process that reads in a file and, for each line, creates or
>>>> edits objects in the object graph. We only want to commit to the database
>>>> once at the end.
>>>>
>>>> We have a finite set of lines, so memory is not an issue. We need to save
>>>> only once because saving will actually fire triggers that will start doing
>>>> other things to the database, which will then lead to optimistic lock
>>>> exceptions for us if we have data that overlaps (which we do).
>>>>
>>>> Please don’t suggest we change how the trigger pattern works – it’s a big
>>>> system and we don’t have control over it.
>>>>
>>>> So, what we’ve toyed with is using a parent/child context arrangement,
>>>> where each line is processed in a child, and assuming everything goes OK,
>>>> we commit only to the parent. This works well as long as we don’t need to
>>>> access the same objects multiple times, but unfortunately, we do. We can
>>>> reach into the parent context’s unsaved objects, but those objects do not
>>>> have any relationships since they were built in the child context. This
>>>> makes things painful.
>>>>
>>>> In EOF, I might consider using a single context and undo, but it doesn’t
>>>> seem like Cayenne has this kind of functionality.
>>>>
>>>> Thoughts? Suggestions? In EOF, I had once written a layer that
>>>> intercepted all queries and tried to find the correct object in unsaved
>>>> objects, but I don’t have nearly enough experience with Cayenne to do that.
>>>>
>>>> Thanks!
>>>> Ken
>>>>
>>>> Confidentiality Notice: This e-mail and accompanying documents contain
>>>> confidential information intended for a specific individual and purpose.
>>>> This e-mailed information is private and protected by law. If you are not
>>>> the intended recipient, you are hereby notified that any disclosure,
>>>> copying, or distribution, or the taking of any action based on the contents
>>>> of this information, is strictly prohibited.
>>>>
>>>
>>>
>>> Confidentiality Notice: This e-mail and accompanying documents contain confidential information intended for a specific individual and purpose. This e-mailed information is private and protected by law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, or distribution, or the taking of any action based on the contents of this information, is strictly prohibited.
>>
>
Confidentiality Notice: This e-mail and accompanying documents contain confidential information intended for a specific individual and purpose. This e-mailed information is private and protected by law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, or distribution, or the taking of any action based on the contents of this information, is strictly prohibited.
Re: Staged saves pattern ?
Posted by Mike Kienenberger <mk...@gmail.com>.
Instead of having each line as a child context under a parent context,
parent
- child1 (line 1)
- child2 (line 2)
- child3 (line 3)
could you have each line processed as a child context of the previous line?
- child1 (line 1)
-- child2 (line 2)
--- child3 (line 3)
---- child4 (line 4)
If you want to "undo" the current line,
child4.rollbackChangesLocally(); and start on the next line.
Note that I have not done anything with child contexts, but this would
be how I'd try to solve it.
On Wed, Mar 28, 2018 at 9:29 PM, Ken Anderson <ke...@anderhome.com> wrote:
> Hugi,
>
> That’s correct - it’s not like we’re just importing a bunch of records. Each row in the file could affect the same set of objects.
>
> So, we did the child context, but obviously if we created an object in a prior child and then saved it to the parent, we won’t be able to easily find it in the next child context. If you get a localObject in your new child context, it is “hollow”, so not connected to all the other objects floating around in the parent. We also can’t fire relationships, because those relationships will go to the database instead of the parent context.
>
> Ken
>
>> On Mar 28, 2018, at 7:11 PM, Hugi Thordarson <hu...@karlmenn.is> wrote:
>>
>>> That's exactly what we want to do - save once at the end. However, we have 2 problems:
>>>
>>> 1. How do we find the objects that we already created but haven't saved yet
>>
>> You can go through yur ObjectContext's newObjects() and filter that to your liking—pretty much the same as you'd do with EOF.
>>
>>> 2. How do we roll back each line if there's an error? Not a DB error, but the logic gets so far, and then determines that there's no way to continue so we must skip this line.
>>
>> As you tried yourself, I'd use a child context and commit to the parent once you're sure everything is in place. Can you explain further what was problematic with that (that you need to "access the same objects multiple times")? Do you mean that each row of the file is in some way looking at data from other rows?
>>
>> Cheers,
>> - hugi
>>
>>
>>>
>>> Ken
>>>
>>> On 3/28/18, 6:07 PM, "John Huss" <jo...@gmail.com> wrote:
>>>
>>> Well, you could just save once at the end. Why do you need to save
>>> multiple times during the processing? Validation exceptions and Optimistic
>>> Locking errors could be handled in the save with some custom logic and a
>>> retry.
>>>
>>> Or if this isn't a super long process you can use a database transaction to
>>> allow saving multiple times without actually having that data be visible
>>> outside of the transaction.
>>>
>>>
>>> On Wed, Mar 28, 2018 at 6:56 AM Ken Anderson <ke...@amphorainc.com>
>>> wrote:
>>>
>>>> All,
>>>>
>>>> We have a process that reads in a file and, for each line, creates or
>>>> edits objects in the object graph. We only want to commit to the database
>>>> once at the end.
>>>>
>>>> We have a finite set of lines, so memory is not an issue. We need to save
>>>> only once because saving will actually fire triggers that will start doing
>>>> other things to the database, which will then lead to optimistic lock
>>>> exceptions for us if we have data that overlaps (which we do).
>>>>
>>>> Please don’t suggest we change how the trigger pattern works – it’s a big
>>>> system and we don’t have control over it.
>>>>
>>>> So, what we’ve toyed with is using a parent/child context arrangement,
>>>> where each line is processed in a child, and assuming everything goes OK,
>>>> we commit only to the parent. This works well as long as we don’t need to
>>>> access the same objects multiple times, but unfortunately, we do. We can
>>>> reach into the parent context’s unsaved objects, but those objects do not
>>>> have any relationships since they were built in the child context. This
>>>> makes things painful.
>>>>
>>>> In EOF, I might consider using a single context and undo, but it doesn’t
>>>> seem like Cayenne has this kind of functionality.
>>>>
>>>> Thoughts? Suggestions? In EOF, I had once written a layer that
>>>> intercepted all queries and tried to find the correct object in unsaved
>>>> objects, but I don’t have nearly enough experience with Cayenne to do that.
>>>>
>>>> Thanks!
>>>> Ken
>>>>
>>>> Confidentiality Notice: This e-mail and accompanying documents contain
>>>> confidential information intended for a specific individual and purpose.
>>>> This e-mailed information is private and protected by law. If you are not
>>>> the intended recipient, you are hereby notified that any disclosure,
>>>> copying, or distribution, or the taking of any action based on the contents
>>>> of this information, is strictly prohibited.
>>>>
>>>
>>>
>>> Confidentiality Notice: This e-mail and accompanying documents contain confidential information intended for a specific individual and purpose. This e-mailed information is private and protected by law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, or distribution, or the taking of any action based on the contents of this information, is strictly prohibited.
>>
>
Re: Staged saves pattern ?
Posted by Ken Anderson <ke...@anderhome.com>.
Hugi,
That’s correct - it’s not like we’re just importing a bunch of records. Each row in the file could affect the same set of objects.
So, we did the child context, but obviously if we created an object in a prior child and then saved it to the parent, we won’t be able to easily find it in the next child context. If you get a localObject in your new child context, it is “hollow”, so not connected to all the other objects floating around in the parent. We also can’t fire relationships, because those relationships will go to the database instead of the parent context.
Ken
> On Mar 28, 2018, at 7:11 PM, Hugi Thordarson <hu...@karlmenn.is> wrote:
>
>> That's exactly what we want to do - save once at the end. However, we have 2 problems:
>>
>> 1. How do we find the objects that we already created but haven't saved yet
>
> You can go through yur ObjectContext's newObjects() and filter that to your liking—pretty much the same as you'd do with EOF.
>
>> 2. How do we roll back each line if there's an error? Not a DB error, but the logic gets so far, and then determines that there's no way to continue so we must skip this line.
>
> As you tried yourself, I'd use a child context and commit to the parent once you're sure everything is in place. Can you explain further what was problematic with that (that you need to "access the same objects multiple times")? Do you mean that each row of the file is in some way looking at data from other rows?
>
> Cheers,
> - hugi
>
>
>>
>> Ken
>>
>> On 3/28/18, 6:07 PM, "John Huss" <jo...@gmail.com> wrote:
>>
>> Well, you could just save once at the end. Why do you need to save
>> multiple times during the processing? Validation exceptions and Optimistic
>> Locking errors could be handled in the save with some custom logic and a
>> retry.
>>
>> Or if this isn't a super long process you can use a database transaction to
>> allow saving multiple times without actually having that data be visible
>> outside of the transaction.
>>
>>
>> On Wed, Mar 28, 2018 at 6:56 AM Ken Anderson <ke...@amphorainc.com>
>> wrote:
>>
>>> All,
>>>
>>> We have a process that reads in a file and, for each line, creates or
>>> edits objects in the object graph. We only want to commit to the database
>>> once at the end.
>>>
>>> We have a finite set of lines, so memory is not an issue. We need to save
>>> only once because saving will actually fire triggers that will start doing
>>> other things to the database, which will then lead to optimistic lock
>>> exceptions for us if we have data that overlaps (which we do).
>>>
>>> Please don’t suggest we change how the trigger pattern works – it’s a big
>>> system and we don’t have control over it.
>>>
>>> So, what we’ve toyed with is using a parent/child context arrangement,
>>> where each line is processed in a child, and assuming everything goes OK,
>>> we commit only to the parent. This works well as long as we don’t need to
>>> access the same objects multiple times, but unfortunately, we do. We can
>>> reach into the parent context’s unsaved objects, but those objects do not
>>> have any relationships since they were built in the child context. This
>>> makes things painful.
>>>
>>> In EOF, I might consider using a single context and undo, but it doesn’t
>>> seem like Cayenne has this kind of functionality.
>>>
>>> Thoughts? Suggestions? In EOF, I had once written a layer that
>>> intercepted all queries and tried to find the correct object in unsaved
>>> objects, but I don’t have nearly enough experience with Cayenne to do that.
>>>
>>> Thanks!
>>> Ken
>>>
>>> Confidentiality Notice: This e-mail and accompanying documents contain
>>> confidential information intended for a specific individual and purpose.
>>> This e-mailed information is private and protected by law. If you are not
>>> the intended recipient, you are hereby notified that any disclosure,
>>> copying, or distribution, or the taking of any action based on the contents
>>> of this information, is strictly prohibited.
>>>
>>
>>
>> Confidentiality Notice: This e-mail and accompanying documents contain confidential information intended for a specific individual and purpose. This e-mailed information is private and protected by law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, or distribution, or the taking of any action based on the contents of this information, is strictly prohibited.
>
Re: Staged saves pattern ?
Posted by Hugi Thordarson <hu...@karlmenn.is>.
> That's exactly what we want to do - save once at the end. However, we have 2 problems:
>
> 1. How do we find the objects that we already created but haven't saved yet
You can go through yur ObjectContext's newObjects() and filter that to your liking—pretty much the same as you'd do with EOF.
> 2. How do we roll back each line if there's an error? Not a DB error, but the logic gets so far, and then determines that there's no way to continue so we must skip this line.
As you tried yourself, I'd use a child context and commit to the parent once you're sure everything is in place. Can you explain further what was problematic with that (that you need to "access the same objects multiple times")? Do you mean that each row of the file is in some way looking at data from other rows?
Cheers,
- hugi
>
> Ken
>
> On 3/28/18, 6:07 PM, "John Huss" <jo...@gmail.com> wrote:
>
> Well, you could just save once at the end. Why do you need to save
> multiple times during the processing? Validation exceptions and Optimistic
> Locking errors could be handled in the save with some custom logic and a
> retry.
>
> Or if this isn't a super long process you can use a database transaction to
> allow saving multiple times without actually having that data be visible
> outside of the transaction.
>
>
> On Wed, Mar 28, 2018 at 6:56 AM Ken Anderson <ke...@amphorainc.com>
> wrote:
>
>> All,
>>
>> We have a process that reads in a file and, for each line, creates or
>> edits objects in the object graph. We only want to commit to the database
>> once at the end.
>>
>> We have a finite set of lines, so memory is not an issue. We need to save
>> only once because saving will actually fire triggers that will start doing
>> other things to the database, which will then lead to optimistic lock
>> exceptions for us if we have data that overlaps (which we do).
>>
>> Please don’t suggest we change how the trigger pattern works – it’s a big
>> system and we don’t have control over it.
>>
>> So, what we’ve toyed with is using a parent/child context arrangement,
>> where each line is processed in a child, and assuming everything goes OK,
>> we commit only to the parent. This works well as long as we don’t need to
>> access the same objects multiple times, but unfortunately, we do. We can
>> reach into the parent context’s unsaved objects, but those objects do not
>> have any relationships since they were built in the child context. This
>> makes things painful.
>>
>> In EOF, I might consider using a single context and undo, but it doesn’t
>> seem like Cayenne has this kind of functionality.
>>
>> Thoughts? Suggestions? In EOF, I had once written a layer that
>> intercepted all queries and tried to find the correct object in unsaved
>> objects, but I don’t have nearly enough experience with Cayenne to do that.
>>
>> Thanks!
>> Ken
>>
>> Confidentiality Notice: This e-mail and accompanying documents contain
>> confidential information intended for a specific individual and purpose.
>> This e-mailed information is private and protected by law. If you are not
>> the intended recipient, you are hereby notified that any disclosure,
>> copying, or distribution, or the taking of any action based on the contents
>> of this information, is strictly prohibited.
>>
>
>
> Confidentiality Notice: This e-mail and accompanying documents contain confidential information intended for a specific individual and purpose. This e-mailed information is private and protected by law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, or distribution, or the taking of any action based on the contents of this information, is strictly prohibited.
Re: Staged saves pattern ?
Posted by Ken Anderson <ke...@amphorainc.com>.
That's exactly what we want to do - save once at the end. However, we have 2 problems:
1. How do we find the objects that we already created but haven't saved yet
2. How do we roll back each line if there's an error? Not a DB error, but the logic gets so far, and then determines that there's no way to continue so we must skip this line.
Ken
On 3/28/18, 6:07 PM, "John Huss" <jo...@gmail.com> wrote:
Well, you could just save once at the end. Why do you need to save
multiple times during the processing? Validation exceptions and Optimistic
Locking errors could be handled in the save with some custom logic and a
retry.
Or if this isn't a super long process you can use a database transaction to
allow saving multiple times without actually having that data be visible
outside of the transaction.
On Wed, Mar 28, 2018 at 6:56 AM Ken Anderson <ke...@amphorainc.com>
wrote:
> All,
>
> We have a process that reads in a file and, for each line, creates or
> edits objects in the object graph. We only want to commit to the database
> once at the end.
>
> We have a finite set of lines, so memory is not an issue. We need to save
> only once because saving will actually fire triggers that will start doing
> other things to the database, which will then lead to optimistic lock
> exceptions for us if we have data that overlaps (which we do).
>
> Please don’t suggest we change how the trigger pattern works – it’s a big
> system and we don’t have control over it.
>
> So, what we’ve toyed with is using a parent/child context arrangement,
> where each line is processed in a child, and assuming everything goes OK,
> we commit only to the parent. This works well as long as we don’t need to
> access the same objects multiple times, but unfortunately, we do. We can
> reach into the parent context’s unsaved objects, but those objects do not
> have any relationships since they were built in the child context. This
> makes things painful.
>
> In EOF, I might consider using a single context and undo, but it doesn’t
> seem like Cayenne has this kind of functionality.
>
> Thoughts? Suggestions? In EOF, I had once written a layer that
> intercepted all queries and tried to find the correct object in unsaved
> objects, but I don’t have nearly enough experience with Cayenne to do that.
>
> Thanks!
> Ken
>
> Confidentiality Notice: This e-mail and accompanying documents contain
> confidential information intended for a specific individual and purpose.
> This e-mailed information is private and protected by law. If you are not
> the intended recipient, you are hereby notified that any disclosure,
> copying, or distribution, or the taking of any action based on the contents
> of this information, is strictly prohibited.
>
Confidentiality Notice: This e-mail and accompanying documents contain confidential information intended for a specific individual and purpose. This e-mailed information is private and protected by law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, or distribution, or the taking of any action based on the contents of this information, is strictly prohibited.
Re: Staged saves pattern ?
Posted by John Huss <jo...@gmail.com>.
Well, you could just save once at the end. Why do you need to save
multiple times during the processing? Validation exceptions and Optimistic
Locking errors could be handled in the save with some custom logic and a
retry.
Or if this isn't a super long process you can use a database transaction to
allow saving multiple times without actually having that data be visible
outside of the transaction.
On Wed, Mar 28, 2018 at 6:56 AM Ken Anderson <ke...@amphorainc.com>
wrote:
> All,
>
> We have a process that reads in a file and, for each line, creates or
> edits objects in the object graph. We only want to commit to the database
> once at the end.
>
> We have a finite set of lines, so memory is not an issue. We need to save
> only once because saving will actually fire triggers that will start doing
> other things to the database, which will then lead to optimistic lock
> exceptions for us if we have data that overlaps (which we do).
>
> Please don’t suggest we change how the trigger pattern works – it’s a big
> system and we don’t have control over it.
>
> So, what we’ve toyed with is using a parent/child context arrangement,
> where each line is processed in a child, and assuming everything goes OK,
> we commit only to the parent. This works well as long as we don’t need to
> access the same objects multiple times, but unfortunately, we do. We can
> reach into the parent context’s unsaved objects, but those objects do not
> have any relationships since they were built in the child context. This
> makes things painful.
>
> In EOF, I might consider using a single context and undo, but it doesn’t
> seem like Cayenne has this kind of functionality.
>
> Thoughts? Suggestions? In EOF, I had once written a layer that
> intercepted all queries and tried to find the correct object in unsaved
> objects, but I don’t have nearly enough experience with Cayenne to do that.
>
> Thanks!
> Ken
>
> Confidentiality Notice: This e-mail and accompanying documents contain
> confidential information intended for a specific individual and purpose.
> This e-mailed information is private and protected by law. If you are not
> the intended recipient, you are hereby notified that any disclosure,
> copying, or distribution, or the taking of any action based on the contents
> of this information, is strictly prohibited.
>