You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-dev@jackrabbit.apache.org by "Michael Dürig (JIRA)" <ji...@apache.org> on 2012/05/02 14:02:48 UTC
[jira] [Created] (OAK-80) Implement batched writing for
KernelNodeStore
Michael Dürig created OAK-80:
--------------------------------
Summary: Implement batched writing for KernelNodeStore
Key: OAK-80
URL: https://issues.apache.org/jira/browse/OAK-80
Project: Jackrabbit Oak
Issue Type: Improvement
Components: core
Affects Versions: 0.1
Reporter: Michael Dürig
Assignee: Michael Dürig
Currently KernelNodeStore and KernelNodeStateBuilder directly apply every operation on the content tree to the private branch of the Microkernel. There have been some concerns re. performance hits due to network latency in the case where the Microkernel is not co-located.
I suggest to add batching capabilities such that operations are only written through to the Microkernel on certain limits.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OAK-80) Implement batched writing for
KernelNodeStore
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OAK-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266506#comment-13266506 ]
Jukka Zitting commented on OAK-80:
----------------------------------
Personally I'd rather see such batching happening under the MicroKernel API where the implementation actually knows about things like whether network roundtrips are involved and what kind of batch sizes are most useful.
> Implement batched writing for KernelNodeStore
> ---------------------------------------------
>
> Key: OAK-80
> URL: https://issues.apache.org/jira/browse/OAK-80
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.1
> Reporter: Michael Dürig
> Assignee: Michael Dürig
>
> Currently KernelNodeStore and KernelNodeStateBuilder directly apply every operation on the content tree to the private branch of the Microkernel. There have been some concerns re. performance hits due to network latency in the case where the Microkernel is not co-located.
> I suggest to add batching capabilities such that operations are only written through to the Microkernel on certain limits.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OAK-80) Implement batched writing for
KernelNodeStore
Posted by "Thomas Mueller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OAK-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266605#comment-13266605 ]
Thomas Mueller commented on OAK-80:
-----------------------------------
>> the long term goal
> ... Anyway, I don't see how that's relevant to this issue.
The question is how would you build an an efficient oak-core remoting, if every Node.setProperty requires a TCP/IP roundtrip, because it goes through oak-core and down to oak-mk?
But I see the advantage to use the MicroKernel API if that simplifies the implementation, and allows large transactions (that don't fit in memory).
> let's just forget about pushing this below the MK interface for now.
OK, just want to make sure we are on the same page.
> Implement batched writing for KernelNodeStore
> ---------------------------------------------
>
> Key: OAK-80
> URL: https://issues.apache.org/jira/browse/OAK-80
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.1
> Reporter: Michael Dürig
> Assignee: Michael Dürig
>
> Currently KernelNodeStore and KernelNodeStateBuilder directly apply every operation on the content tree to the private branch of the Microkernel. There have been some concerns re. performance hits due to network latency in the case where the Microkernel is not co-located.
> I suggest to add batching capabilities such that operations are only written through to the Microkernel on certain limits.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OAK-80) Implement batched writing for
KernelNodeStore
Posted by "Thomas Mueller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OAK-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266541#comment-13266541 ]
Thomas Mueller commented on OAK-80:
-----------------------------------
>> remoting between oak-jcr and oak-core
> I don't think we have current plans for that
I was the opinion that's the long term goal (allow other programming languages such as PHP to use the oak-core API), and the spi-based remoting was only a temporary solution until oak-core remoting is available. If that's not the plan, then I wonder why we would need a separation between oak-jcr and oak-core.
> Implement batched writing for KernelNodeStore
> ---------------------------------------------
>
> Key: OAK-80
> URL: https://issues.apache.org/jira/browse/OAK-80
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.1
> Reporter: Michael Dürig
> Assignee: Michael Dürig
>
> Currently KernelNodeStore and KernelNodeStateBuilder directly apply every operation on the content tree to the private branch of the Microkernel. There have been some concerns re. performance hits due to network latency in the case where the Microkernel is not co-located.
> I suggest to add batching capabilities such that operations are only written through to the Microkernel on certain limits.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OAK-80) Implement batched writing for
KernelNodeStore
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OAK-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266523#comment-13266523 ]
Jukka Zitting commented on OAK-80:
----------------------------------
OK, makes sense.
> Implement batched writing for KernelNodeStore
> ---------------------------------------------
>
> Key: OAK-80
> URL: https://issues.apache.org/jira/browse/OAK-80
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.1
> Reporter: Michael Dürig
> Assignee: Michael Dürig
>
> Currently KernelNodeStore and KernelNodeStateBuilder directly apply every operation on the content tree to the private branch of the Microkernel. There have been some concerns re. performance hits due to network latency in the case where the Microkernel is not co-located.
> I suggest to add batching capabilities such that operations are only written through to the Microkernel on certain limits.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OAK-80) Implement batched writing for
KernelNodeStore
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OAK-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266587#comment-13266587 ]
Jukka Zitting commented on OAK-80:
----------------------------------
bq. the long term goal
Ah yes, you're right. Anyway, I don't see how that's relevant to this issue.
bq. the implementation in the MK would be quite complicated
The complexity is more or less the same regardless of whether we implement this above or below the MK interface.
Anyway, Michael made a good point above, so let's just forget about pushing this below the MK interface for now.
> Implement batched writing for KernelNodeStore
> ---------------------------------------------
>
> Key: OAK-80
> URL: https://issues.apache.org/jira/browse/OAK-80
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.1
> Reporter: Michael Dürig
> Assignee: Michael Dürig
>
> Currently KernelNodeStore and KernelNodeStateBuilder directly apply every operation on the content tree to the private branch of the Microkernel. There have been some concerns re. performance hits due to network latency in the case where the Microkernel is not co-located.
> I suggest to add batching capabilities such that operations are only written through to the Microkernel on certain limits.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OAK-80) Implement batched writing for
KernelNodeStore
Posted by "Stefan Guggisberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OAK-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266538#comment-13266538 ]
Stefan Guggisberg commented on OAK-80:
--------------------------------------
FWIW:
> Personally I'd rather see such batching happening under the MicroKernel API where the implementation actually knows about things like whether network roundtrips are involved and what kind of batch sizes are most useful.
i imagine that the implementation in the MK would be quite complicated (due to the MVCC contract) whereas supporting batched writes in the upper layer is probably not that difficult.
> Implement batched writing for KernelNodeStore
> ---------------------------------------------
>
> Key: OAK-80
> URL: https://issues.apache.org/jira/browse/OAK-80
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.1
> Reporter: Michael Dürig
> Assignee: Michael Dürig
>
> Currently KernelNodeStore and KernelNodeStateBuilder directly apply every operation on the content tree to the private branch of the Microkernel. There have been some concerns re. performance hits due to network latency in the case where the Microkernel is not co-located.
> I suggest to add batching capabilities such that operations are only written through to the Microkernel on certain limits.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OAK-80) Implement batched writing for
KernelNodeStore
Posted by "Michael Dürig (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OAK-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266516#comment-13266516 ]
Michael Dürig commented on OAK-80:
----------------------------------
Yes this would be another option. However, I'd rather push NodeStore down and make it the Microkernel API. To that respect batching is in the right place here.
Anyway, having batching capabilities at that level does not hurt since it can easily be disabled.
> Implement batched writing for KernelNodeStore
> ---------------------------------------------
>
> Key: OAK-80
> URL: https://issues.apache.org/jira/browse/OAK-80
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.1
> Reporter: Michael Dürig
> Assignee: Michael Dürig
>
> Currently KernelNodeStore and KernelNodeStateBuilder directly apply every operation on the content tree to the private branch of the Microkernel. There have been some concerns re. performance hits due to network latency in the case where the Microkernel is not co-located.
> I suggest to add batching capabilities such that operations are only written through to the Microkernel on certain limits.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (OAK-80) Implement batched writing for
KernelNodeStore
Posted by "Michael Dürig (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OAK-80?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Dürig resolved OAK-80.
------------------------------
Resolution: Fixed
Fix Version/s: 0.3
Fixed at revision 1334059
> Implement batched writing for KernelNodeStore
> ---------------------------------------------
>
> Key: OAK-80
> URL: https://issues.apache.org/jira/browse/OAK-80
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.1
> Reporter: Michael Dürig
> Assignee: Michael Dürig
> Fix For: 0.3
>
>
> Currently KernelNodeStore and KernelNodeStateBuilder directly apply every operation on the content tree to the private branch of the Microkernel. There have been some concerns re. performance hits due to network latency in the case where the Microkernel is not co-located.
> I suggest to add batching capabilities such that operations are only written through to the Microkernel on certain limits.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OAK-80) Implement batched writing for
KernelNodeStore
Posted by "Thomas Mueller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OAK-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266510#comment-13266510 ]
Thomas Mueller commented on OAK-80:
-----------------------------------
If we want to support an efficient remoting between the oak-jcr (the 'upper bun') and oak-core (the 'patty'), then I guess it makes sense to keep the transient space within oak-jcr, at least for changes that fit in memory. That means oak-mk (the 'lower bun') wouldn't be involved for such changes, until session.save(). And session.save() would (in the best case) only result in one call to oak-core.
Unless if the 'upper bun' includes its own MicroKernel implementation for the transient space.
> Implement batched writing for KernelNodeStore
> ---------------------------------------------
>
> Key: OAK-80
> URL: https://issues.apache.org/jira/browse/OAK-80
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.1
> Reporter: Michael Dürig
> Assignee: Michael Dürig
>
> Currently KernelNodeStore and KernelNodeStateBuilder directly apply every operation on the content tree to the private branch of the Microkernel. There have been some concerns re. performance hits due to network latency in the case where the Microkernel is not co-located.
> I suggest to add batching capabilities such that operations are only written through to the Microkernel on certain limits.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OAK-80) Implement batched writing for
KernelNodeStore
Posted by "Michael Dürig (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OAK-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266503#comment-13266503 ]
Michael Dürig commented on OAK-80:
----------------------------------
Revision 1333010 adds new implementations for NodeStateBuilder and NodeStore named KernelNodeStateBuilder2 and KernelNodeStore2. These implementations keep the transient space partially in memory and batch it back to the Microkernel as soon as as the commit size (jsop string) exceed 1024 characters.
The new implementations are not yet wired to the rest of oak-core yet. I will do that in subsequent commits and remove the old implementations.
> Implement batched writing for KernelNodeStore
> ---------------------------------------------
>
> Key: OAK-80
> URL: https://issues.apache.org/jira/browse/OAK-80
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.1
> Reporter: Michael Dürig
> Assignee: Michael Dürig
>
> Currently KernelNodeStore and KernelNodeStateBuilder directly apply every operation on the content tree to the private branch of the Microkernel. There have been some concerns re. performance hits due to network latency in the case where the Microkernel is not co-located.
> I suggest to add batching capabilities such that operations are only written through to the Microkernel on certain limits.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OAK-80) Implement batched writing for
KernelNodeStore
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OAK-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266515#comment-13266515 ]
Jukka Zitting commented on OAK-80:
----------------------------------
bq. remoting between oak-jcr and oak-core
I don't think we have current plans for that. Do we need it (instead of the existing spi-based remoting we already have in Jackrabbit 2.x)?
> Implement batched writing for KernelNodeStore
> ---------------------------------------------
>
> Key: OAK-80
> URL: https://issues.apache.org/jira/browse/OAK-80
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.1
> Reporter: Michael Dürig
> Assignee: Michael Dürig
>
> Currently KernelNodeStore and KernelNodeStateBuilder directly apply every operation on the content tree to the private branch of the Microkernel. There have been some concerns re. performance hits due to network latency in the case where the Microkernel is not co-located.
> I suggest to add batching capabilities such that operations are only written through to the Microkernel on certain limits.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira