You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jackrabbit.apache.org by "Jukka Zitting (JIRA)" <ji...@apache.org> on 2010/09/29 13:46:32 UTC
[jira] Created: (JCR-2762) Optimize bundle serialization
Optimize bundle serialization
-----------------------------
Key: JCR-2762
URL: https://issues.apache.org/jira/browse/JCR-2762
Project: Jackrabbit Content Repository
Issue Type: Improvement
Components: jackrabbit-core
Reporter: Jukka Zitting
Priority: Minor
There are a number of ways we could use to make bundle serialization more optimized. Thomas has already done some work on this in the Jackrabbit 3 sandbox, and I'd like to apply some of the optimizations also to the trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (JCR-2762) Optimize bundle serialization
Posted by "Tobias Bocanegra (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921320#action_12921320 ]
Tobias Bocanegra commented on JCR-2762:
---------------------------------------
are those changes backward compatible? i.e. does it read version2 and write version3? or does it need a complete upgrade of all bundles upon upgrade?
> Optimize bundle serialization
> -----------------------------
>
> Key: JCR-2762
> URL: https://issues.apache.org/jira/browse/JCR-2762
> Project: Jackrabbit Content Repository
> Issue Type: Improvement
> Components: jackrabbit-core
> Reporter: Jukka Zitting
> Assignee: Jukka Zitting
> Priority: Minor
> Fix For: 2.2.0
>
> Attachments: 0001-JCR-2762-Optimize-bundle-serialization.patch, 0002-JCR-2762-Optimize-bundle-serialization.patch, 0003-JCR-2762-Optimize-bundle-serialization.patch, 0004-JCR-2762-Optimize-bundle-serialization.patch
>
>
> There are a number of ways we could use to make bundle serialization more optimized. Thomas has already done some work on this in the Jackrabbit 3 sandbox, and I'd like to apply some of the optimizations also to the trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (JCR-2762) Optimize bundle serialization
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jukka Zitting updated JCR-2762:
-------------------------------
Attachment: 0004-JCR-2762-Optimize-bundle-serialization.patch
Good points! The 0004 patch renames the read/writeInt() methods to read/writeVarInt() and contains a bunch of other minor improvements, most notably that of including names used by access control in the list of common names.
> Optimize bundle serialization
> -----------------------------
>
> Key: JCR-2762
> URL: https://issues.apache.org/jira/browse/JCR-2762
> Project: Jackrabbit Content Repository
> Issue Type: Improvement
> Components: jackrabbit-core
> Reporter: Jukka Zitting
> Priority: Minor
> Attachments: 0001-JCR-2762-Optimize-bundle-serialization.patch, 0002-JCR-2762-Optimize-bundle-serialization.patch, 0003-JCR-2762-Optimize-bundle-serialization.patch, 0004-JCR-2762-Optimize-bundle-serialization.patch
>
>
> There are a number of ways we could use to make bundle serialization more optimized. Thomas has already done some work on this in the Jackrabbit 3 sandbox, and I'd like to apply some of the optimizations also to the trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (JCR-2762) Optimize bundle serialization
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jukka Zitting updated JCR-2762:
-------------------------------
Attachment: 0002-JCR-2762-Optimize-bundle-serialization.patch
Patch 0002 further optimizes the name serialization format by only storing most custom namespace URIs once in a bundle.
> Optimize bundle serialization
> -----------------------------
>
> Key: JCR-2762
> URL: https://issues.apache.org/jira/browse/JCR-2762
> Project: Jackrabbit Content Repository
> Issue Type: Improvement
> Components: jackrabbit-core
> Reporter: Jukka Zitting
> Priority: Minor
> Attachments: 0001-JCR-2762-Optimize-bundle-serialization.patch, 0002-JCR-2762-Optimize-bundle-serialization.patch
>
>
> There are a number of ways we could use to make bundle serialization more optimized. Thomas has already done some work on this in the Jackrabbit 3 sandbox, and I'd like to apply some of the optimizations also to the trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (JCR-2762) Optimize bundle serialization
Posted by "Stefan Guggisberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920468#action_12920468 ]
Stefan Guggisberg commented on JCR-2762:
----------------------------------------
> The question therefore is, should we do this now for Jackrabbit 2.2 or postpone this for 2.3? I
+1 for 2.2
> Optimize bundle serialization
> -----------------------------
>
> Key: JCR-2762
> URL: https://issues.apache.org/jira/browse/JCR-2762
> Project: Jackrabbit Content Repository
> Issue Type: Improvement
> Components: jackrabbit-core
> Reporter: Jukka Zitting
> Priority: Minor
> Attachments: 0001-JCR-2762-Optimize-bundle-serialization.patch, 0002-JCR-2762-Optimize-bundle-serialization.patch, 0003-JCR-2762-Optimize-bundle-serialization.patch, 0004-JCR-2762-Optimize-bundle-serialization.patch
>
>
> There are a number of ways we could use to make bundle serialization more optimized. Thomas has already done some work on this in the Jackrabbit 3 sandbox, and I'd like to apply some of the optimizations also to the trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (JCR-2762) Optimize bundle serialization
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920249#action_12920249 ]
Jukka Zitting commented on JCR-2762:
------------------------------------
There's a few other optimizations I've been working on, most notably improved handling of date and long values, but patches 1-4 already achieve most of the potential size reductions. With these patches the size of typical bundles is down about 30-40% from before. The main benefits of the smaller bundle size are reduced disk usage and therefore faster I/O and more optimal use of disk caches.
Note that all persistence managers necessarily have some storage overhead over the plain bundle size, so the total improvements will not be nearly as big as suggested by the bundle size reduction. For example, with these patches the size of the default workspace of a fresh installation of our proprietary CQ product drops from 154MB to 141MB (-9%). The time it took to fill the workspace (including writes to data store, etc.) went down from 85s to 72s (-16%), though this was just a single try so the performance numbers are fairly uncertain.
I didn't yet commit these changes since changing the serialization format always carries some risk (though we now have a fairly good serialization test cases) and prevents switching back to an older version after an upgrade. The question therefore is, should we do this now for Jackrabbit 2.2 or postpone this for 2.3? I'm inclined to do this already in 2.2.
> Optimize bundle serialization
> -----------------------------
>
> Key: JCR-2762
> URL: https://issues.apache.org/jira/browse/JCR-2762
> Project: Jackrabbit Content Repository
> Issue Type: Improvement
> Components: jackrabbit-core
> Reporter: Jukka Zitting
> Priority: Minor
> Attachments: 0001-JCR-2762-Optimize-bundle-serialization.patch, 0002-JCR-2762-Optimize-bundle-serialization.patch, 0003-JCR-2762-Optimize-bundle-serialization.patch, 0004-JCR-2762-Optimize-bundle-serialization.patch
>
>
> There are a number of ways we could use to make bundle serialization more optimized. Thomas has already done some work on this in the Jackrabbit 3 sandbox, and I'd like to apply some of the optimizations also to the trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (JCR-2762) Optimize bundle serialization
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jukka Zitting updated JCR-2762:
-------------------------------
Attachment: 0003-JCR-2762-Optimize-bundle-serialization.patch
Patch 0003 introduces variable-length encoding for internal counts and some other minor optimizations.
> Optimize bundle serialization
> -----------------------------
>
> Key: JCR-2762
> URL: https://issues.apache.org/jira/browse/JCR-2762
> Project: Jackrabbit Content Repository
> Issue Type: Improvement
> Components: jackrabbit-core
> Reporter: Jukka Zitting
> Priority: Minor
> Attachments: 0001-JCR-2762-Optimize-bundle-serialization.patch, 0002-JCR-2762-Optimize-bundle-serialization.patch, 0003-JCR-2762-Optimize-bundle-serialization.patch
>
>
> There are a number of ways we could use to make bundle serialization more optimized. Thomas has already done some work on this in the Jackrabbit 3 sandbox, and I'd like to apply some of the optimizations also to the trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (JCR-2762) Optimize bundle serialization
Posted by "Thomas Mueller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12919754#action_12919754 ]
Thomas Mueller commented on JCR-2762:
-------------------------------------
I would rename the methods from readInt() / writeInt() to readVarInt() / writeVarInt(). The "var" stands for "variable size". Reason: this following code looks confusing until you understand readInt() doesn't do the same as in.readInt():
+ if (version >= BundleBinding.VERSION_3) {
+ len = readInt();
+ } else {
+ len = in.readInt();
+ }
I would avoid recursion here (the same for writeInt):
+ private int readInt() throws IOException {
+ int b = in.readUnsignedByte();
+ if ((b & 0x80) == 0) {
+ return b;
+ } else {
+ return readInt() << 7 | b & 0x7f;
+ }
+ }
Mainly for performance (but also to avoid reading too much if the data is corrupt). Instead, I would use the J3 implementations at http://svn.apache.org/repos/asf/jackrabbit/sandbox/jackrabbit-j3/src/main/java/org/apache/jackrabbit/j3/mc/Bundle.java
> Optimize bundle serialization
> -----------------------------
>
> Key: JCR-2762
> URL: https://issues.apache.org/jira/browse/JCR-2762
> Project: Jackrabbit Content Repository
> Issue Type: Improvement
> Components: jackrabbit-core
> Reporter: Jukka Zitting
> Priority: Minor
> Attachments: 0001-JCR-2762-Optimize-bundle-serialization.patch, 0002-JCR-2762-Optimize-bundle-serialization.patch, 0003-JCR-2762-Optimize-bundle-serialization.patch
>
>
> There are a number of ways we could use to make bundle serialization more optimized. Thomas has already done some work on this in the Jackrabbit 3 sandbox, and I'd like to apply some of the optimizations also to the trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (JCR-2762) Optimize bundle serialization
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jukka Zitting resolved JCR-2762.
--------------------------------
Resolution: Fixed
Fix Version/s: 2.2.0
Assignee: Jukka Zitting
I committed the attached patches in revisions 1022485 - 1022488 and added more efficient serialization of LONG and DATE values in revision 1022864.
> Optimize bundle serialization
> -----------------------------
>
> Key: JCR-2762
> URL: https://issues.apache.org/jira/browse/JCR-2762
> Project: Jackrabbit Content Repository
> Issue Type: Improvement
> Components: jackrabbit-core
> Reporter: Jukka Zitting
> Assignee: Jukka Zitting
> Priority: Minor
> Fix For: 2.2.0
>
> Attachments: 0001-JCR-2762-Optimize-bundle-serialization.patch, 0002-JCR-2762-Optimize-bundle-serialization.patch, 0003-JCR-2762-Optimize-bundle-serialization.patch, 0004-JCR-2762-Optimize-bundle-serialization.patch
>
>
> There are a number of ways we could use to make bundle serialization more optimized. Thomas has already done some work on this in the Jackrabbit 3 sandbox, and I'd like to apply some of the optimizations also to the trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (JCR-2762) Optimize bundle serialization
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921327#action_12921327 ]
Jukka Zitting commented on JCR-2762:
------------------------------------
All old bundles can still be read, and they will automatically get upgraded to version 3 when they are next modified. No special migration is needed.
> Optimize bundle serialization
> -----------------------------
>
> Key: JCR-2762
> URL: https://issues.apache.org/jira/browse/JCR-2762
> Project: Jackrabbit Content Repository
> Issue Type: Improvement
> Components: jackrabbit-core
> Reporter: Jukka Zitting
> Assignee: Jukka Zitting
> Priority: Minor
> Fix For: 2.2.0
>
> Attachments: 0001-JCR-2762-Optimize-bundle-serialization.patch, 0002-JCR-2762-Optimize-bundle-serialization.patch, 0003-JCR-2762-Optimize-bundle-serialization.patch, 0004-JCR-2762-Optimize-bundle-serialization.patch
>
>
> There are a number of ways we could use to make bundle serialization more optimized. Thomas has already done some work on this in the Jackrabbit 3 sandbox, and I'd like to apply some of the optimizations also to the trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (JCR-2762) Optimize bundle serialization
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jukka Zitting updated JCR-2762:
-------------------------------
Attachment: 0001-JCR-2762-Optimize-bundle-serialization.patch
I started this by some refactoring and by adding test cases designed to ensure backwards compatibility of any improvements.
The attached 0001 patch is my first take at actually changing the serialization format. The patch introduces format version 3 that for now only changes the way JCR names are serialized. Instead of the index-based serialization that always used eight bytes for all names, I use a variable-length encoding that uses just a single byte for most of the standard names, and stores other names inline in the bundle itself. Standard namespaces (including the default namespace) only take three bits of storage. Typical non-standard names like "content" or "author" still end up taking about as much space as before.
> Optimize bundle serialization
> -----------------------------
>
> Key: JCR-2762
> URL: https://issues.apache.org/jira/browse/JCR-2762
> Project: Jackrabbit Content Repository
> Issue Type: Improvement
> Components: jackrabbit-core
> Reporter: Jukka Zitting
> Priority: Minor
> Attachments: 0001-JCR-2762-Optimize-bundle-serialization.patch
>
>
> There are a number of ways we could use to make bundle serialization more optimized. Thomas has already done some work on this in the Jackrabbit 3 sandbox, and I'd like to apply some of the optimizations also to the trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.