You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fury.apache.org by "chaokunyang (via GitHub)" <gi...@apache.org> on 2024/05/30 08:57:56 UTC
[PR] feat(java): support meta compression by Deflater [incubator-fury]
chaokunyang opened a new pull request, #1663:
URL: https://github.com/apache/incubator-fury/pull/1663
## What does this PR do?
This PR support meta compression and add Deflater as default compressor.
In our test, it can compress meta by reduce size of **243** :
```
before:
2024-05-30 02:01:52 INFO FuryState:157 [fury_deserialize_compatible-jmh-worker-1] - ======> Fury | STRUCT | false | array | 1227 |
after
2024-05-30 04:47:15 INFO FuryState:157 [fury_deserialize_compatible-jmh-worker-1] - ======> Fury | STRUCT | false | array | 984 |
```
## Related issues
#1660
## Does this PR introduce any user-facing change?
<!--
If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary.
-->
- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?
## Benchmark
<!--
When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): support meta compression by Deflater [incubator-fury]
Posted by "LiangliangSui (via GitHub)" <gi...@apache.org>.
LiangliangSui commented on code in PR #1663:
URL: https://github.com/apache/incubator-fury/pull/1663#discussion_r1622450167
##########
java/fury-core/src/main/java/org/apache/fury/config/Config.java:
##########
@@ -192,6 +195,10 @@ public boolean isScopedMetaShareEnabled() {
return scopedMetaShareEnabled;
}
+ public MetaCompressor getMetaCompressor() {
Review Comment:
Could we add javadoc comments here?
##########
java/fury-core/src/main/java/org/apache/fury/meta/ClassDefEncoder.java:
##########
@@ -146,36 +135,63 @@ static MemoryBuffer encodeClassDef(
int currentClassHeader = (fields.size() << 1);
if (classResolver.isRegistered(type)) {
currentClassHeader |= 1;
- buffer.writeVarUint32Small7(currentClassHeader);
- buffer.writeVarUint32Small7(classResolver.getRegisteredClassId(type));
+ classDefBuf.writeVarUint32Small7(currentClassHeader);
+ classDefBuf.writeVarUint32Small7(classResolver.getRegisteredClassId(type));
} else {
- buffer.writeVarUint32Small7(currentClassHeader);
+ classDefBuf.writeVarUint32Small7(currentClassHeader);
Class<?> currentType = getType(type, className);
Tuple2<String, String> encoded = Encoders.encodePkgAndClass(currentType);
- writePkgName(buffer, encoded.f0);
- writeTypeName(buffer, encoded.f1);
+ writePkgName(classDefBuf, encoded.f0);
+ writeTypeName(classDefBuf, encoded.f1);
}
- writeFieldsInfo(buffer, fields);
+ writeFieldsInfo(classDefBuf, fields);
+ }
+ byte[] compressed =
+ classResolver
+ .getFury()
Review Comment:
I had an idea when I was looking at the code recently.
Could `org.apache.fury.config.Config` be set to `org.apache.fury.resolver.SerializationContext`?
I think it is more reasonable to get `Config` from `SerializationContext` during runtime. It feels a bit weird to get it from `Fury` object.
This will reduce the burden on the `Fury` class.
##########
java/fury-core/src/main/java/org/apache/fury/config/FuryBuilder.java:
##########
@@ -251,6 +254,37 @@ public FuryBuilder withScopedMetaShare(boolean scoped) {
return this;
}
+ /**
+ * Set a compressor for meta compression. Note that the passed {@link MetaCompressor} should be
+ * thread-safe. By default, a `Deflater` based compressor {@link DeflaterMetaCompressor} will be
+ * used. Users can pass other compressor such as `zstd` for better compression rate.
+ */
+ public FuryBuilder withMetaCompressor(MetaCompressor metaCompressor) {
+ Class<?> clz = metaCompressor.getClass();
+ if (clz != DeflaterMetaCompressor.class) {
+ while (clz != null) {
+ try {
+ clz.getDeclaredMethod("hashCode");
+ if (clz == Object.class) {
+ LOG.warn(
+ "{} should implement equals/hashCode method, "
+ + "otherwise compile cache may won't work. "
+ + "Use type to check MetaCompressor identity instead, but this"
+ + "may be incorrect if different compressor instance of same type "
+ + "indicates different compressor.",
+ metaCompressor);
+ metaCompressor = MetaCompressor.typeEqualMetaCompressor(metaCompressor);
+ }
+ break;
+ } catch (NoSuchMethodException e) {
+ clz = clz.getSuperclass();
+ }
+ }
Review Comment:
Could this part of the logic be moved to `MetaCompressor`? This is strongly related to `MetaCompressor` rather than `FuryBuilder`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): support meta compression by Deflater [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #1663:
URL: https://github.com/apache/incubator-fury/pull/1663#discussion_r1623138835
##########
java/fury-core/src/main/java/org/apache/fury/config/FuryBuilder.java:
##########
@@ -251,6 +254,37 @@ public FuryBuilder withScopedMetaShare(boolean scoped) {
return this;
}
+ /**
+ * Set a compressor for meta compression. Note that the passed {@link MetaCompressor} should be
+ * thread-safe. By default, a `Deflater` based compressor {@link DeflaterMetaCompressor} will be
+ * used. Users can pass other compressor such as `zstd` for better compression rate.
+ */
+ public FuryBuilder withMetaCompressor(MetaCompressor metaCompressor) {
+ Class<?> clz = metaCompressor.getClass();
+ if (clz != DeflaterMetaCompressor.class) {
+ while (clz != null) {
+ try {
+ clz.getDeclaredMethod("hashCode");
+ if (clz == Object.class) {
+ LOG.warn(
+ "{} should implement equals/hashCode method, "
+ + "otherwise compile cache may won't work. "
+ + "Use type to check MetaCompressor identity instead, but this"
+ + "may be incorrect if different compressor instance of same type "
+ + "indicates different compressor.",
+ metaCompressor);
+ metaCompressor = MetaCompressor.typeEqualMetaCompressor(metaCompressor);
+ }
+ break;
+ } catch (NoSuchMethodException e) {
+ clz = clz.getSuperclass();
+ }
+ }
Review Comment:
good suggestion, updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): support meta compression by Deflater [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #1663:
URL: https://github.com/apache/incubator-fury/pull/1663#discussion_r1623139137
##########
java/fury-core/src/main/java/org/apache/fury/meta/ClassDefEncoder.java:
##########
@@ -146,36 +135,63 @@ static MemoryBuffer encodeClassDef(
int currentClassHeader = (fields.size() << 1);
if (classResolver.isRegistered(type)) {
currentClassHeader |= 1;
- buffer.writeVarUint32Small7(currentClassHeader);
- buffer.writeVarUint32Small7(classResolver.getRegisteredClassId(type));
+ classDefBuf.writeVarUint32Small7(currentClassHeader);
+ classDefBuf.writeVarUint32Small7(classResolver.getRegisteredClassId(type));
} else {
- buffer.writeVarUint32Small7(currentClassHeader);
+ classDefBuf.writeVarUint32Small7(currentClassHeader);
Class<?> currentType = getType(type, className);
Tuple2<String, String> encoded = Encoders.encodePkgAndClass(currentType);
- writePkgName(buffer, encoded.f0);
- writeTypeName(buffer, encoded.f1);
+ writePkgName(classDefBuf, encoded.f0);
+ writeTypeName(classDefBuf, encoded.f1);
}
- writeFieldsInfo(buffer, fields);
+ writeFieldsInfo(classDefBuf, fields);
+ }
+ byte[] compressed =
+ classResolver
+ .getFury()
Review Comment:
I think it's not related to this PR, we can discuss it in another issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): support meta compression by Deflater [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang merged PR #1663:
URL: https://github.com/apache/incubator-fury/pull/1663
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): support meta compression by Deflater [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #1663:
URL: https://github.com/apache/incubator-fury/pull/1663#discussion_r1622737447
##########
java/fury-core/src/main/java/org/apache/fury/meta/ClassDefEncoder.java:
##########
@@ -146,36 +135,63 @@ static MemoryBuffer encodeClassDef(
int currentClassHeader = (fields.size() << 1);
if (classResolver.isRegistered(type)) {
currentClassHeader |= 1;
- buffer.writeVarUint32Small7(currentClassHeader);
- buffer.writeVarUint32Small7(classResolver.getRegisteredClassId(type));
+ classDefBuf.writeVarUint32Small7(currentClassHeader);
+ classDefBuf.writeVarUint32Small7(classResolver.getRegisteredClassId(type));
} else {
- buffer.writeVarUint32Small7(currentClassHeader);
+ classDefBuf.writeVarUint32Small7(currentClassHeader);
Class<?> currentType = getType(type, className);
Tuple2<String, String> encoded = Encoders.encodePkgAndClass(currentType);
- writePkgName(buffer, encoded.f0);
- writeTypeName(buffer, encoded.f1);
+ writePkgName(classDefBuf, encoded.f0);
+ writeTypeName(classDefBuf, encoded.f1);
}
- writeFieldsInfo(buffer, fields);
+ writeFieldsInfo(classDefBuf, fields);
+ }
+ byte[] compressed =
+ classResolver
+ .getFury()
Review Comment:
Good suggestion, I like the idea to get config from context. But other get context, user must get fury first, then get config from context. It will introduce an extra indirection
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org