You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fury.apache.org by "yihwang6 (via GitHub)" <gi...@apache.org> on 2024/04/19 15:14:28 UTC
[I] [Java] Performance issue when deserializing from FuryInputStream with ZstdInputStream [incubator-fury]
yihwang6 opened a new issue, #1548:
URL: https://github.com/apache/incubator-fury/issues/1548
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-fury/issues) and found no similar issues.
### Version
Version: 0.5.0-SNAPSHOT
OS: Windows
JDK: 8
CommitId：bb419ddf8d4983f438e4b1527c7cb9da74477022
### Component(s)
Java
### Minimal reproduce step
public static final Fury FURY_RELEASE = Fury.builder()
.withLanguage(Language.JAVA)
.requireClassRegistration(false)
.build();
public static <T> ByteArrayOutputStream zstdFurySerialize_Release(T origin) {
try (ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
ZstdOutputStream zstdOutputStream = new ZstdOutputStream(outputStream)) {
FURY_RELEASE.serializeJavaObject(zstdOutputStream, origin);
return outputStream;
} catch (Exception e) {
System.out.println(e);
return null;
}
}
public static <T> T zstdFuryDeserialize_Release(ByteArrayInputStream inputStream, Class<T> clazz) {
try (ZstdInputStream zstdInputStream = new ZstdInputStream(inputStream)) {
return FURY_RELEASE.deserializeJavaObject(zstdInputStream, clazz);
} catch (Exception e) {
System.out.println(e);
return null;
}
}
public static final org.apache.fury.Fury FURY_SNAPSHOT = org.apache.fury.Fury.builder()
.withLanguage(org.apache.fury.config.Language.JAVA)
.requireClassRegistration(false)
.build();
public static <T> ByteArrayOutputStream zstdFurySerialize_Snapshot(T origin) {
try (ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
ZstdOutputStream zstdOutputStream = new ZstdOutputStream(outputStream)) {
FURY_SNAPSHOT.serializeJavaObject(zstdOutputStream, origin);
return outputStream;
} catch (Exception e) {
System.out.println(e);
return null;
}
}
public static <T> T zstdFuryDeserialize_Snapshot(ByteArrayInputStream inputStream, Class<T> clazz) {
try (ZstdInputStream zstdInputStream = new ZstdInputStream(inputStream);
FuryInputStream furyInputStream = new FuryInputStream(zstdInputStream)) {
return FURY_SNAPSHOT.deserializeJavaObject(furyInputStream, clazz);
} catch (Exception e) {
System.out.println(e);
return null;
}
}
### What did you expect to see?
I tested Fury with zstd and found that using FuryInputStream as input for deserializeJavaObject(...) in the latest snapshot version 0.5.0-SNAPSHOT resulted in significantly lower performance compared to the release version 0.4.0..
### What did you see instead?
### Anything Else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] [Java] Performance issue when deserializing from FuryInputStream with ZstdInputStream [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #1548:
URL: https://github.com/apache/incubator-fury/issues/1548#issuecomment-2067029452
I can't run this script on mac, is there somethign extra to configure?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] [Java] Performance issue when deserializing from FuryInputStream with ZstdInputStream [incubator-fury]
Posted by "theweipeng (via GitHub)" <gi...@apache.org>.
theweipeng closed issue #1548: [Java] Performance issue when deserializing from FuryInputStream with ZstdInputStream
URL: https://github.com/apache/incubator-fury/issues/1548
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] [Java] Performance issue when deserializing from FuryInputStream with ZstdInputStream [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #1548:
URL: https://github.com/apache/incubator-fury/issues/1548#issuecomment-2067531898
The `com.github.luben.zstd.ZstdInputStreamNoFinalizer#available` return `1` instead of actual available data, which make the read slow.
I do think it should return acutal available data:
```java
public synchronized int available() throws IOException {
if (this.isClosed) {
throw new IOException("Stream closed");
} else {
return !this.needRead ? 1 : this.in.available();
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] [Java] Performance issue when deserializing from FuryInputStream with ZstdInputStream [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #1548:
URL: https://github.com/apache/incubator-fury/issues/1548#issuecomment-2067071891
How big is your data? Could you add a flamegraph here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] [Java] Performance issue when deserializing from FuryInputStream with ZstdInputStream [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #1548:
URL: https://github.com/apache/incubator-fury/issues/1548#issuecomment-2067544347
Hi @yihwang6 , could you use this https://github.com/apache/incubator-fury/pull/1551 to rerun your benchmark?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org