You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fury.apache.org by "Munoon (via GitHub)" <gi...@apache.org> on 2024/03/30 09:29:51 UTC
[I] Fury fails when deserializing from InputStream [incubator-fury]
Munoon opened a new issue, #1435:
URL: https://github.com/apache/incubator-fury/issues/1435
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-fury/issues) and found no similar issues.
### Version
0.5.0-SNAPSHOT, 0.4.1
### Component(s)
Java
### Minimal reproduce step
```java
import org.apache.fury.Fury;
import org.apache.fury.config.Language;
import java.io.BufferedInputStream;
import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.util.Arrays;
public class Reproducer {
public static void main(String[] args) {
Fury fury = Fury.builder()
.withLanguage(Language.JAVA)
.requireClassRegistration(false)
.withMetaContextShare(false)
.build();
byte[] data = fury.serializeJavaObject(new TestRecord("Some string", 123));
try {
TestRecord result = fury.deserializeJavaObject(Arrays.copyOf(data, data.length), TestRecord.class);
System.out.println("Byte array result: " + result);
} catch (Exception e) {
System.out.println("Byte array result: " + e.getClass().getSimpleName());
e.printStackTrace();
}
ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(Arrays.copyOf(data, data.length));
try {
TestRecord result = fury.deserializeJavaObject(byteArrayInputStream, TestRecord.class);
System.out.println("ByteArrayInputStream result: " + result);
} catch (Exception e) {
System.out.println("ByteArrayInputStream result: " + e.getClass().getSimpleName());
e.printStackTrace();
}
// hack: Fury have an optimisation for ByteArrayInputStream, which we want to avoid
InputStream inputStream = new BufferedInputStream(new ByteArrayInputStream(Arrays.copyOf(data, data.length)));
try {
TestRecord result = fury.deserializeJavaObject(inputStream, TestRecord.class);
System.out.println("InputStream result: " + result);
} catch (Exception e) {
System.out.println("InputStream result: " + e.getClass().getSimpleName());
e.printStackTrace();
}
}
public static class TestRecord {
public final String a;
public final int b;
public TestRecord(String a, int b) {
this.a = a;
this.b = b;
}
@Override
public String toString() {
return "TestRecord{" +
"a='" + a + '\'' +
", b=" + b +
'}';
}
}
}
```
### What did you expect to see?
No exception to be throwed, the data should be deserialized correctly.
### What did you see instead?
`java.lang.UnsupportedOperationException: Unsupported coder 111` when deserializing with `ByteArrayInputStream` and `java.lang.IllegalArgumentException` when deserializing with any other `InputStream`.
### Anything Else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Fury fails when deserializing from InputStream [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #1435:
URL: https://github.com/apache/incubator-fury/issues/1435#issuecomment-2028020399
Hi @Munoon, `fury.deserializeJavaObject(intputstream, class)` should only be used with `fury.serializeJavaObject(outputstream, object)`. You can't use it to deserialize the data from `fury.serializeJavaObject(object)`.
Our document didn't pointed out this, sorry for the inconvenience. We do have plans to align the implementation between those methods, but that may take a little time to implement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Fury fails when deserializing from InputStream [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #1435:
URL: https://github.com/apache/incubator-fury/issues/1435#issuecomment-2032670945
In those days, objects are getting bigger and bigger, the steaming is important to reduce latency for big objects, especially for big data and tensor computing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Fury fails when deserializing from InputStream [incubator-fury]
Posted by "Munoon (via GitHub)" <gi...@apache.org>.
Munoon commented on issue #1435:
URL: https://github.com/apache/incubator-fury/issues/1435#issuecomment-2028024181
> Hi @Munoon, `fury.deserializeJavaObject(intputstream, class)` should only be used with `fury.serializeJavaObject(outputstream, object)`. You can't use it to deserialize the data from `fury.serializeJavaObject(object)`.
Can you please explain what's the benefit of writing this size?
In my app I am using Fury to write the data in to the file (currently it is implemented using `Files.writeAllBytes(fury.deserializeJavaObject(obj))`) and then read it later.
So I'm trying to understand which case suits better here.
Also, I found out that the fastest way to read the data from the file with Fury is by passing the `FileChannel -> ByteBuffer -> MemoryBuffer` wrapper. Since, I guess, this is a very popular use case in using Fury, I guess such API may be added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Fury fails when deserializing from InputStream [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #1435:
URL: https://github.com/apache/incubator-fury/issues/1435#issuecomment-2032622165
> Fury seems like an interesting project to me, so I'd like to contribute, actually. If you think, that this feature worth implementing - I'd like to make a try.
>
> Also, I think that MemoryBuffer could become interface now, where different implementations holds different data sources (native address, byte array, `InputStream`, channel, etc)
That would be very great, I try to implement the native streaming deserialization in #1451 . It's a little complicated due to we don't employ a polymorphic design for `MemoryBuffer`. The streaming reading for `InputStream` are basiclly finished, but the support for `java.nio.Channel` are not included in that PR. Maybe it can be taken as an example to implement the streaming channel deserializaiton.
I thought about make `MemoryBuffer` polymorphic when I create fury several years ago, but polymorphism will make the invocation and other performance optimization such as bound checks elimination infeasible.
And considering most cases are small object deserializaiton, the streaming are unnecessary, so I made it as a final classes in the end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Fury fails when deserializing from InputStream [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang closed issue #1435: Fury fails when deserializing from InputStream
URL: https://github.com/apache/incubator-fury/issues/1435
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Fury fails when deserializing from InputStream [incubator-fury]
Posted by "Munoon (via GitHub)" <gi...@apache.org>.
Munoon commented on issue #1435:
URL: https://github.com/apache/incubator-fury/issues/1435#issuecomment-2028008555
After hours of coding I finally came up that I have to manually prefix the byte array with it's original size. However I wasn't able to find any information about this on the documentation.
I think, either documentation have to be updated, either InputStream size read should be refactored with `available()` method, either this size should be written by Fury method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Fury fails when deserializing from InputStream [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #1435:
URL: https://github.com/apache/incubator-fury/issues/1435#issuecomment-2028522250
Currently fury doesn't support streaming read, so we prefix the size in fury.serializeJavaObject(outputstream, object. Then when deserialization, we can read size first and read binary from the stream to deserialize the object like you did manually. If you want to fury do this for you, you must use fury.serializeJavaObject(outputstream, object) for serialization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Fury fails when deserializing from InputStream [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #1435:
URL: https://github.com/apache/incubator-fury/issues/1435#issuecomment-2028522511
It is possible to support streaming natively. We can make MemoryBuffer hold a read channel Or input stream object, and read data only in checkReadableBytes method.
It's feasible, but we don't have time for this currently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Fury fails when deserializing from InputStream [incubator-fury]
Posted by "Munoon (via GitHub)" <gi...@apache.org>.
Munoon commented on issue #1435:
URL: https://github.com/apache/incubator-fury/issues/1435#issuecomment-2032048074
Fury seems like an interesting project to me, so I'd like to contribute, actually.
If you think, that this feature worth implementing - I'd like to make a try.
Also, I think that MemoryBuffer could become interface now, where different implementations holds different data sources (native address, byte array, `InputStream`, channel, etc)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org