You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fury.apache.org by "andyczerwonka (via GitHub)" <gi...@apache.org> on 2024/01/09 17:35:02 UTC
[I] llegalArgumentException: Expected AbstractCollectionSerializer but got io.fury.serializer.Serializer [incubator-fury]
andyczerwonka opened a new issue, #1325:
URL: https://github.com/apache/incubator-fury/issues/1325
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-fury/issues) and found no similar issues.
### Version
0.4.1
### Component(s)
Java
### Minimal reproduce step
Run the serializer against our object graph.
### What did you expect to see?
No runtime exception.
### What did you see instead?
class: class io.citrine.lolo.bags.RegressionBaggerTrainingResult
17:02:55 at io.fury.serializer.CodegenSerializer.loadCodegenSerializer(CodegenSerializer.java:48)
17:02:55 at io.fury.resolver.ClassResolver.lambda$getObjectSerializerClass$2(ClassResolver.java:963)
17:02:55 at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
17:02:55 at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
17:02:55 at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
17:02:55 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
17:02:55 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
17:02:55 at java.base/java.lang.Thread.run(Thread.java:829)
17:02:55 Caused by: java.lang.IllegalArgumentException: Expected AbstractCollectionSerializer but got io.fury.serializer.Serializer
17:02:55 at io.fury.util.Preconditions.checkArgument(Preconditions.java:78)
17:02:55 at io.fury.builder.BaseObjectCodecBuilder.deserializeForCollection(BaseObjectCodecBuilder.java:1210)
17:02:55 at io.fury.builder.BaseObjectCodecBuilder.deserializeForNotNull(BaseObjectCodecBuilder.java:1162)
17:02:55 at io.fury.builder.BaseObjectCodecBuilder.lambda$readContainerElement$12(BaseObjectCodecBuilder.java:1401)
17:02:55 at io.fury.builder.BaseObjectCodecBuilder.readRef(BaseObjectCodecBuilder.java:1093)
17:02:55 at io.fury.builder.BaseObjectCodecBuilder.readContainerElement(BaseObjectCodecBuilder.java:1398)
17:02:55 at io.fury.builder.BaseObjectCodecBuilder.lambda$readContainerElements$2223955c$1(BaseObjectCodecBuilder.java:1360)
17:02:55 at io.fury.codegen.Expression$ForLoop.doGenCode(Expression.java:2363)
17:02:55 at io.fury.codegen.Expression.genCode(Expression.java:102)
17:02:55 at io.fury.codegen.Expression$ListExpression.doGenCode(Expression.java:181)
17:02:55 at io.fury.codegen.Expression.genCode(Expression.java:102)
17:02:55 at io.fury.codegen.ExpressionOptimizer.invokeGenerated(ExpressionOptimizer.java:120)
17:02:55 at io.fury.codegen.ExpressionOptimizer.invokeGenerated(ExpressionOptimizer.java:68)
17:02:55 at io.fury.builder.BaseObjectCodecBuilder.readCollectionCodegen(BaseObjectCodecBuilder.java:1298)
17:02:55 at io.fury.builder.BaseObjectCodecBuilder.deserializeForCollection(BaseObjectCodecBuilder.java:1220)
17:02:55 at io.fury.builder.BaseObjectCodecBuilder.deserializeForNotNull(BaseObjectCodecBuilder.java:1162)
17:02:55 at io.fury.builder.BaseObjectCodecBuilder.deserializeForNotNull(BaseObjectCodecBuilder.java:1122)
17:02:55 at io.fury.builder.BaseObjectCodecBuilder.lambda$deserializeFor$7(BaseObjectCodecBuilder.java:1073)
17:02:55 at io.fury.builder.BaseObjectCodecBuilder.readRef(BaseObjectCodecBuilder.java:1093)
17:02:55 at io.fury.builder.BaseObjectCodecBuilder.deserializeFor(BaseObjectCodecBuilder.java:1073)
17:02:55 at io.fury.builder.BaseObjectCodecBuilder.deserializeFor(BaseObjectCodecBuilder.java:1057)
17:02:55 at io.fury.builder.ObjectCodecBuilder.lambda$deserializeGroup$37fcf467$1(ObjectCodecBuilder.java:522)
17:02:55 at io.fury.codegen.ExpressionOptimizer.invokeGenerated(ExpressionOptimizer.java:48)
17:02:55 at io.fury.builder.ObjectCodecOptimizer.invokeGenerated(ObjectCodecOptimizer.java:134)
17:02:55 at io.fury.builder.ObjectCodecBuilder.deserializeGroup(ObjectCodecBuilder.java:538)
17:02:55 at io.fury.builder.ObjectCodecBuilder.buildDecodeExpression(ObjectCodecBuilder.java:439)
17:02:55 at io.fury.builder.BaseObjectCodecBuilder.genCode(BaseObjectCodecBuilder.java:204)
17:02:55 at io.fury.codegen.CompileUnit.getCode(CompileUnit.java:54)
17:02:55 at io.fury.codegen.JaninoUtils.toBytecode(JaninoUtils.java:72)
17:02:55 at io.fury.codegen.JaninoUtils.toBytecode(JaninoUtils.java:64)
17:02:55 at io.fury.codegen.CodeGenerator.compile(CodeGenerator.java:144)
17:02:55 at io.fury.builder.CodecUtils.loadOrGenCodecClass(CodecUtils.java:91)
17:02:55 at io.fury.builder.CodecUtils.loadOrGenObjectCodecClass(CodecUtils.java:42)
17:02:55 at io.fury.serializer.CodegenSerializer.loadCodegenSerializer(CodegenSerializer.java:45)
17:02:55 ... 7 more```
```
### Anything Else?
I might be doing something wrong, but would like to help to triage.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "andyczerwonka (via GitHub)" <gi...@apache.org>.
andyczerwonka commented on issue #1325:
URL: https://github.com/apache/incubator-fury/issues/1325#issuecomment-1894882630
> Please note latest snapshot is released under ASF, and the package name has been renamed to `org.apache.fury` instead of `io.fury`
Which is probably why it didn’t get pulled in. When can we expect a non-snapshot release?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #1325:
URL: https://github.com/apache/incubator-fury/issues/1325#issuecomment-1898844103
Could you build a jar nad use that jar? We may need some time before we release a jar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #1325:
URL: https://github.com/apache/incubator-fury/issues/1325#issuecomment-1893077940
Can your reproduction code be converted into a unit test? Would be great if we can include it into fury unit test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #1325:
URL: https://github.com/apache/incubator-fury/issues/1325#issuecomment-1892981545
Hi @andyczerwonka, has this issue been addressed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang closed issue #1325: Create sequential serializer failed
URL: https://github.com/apache/incubator-fury/issues/1325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "andyczerwonka (via GitHub)" <gi...@apache.org>.
andyczerwonka commented on issue #1325:
URL: https://github.com/apache/incubator-fury/issues/1325#issuecomment-1893087457
I have tested it with the `0.5.0-SNAPSHOT` version and it continues to fail.
```
Exception in thread "fury-jit-compiler-10" java.lang.RuntimeException: Create sequential serializer failed,
class: class io.citrine.lolo.bags.RegressionBaggerTrainingResult
at io.fury.serializer.CodegenSerializer.loadCodegenSerializer(CodegenSerializer.java:51)
at io.fury.resolver.ClassResolver.lambda$getObjectSerializerClass$2(ClassResolver.java:966)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalArgumentException: Expected AbstractCollectionSerializer but got io.fury.serializer.Serializer
at io.fury.util.Preconditions.checkArgument(Preconditions.java:81)
at io.fury.builder.BaseObjectCodecBuilder.deserializeForCollection(BaseObjectCodecBuilder.java:1210)
at io.fury.builder.BaseObjectCodecBuilder.deserializeForNotNull(BaseObjectCodecBuilder.java:1162)
at io.fury.builder.BaseObjectCodecBuilder.lambda$readContainerElement$12(BaseObjectCodecBuilder.java:1400)
at io.fury.builder.BaseObjectCodecBuilder.readRef(BaseObjectCodecBuilder.java:1093)
at io.fury.builder.BaseObjectCodecBuilder.readContainerElement(BaseObjectCodecBuilder.java:1397)
at io.fury.builder.BaseObjectCodecBuilder.lambda$readContainerElements$2223955c$1(BaseObjectCodecBuilder.java:1359)
at io.fury.codegen.Expression$ForLoop.doGenCode(Expression.java:2366)
at io.fury.codegen.Expression.genCode(Expression.java:105)
at io.fury.codegen.Expression$ListExpression.doGenCode(Expression.java:184)
at io.fury.codegen.Expression.genCode(Expression.java:105)
at io.fury.codegen.ExpressionOptimizer.invokeGenerated(ExpressionOptimizer.java:123)
at io.fury.codegen.ExpressionOptimizer.invokeGenerated(ExpressionOptimizer.java:71)
at io.fury.builder.BaseObjectCodecBuilder.readCollectionCodegen(BaseObjectCodecBuilder.java:1297)
at io.fury.builder.BaseObjectCodecBuilder.deserializeForCollection(BaseObjectCodecBuilder.java:1220)
at io.fury.builder.BaseObjectCodecBuilder.deserializeForNotNull(BaseObjectCodecBuilder.java:1162)
at io.fury.builder.BaseObjectCodecBuilder.deserializeForNotNull(BaseObjectCodecBuilder.java:1122)
at io.fury.builder.BaseObjectCodecBuilder.lambda$deserializeFor$7(BaseObjectCodecBuilder.java:1073)
at io.fury.builder.BaseObjectCodecBuilder.readRef(BaseObjectCodecBuilder.java:1093)
at io.fury.builder.BaseObjectCodecBuilder.deserializeFor(BaseObjectCodecBuilder.java:1073)
at io.fury.builder.BaseObjectCodecBuilder.deserializeFor(BaseObjectCodecBuilder.java:1057)
at io.fury.builder.ObjectCodecBuilder.lambda$deserializeGroup$37fcf467$1(ObjectCodecBuilder.java:525)
at io.fury.codegen.ExpressionOptimizer.invokeGenerated(ExpressionOptimizer.java:51)
at io.fury.builder.ObjectCodecOptimizer.invokeGenerated(ObjectCodecOptimizer.java:137)
at io.fury.builder.ObjectCodecBuilder.deserializeGroup(ObjectCodecBuilder.java:541)
at io.fury.builder.ObjectCodecBuilder.buildDecodeExpression(ObjectCodecBuilder.java:442)
at io.fury.builder.BaseObjectCodecBuilder.genCode(BaseObjectCodecBuilder.java:208)
at io.fury.codegen.CompileUnit.getCode(CompileUnit.java:57)
at io.fury.codegen.JaninoUtils.toBytecode(JaninoUtils.java:75)
at io.fury.codegen.JaninoUtils.toBytecode(JaninoUtils.java:67)
at io.fury.codegen.CodeGenerator.compile(CodeGenerator.java:147)
at io.fury.builder.CodecUtils.loadOrGenCodecClass(CodecUtils.java:94)
at io.fury.builder.CodecUtils.loadOrGenObjectCodecClass(CodecUtils.java:45)
at io.fury.serializer.CodegenSerializer.loadCodegenSerializer(CodegenSerializer.java:48)
... 7 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "andyczerwonka (via GitHub)" <gi...@apache.org>.
andyczerwonka commented on issue #1325:
URL: https://github.com/apache/incubator-fury/issues/1325#issuecomment-1893071585
@chaokunyang I have been able to reproduce. I now need instructions as to how to build the jar with your fix to validate the fix. Or you can attach it here if that's easier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "andyczerwonka (via GitHub)" <gi...@apache.org>.
andyczerwonka commented on issue #1325:
URL: https://github.com/apache/incubator-fury/issues/1325#issuecomment-1896169611
@chaokunyang I have confirmed that the issue has been resolved in 0.5.0.
Is there a work-around for this issue? Can it be back-ported to something that can be released earlier? We are in a non-working state and can't move forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #1325:
URL: https://github.com/apache/incubator-fury/issues/1325#issuecomment-1887303862
https://github.com/chaokunyang/fury/tree/refine_collection_serializer_cast
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "andyczerwonka (via GitHub)" <gi...@apache.org>.
andyczerwonka commented on issue #1325:
URL: https://github.com/apache/incubator-fury/issues/1325#issuecomment-1894876657
Is the nested collection issue fixed in the latest snapshot?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #1325:
URL: https://github.com/apache/incubator-fury/issues/1325#issuecomment-1893076995
You can just use
```xml
<repositories>
<repository>
<id>apache</id>
<url>https://repository.apache.org/snapshots/</url>
<releases>
<enabled>false</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
<dependency>
<groupId>org.apache.fury</groupId>
<artifactId>fury-core</artifactId>
<version>0.5.0-SNAPSHOT</version>
</dependency>
<!-- row/arrow format support -->
<!-- <dependency>
<groupId>org.apache.fury</groupId>
<artifactId>fury-format</artifactId>
<version>0.5.0-SNAPSHOT</version>
</dependency> -->
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "andyczerwonka (via GitHub)" <gi...@apache.org>.
andyczerwonka commented on issue #1325:
URL: https://github.com/apache/incubator-fury/issues/1325#issuecomment-1894125870
I have created an example of the failure.
https://gist.github.com/andyczerwonka/23da899e21d3f69618360024038e4be4#file-serdethreadingtest-scala-L20-L22
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "andyczerwonka (via GitHub)" <gi...@apache.org>.
andyczerwonka commented on issue #1325:
URL: https://github.com/apache/incubator-fury/issues/1325#issuecomment-1898959480
Defect is addressed in the latest snapshot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "andyczerwonka (via GitHub)" <gi...@apache.org>.
andyczerwonka commented on issue #1325:
URL: https://github.com/apache/incubator-fury/issues/1325#issuecomment-1887298572
> @andyczerwonka I try to fix it in #1333, but I'm not sure whether they are same issue. Could you try this branch in your environment?
Which branch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "andyczerwonka (via GitHub)" <gi...@apache.org>.
andyczerwonka commented on issue #1325:
URL: https://github.com/apache/incubator-fury/issues/1325#issuecomment-1884133976
> Hi @andyczerwonka , thanks for reporting this bug. Can this bug be reproduced locally?
No, we’re seeing it intermittently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #1325:
URL: https://github.com/apache/incubator-fury/issues/1325#issuecomment-1894863166
Hi @andyczerwonka , I tested with your code, Fury succeeds in serialization:
```scala
package org.apache.fury.serializer;
import org.apache.fury.Fury
import org.apache.fury.config.Language
import java.io.ByteArrayInputStream
import java.io.ByteArrayOutputStream
import java.util.Base64
import java.util.zip.GZIPInputStream
import java.util.zip.GZIPOutputStream
import _root_.scala
import _root_.scala.util.{Success, Try}
import _root_.scala.concurrent.Future
import _root_.scala.concurrent.ExecutionContext
import java.util.concurrent.ThreadLocalRandom
import _root_.scala.concurrent.Await
import org.apache.fury.ThreadLocalFury
import org.scalatest.matchers.should.Matchers
import org.scalatest.wordspec.AnyWordSpec
// What makes this fail is the nested collection in the case class. If you change it to
// as 1-dimensional collection, we no longer see the exception
case class SampleData(label: String, data: Seq[Seq[Int]])
class SerdeThreadingTest extends AnyWordSpec with Matchers {
def threadLocalFury =
new ThreadLocalFury(classLoader => {
Fury
.builder()
.withLanguage(Language.JAVA)
.requireClassRegistration(false)
.withScalaOptimizationEnabled(true)
.withRefTracking(true)
.withStringCompressed(true)
.withLongCompressed(true)
.withIntCompressed(true)
.withAsyncCompilation(true)
.withClassLoader(classLoader)
.build()
})
private val fury = Fury
.builder()
.withLanguage(Language.JAVA)
.requireClassRegistration(false)
.withScalaOptimizationEnabled(true)
.withRefTracking(true)
.withStringCompressed(true)
.withLongCompressed(true)
.withIntCompressed(true)
.withAsyncCompilation(true)
.buildThreadSafeFury()
def encode(sampleData: SampleData) = {
val raw = fury.serialize(sampleData)
val bos = new ByteArrayOutputStream(raw.length)
val zos = new GZIPOutputStream(bos)
zos.write(raw)
zos.flush()
zos.close()
bos.close()
sleepBetween(500, 1000)
Base64.getEncoder.encodeToString(bos.toByteArray)
}
def decode(encoded: String) =
Try {
val bis = new ByteArrayInputStream(Base64.getDecoder.decode(encoded))
val zis = new GZIPInputStream(bis)
val uncompressed = zis.readAllBytes()
val result = fury.deserialize(uncompressed).asInstanceOf[SampleData]
zis.close()
bis.close()
sleepBetween(500, 1000)
result
}
def sleepBetween(min: Int, max: Int) = {
val sleepTime = ThreadLocalRandom.current().nextInt(min, max)
Thread.sleep(sleepTime.toLong)
}
"fury scala object support" should {
"testNonThreadedSerde" in {
val data = SampleData("single sample", Seq.empty)
val encoded = encode(data)
val decoded = decode(encoded)
println(decoded)
assert(decoded == Success(data))
}
"testNestedCollectionThreadedSerde" in {
import scala.concurrent.duration._
implicit val ec = ExecutionContext.global
val tasks = for (i <- 1 to 1) yield Future {
val data = SampleData(i.toString, Seq.empty)
val encoded = encode(data)
encoded
}
val decodedFuture = for {
f <- Future.sequence(tasks)
} yield for {
encoded <- f
} yield {
val Success(decoded) = decode(encoded)
decoded
}
val result = Await.result(decodedFuture, 20.seconds)
println(result)
assert(result.size == 1)
}
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #1325:
URL: https://github.com/apache/incubator-fury/issues/1325#issuecomment-1894881265
> Is the nested collection issue fixed in the latest snapshot?
Yep, it has been fixed in latest snapshot:
Please note latest snapshot is released under ASF, and the package name has been renamed to `org.apache.fury` instead of `io.fury`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #1325:
URL: https://github.com/apache/incubator-fury/issues/1325#issuecomment-1884117771
Hi @andyczerwonka , thanks for reporting this bug. Can can reproduce it locally?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #1325:
URL: https://github.com/apache/incubator-fury/issues/1325#issuecomment-1894903229
You need to add apache snapshot repo:
```xml
<repositories>
<repository>
<id>apache</id>
<url>https://repository.apache.org/snapshots/</url>
<releases>
<enabled>false</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
<dependency>
<groupId>org.apache.fury</groupId>
<artifactId>fury-core</artifactId>
<version>0.5.0-SNAPSHOT</version>
</dependency>
```
The non-snapshot release may need a month before we release it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #1325:
URL: https://github.com/apache/incubator-fury/issues/1325#issuecomment-1894864270
I guess you didn't use latest snapshot version of Fury. The package name has been changed from `io.fury` to `org.apache.fury`. And nested collection serialization in `case class SampleData(label: String, data: Seq[Seq[Int]])` do be fixed in #1333
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "andyczerwonka (via GitHub)" <gi...@apache.org>.
andyczerwonka closed issue #1325: Create sequential serializer failed
URL: https://github.com/apache/incubator-fury/issues/1325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #1325:
URL: https://github.com/apache/incubator-fury/issues/1325#issuecomment-1886831821
@andyczerwonka I try to fix it in #1333, but I'm not sure whether they are same issue. Could you try this branch in your environment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #1325:
URL: https://github.com/apache/incubator-fury/issues/1325#issuecomment-1886648178
@andyczerwonka Coud you use lastest snapshot jar? Dose it work for you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "andyczerwonka (via GitHub)" <gi...@apache.org>.
andyczerwonka commented on issue #1325:
URL: https://github.com/apache/incubator-fury/issues/1325#issuecomment-1893089719
It's late here, so first thing tomorrow I will try to share the code that reproduces the error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #1325:
URL: https://github.com/apache/incubator-fury/issues/1325#issuecomment-1888754978
Do it work for you ? @andyczerwonka
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "andyczerwonka (via GitHub)" <gi...@apache.org>.
andyczerwonka commented on issue #1325:
URL: https://github.com/apache/incubator-fury/issues/1325#issuecomment-1890835765
I’m still working on reproduction steps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [I] Create sequential serializer failed [incubator-fury]
Posted by "andyczerwonka (via GitHub)" <gi...@apache.org>.
andyczerwonka commented on issue #1325:
URL: https://github.com/apache/incubator-fury/issues/1325#issuecomment-1889308672
I haven’t been able to test it yet, but I will be doing it either today or
tomorrow morning.
On Fri, Jan 12, 2024 at 2:40 AM Shawn Yang ***@***.***> wrote:
> Do it work for you ? @andyczerwonka <https://github.com/andyczerwonka>
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/incubator-fury/issues/1325#issuecomment-1888754978>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AABB5ETZ2ZP25FCUPAB2RO3YOEAKNAVCNFSM6AAAAABBTPJ37KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBYG42TIOJXHA>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org