You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "ahmedabu98 (via GitHub)" <gi...@apache.org> on 2024/10/08 19:11:07 UTC
[I] [Bug]: ByteBuddy doesn't do well with Map [beam]
ahmedabu98 opened a new issue, #32701:
URL: https://github.com/apache/beam/issues/32701
### What happened?
I have an @AutoValue class with the `@DefaultSchema(AutoValueSchema.class)`. One of the getters/setters is a `Map<Integer, ByteBuffer>`. When using elements of this class in my pipeline, I get the following error:
<details>
<summary>Error stack trace</summary>
```
java.lang.ClassCastException: class [B cannot be cast to class java.nio.ByteBuffer ([B and java.nio.ByteBuffer are in module java.base of loader 'bootstrap')
at org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Function$SchemaCodeGen$sIwEd1pP.apply(Unknown Source)
at org.apache.beam.sdk.schemas.utils.ByteBuddyUtils$TransformingMap.<init>(ByteBuddyUtils.java:525)
at org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.getTransformingMap(ByteBuddyUtils.java:508)
at org.apache.beam.sdk.io.iceberg.FieldValueGetter$SchemaCodeGen$c6I5wjZP.get(Unknown Source)
at org.apache.beam.sdk.io.iceberg.FieldValueGetter$SchemaCodeGen$c6I5wjZP.get(Unknown Source)
at org.apache.beam.sdk.values.RowWithGetters$1.apply(RowWithGetters.java:82)
at org.apache.beam.sdk.values.RowWithGetters$1.apply(RowWithGetters.java:75)
at java.base/java.util.Map.computeIfAbsent(Map.java:1003)
at org.apache.beam.sdk.values.RowWithGetters.getValue(RowWithGetters.java:73)
at org.apache.beam.sdk.coders.RowCoderGenerator$EncodeInstruction.encodeDelegate(RowCoderGenerator.java:419)
at org.apache.beam.sdk.coders.Coder$ByteBuddy$w90nPOHK.encode(Unknown Source)
at org.apache.beam.sdk.coders.Coder$ByteBuddy$w90nPOHK.encode(Unknown Source)
at org.apache.beam.sdk.schemas.SchemaCoder.encode(SchemaCoder.java:121)
at org.apache.beam.sdk.coders.RowCoderGenerator$EncodeInstruction.encodeDelegate(RowCoderGenerator.java:437)
at org.apache.beam.sdk.coders.Coder$ByteBuddy$1ANxtLai.encode(Unknown Source)
at org.apache.beam.sdk.coders.Coder$ByteBuddy$1ANxtLai.encode(Unknown Source)
at org.apache.beam.sdk.schemas.SchemaCoder.encode(SchemaCoder.java:121)
at org.apache.beam.sdk.coders.Coder.encode(Coder.java:135)
at org.apache.beam.sdk.util.CoderUtils.encodeToSafeStream(CoderUtils.java:86)
at org.apache.beam.sdk.util.CoderUtils.encodeToByteArray(CoderUtils.java:70)
at org.apache.beam.sdk.util.CoderUtils.encodeToByteArray(CoderUtils.java:55)
at org.apache.beam.sdk.util.CoderUtils.clone(CoderUtils.java:168)
at org.apache.beam.runners.direct.CloningBundleFactory$CloningBundle.add(CloningBundleFactory.java:87)
at org.apache.beam.runners.direct.FlattenEvaluatorFactory$FlattenEvaluator.processElement(FlattenEvaluatorFactory.java:75)
at org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:165)
at org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:129)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
```
</details>
I think I was able to track the problem down to this code in ByteBuddy:
https://github.com/apache/beam/blob/e4ab66da4cc716f25f2f17c4183ed810c56ea269/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ByteBuddyUtils.java#L1113-L1125
From what I understand, Beam does some conversions between `ByteBuffer` and `byte[]` (for reasons I am not aware of). In the above code, the Map's key and value element types are not transparent. Instead, they show up as generics. Because of this, we never enter the `if` block, which contains code that applies conversion logic.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: ByteBuddy doesn't do well with Map [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on issue #32701:
URL: https://github.com/apache/beam/issues/32701#issuecomment-2465412795
Not a blocker, removed the milestone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: ByteBuddy doesn't do well with Map [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #32701:
URL: https://github.com/apache/beam/issues/32701#issuecomment-2465582262
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: ByteBuddy doesn't do well with Map [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #32701:
URL: https://github.com/apache/beam/issues/32701#issuecomment-3738392251
This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: ByteBuddy doesn't do well with Map [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #32701:
URL: https://github.com/apache/beam/issues/32701#issuecomment-2437929579
I saw #32757 was reverted in #32887, is this Issue still at resolved stage? For same reason, is #32821 ready to get in?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: ByteBuddy doesn't do well with Map [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on issue #32701:
URL: https://github.com/apache/beam/issues/32701#issuecomment-2438736209
Reopening because we decided to go with the solution in #32081
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: ByteBuddy doesn't do well with Map [beam]
Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax commented on issue #32701:
URL: https://github.com/apache/beam/issues/32701#issuecomment-2417382942
This may fixed now that we are resolving all generic types
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: ByteBuddy doesn't do well with Map [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on issue #32701:
URL: https://github.com/apache/beam/issues/32701#issuecomment-2417508952
This is indeed fixed by both #32757 and #32810. Will close this issue after the latter is merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: ByteBuddy doesn't do well with Map [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #32701:
URL: https://github.com/apache/beam/issues/32701#issuecomment-2465233078
Looks like this got the 2.61.0 milestone added when it was closed/reopened; should this be a release blocker?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: ByteBuddy doesn't do well with Map [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #32701:
URL: https://github.com/apache/beam/issues/32701#issuecomment-3023873467
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: ByteBuddy doesn't do well with Map [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 closed issue #32701: [Bug]: ByteBuddy doesn't do well with Map<T, ByteBuffer>
URL: https://github.com/apache/beam/issues/32701
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: ByteBuddy doesn't do well with Map [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on issue #32701:
URL: https://github.com/apache/beam/issues/32701#issuecomment-2613070931
Is there a plan to roll #32757 forward? It seems we are starting to put bandaids in place because we don't have it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: ByteBuddy doesn't do well with Map [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on issue #32701:
URL: https://github.com/apache/beam/issues/32701#issuecomment-2613176388
@reuvenlax mentioned that the intended fix for #32757 was covered more aptly by #32081. I think he doesn't plan to roll it forward, but it would be helpful to know why the new PR doesn't also address this bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: ByteBuddy doesn't do well with Map [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on issue #32701:
URL: https://github.com/apache/beam/issues/32701#issuecomment-3075009000
Let's keep this open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: ByteBuddy doesn't do well with Map [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #32701:
URL: https://github.com/apache/beam/issues/32701#issuecomment-3649380653
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: ByteBuddy doesn't do well with Map [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #32701: [Bug]: ByteBuddy doesn't do well with Map<T, ByteBuffer>
URL: https://github.com/apache/beam/issues/32701
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org