You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Polber (via GitHub)" <gi...@apache.org> on 2024/03/27 22:38:44 UTC
[I] [Bug]: Beam YAML WriteToJson fails on Beam 2.55 [beam]
Polber opened a new issue, #30776:
URL: https://github.com/apache/beam/issues/30776
### What happened?
Since Beam 2.55 was released, the Cross-language transform for JsonWrite does not work on Beam YAML (or Beam Python when using ExternalTransform)
A change to https://github.com/apache/beam/blob/master/sdks/java/io/json/build.gradle removed a dependency on everit -
`implementation library.java.everit_json_schema`
PR: https://github.com/apache/beam/pull/29924
This also removed the library from being packaged into the beam-sdks-java-extensions-sql-expansion-service-2.55.0.jar, (sdks:java:extensions:schemaio-expansion-service:shadowJar)
So, when using xlang JsonWrite - https://github.com/apache/beam/blob/master/sdks/java/io/json/src/main/java/org/apache/beam/sdk/io/json/providers/JsonWriteTransformProvider.java
the expansion will fail complaining about `java.lang.ClassNotFoundException: org.everit.json.schema.Schema$Builder`
### Issue Priority
Priority: 1 (data loss / total loss of function)
### Issue Components
- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [X] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Beam YAML WriteToJson fails on Beam 2.55 [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on issue #30776:
URL: https://github.com/apache/beam/issues/30776#issuecomment-2024198771
I notice that [sdks/java/extensions/schemaio-expansion-service/build.gradle is](https://github.com/apache/beam/blob/master/sdks/java/extensions/schemaio-expansion-service/build.gradle) has suppressed all dependency configuration warnings.
I presume this is because it does not directly depend on any of those things, but wants them in the uber jar. I have to believe there is a more principled way of achieving that, for example a `runtime` scope or something to do with shadow jar configuration?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Beam YAML WriteToJson fails on Beam 2.55 [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on issue #30776:
URL: https://github.com/apache/beam/issues/30776#issuecomment-2024194088
Seems like there's an issue in how dependencies are specified. Searched for uses of it: https://github.com/search?q=repo%3Aapache%2Fbeam+org.everit+language%3AJava&type=code
It looks like the core SDK depends on it but requires users to add it as a dependency: https://github.com/apache/beam/blob/62200da9df8fee2835ab2b7731c33bd8e243c502/sdks/java/core/build.gradle#L104
You were getting lucky that it was also added as a firm dependency, despite sdks/java/io/json/ not actually depending on it. I bet the reason I removed it was that I got an IWYU error. There are two good fixes: (1) add a dep directly at the point of bundling the expansion service jar or (2) just add the dep to the core SDK. And I guess there is fix (3) which is cludge to check if it is present and don't validate if it is not available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Beam YAML WriteToJson fails on Beam 2.55 [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #30776: [Bug]: Beam YAML WriteToJson fails on Beam 2.55
URL: https://github.com/apache/beam/issues/30776
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org