You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "bvolpato (via GitHub)" <gi...@apache.org> on 2023/11/11 18:11:39 UTC
[PR] Relocate avro dependencies in harness jar [beam]
bvolpato opened a new pull request, #29407:
URL: https://github.com/apache/beam/pull/29407
After https://github.com/apache/beam/pull/27851, user code that depends on versions newer than Avro 1.8.2 are having problems running on Dataflow.
For example in https://github.com/GoogleCloudPlatform/DataflowTemplates, where we moved on to Avro 1.11.3, there were incompatibility errors:
> Caused by: java.io.InvalidClassException: org.apache.avro.specific.SpecificRecordBase; local class incompatible: stream classdesc serialVersionUID = -1463700717714793795, local class serialVersionUID = 189988654766568477
and
> Caused by: java.lang.NoSuchMethodError: 'boolean org.apache.avro.generic.GenericRecord.hasField(java.lang.String)' com.google.cloud.teleport.v2.transforms.FormatDatastreamRecordToJson.getMetadataIsDeleted(FormatDatastreamRecordToJson.java:258) com.google.cloud.teleport.v2.transforms.FormatDatastreamRecordToJson.apply(FormatDatastreamRecordToJson.java:123) com.google.cloud.teleport.v2.transforms.FormatDatastreamRecordToJson.apply(FormatDatastreamRecordToJson.java:51) org.apache.beam.sdk.extensions.avro.io.AvroSource$AvroBlock.readNextRecord(AvroSource.java:610)
The root cause is that Avro classes are now being shipped along with the `/opt/apache/beam/jars/beam-sdks-java-harness.jar`, which wasn't the case before.
Relocating `org.apache.avro` to `org.apache.beam.fn.harness.repackaged.org.apache.avro` will solve the problem and allow users to control their Avro.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Relocate Avro dependencies in Fn Harness JAR [beam]
Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on PR #29407:
URL: https://github.com/apache/beam/pull/29407#issuecomment-1808255635
Closing in favor of https://github.com/apache/beam/pull/29412
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Relocate Avro dependencies in Fn Harness JAR [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29407:
URL: https://github.com/apache/beam/pull/29407#issuecomment-1806893786
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Relocate Avro dependencies in Fn Harness JAR [beam]
Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on PR #29407:
URL: https://github.com/apache/beam/pull/29407#issuecomment-1807251530
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Relocate Avro dependencies in Fn Harness JAR [beam]
Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato closed pull request #29407: Relocate Avro dependencies in Fn Harness JAR
URL: https://github.com/apache/beam/pull/29407
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Relocate avro dependencies in harness jar [beam]
Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on PR #29407:
URL: https://github.com/apache/beam/pull/29407#issuecomment-1806881952
@aromanenko-dev, was the inclusion of Avro intentional in the harness JAR?
@damccorm @Abacn I think this is a blocker for 2.52.0, as there will be pipeline failures if someone depends on anything after Avro 1.8.2 (which was the case for Dataflow Templates, and would require code refactoring).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Relocate Avro dependencies in Fn Harness JAR [beam]
Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on PR #29407:
URL: https://github.com/apache/beam/pull/29407#issuecomment-1806883664
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Relocate Avro dependencies in Fn Harness JAR [beam]
Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on PR #29407:
URL: https://github.com/apache/beam/pull/29407#issuecomment-1808189912
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Relocate Avro dependencies in Fn Harness JAR [beam]
Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on PR #29407:
URL: https://github.com/apache/beam/pull/29407#issuecomment-1806883803
Java PreCommit is getting:
```
13:19:07 > Configure project :sdks:java:container:java21
13:19:07 docker-pull-licenses set, creating go-licenses
13:19:17 System Go installation: /snap/bin/go is go version go1.16.15 linux/amd64; Preparing to use /home/jenkins/go/bin/go1.21.4
13:19:17 /home/jenkins/go/pkg/mod/golang.org/dl@v0.0.0-20231107201224-8165dcd93058/go1.21.4/main.go:20:8: //go:build comment without // +build comment
13:19:17
13:19:17 FAILURE: Build failed with an exception.
13:19:17
13:19:17 * What went wrong:
13:19:17 Could not determine the dependencies of task ':sdks:java:container:goBuild'.
13:19:17 > Could not create task ':sdks:java:container:goPrepare'.
13:19:17 > Process 'command 'sh'' finished with non-zero exit value 1
```
Which I think is unrelated to the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Relocate Avro dependencies in Fn Harness JAR [beam]
Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on PR #29407:
URL: https://github.com/apache/beam/pull/29407#issuecomment-1806883118
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org