You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Abacn (via GitHub)" <gi...@apache.org> on 2023/02/23 01:58:04 UTC
[GitHub] [beam] Abacn opened a new issue, #25601: [Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice
Abacn opened a new issue, #25601:
URL: https://github.com/apache/beam/issues/25601
### What happened?
Python PostCommit jdbcio_xlang_it_test failing. Jenkins log does not show much message, but running locally the actual error shows:
```
ERROR:apache_beam.utils.subprocess_server:Starting job service with ['java', '-jar', '/Users/yathu/.apache_beam/cache/jars/composite-jars/9b6e2cb01bd723cbd87a5e71462f25664f165993697afc98e1c64c34bf814f98.jar', '50709', '--filesToStage=/Users/yathu/dev/virtualenv/py38beam/lib/sdks/java/extensions/schemaio-expansion-service/build/libs/beam-sdks-java-extensions-schemaio-expansion-service-2.47.0-SNAPSHOT.jar,/Users/yathu/.apache_beam/cache/jars/postgresql-42.2.16.jar']
ERROR:apache_beam.utils.subprocess_server:Error bringing up service
Traceback (most recent call last):
File "/Users/yathu/dev/virtualenv/py38beam/lib/python3.8/site-packages/apache_beam/utils/subprocess_server.py", line 88, in start
raise RuntimeError(
RuntimeError: Service failed to start up with error 1
Traceback (most recent call last):
```
This is because there are two `@AutoService AvroSchemaIOProvider` classes so the schemaio expansion service fails to start.
The fix could either be similar to the workaround here: https://github.com/apache/beam/blob/011296c14659f80c8ecbeefda79ecc3f1113bd95/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/io/Providers.java#L45
Or simply remove AvroSchemaIOProvider in core.
### Issue Failure
Failure: Test is continually failing
### Issue Priority
Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)
### Issue Components
- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on issue #25601: [Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on issue #25601:
URL: https://github.com/apache/beam/issues/25601#issuecomment-1442375642
@damccorm Cherry-pick is ok, I just asked what is easier for you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on issue #25601: [Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on issue #25601:
URL: https://github.com/apache/beam/issues/25601#issuecomment-1443548177
Thanks @damccorm !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on issue #25601: [Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on issue #25601:
URL: https://github.com/apache/beam/issues/25601#issuecomment-1442192686
@mosche I added logging for providers registering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on issue #25601: [Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on issue #25601:
URL: https://github.com/apache/beam/issues/25601#issuecomment-1442381009
Ok, so, let's wait for tests passed and if it's ok, I'll merge it. Then, we can cherry-pick it to release branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on issue #25601: [Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice
Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on issue #25601:
URL: https://github.com/apache/beam/issues/25601#issuecomment-1441978244
@Abacn @aromanenko-dev I was just looking at this as well. I agree with @aromanenko-dev that removing anything from core might be problematic. Though, also, it turns out that `org.apache.beam.sdk.extensions.avro.io.AvroSchemaIOProvider` isn't even part of the uber expansion jar, there's a separate one build for testing (`:sdks:java:testing:expansion-service`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on issue #25601: [Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice
Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on issue #25601:
URL: https://github.com/apache/beam/issues/25601#issuecomment-1441998307
Disussing with @aromanenko-dev, fixing the duplicate service should be trivial by allowing the avro extension to override any provider with `avro` as identifier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on issue #25601: [Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on issue #25601:
URL: https://github.com/apache/beam/issues/25601#issuecomment-1442360487
@damccorm I see that you already cut a branch for release. If fix from #25611 works and there are no other issues, could you cut it again to include this fix and not cherry-pick?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on issue #25601: [Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #25601:
URL: https://github.com/apache/beam/issues/25601#issuecomment-1442366418
AFAICT none of the changes since the cut should prevent an easy cherry-pick
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on issue #25601: [Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #25601:
URL: https://github.com/apache/beam/issues/25601#issuecomment-1442173483
Note that schemaio expansion-service does not depend on avro extension, but the uber jar includes it. I checked that unzip the jar I see class of sdk/extensions/avro and class files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on issue #25601: [Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #25601:
URL: https://github.com/apache/beam/issues/25601#issuecomment-1442376479
Ok cool - thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm closed issue #25601: [Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm closed issue #25601: [Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice
URL: https://github.com/apache/beam/issues/25601
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on issue #25601: [Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice
Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on issue #25601:
URL: https://github.com/apache/beam/issues/25601#issuecomment-1442011123
@aromanenko-dev I think there's two more follow ups required:
- Log deprecation warnings when using the Avro related classes from core are used. `AvroSchemaIOProvider` is a good example where users might never notice such a deprecation as there's no direct dependency on the class.
- Investigate / Discuss if the schemaio expansion-service should depend on the avro extension, currently it doesn't. Thinking ahead, the moment the deprecated `AvroSchemaIOProvider` is removed from `core` any x-lang pipeline using it would break.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on issue #25601: [Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice
Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on issue #25601:
URL: https://github.com/apache/beam/issues/25601#issuecomment-1442132121
> Good point about logging. Though, since these classes are already annotated with @Deprecated annotation. Shouldn't they be automatically added to build log? Or you are talking mostly about runtime logs?
Of course I'm talking about runtime ... Users won't ever directly interact with AvroSchemaIOProvider. Because of that they are not going to notice a deprecation warning at build time.
> IIUC, AvroSchemaIOProvider should be loaded only if it's used in one of it's dependent modules and it doesn't require direct extensions/avro dependency?
This isn't as simple ... AvroSchemaIOProvider exposes Avro sources in Beam SQL / xlang in a rather dynamic way to the user.
And currently this works as Avro is still part of core. Once removed, the behavior will suddenly break for users that have successfully used it before unless the Avro extension is always added to the expansion service jar :/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on issue #25601: [Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #25601:
URL: https://github.com/apache/beam/issues/25601#issuecomment-1442365027
Is there a reason not to cherry pick? Recutting isn't supported by the current scripts and might lead to issues
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on issue #25601: [Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #25601:
URL: https://github.com/apache/beam/issues/25601#issuecomment-1442375469
If its just a matter of making time to do the CP promptly, I'm happy to help out there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on issue #25601: [Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on issue #25601:
URL: https://github.com/apache/beam/issues/25601#issuecomment-1442113756
@mosche
* Good point about logging. Though, since these classes are already annotated with `@Deprecated` annotation. Shouldn't they be automatically added to build log? Or you are talking mostly about runtime logs?
* IIUC, `AvroSchemaIOProvider` should be loaded only if it's used in one of it's dependent modules and it doesn't require direct `extensions/avro` dependency?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on issue #25601: [Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice
Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on issue #25601:
URL: https://github.com/apache/beam/issues/25601#issuecomment-1442249214
thx @Abacn, i checked a jar of an earlier version before the extension existed 🤦
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on issue #25601: [Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #25601:
URL: https://github.com/apache/beam/issues/25601#issuecomment-1441125846
CC: @aromanenko-dev
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on issue #25601: [Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #25601:
URL: https://github.com/apache/beam/issues/25601#issuecomment-1441971809
@aromanenko-dev this is currently the only known release blocker - do you think you'll be able to get a fix in quickly, or should we consider temporarily reverting https://github.com/apache/beam/pull/25534 (since if I understand correctly, that is the cause of the failures)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on issue #25601: [Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #25601:
URL: https://github.com/apache/beam/issues/25601#issuecomment-1441985620
another quick fix for 2.46.0 would be remove the SchemaIOProvider AutoService in extension-avro, or even just comment out the `@AutoService` annotation. This would not introduce any breaking change and also let expansion service working properly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on issue #25601: [Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on issue #25601:
URL: https://github.com/apache/beam/issues/25601#issuecomment-1441998372
@damccorm @Abacn Before any reverts or deletes, I'd like to try a fix proposed by @mosche if it will work. Let me create a PR for that and check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on issue #25601: [Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on issue #25601:
URL: https://github.com/apache/beam/issues/25601#issuecomment-1442181859
I think this is why we have this issue actually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on issue #25601: [Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #25601:
URL: https://github.com/apache/beam/issues/25601#issuecomment-1442681030
I merged the PR and put up a cherry-pick, could someone please approve?
https://github.com/apache/beam/pull/25618
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org