You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "aromanenko-dev (via GitHub)" <gi...@apache.org> on 2023/02/01 11:17:27 UTC
[GitHub] [beam] aromanenko-dev opened a new issue, #25252: [Task]: [Avro] Don't use Avro in "core"
aromanenko-dev opened a new issue, #25252:
URL: https://github.com/apache/beam/issues/25252
### What needs to happen?
After extracting all Avro-related class into a separate Avro extension and make all other modules, that leverage Avro in some way, depend on this extension, we need to make sure that "core" doesn't rely on Avro anymore.
### Issue Priority
Priority: 2 (default / most normal work should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on issue #25252: [Task]: [Avro] Stop using Avro in "core"
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on issue #25252:
URL: https://github.com/apache/beam/issues/25252#issuecomment-1514849614
Hi @bvolpato !
Actually, this task is suspended by the policy that we have for breaking changes in Beam. This one is a part of epic task #24292 where we did quite a lot of progress by introducing a new Avro extension for Java SDK (it was tested and finally should support different Avro versions), transitioning all other modules, that leverages Avro from `core`, to this extension and deprecating Avro-related classed in `core`. Though, we can't just drop an Avro support from core to avoid instant breaking changes, so we need to wait several minor Beam releases (at least three, iirc) and give the user some time to switching and test this new extension.
Do you think you can also start using `extensions/avro` with a more recent Avro version and in this case I think 1.8.2 will be dropped from classpath and won't bring the other out-dated dependencies?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: [Avro] Stop using Avro in "core" [beam]
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev closed issue #25252: [Task]: [Avro] Stop using Avro in "core"
URL: https://github.com/apache/beam/issues/25252
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] shunping-google commented on issue #25252: [Task]: [Avro] Stop using Avro in "core"
Posted by "shunping-google (via GitHub)" <gi...@apache.org>.
shunping-google commented on issue #25252:
URL: https://github.com/apache/beam/issues/25252#issuecomment-1589928265
Hi @aromanenko-dev, I went over most of the Beam issues and PRs related to the avro dependency (dated back from 2019!) and have now reached this task. Thank you again for leading the entire epic task!
I have two questions to ask if you don't mind.
* I notice we still have avro versions defined in
https://github.com/apache/beam/blob/v2.48.0/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L589-L590. Are we planning to clean them up?
* Do we have a timeline for when it will be released? I have included some upcoming releases and cut-off dates below for your reference:
2.50.0: Aug 9, 2023
2.51.0: Sep 20, 2023
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bvolpato commented on issue #25252: [Task]: [Avro] Stop using Avro in "core"
Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on issue #25252:
URL: https://github.com/apache/beam/issues/25252#issuecomment-1514114573
Hi @aromanenko-dev! Thanks for the effort here!
Do we have any plans to move this forward? We're noticing concerns regarding vulnerabilities/CVEs on jackson-mapper-asl (which comes from Avro 1.8.2) -- so wondering if we should put any effort on upgrading that dependency, or if it's best to proceed here and get rid of it in "core".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bvolpato commented on issue #25252: [Task]: [Avro] Stop using Avro in "core"
Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on issue #25252:
URL: https://github.com/apache/beam/issues/25252#issuecomment-1515616700
Thanks for the response @aromanenko-dev! I'll look into using `extensions/avro`. Although vulnerability scanners will still dislike having Avro 1.8.2 packaged, it sounds like we have a good plan to move forward and eventually get rid of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on issue #25252: [Task]: [Avro] Stop using Avro in "core"
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on issue #25252:
URL: https://github.com/apache/beam/issues/25252#issuecomment-1593191366
@shunping-google
Thanks, I'm happy to help!
1) We will update Avro version to more recent one.
2) We deprecated Avro in "core" in Beam 2.46.0 and we usually remove deprecated stuff, at least, after 3 Beam releases if there are no issues discovered during this period. So, for the moment, I don't see any blockers to remove it in, say, 2.50.0. Anyway, I'll send an email to mailing list before to make sure that there are no other objections for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org