You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Abacn (via GitHub)" <gi...@apache.org> on 2024/08/15 20:38:45 UTC
[PR] More Java11 test fix [beam]
Abacn opened a new pull request, #32209:
URL: https://github.com/apache/beam/pull/32209
* Fix Java IOs/HCatalogIO PreCommit typo
* Run Spark3 Version PreCommit on Java8
* Bump Samza version to support Java8 / Java11 and also M1 Mac
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] More Java11 test fix [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm merged PR #32209:
URL: https://github.com/apache/beam/pull/32209
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] More Java11 test fix [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #32209:
URL: https://github.com/apache/beam/pull/32209#discussion_r1719034105
##########
runners/samza/build.gradle:
##########
@@ -57,12 +58,12 @@ dependencies {
runtimeOnly "org.rocksdb:rocksdbjni:6.15.2"
runtimeOnly "org.scala-lang:scala-library:2.11.8"
implementation "org.apache.samza:samza-api:$samza_version"
- implementation "org.apache.samza:samza-core_2.11:$samza_version"
- runtimeOnly "org.apache.samza:samza-kafka_2.11:$samza_version"
- runtimeOnly "org.apache.samza:samza-kv_2.11:$samza_version"
- implementation "org.apache.samza:samza-kv-rocksdb_2.11:$samza_version"
- implementation "org.apache.samza:samza-kv-inmemory_2.11:$samza_version"
- implementation "org.apache.samza:samza-yarn_2.11:$samza_version"
+ implementation "org.apache.samza:samza-core_$samza_scala:$samza_version"
Review Comment:
Revert Samza changes because actually the version bump is non-trivial. Python Validates Runner Samza test now failing with various reasons. First it is
```
java.lang.UnsupportedOperationException: SamzaImpulseSystem doesn't support producing
at org.apache.beam.runners.samza.translation.SamzaImpulseSystemFactory.getProducer(SamzaImpulseSystemFactory.java:59)
at org.apache.samza.system.SystemFactory.getProducer(SystemFactory.java:66)
at org.apache.samza.container.SamzaContainer$.$anonfun$apply$27(SamzaContainer.scala:248)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
...
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at org.apache.samza.container.SamzaContainer$.apply(SamzaContainer.scala:245)
at org.apache.samza.container.SamzaContainer.apply(SamzaContainer.scala)
at org.apache.samza.processor.StreamProcessor.createSamzaContainer(StreamProcessor.java:412)
```
after change `SamzaImpulseSystemFactory.getProducer` to return null (as other getProducer), tests then fail with
```
024-08-15T21:18:38.5903463Z [grpc-default-executor-1] INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService - getProcessBundleDescriptor request with id 18-4
Traceback (most recent call last):
File "/runner/_work/beam/beam/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/apache_beam/runners/common.py", line 1495, in process
return self.do_fn_invoker.invoke_process(windowed_value)
File "/runner/_work/beam/beam/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/apache_beam/runners/common.py", line 687, in invoke_process
self.output_handler.handle_process_outputs(
File "/runner/_work/beam/beam/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/apache_beam/runners/common.py", line 1681, in handle_process_outputs
self._write_value_to_tag(tag, windowed_value, watermark_estimator)
...
File "/runner/_work/beam/beam/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/apache_beam/coders/coder_impl.py", line 1259, in get_estimated_size_and_observables
self._elem_coder.get_estimated_size_and_observables(
File "/runner/_work/beam/beam/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/apache_beam/coders/coder_impl.py", line 291, in get_estimated_size_and_observables
return self.estimate_size(value, nested), []
File "/runner/_work/beam/beam/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/apache_beam/coders/coder_impl.py", line 282, in estimate_size
return self._get_nested_size(self._size_estimator(value), nested)
File "/runner/_work/beam/beam/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/apache_beam/coders/coders.py", line 209, in estimate_size
return len(self.encode(value))
File "/runner/_work/beam/beam/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/apache_beam/coders/coders.py", line 407, in encode
return value.encode('utf-8')
AttributeError: 'numpy.int64' object has no attribute 'encode'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] More Java11 test fix [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32209:
URL: https://github.com/apache/beam/pull/32209#issuecomment-2292790348
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] More Java11 test fix [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32209:
URL: https://github.com/apache/beam/pull/32209#issuecomment-2293597813
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] More Java11 test fix [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #32209:
URL: https://github.com/apache/beam/pull/32209#issuecomment-2292786485
Involved tests:
PostCommit Java ValidatesRunner Samza: https://github.com/apache/beam/actions/runs/10411089794
PostCommit Python ValidatesRunner Samza: https://github.com/apache/beam/actions/runs/10411090450
PostCommit XVR Samza: https://github.com/apache/beam/actions/runs/10411091254
PreCommit Java Spark3 Versions: https://github.com/apache/beam/actions/runs/10411096758
PreCommit Java HCatalog IO Direct: https://github.com/apache/beam/actions/runs/10411100175 show 18 tests
PostCommit Java Hadoop Versions: https://github.com/apache/beam/actions/runs/10411102961
PostCommit Java PVR Samza: https://github.com/apache/beam/actions/runs/10411552878
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] More Java11 test fix [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #32209:
URL: https://github.com/apache/beam/pull/32209#issuecomment-2293596698
R: @lostluck
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] More Java11 test fix [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #32209:
URL: https://github.com/apache/beam/pull/32209#discussion_r1718967608
##########
runners/samza/build.gradle:
##########
@@ -57,12 +58,12 @@ dependencies {
runtimeOnly "org.rocksdb:rocksdbjni:6.15.2"
runtimeOnly "org.scala-lang:scala-library:2.11.8"
implementation "org.apache.samza:samza-api:$samza_version"
- implementation "org.apache.samza:samza-core_2.11:$samza_version"
- runtimeOnly "org.apache.samza:samza-kafka_2.11:$samza_version"
- runtimeOnly "org.apache.samza:samza-kv_2.11:$samza_version"
- implementation "org.apache.samza:samza-kv-rocksdb_2.11:$samza_version"
- implementation "org.apache.samza:samza-kv-inmemory_2.11:$samza_version"
- implementation "org.apache.samza:samza-yarn_2.11:$samza_version"
+ implementation "org.apache.samza:samza-core_$samza_scala:$samza_version"
Review Comment:
this may affect users, however given that samza 1.6.0 is 3 years old I would consider this more like a long overdue update.
tested validatesRunner still passes on both Java8 and 11
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] More Java11 test fix [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #32209:
URL: https://github.com/apache/beam/pull/32209#issuecomment-2293595685
R: @damccorm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org