You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Abacn (via GitHub)" <gi...@apache.org> on 2024/04/25 18:08:36 UTC
[PR] Revert "python sdk: fix several bugs regarding avto <-> beam schema conversion" [beam]
Abacn opened a new pull request, #31107:
URL: https://github.com/apache/beam/pull/31107
Reverts apache/beam#30770
the newly added test `TestFastAvro::test_avro_schema_to_beam_schema_with_nullable_atomic_fields` failing XVR test:
https://github.com/apache/beam/pull/30770#issuecomment-2077343964
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "python sdk: fix several bugs regarding avto <-> beam schema conversion" [beam]
Posted by "benkonz (via GitHub)" <gi...@apache.org>.
benkonz commented on PR #31107:
URL: https://github.com/apache/beam/pull/31107#issuecomment-2077961604
yeah, for some reason, in the flink runner specifically, after the `SqlTransform("SELECT * FROM PCOLLECTION")` in the `test_avro_schema_to_beam_schema_with_nullable_atomic_fields` test, there aren't any records outputted. I verified that the test works locally when I run `pytest -v apache_beam/io/avroio_test.py::TestFastAvro::test_avro_schema_to_beam_schema_w
ith_nullable_atomic_fields`, but with flink it just outputs an empty PCollection, still looking into this, but feel free to merge the revert. I'll post a new PR once I can get the test working, also open to feedback if anyone knows what's causing this behavior!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "python sdk: fix several bugs regarding avto <-> beam schema conversion" [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31107:
URL: https://github.com/apache/beam/pull/31107#issuecomment-2077877180
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [test only] Test PostCommit runs for #31116 [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed pull request #31107: [test only] Test PostCommit runs for #31116
URL: https://github.com/apache/beam/pull/31107
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "python sdk: fix several bugs regarding avto <-> beam schema conversion" [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31107:
URL: https://github.com/apache/beam/pull/31107#issuecomment-2077874345
[The Workflow run](https://github.com/apache/beam/actions/runs/8837344247) is cancelling this PR. It is an earlier duplicate of 2083803 run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "python sdk: fix several bugs regarding avto <-> beam schema conversion" [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #31107:
URL: https://github.com/apache/beam/pull/31107#issuecomment-2077870700
R: @benkonz @ahmedabu98
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "python sdk: fix several bugs regarding avto <-> beam schema conversion" [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed pull request #31107: Revert "python sdk: fix several bugs regarding avto <-> beam schema conversion"
URL: https://github.com/apache/beam/pull/31107
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "python sdk: fix several bugs regarding avto <-> beam schema conversion" [beam]
Posted by "benkonz (via GitHub)" <gi...@apache.org>.
benkonz commented on PR #31107:
URL: https://github.com/apache/beam/pull/31107#issuecomment-2079426790
@Abacn I found the cause of the issue in the test. The Flink runner wasn't able to access the local file-system with `avroio.ReadFromAvro(...)` and `avroio.WriteToAvro(...)`. I refactored the test in https://github.com/apache/beam/pull/31116 to create a PCollection from memory, rather than using temp directories, and verified that the previously failing gradle command works (`./gradlew :runners:flink:1.15:job-server:validatesCrossLanguageRunnerPythonUsingSql`). Could you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "python sdk: fix several bugs regarding avto <-> beam schema conversion" [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31107:
URL: https://github.com/apache/beam/pull/31107#issuecomment-2077874473
[The Workflow run](https://github.com/apache/beam/actions/runs/8837344242) is cancelling this PR. It is an earlier duplicate of 1729654 run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "python sdk: fix several bugs regarding avto <-> beam schema conversion" [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #31107:
URL: https://github.com/apache/beam/pull/31107#issuecomment-2079646424
@benkonz thanks for the fix. The test wasn't only failing on Flink runner, it's also failing on direct runner (and all runners)
See https://github.com/apache/beam/actions/workflows/beam_PostCommit_XVR_Spark3.yml?query=event%3Aschedule https://github.com/apache/beam/actions/workflows/beam_PostCommit_XVR_Samza.yml?query=event%3Aschedule https://github.com/apache/beam/actions/runs/8847577228/job/24295682944
Let me trigger tests for #31116 to see if the test is fixed on all runners
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "python sdk: fix several bugs regarding avto <-> beam schema conversion" [beam]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #31107:
URL: https://github.com/apache/beam/pull/31107#issuecomment-2078013846
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/31107?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `25.00000%` with `6 lines` in your changes are missing coverage. Please review.
> Project coverage is 71.01%. Comparing base [(`2db9b80`)](https://app.codecov.io/gh/apache/beam/commit/2db9b8097b79618c2d3d8c76c45a2fc979196ab3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`0dfdcc2`)](https://app.codecov.io/gh/apache/beam/pull/31107?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 5 commits behind head on master.
| [Files](https://app.codecov.io/gh/apache/beam/pull/31107?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [sdks/python/apache\_beam/io/avroio.py](https://app.codecov.io/gh/apache/beam/pull/31107?src=pr&el=tree&filepath=sdks%2Fpython%2Fapache_beam%2Fio%2Favroio.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXZyb2lvLnB5) | 25.00% | [6 Missing :warning: ](https://app.codecov.io/gh/apache/beam/pull/31107?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #31107 +/- ##
=============================================
+ Coverage 60.36% 71.01% +10.64%
Complexity 2980 2980
=============================================
Files 659 1064 +405
Lines 66058 133212 +67154
Branches 3231 3231
=============================================
+ Hits 39875 94595 +54720
- Misses 23083 35517 +12434
Partials 3100 3100
```
| [Flag](https://app.codecov.io/gh/apache/beam/pull/31107/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [python](https://app.codecov.io/gh/apache/beam/pull/31107/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.48% <25.00%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/beam/pull/31107?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "python sdk: fix several bugs regarding avto <-> beam schema conversion" [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #31107:
URL: https://github.com/apache/beam/pull/31107#issuecomment-2077870168
#30770 was several bug fix together. We can add them back one at a time for those did not have issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org