You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/03/31 22:26:12 UTC
[GitHub] [beam] blcksrx opened a new pull request #11279: [BEAM-9653] remove
_AvroSource in favor of using _FastAvroSource
blcksrx opened a new pull request #11279: [BEAM-9653] remove _AvroSource in favor of using _FastAvroSource
URL: https://github.com/apache/beam/pull/11279
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [x] Update `CHANGES.md` with noteworthy changes.
- [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on a change in pull request #11279:
[BEAM-9653] remove _AvroSource in favor of using _FastAvroSource
Posted by GitBox <gi...@apache.org>.
mxm commented on a change in pull request #11279: [BEAM-9653] remove _AvroSource in favor of using _FastAvroSource
URL: https://github.com/apache/beam/pull/11279#discussion_r401445445
##########
File path: sdks/python/apache_beam/io/avroio.py
##########
@@ -451,7 +404,6 @@ class _FastAvroSource(filebasedsource.FileBasedSource):
common to all file-based sources such as file-pattern expansion and splitting
into bundles for parallel processing.
- TODO: remove ``_AvroSource`` in favor of using ``_FastAvroSource``
everywhere once it has been more widely tested
Review comment:
Perhaps @chamikaramj or @aaltay have an opinion on how mature the fast Avro IO is?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay commented on a change in pull request #11279:
[BEAM-9653] remove _AvroSource in favor of using _FastAvroSource
Posted by GitBox <gi...@apache.org>.
aaltay commented on a change in pull request #11279: [BEAM-9653] remove _AvroSource in favor of using _FastAvroSource
URL: https://github.com/apache/beam/pull/11279#discussion_r401898368
##########
File path: sdks/python/apache_beam/io/avroio.py
##########
@@ -451,7 +404,6 @@ class _FastAvroSource(filebasedsource.FileBasedSource):
common to all file-based sources such as file-pattern expansion and splitting
into bundles for parallel processing.
- TODO: remove ``_AvroSource`` in favor of using ``_FastAvroSource``
everywhere once it has been more widely tested
Review comment:
I believe most of the users are still on the regular avro. If I remember correctly, for python3 the default is fastavro, for python2 the default is avro with a flag to change it.
@chamikaramj - what do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] blcksrx commented on issue #11279: [BEAM-9653] remove
_AvroSource in favor of using _FastAvroSource
Posted by GitBox <gi...@apache.org>.
blcksrx commented on issue #11279: [BEAM-9653] remove _AvroSource in favor of using _FastAvroSource
URL: https://github.com/apache/beam/pull/11279#issuecomment-606917166
R: @mxm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on a change in pull request #11279:
[BEAM-9653] remove _AvroSource in favor of using _FastAvroSource
Posted by GitBox <gi...@apache.org>.
mxm commented on a change in pull request #11279: [BEAM-9653] remove _AvroSource in favor of using _FastAvroSource
URL: https://github.com/apache/beam/pull/11279#discussion_r401441983
##########
File path: sdks/python/apache_beam/io/avroio.py
##########
@@ -451,7 +404,6 @@ class _FastAvroSource(filebasedsource.FileBasedSource):
common to all file-based sources such as file-pattern expansion and splitting
into bundles for parallel processing.
- TODO: remove ``_AvroSource`` in favor of using ``_FastAvroSource``
everywhere once it has been more widely tested
Review comment:
```suggestion
```
has it been more widely tested? :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11279: [BEAM-9653] remove
_AvroSource in favor of using _FastAvroSource
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11279: [BEAM-9653] remove _AvroSource in favor of using _FastAvroSource
URL: https://github.com/apache/beam/pull/11279#issuecomment-607493149
Yes, FastAvro is already the default for Python 3. I suggest holding off this change till Python 2 is fully decommissioned.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services