You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/03/26 01:09:41 UTC
[GitHub] [beam] udim opened a new pull request #11232: [BEAM-8280] Document
Python 3 annotations support
udim opened a new pull request #11232: [BEAM-8280] Document Python 3 annotations support
URL: https://github.com/apache/beam/pull/11232
Also updates snippets, but since snippets are taken from Beam's Github
repo from master please imagine they look like they do in this PR.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] udim commented on a change in pull request #11232:
[BEAM-8280] Document Python 3 annotations support
Posted by GitBox <gi...@apache.org>.
udim commented on a change in pull request #11232: [BEAM-8280] Document Python 3 annotations support
URL: https://github.com/apache/beam/pull/11232#discussion_r398765090
##########
File path: website/src/documentation/sdks/python-type-safety.md
##########
@@ -64,22 +84,75 @@ To specify type hints as properties of a `DoFn` or `PTransform`, use the decorat
The following code declares an `int` type hint on `FilterEvensDoFn`, using the decorator `@with_input_types()`.
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:type_hints_do_fn %}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:type_hints_do_fn %}```
Decorators receive an arbitrary number of positional and/or keyword arguments, typically interpreted in the context of the function they're wrapping. Generally the first argument is a type hint for the main input, and additional arguments are type hints for side inputs.
-### Defining Generic Types
+### Declaring Type Hints Using Annotations
+
+_New in version 2.21.0._
Review comment:
Yes: https://github.com/apache/beam/pull/10717/files#diff-8b1c3fd0d4a6765c16dfd18509182f9d
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] udim merged pull request #11232: [BEAM-8280] Document Python
3 annotations support
Posted by GitBox <gi...@apache.org>.
udim merged pull request #11232: [BEAM-8280] Document Python 3 annotations support
URL: https://github.com/apache/beam/pull/11232
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] udim commented on issue #11232: [BEAM-8280] Document Python
3 annotations support
Posted by GitBox <gi...@apache.org>.
udim commented on issue #11232: [BEAM-8280] Document Python 3 annotations support
URL: https://github.com/apache/beam/pull/11232#issuecomment-604170679
R: @robertwb @aaltay
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] udim commented on a change in pull request #11232:
[BEAM-8280] Document Python 3 annotations support
Posted by GitBox <gi...@apache.org>.
udim commented on a change in pull request #11232: [BEAM-8280] Document Python 3 annotations support
URL: https://github.com/apache/beam/pull/11232#discussion_r405715326
##########
File path: website/src/documentation/sdks/python-type-safety.md
##########
@@ -23,38 +23,58 @@ Python is a dynamically-typed language with no static type checking. Because of
The Apache Beam SDK for Python uses **type hints** during pipeline construction and runtime to try to emulate the correctness guarantees achieved by true static typing. Additionally, using type hints lays some groundwork that allows the backend service to perform efficient type deduction and registration of `Coder` objects.
-Python version 3.5 introduces a module called **typing** to provide hints for type validators in the language. The Beam SDK for Python, based on Python version 2.7, implements a subset of [PEP 484](https://www.python.org/dev/peps/pep-0484/) and aims to follow it as closely as possible in its own typehints module.
+Python version 3.5 introduces a module called **typing** to provide hints for type validators in the language.
+The Beam SDK for Python implements a subset of [PEP 484](https://www.python.org/dev/peps/pep-0484/) and aims to follow it as closely as possible in its own typehints module.
+
+These flags control Beam type safety:
+- `--no_pipeline_type_check`
+
+ Disables type checking during pipeline construction.
+ Default is to perform these checks.
+- `--runtime_type_check`
+
+ Enables runtime type checking of every element.
+ This may affect pipeline performance, so the default is to skip these checks.
## Benefits of Type Hints
-The Beam SDK for Python includes some automatic type checking: for example, some `PTransform`s, such as `Create` and simple `ParDo` transforms, attempt to deduce their output type given their input. However, the Beam cannot infer types in all cases. Therefore, the recommendation is that you declare type hints to aid you in performing your own type checks if necessary.
+The Beam SDK for Python includes some automatic type checking: for example, some `PTransforms`, such as `Create` and simple `ParDo` transforms, attempt to deduce their output type given their input. However, the Beam cannot infer types in all cases. Therefore, the recommendation is that you declare type hints to aid you in performing your own type checks if necessary.
-When you use type hints, the runner raises exceptions during pipeline construction time, rather than runtime. For example, the runner generates an exception if it detects that your pipeline applies mismatched `PTransforms` (where the expected outputs of one transform do not match the expected inputs of the following transform). These exceptions are raised at pipeline construction time, regardless of where your pipeline will execute. Introducing type hints for the `PTransform`s you define allows you to catch potential bugs up front in the local runner, rather than after minutes of execution into a deep, complex pipeline.
+When you use type hints, the runner raises exceptions during pipeline construction time, rather than runtime. For example, the runner generates an exception if it detects that your pipeline applies mismatched `PTransforms` (where the expected outputs of one transform do not match the expected inputs of the following transform). These exceptions are raised at pipeline construction time, regardless of where your pipeline will execute. Introducing type hints for the `PTransforms` you define allows you to catch potential bugs up front in the local runner, rather than after minutes of execution into a deep, complex pipeline.
Review comment:
Done. Also reworded this paragraph a bit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] robertwb commented on a change in pull request #11232:
[BEAM-8280] Document Python 3 annotations support
Posted by GitBox <gi...@apache.org>.
robertwb commented on a change in pull request #11232: [BEAM-8280] Document Python 3 annotations support
URL: https://github.com/apache/beam/pull/11232#discussion_r399554700
##########
File path: website/src/documentation/sdks/python-type-safety.md
##########
@@ -64,22 +84,75 @@ To specify type hints as properties of a `DoFn` or `PTransform`, use the decorat
The following code declares an `int` type hint on `FilterEvensDoFn`, using the decorator `@with_input_types()`.
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:type_hints_do_fn %}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:type_hints_do_fn %}```
Decorators receive an arbitrary number of positional and/or keyword arguments, typically interpreted in the context of the function they're wrapping. Generally the first argument is a type hint for the main input, and additional arguments are type hints for side inputs.
-### Defining Generic Types
+### Declaring Type Hints Using Annotations
Review comment:
"Declaring Type Hints Using Type Annotations" (might be a bit redundant, but is clearer).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] robertwb commented on a change in pull request #11232:
[BEAM-8280] Document Python 3 annotations support
Posted by GitBox <gi...@apache.org>.
robertwb commented on a change in pull request #11232: [BEAM-8280] Document Python 3 annotations support
URL: https://github.com/apache/beam/pull/11232#discussion_r399554554
##########
File path: website/src/documentation/sdks/python-type-safety.md
##########
@@ -64,22 +84,75 @@ To specify type hints as properties of a `DoFn` or `PTransform`, use the decorat
The following code declares an `int` type hint on `FilterEvensDoFn`, using the decorator `@with_input_types()`.
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:type_hints_do_fn %}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:type_hints_do_fn %}```
Decorators receive an arbitrary number of positional and/or keyword arguments, typically interpreted in the context of the function they're wrapping. Generally the first argument is a type hint for the main input, and additional arguments are type hints for side inputs.
-### Defining Generic Types
+### Declaring Type Hints Using Annotations
Review comment:
This should be the preferred way; put it first.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] udim commented on a change in pull request #11232:
[BEAM-8280] Document Python 3 annotations support
Posted by GitBox <gi...@apache.org>.
udim commented on a change in pull request #11232: [BEAM-8280] Document Python 3 annotations support
URL: https://github.com/apache/beam/pull/11232#discussion_r405723061
##########
File path: website/src/documentation/sdks/python-type-safety.md
##########
@@ -64,22 +84,75 @@ To specify type hints as properties of a `DoFn` or `PTransform`, use the decorat
The following code declares an `int` type hint on `FilterEvensDoFn`, using the decorator `@with_input_types()`.
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:type_hints_do_fn %}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:type_hints_do_fn %}```
Decorators receive an arbitrary number of positional and/or keyword arguments, typically interpreted in the context of the function they're wrapping. Generally the first argument is a type hint for the main input, and additional arguments are type hints for side inputs.
-### Defining Generic Types
+### Declaring Type Hints Using Annotations
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] udim commented on a change in pull request #11232:
[BEAM-8280] Document Python 3 annotations support
Posted by GitBox <gi...@apache.org>.
udim commented on a change in pull request #11232: [BEAM-8280] Document Python 3 annotations support
URL: https://github.com/apache/beam/pull/11232#discussion_r405722674
##########
File path: website/src/documentation/sdks/python-type-safety.md
##########
@@ -64,22 +84,75 @@ To specify type hints as properties of a `DoFn` or `PTransform`, use the decorat
The following code declares an `int` type hint on `FilterEvensDoFn`, using the decorator `@with_input_types()`.
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:type_hints_do_fn %}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:type_hints_do_fn %}```
Decorators receive an arbitrary number of positional and/or keyword arguments, typically interpreted in the context of the function they're wrapping. Generally the first argument is a type hint for the main input, and additional arguments are type hints for side inputs.
-### Defining Generic Types
+### Declaring Type Hints Using Annotations
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay commented on a change in pull request #11232:
[BEAM-8280] Document Python 3 annotations support
Posted by GitBox <gi...@apache.org>.
aaltay commented on a change in pull request #11232: [BEAM-8280] Document Python 3 annotations support
URL: https://github.com/apache/beam/pull/11232#discussion_r398281648
##########
File path: website/src/documentation/sdks/python-type-safety.md
##########
@@ -64,22 +84,75 @@ To specify type hints as properties of a `DoFn` or `PTransform`, use the decorat
The following code declares an `int` type hint on `FilterEvensDoFn`, using the decorator `@with_input_types()`.
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:type_hints_do_fn %}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:type_hints_do_fn %}```
Decorators receive an arbitrary number of positional and/or keyword arguments, typically interpreted in the context of the function they're wrapping. Generally the first argument is a type hint for the main input, and additional arguments are type hints for side inputs.
-### Defining Generic Types
+### Declaring Type Hints Using Annotations
+
+_New in version 2.21.0._
Review comment:
Is this also added to the changes.md ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] robertwb commented on a change in pull request #11232:
[BEAM-8280] Document Python 3 annotations support
Posted by GitBox <gi...@apache.org>.
robertwb commented on a change in pull request #11232: [BEAM-8280] Document Python 3 annotations support
URL: https://github.com/apache/beam/pull/11232#discussion_r399553756
##########
File path: website/src/documentation/sdks/python-type-safety.md
##########
@@ -23,38 +23,58 @@ Python is a dynamically-typed language with no static type checking. Because of
The Apache Beam SDK for Python uses **type hints** during pipeline construction and runtime to try to emulate the correctness guarantees achieved by true static typing. Additionally, using type hints lays some groundwork that allows the backend service to perform efficient type deduction and registration of `Coder` objects.
-Python version 3.5 introduces a module called **typing** to provide hints for type validators in the language. The Beam SDK for Python, based on Python version 2.7, implements a subset of [PEP 484](https://www.python.org/dev/peps/pep-0484/) and aims to follow it as closely as possible in its own typehints module.
+Python version 3.5 introduces a module called **typing** to provide hints for type validators in the language.
+The Beam SDK for Python implements a subset of [PEP 484](https://www.python.org/dev/peps/pep-0484/) and aims to follow it as closely as possible in its own typehints module.
+
+These flags control Beam type safety:
+- `--no_pipeline_type_check`
+
+ Disables type checking during pipeline construction.
+ Default is to perform these checks.
+- `--runtime_type_check`
+
+ Enables runtime type checking of every element.
+ This may affect pipeline performance, so the default is to skip these checks.
## Benefits of Type Hints
-The Beam SDK for Python includes some automatic type checking: for example, some `PTransform`s, such as `Create` and simple `ParDo` transforms, attempt to deduce their output type given their input. However, the Beam cannot infer types in all cases. Therefore, the recommendation is that you declare type hints to aid you in performing your own type checks if necessary.
+The Beam SDK for Python includes some automatic type checking: for example, some `PTransforms`, such as `Create` and simple `ParDo` transforms, attempt to deduce their output type given their input. However, the Beam cannot infer types in all cases. Therefore, the recommendation is that you declare type hints to aid you in performing your own type checks if necessary.
-When you use type hints, the runner raises exceptions during pipeline construction time, rather than runtime. For example, the runner generates an exception if it detects that your pipeline applies mismatched `PTransforms` (where the expected outputs of one transform do not match the expected inputs of the following transform). These exceptions are raised at pipeline construction time, regardless of where your pipeline will execute. Introducing type hints for the `PTransform`s you define allows you to catch potential bugs up front in the local runner, rather than after minutes of execution into a deep, complex pipeline.
+When you use type hints, the runner raises exceptions during pipeline construction time, rather than runtime. For example, the runner generates an exception if it detects that your pipeline applies mismatched `PTransforms` (where the expected outputs of one transform do not match the expected inputs of the following transform). These exceptions are raised at pipeline construction time, regardless of where your pipeline will execute. Introducing type hints for the `PTransforms` you define allows you to catch potential bugs up front in the local runner, rather than after minutes of execution into a deep, complex pipeline.
Review comment:
I might lead with this paragraph, maybe even giving a very concrete example ("for example, giving an error when trying to apply a PTransform that expects a PCollection of strings to a PCollection of ints"). Then put the other paragraph about Beam not always being able to infer types next.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services