[jira] [Comment Edited] (BEAM-7060) Design Py3-compatible typehints annotation support in Beam 3.

["Chad Dombrova (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16846309#comment-16846309 ] 

Chad Dombrova edited comment on BEAM-7060 at 5/23/19 12:19 AM:
---------------------------------------------------------------

Hi all, I'm quite interested in this topic and I have quite a bit of experience with python typing and mypy that I would love to contribute to this project. 

Firstly, I think it's definitely the right choice to move from the in-house typehints to the official typing module. 

Do you plan on keeping runtime type checks, or to switch over entirely to static type checks, e.g. using mypy?  The advantage of the latter is that users who are using annotations throughout their code get consistency between the type validation applied at the interfaces to their transforms (currently handled by beam typehints) and within their transform functions/classes. Also, as you're probably well aware, type analysis is fraught with corner cases that are really hard to get right, so it's nice to be able to lean on a project with dedicated experts in this area. 

If you go with the static approach, you should probably start by adding annotations to the beam python sdk first, either as .pyi stubs of the public API or inline.  I recommend the latter, so that you can get the benefits in your own development process. Then, if necessary, you can write a plugin for mypy to give it insight into any dynamic/magic code that it cannot properly inspect.  As an example, mypy ships with [a plugin|[https://github.com/python/mypy/blob/master/mypy/plugins/attrs.py]] for [attrs|[https://github.com/python-attrs/attrs]].   It is also possible for mypy plugins to be bundled with third-party code and registered with mypy rather than added directly to the mypy project.

I'm happy to help with the addition of type annotations into the current code base and add mypy tests, if you all agree that's the direction you'd like to go!

 

 


was (Author: chadrik):
Hi all, I'm quite interested in this topic and I have quite a bit of experience with python typing and mypy. 

Firstly, I think it's definitely the right choice to move from the in-house typehints to the official typing module. 

Do you plan on keeping runtime type checks, or to switch over entirely to static type checks, e.g. using mypy?  The advantage of the latter is that users who are using annotations throughout their code get consistency between the type validation applied at the interfaces to their transforms (currently handled by beam typehints) and within their transform functions/classes. Also, as you're probably well aware, type analysis is fraught with corner cases that are really hard to get right, so it's nice to be able to lean on a project with dedicated experts in this area. 

If you go with the static approach, you should probably start by adding annotations to the beam python sdk first, either as .pyi stubs of the public API or inline.  I recommend the latter, so that you can get the benefits in your own development process. Then, if necessary, you can write a plugin for mypy to give it insight into any dynamic/magic code that it cannot properly inspect.  As an example, mypy ships with [a plugin|[https://github.com/python/mypy/blob/master/mypy/plugins/attrs.py]] for [attrs|[https://github.com/python-attrs/attrs]].   It is also possible for mypy plugins to be bundled with third-party code and registered with mypy rather than added directly to the mypy project.

I'm happy to help with the addition of type annotations into the current code base and add mypy tests, if you all agree that's the direction you'd like to go!

 

 

> Design Py3-compatible typehints annotation support in Beam 3.
> -------------------------------------------------------------
>
>                 Key: BEAM-7060
>                 URL: https://issues.apache.org/jira/browse/BEAM-7060
>             Project: Beam
>          Issue Type: Sub-task
>          Components: sdk-py-core
>            Reporter: Valentyn Tymofieiev
>            Assignee: Udi Meiri
>            Priority: Major
>
> Existing [Typehints implementaiton in Beam|[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/typehints/
> ] heavily relies on internal details of CPython implementation, and some of the assumptions of this implementation broke as of Python 3.6, see for example: https://issues.apache.org/jira/browse/BEAM-6877, which makes  typehints support unusable on Python 3.6 as of now. [Python 3 Kanban Board|https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245&view=detail] lists several specific typehints-related breakages, prefixed with "TypeHints Py3 Error".
> We need to decide whether to:
> - Deprecate in-house typehints implementation.
> - Continue to support in-house implementation, which at this point is a stale code and has other known issues.
> - Attempt to use some off-the-shelf libraries for supporting type-annotations, like  Pytype, Mypy, PyAnnotate.
> WRT to this decision we also need to plan on immediate next steps to unblock adoption of Beam for  Python 3.6+ users. One potential option may be to have Beam SDK ignore any typehint annotations on Py 3.6+.
> cc: [~udim], [~altay], [~robertwb].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)