You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "jrmccluskey (via GitHub)" <gi...@apache.org> on 2023/03/10 17:26:21 UTC
[GitHub] [beam] jrmccluskey opened a new pull request, #25795: Add support for CPython 3.11 opcodes
jrmccluskey opened a new pull request, #25795:
URL: https://github.com/apache/beam/pull/25795
Adds typehinting support for CPython 3.11 opcodes, aiming to unblock adding support for Python 3.11.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey merged pull request #25795: Add support for CPython 3.11 opcodes
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey merged PR #25795:
URL: https://github.com/apache/beam/pull/25795
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #25795: Add support for CPython 3.11 opcodes
Posted by "AnandInguva (via GitHub)" <gi...@apache.org>.
AnandInguva commented on PR #25795:
URL: https://github.com/apache/beam/pull/25795#issuecomment-1464140955
cc: @tvalentyn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on a diff in pull request #25795: Add support for CPython 3.11 opcodes
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #25795:
URL: https://github.com/apache/beam/pull/25795#discussion_r1136210202
##########
sdks/python/apache_beam/typehints/trivial_inference.py:
##########
@@ -498,6 +518,39 @@ def infer_return_type_func(f, input_types, debug=False, depth=0):
else:
return_type = typehints.Any
state.stack[-pop_count:] = [return_type]
+ elif opname == 'CALL':
+ pop_count = 1 + arg
+ # Keyword Args case
+ if state.kw_names is not None:
+ if isinstance(state.stack[-pop_count], Const):
+ from apache_beam.pvalue import Row
+ if state.stack[-pop_count].value == Row:
Review Comment:
This is doable but a little finnicky since the `CALL` version and the `CALL_FUNCTION_KW` version have slightly different accessed indices in the stack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on a diff in pull request #25795: Add support for CPython 3.11 opcodes
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #25795:
URL: https://github.com/apache/beam/pull/25795#discussion_r1136072403
##########
sdks/python/apache_beam/typehints/opcodes.py:
##########
@@ -440,18 +450,30 @@ def make_function(state, arg):
"""
# TODO(luke-zhu): Handle default argument types
globals = state.f.__globals__ # Inherits globals from the current frame
- func_name = state.stack[-1].value
- func_code = state.stack[-2].value
- pop_count = 2
+ tos = state.stack[-1].value
+ if isinstance(tos, types.CodeType):
+ func_name = None
Review Comment:
This is related to the lack of a fully qualified name being pushed onto the stack for lambdas in 3.11 on. For pre-3.11 versions TOS in calls to `make_function()` will always be a string containing the fully qualified name of the function being made, but post-3.11 this could just be the code for the lambda. We have to check for this case and adjust the indices we access on the stack accordingly to avoid index-out-of-range issues
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on a diff in pull request #25795: Add support for CPython 3.11 opcodes
Posted by "AnandInguva (via GitHub)" <gi...@apache.org>.
AnandInguva commented on code in PR #25795:
URL: https://github.com/apache/beam/pull/25795#discussion_r1134375975
##########
sdks/python/apache_beam/typehints/trivial_inference.py:
##########
@@ -408,17 +418,27 @@ def infer_return_type_func(f, input_types, debug=False, depth=0):
if op in dis.hasconst:
print('(' + repr(co.co_consts[arg]) + ')', end=' ')
elif op in dis.hasname:
- print('(' + co.co_names[arg] + ')', end=' ')
+ if (sys.version_info.major, sys.version_info.minor) >= (3, 11):
+ # Pre-emptively bit-shift so the print doesn't go out of index
+ printArg = arg >> 1
Review Comment:
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on a diff in pull request #25795: Add support for CPython 3.11 opcodes
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #25795:
URL: https://github.com/apache/beam/pull/25795#discussion_r1136068126
##########
sdks/python/apache_beam/typehints/trivial_inference.py:
##########
@@ -498,6 +518,39 @@ def infer_return_type_func(f, input_types, debug=False, depth=0):
else:
return_type = typehints.Any
state.stack[-pop_count:] = [return_type]
+ elif opname == 'CALL':
+ pop_count = 1 + arg
+ # Keyword Args case
+ if state.kw_names is not None:
+ if isinstance(state.stack[-pop_count], Const):
+ from apache_beam.pvalue import Row
+ if state.stack[-pop_count].value == Row:
+ fields = state.kw_names
+ return_type = row_type.RowTypeConstraint.from_fields(
+ list(
+ zip(fields,
+ Const.unwrap_all(state.stack[-pop_count + 1:]))))
+ else:
+ return_type = Any
+ state.kw_names = None
+ else:
+ # Catch lambdas not passing in arg values properly
Review Comment:
Okay so this will be a bit of a lengthy comment. For a typical function call the stack winds up being something like
Keyword Args
Positional Args
Fully-Qualified Name of Function
Callable
and the arg to the `CALL` (or pre-3.11, `CALL_FUNCTION`) opcode is an integer representing the number of elements on the stack above the callable itself. This behavior was the same for defined functions and lambdas. In 3.11, however, lambdas now produce the following:
Keyword Args
Positional Args
Callable
with the opcode and arg always being `CALL 0`. I don't have a handle on why lambdas are getting a 0 arg from the CPython side yet, but this isn't documented anywhere I can find. So, as a result, I have to add a decent chunk of logic to iterate through the stack and find the callable in the event that we get a `CALL 0` op.
This gets used pretty extensively because a ton of the test cases pass lambdas into the type hinting functions rather than pre-defining a function (although we do use both, so we cover both cases in the unit tests.) I should probably update that comment to have a little bit more context and a link to an issue I filed to the CPython repo asking about this behavior since it goes against the documented behavior of `CALL`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #25795: Add support for CPython 3.11 opcodes
Posted by "AnandInguva (via GitHub)" <gi...@apache.org>.
AnandInguva commented on PR #25795:
URL: https://github.com/apache/beam/pull/25795#issuecomment-1468117895
>
I am testing Python 3.11 here https://github.com/apache/beam/pull/24721. I merged this PR with some other changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #25795: Add support for CPython 3.11 opcodes
Posted by "AnandInguva (via GitHub)" <gi...@apache.org>.
AnandInguva commented on PR #25795:
URL: https://github.com/apache/beam/pull/25795#issuecomment-1474276813
> @AnandInguva I think this PR can be merged on its own if related inference tests on 3.11 are passing in your branch. Do you have any concerns merging this?
yes, it can be merged in its current state. I didn't find unit tests failing on github actions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a diff in pull request #25795: Add support for CPython 3.11 opcodes
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #25795:
URL: https://github.com/apache/beam/pull/25795#discussion_r1134730567
##########
sdks/python/apache_beam/typehints/opcodes.py:
##########
@@ -428,10 +432,16 @@ def gen_start(state, arg):
def load_closure(state, arg):
+ # The arg is no longer offset by len(covar_names) as of 3.11
Review Comment:
nit: it may be use useful to add references to CPython docs such as announcements or bpa- issues.
##########
sdks/python/apache_beam/typehints/opcodes.py:
##########
@@ -440,18 +450,30 @@ def make_function(state, arg):
"""
# TODO(luke-zhu): Handle default argument types
globals = state.f.__globals__ # Inherits globals from the current frame
- func_name = state.stack[-1].value
- func_code = state.stack[-2].value
- pop_count = 2
+ tos = state.stack[-1].value
+ if isinstance(tos, types.CodeType):
+ func_name = None
Review Comment:
for my education, do we ever go in this branch for versions less than 3.11 ? If not, why? if yes: do we have changes in the inference behavior?
##########
sdks/python/apache_beam/typehints/trivial_inference.py:
##########
@@ -498,6 +518,39 @@ def infer_return_type_func(f, input_types, debug=False, depth=0):
else:
return_type = typehints.Any
state.stack[-pop_count:] = [return_type]
+ elif opname == 'CALL':
+ pop_count = 1 + arg
+ # Keyword Args case
+ if state.kw_names is not None:
+ if isinstance(state.stack[-pop_count], Const):
+ from apache_beam.pvalue import Row
+ if state.stack[-pop_count].value == Row:
Review Comment:
does it make sense to move this code into a helper and reuse in CALL_FUNCTION_KW processing?
##########
sdks/python/apache_beam/typehints/trivial_inference.py:
##########
@@ -498,6 +518,39 @@ def infer_return_type_func(f, input_types, debug=False, depth=0):
else:
return_type = typehints.Any
state.stack[-pop_count:] = [return_type]
+ elif opname == 'CALL':
+ pop_count = 1 + arg
+ # Keyword Args case
+ if state.kw_names is not None:
+ if isinstance(state.stack[-pop_count], Const):
+ from apache_beam.pvalue import Row
+ if state.stack[-pop_count].value == Row:
+ fields = state.kw_names
+ return_type = row_type.RowTypeConstraint.from_fields(
+ list(
+ zip(fields,
+ Const.unwrap_all(state.stack[-pop_count + 1:]))))
+ else:
+ return_type = Any
+ state.kw_names = None
+ else:
+ # Catch lambdas not passing in arg values properly
Review Comment:
do we already have tests for this?
For my education, what does not passing properly mean in this context?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on a diff in pull request #25795: Add support for CPython 3.11 opcodes
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #25795:
URL: https://github.com/apache/beam/pull/25795#discussion_r1134164367
##########
sdks/python/apache_beam/typehints/trivial_inference.py:
##########
@@ -408,17 +418,27 @@ def infer_return_type_func(f, input_types, debug=False, depth=0):
if op in dis.hasconst:
print('(' + repr(co.co_consts[arg]) + ')', end=' ')
elif op in dis.hasname:
- print('(' + co.co_names[arg] + ')', end=' ')
+ if (sys.version_info.major, sys.version_info.minor) >= (3, 11):
+ # Pre-emptively bit-shift so the print doesn't go out of index
+ printArg = arg >> 1
Review Comment:
It's actually the mechanics in 3.11 for free vars, prior to this you'd have you'd just get an int arg and return the value out. In 3.11 the low bit that gets shifted out matters on the CPython side for deciding whether or not to push a NULL on the stack. We don't sweat this implementation detail since it doesn't seem to matter much to us, we just shift the value to avoid an index out of range for debug printing and retrieve the proper index in LOAD_GLOBAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] yuya-morimoto commented on pull request #25795: Add support for CPython 3.11 opcodes
Posted by "yuya-morimoto (via GitHub)" <gi...@apache.org>.
yuya-morimoto commented on PR #25795:
URL: https://github.com/apache/beam/pull/25795#issuecomment-1469135784
> I am testing Python 3.11 here https://github.com/apache/beam/pull/24721. I merged this PR with some other changes.
@AnandInguva
ok, thanks.
Python3.11 seems to work fine even with the current release version(apache:master). Do you match?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a diff in pull request #25795: Add support for CPython 3.11 opcodes
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #25795:
URL: https://github.com/apache/beam/pull/25795#discussion_r1140600127
##########
sdks/python/apache_beam/typehints/trivial_inference.py:
##########
@@ -498,6 +518,40 @@ def infer_return_type_func(f, input_types, debug=False, depth=0):
else:
return_type = typehints.Any
state.stack[-pop_count:] = [return_type]
+ elif opname == 'CALL':
+ pop_count = 1 + arg
+ # Keyword Args case
+ if state.kw_names is not None:
+ if isinstance(state.stack[-pop_count], Const):
+ from apache_beam.pvalue import Row
+ if state.stack[-pop_count].value == Row:
+ fields = state.kw_names
+ return_type = row_type.RowTypeConstraint.from_fields(
+ list(
+ zip(fields,
+ Const.unwrap_all(state.stack[-pop_count + 1:]))))
+ else:
+ return_type = Any
Review Comment:
Thanks. is there a FR for that in https://github.com/apache/beam/issues?q=is%3Aopen+label%3Apython+label%3Atypes+sort%3Aupdated-desc+ ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #25795: Add support for CPython 3.11 opcodes
Posted by "AnandInguva (via GitHub)" <gi...@apache.org>.
AnandInguva commented on PR #25795:
URL: https://github.com/apache/beam/pull/25795#issuecomment-1474281334
> > @AnandInguva I think this PR can be merged on its own if related inference tests on 3.11 are passing in your branch. Do you have any concerns merging this?
>
> yes, it can be merged in its current state. I didn't find unit tests failing on github actions. I will run one more test run and confirm everything is green on Py311. Give me some time to do that.
https://github.com/apache/beam/actions/runs/4450700438/jobs/7816480746?pr=24721. This run includes the current PR changes. 3.11 has green state. so I am good for this to get merged on to the master. if something comes up, i will create an issue and let the team know
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #25795: Add support for CPython 3.11 opcodes
Posted by "AnandInguva (via GitHub)" <gi...@apache.org>.
AnandInguva commented on PR #25795:
URL: https://github.com/apache/beam/pull/25795#issuecomment-1466596663
I will pass this to @tvalentyn or @robertwb for further review. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on a diff in pull request #25795: Add support for CPython 3.11 opcodes
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #25795:
URL: https://github.com/apache/beam/pull/25795#discussion_r1134154038
##########
sdks/python/apache_beam/typehints/trivial_inference_test.py:
##########
@@ -299,7 +299,7 @@ def m(self, x):
return x
self.assertReturnType(int, lambda: A().m(3))
- self.assertReturnType(float, lambda: A.m(A(), 3.0))
+ #self.assertReturnType(float, lambda: A.m(A(), 3.0))
Review Comment:
I had commented that out to diagnose an issue with the first test case, just forgot to reinstate it. Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #25795: Add support for CPython 3.11 opcodes
Posted by "AnandInguva (via GitHub)" <gi...@apache.org>.
AnandInguva commented on PR #25795:
URL: https://github.com/apache/beam/pull/25795#issuecomment-1470272472
> > I am testing Python 3.11 here #24721. I merged this PR with some other changes.
> > @AnandInguva
> > ok, thanks.
> > Python3.11 seems to work fine even with the current release version(apache:master). Do you match?
Python 3.11 won'r work with the master. There are some opcodes issue(tackled here) and dill related issues(we need to update dill).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #25795: Add support for CPython 3.11 opcodes
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #25795:
URL: https://github.com/apache/beam/pull/25795#issuecomment-1474275210
@AnandInguva I think this PR can be merged on its own if related inference tests on 3.11 are passing in your branch. Do you have any concerns merging this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] yuya-morimoto commented on pull request #25795: Add support for CPython 3.11 opcodes
Posted by "yuya-morimoto (via GitHub)" <gi...@apache.org>.
yuya-morimoto commented on PR #25795:
URL: https://github.com/apache/beam/pull/25795#issuecomment-1467904669
@jrmccluskey
i am trying to use it with 3.11.
do you also modify setup.py, etc.?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #25795: Add support for CPython 3.11 opcodes
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #25795:
URL: https://github.com/apache/beam/pull/25795#issuecomment-1464160480
## [Codecov](https://codecov.io/gh/apache/beam/pull/25795?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#25795](https://codecov.io/gh/apache/beam/pull/25795?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0cb4998) into [master](https://codecov.io/gh/apache/beam/commit/f44c553478e6c0dccfcde45579334ee500f3add3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f44c553) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #25795 +/- ##
==========================================
- Coverage 72.81% 72.81% -0.01%
==========================================
Files 775 775
Lines 102928 102915 -13
==========================================
- Hits 74946 74933 -13
Misses 26527 26527
Partials 1455 1455
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/25795?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/25795?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `85.14% <ø> (-0.22%)` | :arrow_down: |
| [.../python/apache\_beam/typehints/trivial\_inference.py](https://codecov.io/gh/apache/beam/pull/25795?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3RyaXZpYWxfaW5mZXJlbmNlLnB5) | `95.78% <ø> (-0.12%)` | :arrow_down: |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #25795: Add support for CPython 3.11 opcodes
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25795:
URL: https://github.com/apache/beam/pull/25795#issuecomment-1464132478
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on a diff in pull request #25795: Add support for CPython 3.11 opcodes
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #25795:
URL: https://github.com/apache/beam/pull/25795#discussion_r1134164367
##########
sdks/python/apache_beam/typehints/trivial_inference.py:
##########
@@ -408,17 +418,27 @@ def infer_return_type_func(f, input_types, debug=False, depth=0):
if op in dis.hasconst:
print('(' + repr(co.co_consts[arg]) + ')', end=' ')
elif op in dis.hasname:
- print('(' + co.co_names[arg] + ')', end=' ')
+ if (sys.version_info.major, sys.version_info.minor) >= (3, 11):
+ # Pre-emptively bit-shift so the print doesn't go out of index
+ printArg = arg >> 1
Review Comment:
It's actually the mechanics in 3.11 for free vars, prior to this you'd have you'd just get an int arg and return the value out but in 3.11 the bit that gets shifted out matters on the CPython side for deciding whether or not to push a NULL on the stack. We don't sweat this implementation detail since it doesn't seem to matter much to us, we just shift the value to avoid an index out of range for debug printing and retrieve the proper index in LOAD_GLOBAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a diff in pull request #25795: Add support for CPython 3.11 opcodes
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #25795:
URL: https://github.com/apache/beam/pull/25795#discussion_r1139065133
##########
sdks/python/apache_beam/typehints/trivial_inference.py:
##########
@@ -498,6 +518,40 @@ def infer_return_type_func(f, input_types, debug=False, depth=0):
else:
return_type = typehints.Any
state.stack[-pop_count:] = [return_type]
+ elif opname == 'CALL':
+ pop_count = 1 + arg
+ # Keyword Args case
+ if state.kw_names is not None:
+ if isinstance(state.stack[-pop_count], Const):
+ from apache_beam.pvalue import Row
+ if state.stack[-pop_count].value == Row:
+ fields = state.kw_names
+ return_type = row_type.RowTypeConstraint.from_fields(
+ list(
+ zip(fields,
+ Const.unwrap_all(state.stack[-pop_count + 1:]))))
+ else:
+ return_type = Any
Review Comment:
Re places where we return 'Any', could you comment on why we currently don't attempt any inference there, and whether it would make sense track those as feature requests (we have several issues tagged as 'types', some of these FRs may be already filed).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on pull request #25795: Add support for CPython 3.11 opcodes
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #25795:
URL: https://github.com/apache/beam/pull/25795#issuecomment-1464130993
R: @robertwb @AnandInguva
CC: @damccorm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on a diff in pull request #25795: Add support for CPython 3.11 opcodes
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #25795:
URL: https://github.com/apache/beam/pull/25795#discussion_r1134152750
##########
sdks/python/apache_beam/typehints/trivial_inference.py:
##########
@@ -374,7 +378,12 @@ def infer_return_type_func(f, input_types, debug=False, depth=0):
# In Python 3, use dis library functions to disassemble bytecode and handle
# EXTENDED_ARGs.
ofs_table = {} # offset -> instruction
- for instruction in dis.get_instructions(f):
+ if (sys.version_info.major, sys.version_info.minor) >= (3, 11):
+ disInts = dis.get_instructions(f, show_caches=True)
Review Comment:
I've cleaned up the instances from a quick pass, will update others as I go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on a diff in pull request #25795: Add support for CPython 3.11 opcodes
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #25795:
URL: https://github.com/apache/beam/pull/25795#discussion_r1139108726
##########
sdks/python/apache_beam/typehints/trivial_inference.py:
##########
@@ -498,6 +518,40 @@ def infer_return_type_func(f, input_types, debug=False, depth=0):
else:
return_type = typehints.Any
state.stack[-pop_count:] = [return_type]
+ elif opname == 'CALL':
+ pop_count = 1 + arg
+ # Keyword Args case
+ if state.kw_names is not None:
+ if isinstance(state.stack[-pop_count], Const):
+ from apache_beam.pvalue import Row
+ if state.stack[-pop_count].value == Row:
+ fields = state.kw_names
+ return_type = row_type.RowTypeConstraint.from_fields(
+ list(
+ zip(fields,
+ Const.unwrap_all(state.stack[-pop_count + 1:]))))
+ else:
+ return_type = Any
Review Comment:
We don't attempt type hinting on keyword args beyond rows (historically it's been hard to do that, with the post-3.11 practice of the KW names being stored as a tuple in the state rather than on the stack it may be easier to handle that case) and if we recurse more than 5 times we call it as an Any. Right now I'm trying to keep the behavior consistent with previous behavior, moving forward we'll be able to improve these mechanics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on a diff in pull request #25795: Add support for CPython 3.11 opcodes
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #25795:
URL: https://github.com/apache/beam/pull/25795#discussion_r1140601450
##########
sdks/python/apache_beam/typehints/trivial_inference.py:
##########
@@ -498,6 +518,40 @@ def infer_return_type_func(f, input_types, debug=False, depth=0):
else:
return_type = typehints.Any
state.stack[-pop_count:] = [return_type]
+ elif opname == 'CALL':
+ pop_count = 1 + arg
+ # Keyword Args case
+ if state.kw_names is not None:
+ if isinstance(state.stack[-pop_count], Const):
+ from apache_beam.pvalue import Row
+ if state.stack[-pop_count].value == Row:
+ fields = state.kw_names
+ return_type = row_type.RowTypeConstraint.from_fields(
+ list(
+ zip(fields,
+ Const.unwrap_all(state.stack[-pop_count + 1:]))))
+ else:
+ return_type = Any
Review Comment:
https://github.com/apache/beam/issues/24755
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on a diff in pull request #25795: Add support for CPython 3.11 opcodes
Posted by "AnandInguva (via GitHub)" <gi...@apache.org>.
AnandInguva commented on code in PR #25795:
URL: https://github.com/apache/beam/pull/25795#discussion_r1134013626
##########
sdks/python/apache_beam/typehints/trivial_inference.py:
##########
@@ -374,7 +378,12 @@ def infer_return_type_func(f, input_types, debug=False, depth=0):
# In Python 3, use dis library functions to disassemble bytecode and handle
# EXTENDED_ARGs.
ofs_table = {} # offset -> instruction
- for instruction in dis.get_instructions(f):
+ if (sys.version_info.major, sys.version_info.minor) >= (3, 11):
+ disInts = dis.get_instructions(f, show_caches=True)
Review Comment:
```suggestion
dis_ints = dis.get_instructions(f, show_caches=True)
```
Can we follow PEP-8 for variable naming? [Function and Variable Names](https://peps.python.org/pep-0008/#function-and-variable-names)
##########
sdks/python/apache_beam/typehints/trivial_inference.py:
##########
@@ -408,17 +418,27 @@ def infer_return_type_func(f, input_types, debug=False, depth=0):
if op in dis.hasconst:
print('(' + repr(co.co_consts[arg]) + ')', end=' ')
elif op in dis.hasname:
- print('(' + co.co_names[arg] + ')', end=' ')
+ if (sys.version_info.major, sys.version_info.minor) >= (3, 11):
+ # Pre-emptively bit-shift so the print doesn't go out of index
+ printArg = arg >> 1
Review Comment:
I didn't understand. Why do we do this for 3.11?
##########
sdks/python/apache_beam/typehints/trivial_inference_test.py:
##########
@@ -299,7 +299,7 @@ def m(self, x):
return x
self.assertReturnType(int, lambda: A().m(3))
- self.assertReturnType(float, lambda: A.m(A(), 3.0))
+ #self.assertReturnType(float, lambda: A.m(A(), 3.0))
Review Comment:
why did we comment this test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on a diff in pull request #25795: Add support for CPython 3.11 opcodes
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #25795:
URL: https://github.com/apache/beam/pull/25795#discussion_r1136209282
##########
sdks/python/apache_beam/typehints/opcodes.py:
##########
@@ -428,10 +432,16 @@ def gen_start(state, arg):
def load_closure(state, arg):
+ # The arg is no longer offset by len(covar_names) as of 3.11
Review Comment:
I added some references to documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org