You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/19 12:21:08 UTC
[GitHub] [airflow] potiuk opened a new pull request #11659: Enables splitting tests into smaller chunks
potiuk opened a new pull request #11659:
URL: https://github.com/apache/airflow/pull/11659
We've implemented the capability of running the tests in smaller
chunks and selective running only some of those, but this
capability have been disabled by mistake by default setting of
TEST_TYPE to "All" and not removing it when TEST_TYPES are set
to the sets of tests that should be run.
This should speed up many of our tests and also hopefully
lower the chance of EXIT 137 errors.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11659: Enables splitting tests into smaller chunks
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11659:
URL: https://github.com/apache/airflow/pull/11659#issuecomment-713519441
I also added one more, important change: For the tests, I also put the list of test types that are being executed in the name of the test job. This way - similarly as we see now that the API/Docs are skipped, we will also see which of the test 'types" are being executed and it will be easier to spot any anomalies (such as not executed Core tests on clearly core-related change).
I really think this one will finally give us the boost when it comes to strain on GA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11659: Enables splitting tests into smaller chunks
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11659:
URL: https://github.com/apache/airflow/pull/11659#issuecomment-714667007
[The Workflow run](https://github.com/apache/airflow/actions/runs/322589218) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11659: Enables splitting tests into smaller chunks
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11659:
URL: https://github.com/apache/airflow/pull/11659#issuecomment-712675230
HInt: ``--with-db-init`` in pytest seems to solve the problem (?????). Really strange.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11659: Enables splitting tests into smaller chunks
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11659:
URL: https://github.com/apache/airflow/pull/11659#issuecomment-712670165
Hey @ashb @mik-laj @kaxil @turbaszek - > I need your help. I have this really strange problem and maybe you will be able to help quickly. This is the LAST step to have stable selective tests working.
The MySQL tests in this build fail because of fernet.InvalidToken and I can't figure it out quickly:
It's rather easy to reproduce:
```
./breeze --backend sqlite --db-reset
```
And then:
```
pytest tests/providers/amazon/aws/hooks/test_emr.py::TestEmrHook::test_create_job_flow_extra_args
```
This fails with:
```
____________________________________________________________________________________________________________________________ TestEmrHook.test_create_job_flow_extra_args _____________________________________________________________________________________________________________________________
self = <tests.providers.amazon.aws.hooks.test_emr.TestEmrHook testMethod=test_create_job_flow_extra_args>
@mock_emr
def test_create_job_flow_extra_args(self):
"""
Test that we can add extra arguments to the launch call.
This is useful for when AWS add new options, such as
"SecurityConfiguration" so that we don't have to change our code
"""
client = boto3.client('emr', region_name='us-east-1')
hook = EmrHook(aws_conn_id='aws_default', emr_conn_id='emr_default')
# AmiVersion is really old and almost no one will use it anymore, but
# it's one of the "optional" request params that moto supports - it's
# coverage of EMR isn't 100% it turns out.
> cluster = hook.create_job_flow({'Name': 'test_cluster', 'ReleaseLabel': '', 'AmiVersion': '3.2'})
tests/providers/amazon/aws/hooks/test_emr.py:62:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
airflow/providers/amazon/aws/hooks/emr.py:80: in create_job_flow
config = emr_conn.extra_dejson.copy()
airflow/models/connection.py:377: in extra_dejson
if self.extra:
/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/attributes.py:358: in __get__
retval = self.descriptor.__get__(instance, owner)
airflow/models/connection.py:292: in get_extra
return fernet.decrypt(bytes(self._extra, 'utf-8')).decode()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <cryptography.fernet.MultiFernet object at 0x7fa7aa0ebed0>
msg = b'gAAAAABfjpjdT5OM4KF8yhsX_-nEC3-53b-XDTF4hpZVTzqVrYjEh5AHebgky5PCvt_hogmCInRNWXm5Zhe_zK-x9-TgAYxQlKZ2UwBcTaTi4uufY0ji...Ppw6xt39aGU0rQF_NbZYSXL0Ls9XceeALWIuGjEuxtpEGmlyX1bV2vAS9dU36rzffPcZj0VM8zvpbn3gNaFeZBq0aSSKPHtYOtzsWFj8qVdxQsellPHyI=', ttl = None
def decrypt(self, msg, ttl=None):
for f in self._fernets:
try:
return f.decrypt(msg, ttl)
except InvalidToken:
pass
> raise InvalidToken
E cryptography.fernet.InvalidToken
/usr/local/lib/python3.7/site-packages/cryptography/fernet.py:183: InvalidToken
--------------------------------------------------------------------------------------------------------------------------------------- Captured stdout setup ----------------------------------------------------------------------------------------------------------------------------------------
========================= AIRFLOW ==========================
Home of the user: /root
Airflow home /root/airflow
Skipping initializing of the DB as it was initialized already.
You can re-initialize the database by adding --with-db-init flag when running tests.
---------------------------------------------------------------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------------------------------------------------------------
[2020-10-20 07:59:31,472] {credentials.py:1087} INFO - Found credentials in environment variables.
----------------------------------------------------------------------------------------------------------------------------------------- Captured log call ------------------------------------------------------------------------------------------------------------------------------------------
INFO botocore.credentials:credentials.py:1087 Found credentials in environment variables.
====================================================================================================================================== short test summary info =======================================================================================================================================
FAILED tests/providers/amazon/aws/hooks/test_emr.py::TestEmrHook::test_create_job_flow_extra_args - cryptography.fernet.InvalidToken
```
Does it ring a bell maybe ??????
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11659: Enables splitting tests into smaller chunks
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11659:
URL: https://github.com/apache/airflow/pull/11659#issuecomment-713822610
> > For the tests, I also put the list of test types that are being executed in the name of the test job
>
> I couldn't see (from the code, the jobs haven't been created yet) this anywhere - where would it appear?
In the ci.yml. You can see it in the previous run where Kamil's change to kerberos network creation failed after rebase. I think I fixed it the latest build - but you can take a look at the failed build: https://github.com/apache/airflow/actions/runs/319791128
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11659: Enables splitting tests into smaller chunks
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11659:
URL: https://github.com/apache/airflow/pull/11659#issuecomment-712676138
I really hope when I go back from running this morning, it will be green and we will be able to get rid of the EXIT 137 problems
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11659: Enables splitting tests into smaller chunks
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11659:
URL: https://github.com/apache/airflow/pull/11659#issuecomment-712291163
Actually this one might finally get rid of the memory problems EXIT 137 when I re-joined the jobs, I thought that it does not help with the EXIT 137 but the reason was that I accidentally turned on back the old behaviour where the tests were not split into groups. Now the tests will run group-by-group and they will generally take much less memory per group. Github support confirmed that memory is the most likely culprit for EXIT 137 so we might be able to get back to stable with that one @ashb @kaxil @feluelle @turbaszek @mik-laj
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11659: Enables splitting tests into smaller chunks
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11659:
URL: https://github.com/apache/airflow/pull/11659#issuecomment-714382591
Hey @ashb -> I think this one will finally get green. There was a problem with integrating the kerberos change from @mik-laj but I think I solved it in the last force-push. I'd love to get it in to see the impact on this afternoon's strain on GA coming from the other projects.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #11659: Enables splitting tests into smaller chunks
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #11659:
URL: https://github.com/apache/airflow/pull/11659#issuecomment-713816718
> For the tests, I also put the list of test types that are being executed in the name of the test job
I couldn't see (from the code, the jobs haven't been created yet) this anywhere - where would it appear?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11659: Enables splitting tests into smaller chunks
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11659:
URL: https://github.com/apache/airflow/pull/11659#issuecomment-712450541
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11659: Enables splitting tests into smaller chunks
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11659:
URL: https://github.com/apache/airflow/pull/11659#issuecomment-714268561
[The Workflow run](https://github.com/apache/airflow/actions/runs/321459509) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11659: Enables splitting tests into smaller chunks
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11659:
URL: https://github.com/apache/airflow/pull/11659#issuecomment-712450267
[The Workflow run](https://github.com/apache/airflow/actions/runs/316309594) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11659: Enables splitting tests into smaller chunks
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11659:
URL: https://github.com/apache/airflow/pull/11659#issuecomment-712120350
It looks like the "Sequential" splitting of tests in our tests have not been enabled. While the TEST_TYPES variables is now set properly by the "selective_tests.sh" the TEST_TYPE one was set by default to "All" and the effect of it was that the dynamic selection of tests did not work as intended (this was disabled by accident when special Postgres/MySQL types of tests were added to handle the case where some Postgres/MySQL tests have to be run regardless of the dymamic selection.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #11659: Enables splitting tests into smaller chunks
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #11659:
URL: https://github.com/apache/airflow/pull/11659
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11659: Enables splitting tests into smaller chunks
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11659:
URL: https://github.com/apache/airflow/pull/11659#issuecomment-713061421
[The Workflow run](https://github.com/apache/airflow/actions/runs/318297415) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org