You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/06 11:43:44 UTC
[GitHub] [airflow] IAL32 opened a new pull request, #28149: Migrate amazon provider operator tests from `unittests` to `pytest`
IAL32 opened a new pull request, #28149:
URL: https://github.com/apache/airflow/pull/28149
Related: https://github.com/apache/airflow/pull/28039 https://github.com/apache/airflow/pull/28139 https://github.com/apache/airflow/pull/28145 @Taragolis
Migrate Amazon provider's transfer tests to `pytest`.
All changes are more or less straightforward:
- Get rid of unittests.TestCase class and TestCase.assert* methods
- Convert setUp* and tearDown* methods to [appropriate pytest alternative](https://docs.pytest.org/en/6.2.x/xunit_setup.html#classic-xunit-style-setup)
- Replace decorator `parameterized.expand` by `pytest.mark.parametrize`.
- Renamed `@patch` to `@mock.patch` to conform to other tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #28149: Migrate amazon provider operator tests from `unittests` to `pytest`
Posted by GitBox <gi...@apache.org>.
Taragolis commented on code in PR #28149:
URL: https://github.com/apache/airflow/pull/28149#discussion_r1040899742
##########
tests/providers/amazon/aws/operators/test_s3_file_transform.py:
##########
@@ -34,8 +33,8 @@
from airflow.providers.amazon.aws.operators.s3 import S3FileTransformOperator
-class TestS3FileTransformOperator(unittest.TestCase):
- def setUp(self):
+class TestS3FileTransformOperator:
+ def setup_method(self):
Review Comment:
Just optional suggestion (and reminder) because we could do it later by separate PR, when actually refactor some tests.
We manually create here temporary files and remove it in teardown_method, it could be replaced by new fixture which use `tmp_path_factory` builtin pytest fixture
https://github.com/apache/airflow/pull/27970/files#r1034288640
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #28149: Migrate amazon provider operator tests from `unittests` to `pytest`
Posted by GitBox <gi...@apache.org>.
Taragolis commented on code in PR #28149:
URL: https://github.com/apache/airflow/pull/28149#discussion_r1040895986
##########
tests/providers/amazon/aws/operators/test_eks.py:
##########
@@ -172,8 +171,7 @@ def test_execute_create_cluster(self, mock_create_nodegroup, mock_create_cluster
]
for (operator, parameters) in operator_under_test:
- with self.subTest():
- operator.execute({})
+ operator.execute({})
Review Comment:
As you wish, you could just grub whatever you think appropriate and help full.
But I didn't recommended copy-paste entire files because some changes might be add to this files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #28149: Migrate amazon provider operator tests from `unittests` to `pytest`
Posted by GitBox <gi...@apache.org>.
Taragolis commented on code in PR #28149:
URL: https://github.com/apache/airflow/pull/28149#discussion_r1040883358
##########
tests/providers/amazon/aws/operators/test_batch.py:
##########
@@ -49,7 +48,7 @@ class TestBatchOperator(unittest.TestCase):
@mock.patch.dict("os.environ", AWS_ACCESS_KEY_ID=AWS_ACCESS_KEY_ID)
@mock.patch.dict("os.environ", AWS_SECRET_ACCESS_KEY=AWS_SECRET_ACCESS_KEY)
@mock.patch("airflow.providers.amazon.aws.hooks.batch_client.AwsBaseHook.get_client_type")
- def setUp(self, get_client_type_mock):
+ def setup(self, get_client_type_mock):
Review Comment:
This will fail, if you decorate setup_method/teardown_method or entire class you need to provide `method` argument see discussions:
- https://github.com/apache/airflow/pull/28039#discussion_r1037813189
- https://github.com/apache/airflow/pull/28039#discussion_r1037814650
```suggestion
def setup(self, method, get_client_type_mock):
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #28149: Migrate amazon provider operator tests from `unittests` to `pytest`
Posted by GitBox <gi...@apache.org>.
Taragolis commented on code in PR #28149:
URL: https://github.com/apache/airflow/pull/28149#discussion_r1042595692
##########
tests/providers/amazon/aws/operators/test_datasync.py:
##########
@@ -100,7 +96,7 @@ def setUp(self):
DestinationLocationArn=self.destination_location_arn,
)["TaskArn"]
- def tearDown(self):
+ def teardown(self):
Review Comment:
```suggestion
def teardown_method(self, method):
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] IAL32 commented on a diff in pull request #28149: Migrate amazon provider operator tests from `unittests` to `pytest`
Posted by GitBox <gi...@apache.org>.
IAL32 commented on code in PR #28149:
URL: https://github.com/apache/airflow/pull/28149#discussion_r1040905025
##########
tests/providers/amazon/aws/operators/test_s3_file_transform.py:
##########
@@ -34,8 +33,8 @@
from airflow.providers.amazon.aws.operators.s3 import S3FileTransformOperator
-class TestS3FileTransformOperator(unittest.TestCase):
- def setUp(self):
+class TestS3FileTransformOperator:
+ def setup_method(self):
Review Comment:
I think this is best be done in a separate PR. I was also considering modifying this test, as it is actually untestable under MacOS, triggering an error: https://stackoverflow.com/questions/32115715/os-mknod-fails-on-macos
Thanks for the heads up!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #28149: Migrate amazon provider operator tests from `unittests` to `pytest`
Posted by GitBox <gi...@apache.org>.
Taragolis commented on code in PR #28149:
URL: https://github.com/apache/airflow/pull/28149#discussion_r1040887364
##########
tests/providers/amazon/aws/operators/test_eks.py:
##########
@@ -172,8 +171,7 @@ def test_execute_create_cluster(self, mock_create_nodegroup, mock_create_cluster
]
for (operator, parameters) in operator_under_test:
- with self.subTest():
- operator.execute({})
+ operator.execute({})
Review Comment:
Close builtin replacement to `unittest.TestCase.subTest` is parametrise.
You might have a look how I implemented this in initial PR: https://github.com/apache/airflow/pull/27970/files#r1034285155
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] IAL32 commented on a diff in pull request #28149: Migrate amazon provider operator tests from `unittests` to `pytest`
Posted by GitBox <gi...@apache.org>.
IAL32 commented on code in PR #28149:
URL: https://github.com/apache/airflow/pull/28149#discussion_r1040906532
##########
tests/providers/amazon/aws/operators/test_eks.py:
##########
@@ -172,8 +171,7 @@ def test_execute_create_cluster(self, mock_create_nodegroup, mock_create_cluster
]
for (operator, parameters) in operator_under_test:
- with self.subTest():
- operator.execute({})
+ operator.execute({})
Review Comment:
I added you as Co-Author of [921588e](https://github.com/apache/airflow/pull/28149/commits/921588e128f99bd2be702d54de285fe25e63f5a5), hopefully this will remain when merging the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] IAL32 commented on a diff in pull request #28149: Migrate amazon provider operator tests from `unittests` to `pytest`
Posted by GitBox <gi...@apache.org>.
IAL32 commented on code in PR #28149:
URL: https://github.com/apache/airflow/pull/28149#discussion_r1040893811
##########
tests/providers/amazon/aws/operators/test_eks.py:
##########
@@ -172,8 +171,7 @@ def test_execute_create_cluster(self, mock_create_nodegroup, mock_create_cluster
]
for (operator, parameters) in operator_under_test:
- with self.subTest():
- operator.execute({})
+ operator.execute({})
Review Comment:
I actually didn't see that you were working on exactly this ^^ sorry for that.
Do you mind if I continue and just copy-paste your contribution to this file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #28149: Migrate amazon provider operator tests from `unittests` to `pytest`
Posted by GitBox <gi...@apache.org>.
Taragolis commented on code in PR #28149:
URL: https://github.com/apache/airflow/pull/28149#discussion_r1040890444
##########
tests/providers/amazon/aws/operators/test_eks.py:
##########
@@ -92,8 +91,8 @@ class CreateNodegroupParams(TypedDict):
nodegroup_role_arn: str
-class TestEksCreateClusterOperator(unittest.TestCase):
- def setUp(self) -> None:
+class TestEksCreateClusterOperator:
Review Comment:
Also I remember that I have an issue with configurations mutability between tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #28149: Migrate amazon provider operator tests from `unittests` to `pytest`
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #28149:
URL: https://github.com/apache/airflow/pull/28149
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #28149: Migrate amazon provider operator tests from `unittests` to `pytest`
Posted by GitBox <gi...@apache.org>.
Taragolis commented on code in PR #28149:
URL: https://github.com/apache/airflow/pull/28149#discussion_r1042595692
##########
tests/providers/amazon/aws/operators/test_datasync.py:
##########
@@ -100,7 +96,7 @@ def setUp(self):
DestinationLocationArn=self.destination_location_arn,
)["TaskArn"]
- def tearDown(self):
+ def teardown(self):
Review Comment:
```suggestion
def teardown_method(self):
```
##########
tests/providers/amazon/aws/operators/test_datasync.py:
##########
@@ -69,13 +68,9 @@
@mock_datasync
@mock.patch.object(DataSyncHook, "get_conn")
-class DataSyncTestCaseBase(unittest.TestCase):
- def __init__(self, *args, **kwargs):
- super().__init__(*args, **kwargs)
- self.datasync = None
-
+class DataSyncTestCaseBase:
# Runs once for each test
- def setUp(self):
+ def setup(self):
Review Comment:
I've found that `pytest` actually run methods named **setup**, **teardown**. I don't know might be it some kind of undocumented feature which could remove in future releases. So better keep general notation
```suggestion
def setup_method(self, method):
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] IAL32 commented on a diff in pull request #28149: Migrate amazon provider operator tests from `unittests` to `pytest`
Posted by GitBox <gi...@apache.org>.
IAL32 commented on code in PR #28149:
URL: https://github.com/apache/airflow/pull/28149#discussion_r1040873557
##########
tests/providers/amazon/aws/operators/test_sagemaker_transform.py:
##########
@@ -43,6 +43,16 @@
"TransformOutput": {"S3OutputPath": "output_path"},
"TransformResources": {"InstanceType": "ml.m4.xlarge", "InstanceCount": "3"},
}
+CREATE_TRANSFORM_PARAMS_INTEGER_FIELDS: dict = {
+ "TransformJobName": "job_name",
+ "ModelName": "model_name",
+ "MaxConcurrentTransforms": 12,
+ "MaxPayloadInMB": 6,
+ "BatchStrategy": "MultiRecord",
+ "TransformInput": {"DataSource": {"S3DataSource": {"S3DataType": "S3Prefix", "S3Uri": "s3_uri"}}},
+ "TransformOutput": {"S3OutputPath": "output_path"},
+ "TransformResources": {"InstanceType": "ml.m4.xlarge", "InstanceCount": 3},
+}
Review Comment:
Note: previous implementation relied on the fact that executing the operator actively modified the `CREATE_TRANSFORM_PARAMS` constant, which is bad practice. Here I provide a second dictionary with the expected integer values and modified tests accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] IAL32 commented on pull request #28149: Migrate amazon provider operator tests from `unittests` to `pytest`
Posted by GitBox <gi...@apache.org>.
IAL32 commented on PR #28149:
URL: https://github.com/apache/airflow/pull/28149#issuecomment-1341692144
@Taragolis all done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org