You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/25 10:52:24 UTC
[GitHub] [airflow] bryzgaloff opened a new pull request #10546: Feature/docker operator extra hosts
bryzgaloff opened a new pull request #10546:
URL: https://github.com/apache/airflow/pull/10546
Added support for `extra_hosts` argument ([docs](https://docker-py.readthedocs.io/en/stable/containers.html?highlight=extra_hosts#docker.models.containers.ContainerCollection.run)) which is equivalent to `--add-host` flag for `docker run` command ([docs](https://docs.docker.com/engine/reference/commandline/run/#:~:text=Add%20entries%20to%20container%20hosts%20file%20(%2D%2Dadd%2Dhost)&text=Sometimes%20you%20need%20to%20connect,the%20ip%20addr%20show%20command.)).
Covered with tests. Also includes minor enhancements to `DockerOperator` tests: common code is moved to a `setUp` method which makes code cleaner and reduces duplication.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #10546: DockerOperator extra_hosts argument support added
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #10546:
URL: https://github.com/apache/airflow/pull/10546#discussion_r476363932
##########
File path: tests/providers/docker/operators/test_docker.py
##########
@@ -32,23 +32,32 @@
class TestDockerOperator(unittest.TestCase):
- @mock.patch('airflow.providers.docker.operators.docker.TemporaryDirectory')
- @mock.patch('airflow.providers.docker.operators.docker.APIClient')
- def test_execute(self, client_class_mock, tempdir_mock):
- host_config = mock.Mock()
- tempdir_mock.return_value.__enter__.return_value = '/mkdtemp'
-
- client_mock = mock.Mock(spec=APIClient)
- client_mock.create_container.return_value = {'Id': 'some_id'}
- client_mock.create_host_config.return_value = host_config
- client_mock.images.return_value = []
- client_mock.attach.return_value = ['container log']
- client_mock.logs.return_value = ['container log']
- client_mock.pull.return_value = {"status": "pull log"}
- client_mock.wait.return_value = {"StatusCode": 0}
+ def setUp(self):
+ self.tempdir_patch = mock.patch(
+ 'airflow.providers.docker.operators.docker.TemporaryDirectory',
+ )
Review comment:
```suggestion
self.tempdir_patcher = mock.patch(
'airflow.providers.docker.operators.docker.TemporaryDirectory',
).patcher()
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bryzgaloff commented on a change in pull request #10546: DockerOperator extra_hosts argument support added
Posted by GitBox <gi...@apache.org>.
bryzgaloff commented on a change in pull request #10546:
URL: https://github.com/apache/airflow/pull/10546#discussion_r476442034
##########
File path: tests/providers/docker/operators/test_docker.py
##########
@@ -32,23 +32,32 @@
class TestDockerOperator(unittest.TestCase):
- @mock.patch('airflow.providers.docker.operators.docker.TemporaryDirectory')
- @mock.patch('airflow.providers.docker.operators.docker.APIClient')
- def test_execute(self, client_class_mock, tempdir_mock):
- host_config = mock.Mock()
- tempdir_mock.return_value.__enter__.return_value = '/mkdtemp'
-
- client_mock = mock.Mock(spec=APIClient)
- client_mock.create_container.return_value = {'Id': 'some_id'}
- client_mock.create_host_config.return_value = host_config
- client_mock.images.return_value = []
- client_mock.attach.return_value = ['container log']
- client_mock.logs.return_value = ['container log']
- client_mock.pull.return_value = {"status": "pull log"}
- client_mock.wait.return_value = {"StatusCode": 0}
+ def setUp(self):
+ self.tempdir_patch = mock.patch(
+ 'airflow.providers.docker.operators.docker.TemporaryDirectory',
+ )
Review comment:
Thank you, sir, worked perfectly! There will never come a day when I fully read the mock documentation, always find smth new there :D
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bryzgaloff commented on pull request #10546: DockerOperator extra_hosts argument support added
Posted by GitBox <gi...@apache.org>.
bryzgaloff commented on pull request #10546:
URL: https://github.com/apache/airflow/pull/10546#issuecomment-681872124
Woooo, guys, thank you, it's an awesome experience ^_^
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bryzgaloff commented on pull request #10546: DockerOperator extra_hosts argument support added
Posted by GitBox <gi...@apache.org>.
bryzgaloff commented on pull request #10546:
URL: https://github.com/apache/airflow/pull/10546#issuecomment-681370371
Rerun black to pass CI/CD check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #10546: DockerOperator extra_hosts argument support added
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #10546:
URL: https://github.com/apache/airflow/pull/10546#issuecomment-681868000
Awesome work, congrats on your first merged pull request!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #10546: DockerOperator extra_hosts argument support added
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #10546:
URL: https://github.com/apache/airflow/pull/10546#discussion_r477211678
##########
File path: airflow/providers/docker/operators/docker.py
##########
@@ -167,7 +167,8 @@ def __init__(
shm_size: Optional[int] = None,
tty: Optional[bool] = False,
cap_add: Optional[Iterable[str]] = None,
- **kwargs,
+ extra_hosts: Optional[Dict[str, str]] = None,
+ **kwargs,
Review comment:
Invalid indentation?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #10546: DockerOperator extra_hosts argument support added
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #10546:
URL: https://github.com/apache/airflow/pull/10546#issuecomment-681872936
Good work @bryzgaloff , hope to see more of your contributions and thank you for this PR :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #10546: DockerOperator extra_hosts argument support added
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #10546:
URL: https://github.com/apache/airflow/pull/10546#discussion_r477537654
##########
File path: tests/providers/docker/operators/test_docker.py
##########
@@ -298,3 +249,22 @@ def test_execute_xcom_behavior(self, client_class_mock, tempdir_mock):
self.assertEqual(xcom_push_result, b'container log')
self.assertIs(no_xcom_push_result, None)
+
+ def test_extra_hosts(self):
+ hosts_obj = mock.Mock()
+ operator = \
+ DockerOperator(task_id='test', image='test', extra_hosts=hosts_obj)
+ operator.execute(None)
+ self.client_mock.create_container.assert_called_once()
+ self.assertIn(
+ 'host_config',
+ self.client_mock.create_container.call_args.kwargs,
+ )
+ self.assertIn(
+ 'extra_hosts',
+ self.client_mock.create_host_config.call_args.kwargs,
+ )
+ self.assertIs(
+ hosts_obj,
+ self.client_mock.create_host_config.call_args.kwargs['extra_hosts'],
+ )
Review comment:
Instead of passing mock object to `hosts_obj` why not pass a dictionary and check that it is used in mock_call
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bryzgaloff commented on pull request #10546: DockerOperator extra_hosts argument support added
Posted by GitBox <gi...@apache.org>.
bryzgaloff commented on pull request #10546:
URL: https://github.com/apache/airflow/pull/10546#issuecomment-680806395
@kaxil rebased. I have also fixed up codereview-related commit. So the code is now ready to be merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bryzgaloff commented on a change in pull request #10546: DockerOperator extra_hosts argument support added
Posted by GitBox <gi...@apache.org>.
bryzgaloff commented on a change in pull request #10546:
URL: https://github.com/apache/airflow/pull/10546#discussion_r477229528
##########
File path: airflow/providers/docker/operators/docker.py
##########
@@ -167,7 +167,8 @@ def __init__(
shm_size: Optional[int] = None,
tty: Optional[bool] = False,
cap_add: Optional[Iterable[str]] = None,
- **kwargs,
+ extra_hosts: Optional[Dict[str, str]] = None,
+ **kwargs,
Review comment:
Yes, fixed and also rerun black on test code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bryzgaloff commented on a change in pull request #10546: DockerOperator extra_hosts argument support added
Posted by GitBox <gi...@apache.org>.
bryzgaloff commented on a change in pull request #10546:
URL: https://github.com/apache/airflow/pull/10546#discussion_r478076977
##########
File path: tests/providers/docker/operators/test_docker.py
##########
@@ -298,3 +249,22 @@ def test_execute_xcom_behavior(self, client_class_mock, tempdir_mock):
self.assertEqual(xcom_push_result, b'container log')
self.assertIs(no_xcom_push_result, None)
+
+ def test_extra_hosts(self):
+ hosts_obj = mock.Mock()
+ operator = \
+ DockerOperator(task_id='test', image='test', extra_hosts=hosts_obj)
+ operator.execute(None)
+ self.client_mock.create_container.assert_called_once()
+ self.assertIn(
+ 'host_config',
+ self.client_mock.create_container.call_args.kwargs,
+ )
+ self.assertIn(
+ 'extra_hosts',
+ self.client_mock.create_host_config.call_args.kwargs,
+ )
+ self.assertIs(
+ hosts_obj,
+ self.client_mock.create_host_config.call_args.kwargs['extra_hosts'],
+ )
Review comment:
Because we do not need to know which type `extra_hosts` argument is of.just need to check that it is successfully passed to `create_host_config` call whatever the type is.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil merged pull request #10546: DockerOperator extra_hosts argument support added
Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #10546:
URL: https://github.com/apache/airflow/pull/10546
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #10546: DockerOperator extra_hosts argument support added
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #10546:
URL: https://github.com/apache/airflow/pull/10546#discussion_r476363932
##########
File path: tests/providers/docker/operators/test_docker.py
##########
@@ -32,23 +32,32 @@
class TestDockerOperator(unittest.TestCase):
- @mock.patch('airflow.providers.docker.operators.docker.TemporaryDirectory')
- @mock.patch('airflow.providers.docker.operators.docker.APIClient')
- def test_execute(self, client_class_mock, tempdir_mock):
- host_config = mock.Mock()
- tempdir_mock.return_value.__enter__.return_value = '/mkdtemp'
-
- client_mock = mock.Mock(spec=APIClient)
- client_mock.create_container.return_value = {'Id': 'some_id'}
- client_mock.create_host_config.return_value = host_config
- client_mock.images.return_value = []
- client_mock.attach.return_value = ['container log']
- client_mock.logs.return_value = ['container log']
- client_mock.pull.return_value = {"status": "pull log"}
- client_mock.wait.return_value = {"StatusCode": 0}
+ def setUp(self):
+ self.tempdir_patch = mock.patch(
+ 'airflow.providers.docker.operators.docker.TemporaryDirectory',
+ )
Review comment:
```suggestion
self.tempdir_patcher = mock.patch(
'airflow.providers.docker.operators.docker.TemporaryDirectory',
).patcher()
self.tempdir_patch = self.tempdir_patcher.start()
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #10546: DockerOperator extra_hosts argument support added
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #10546:
URL: https://github.com/apache/airflow/pull/10546#discussion_r476365583
##########
File path: tests/providers/docker/operators/test_docker.py
##########
@@ -32,23 +32,32 @@
class TestDockerOperator(unittest.TestCase):
- @mock.patch('airflow.providers.docker.operators.docker.TemporaryDirectory')
- @mock.patch('airflow.providers.docker.operators.docker.APIClient')
- def test_execute(self, client_class_mock, tempdir_mock):
- host_config = mock.Mock()
- tempdir_mock.return_value.__enter__.return_value = '/mkdtemp'
-
- client_mock = mock.Mock(spec=APIClient)
- client_mock.create_container.return_value = {'Id': 'some_id'}
- client_mock.create_host_config.return_value = host_config
- client_mock.images.return_value = []
- client_mock.attach.return_value = ['container log']
- client_mock.logs.return_value = ['container log']
- client_mock.pull.return_value = {"status": "pull log"}
- client_mock.wait.return_value = {"StatusCode": 0}
+ def setUp(self):
+ self.tempdir_patch = mock.patch(
+ 'airflow.providers.docker.operators.docker.TemporaryDirectory',
+ )
Review comment:
See: https://docs.python.org/3/library/unittest.mock.html#patch-methods-start-and-stop
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #10546: Feature/docker operator extra hosts
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #10546:
URL: https://github.com/apache/airflow/pull/10546#issuecomment-679951908
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, itโs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better ๐.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://apache-airflow-slack.herokuapp.com/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #10546: DockerOperator extra_hosts argument support added
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #10546:
URL: https://github.com/apache/airflow/pull/10546#discussion_r476364282
##########
File path: tests/providers/docker/operators/test_docker.py
##########
@@ -32,23 +32,32 @@
class TestDockerOperator(unittest.TestCase):
- @mock.patch('airflow.providers.docker.operators.docker.TemporaryDirectory')
- @mock.patch('airflow.providers.docker.operators.docker.APIClient')
- def test_execute(self, client_class_mock, tempdir_mock):
- host_config = mock.Mock()
- tempdir_mock.return_value.__enter__.return_value = '/mkdtemp'
-
- client_mock = mock.Mock(spec=APIClient)
- client_mock.create_container.return_value = {'Id': 'some_id'}
- client_mock.create_host_config.return_value = host_config
- client_mock.images.return_value = []
- client_mock.attach.return_value = ['container log']
- client_mock.logs.return_value = ['container log']
- client_mock.pull.return_value = {"status": "pull log"}
- client_mock.wait.return_value = {"StatusCode": 0}
+ def setUp(self):
+ self.tempdir_patch = mock.patch(
+ 'airflow.providers.docker.operators.docker.TemporaryDirectory',
+ )
+ self.tempdir_mock = self.tempdir_patch.__enter__()
+ self.tempdir_mock.return_value.__enter__.return_value = '/mkdtemp'
+
+ self.client_mock = mock.Mock(spec=APIClient)
+ self.client_mock.create_container.return_value = {'Id': 'some_id'}
+ self.client_mock.images.return_value = []
+ self.client_mock.attach.return_value = ['container log']
+ self.client_mock.logs.return_value = ['container log']
+ self.client_mock.pull.return_value = {"status": "pull log"}
+ self.client_mock.wait.return_value = {"StatusCode": 0}
+ self.client_mock.create_host_config.return_value = mock.Mock()
+ self.client_class_patch = mock.patch(
+ 'airflow.providers.docker.operators.docker.APIClient',
+ return_value=self.client_mock,
+ )
+ self.client_class_mock = self.client_class_patch.__enter__()
- client_class_mock.return_value = client_mock
+ def tearDown(self) -> None:
+ self.tempdir_patch.__exit__(None, None, None)
Review comment:
```suggestion
self.tempdir_patcher.stop()
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bryzgaloff commented on pull request #10546: DockerOperator extra_hosts argument support added
Posted by GitBox <gi...@apache.org>.
bryzgaloff commented on pull request #10546:
URL: https://github.com/apache/airflow/pull/10546#issuecomment-681861579
@kaxil @mik-laj will you please leave your review or give an approval? All checks have passed, all the comments are resolved.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org