You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/06 21:05:15 UTC
[GitHub] [airflow] subkanthi opened a new pull request #22027: Fixed dask executor and tests
subkanthi opened a new pull request #22027:
URL: https://github.com/apache/airflow/pull/22027
Fixed dask executor and tests.
Distributed pip package does not ship with tests folder anymore.
And there is a python script in distributed/tests to create the certificates.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #22027: WIP - Fixed dask executor and tests
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #22027:
URL: https://github.com/apache/airflow/pull/22027#discussion_r820316014
##########
File path: tests/executors/test_dask_executor.py
##########
@@ -121,27 +120,27 @@ def setUp(self):
@conf_vars(
{
- ('dask', 'tls_ca'): get_cert('tls-ca-cert.pem'),
- ('dask', 'tls_cert'): get_cert('tls-key-cert.pem'),
- ('dask', 'tls_key'): get_cert('tls-key.pem'),
+ ('dask', 'tls_ca'): 'certs/tls-ca-cert.pem',
+ ('dask', 'tls_cert'): 'certs/tls-key-cert.pem',
+ ('dask', 'tls_key'): 'certs/tls-key.pem',
}
)
- def test_tls(self):
- # These use test certs that ship with dask/distributed and should not be
- # used in production
- with dask_testing_cluster(
- worker_kwargs={'security': tls_security(), "protocol": "tls"},
Review comment:
Let's see. They did not for me when I run them previously.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22027: Fixed dask executor and tests
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22027:
URL: https://github.com/apache/airflow/pull/22027#issuecomment-1061907349
Wohooo!! Thanks @subkanthi !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #22027: Fixed dask executor and tests
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #22027:
URL: https://github.com/apache/airflow/pull/22027#discussion_r821173665
##########
File path: tests/executors/test_dask_executor.py
##########
@@ -121,27 +120,27 @@ def setUp(self):
@conf_vars(
{
- ('dask', 'tls_ca'): get_cert('tls-ca-cert.pem'),
- ('dask', 'tls_cert'): get_cert('tls-key-cert.pem'),
- ('dask', 'tls_key'): get_cert('tls-key.pem'),
+ ('dask', 'tls_ca'): 'certs/tls-ca-cert.pem',
+ ('dask', 'tls_cert'): 'certs/tls-key-cert.pem',
+ ('dask', 'tls_key'): 'certs/tls-key.pem',
}
)
- def test_tls(self):
- # These use test certs that ship with dask/distributed and should not be
- # used in production
- with dask_testing_cluster(
- worker_kwargs={'security': tls_security(), "protocol": "tls"},
Review comment:
Cool @subkanthi ! Looks good. I think you can bring back the tests now - in my change (when you rebase) you will see how I dealt with get_cert to skip those
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #22027: Fixed dask executor and tests
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #22027:
URL: https://github.com/apache/airflow/pull/22027#discussion_r821635422
##########
File path: tests/executors/test_dask_executor.py
##########
@@ -121,27 +120,27 @@ def setUp(self):
@conf_vars(
{
- ('dask', 'tls_ca'): get_cert('tls-ca-cert.pem'),
- ('dask', 'tls_cert'): get_cert('tls-key-cert.pem'),
- ('dask', 'tls_key'): get_cert('tls-key.pem'),
+ ('dask', 'tls_ca'): 'certs/tls-ca-cert.pem',
+ ('dask', 'tls_cert'): 'certs/tls-key-cert.pem',
+ ('dask', 'tls_key'): 'certs/tls-key.pem',
}
)
- def test_tls(self):
- # These use test certs that ship with dask/distributed and should not be
- # used in production
- with dask_testing_cluster(
- worker_kwargs={'security': tls_security(), "protocol": "tls"},
Review comment:
I think you cna ucomment your line comments now :).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on a change in pull request #22027: WIP - Fixed dask executor and tests
Posted by GitBox <gi...@apache.org>.
subkanthi commented on a change in pull request #22027:
URL: https://github.com/apache/airflow/pull/22027#discussion_r820337396
##########
File path: tests/executors/test_dask_executor.py
##########
@@ -121,27 +120,27 @@ def setUp(self):
@conf_vars(
{
- ('dask', 'tls_ca'): get_cert('tls-ca-cert.pem'),
- ('dask', 'tls_cert'): get_cert('tls-key-cert.pem'),
- ('dask', 'tls_key'): get_cert('tls-key.pem'),
+ ('dask', 'tls_ca'): 'certs/tls-ca-cert.pem',
+ ('dask', 'tls_cert'): 'certs/tls-key-cert.pem',
+ ('dask', 'tls_key'): 'certs/tls-key.pem',
}
)
- def test_tls(self):
- # These use test certs that ship with dask/distributed and should not be
- # used in production
- with dask_testing_cluster(
- worker_kwargs={'security': tls_security(), "protocol": "tls"},
Review comment:
Atleast on my local, most of the tests were failing because of errors with pickling the function, the change in [airflow/executors/dask_executor.py], fixed it
- future = self.client.submit(airflow_run, pure=False, resources=resources)
+ future = self.client.submit(subprocess.check_call, command, pure=False, resources=resources)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22027: WIP - Fixed dask executor and tests
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22027:
URL: https://github.com/apache/airflow/pull/22027#issuecomment-1060535403
Can you please rebase @subkanthi - I have just updated the constraints.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22027: WIP - Fixed dask executor and tests
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22027:
URL: https://github.com/apache/airflow/pull/22027#issuecomment-1060043967
Ah, nice. If we can make it works that would be cool :).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #22027: Fixed dask executor and tests
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #22027:
URL: https://github.com/apache/airflow/pull/22027
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on a change in pull request #22027: Fixed dask executor and tests
Posted by GitBox <gi...@apache.org>.
subkanthi commented on a change in pull request #22027:
URL: https://github.com/apache/airflow/pull/22027#discussion_r820286890
##########
File path: airflow/executors/dask_executor.py
##########
@@ -96,7 +96,7 @@ def airflow_run():
raise AirflowException(f"Attempted to submit task to an unavailable queue: '{queue}'")
resources = {queue: 1}
- future = self.client.submit(airflow_run, pure=False, resources=resources)
+ future = self.client.submit(subprocess.check_call, command, pure=False, resources=resources)
Review comment:
Related to #11451
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22027: WIP - Fixed dask executor and tests
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22027:
URL: https://github.com/apache/airflow/pull/22027#issuecomment-1060119416
I think the problem is that constraints have not been refreshed yet. Let's try tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on a change in pull request #22027: WIP - Fixed dask executor and tests
Posted by GitBox <gi...@apache.org>.
subkanthi commented on a change in pull request #22027:
URL: https://github.com/apache/airflow/pull/22027#discussion_r820311960
##########
File path: tests/executors/test_dask_executor.py
##########
@@ -121,27 +120,27 @@ def setUp(self):
@conf_vars(
{
- ('dask', 'tls_ca'): get_cert('tls-ca-cert.pem'),
- ('dask', 'tls_cert'): get_cert('tls-key-cert.pem'),
- ('dask', 'tls_key'): get_cert('tls-key.pem'),
+ ('dask', 'tls_ca'): 'certs/tls-ca-cert.pem',
+ ('dask', 'tls_cert'): 'certs/tls-key-cert.pem',
+ ('dask', 'tls_key'): 'certs/tls-key.pem',
}
)
- def test_tls(self):
- # These use test certs that ship with dask/distributed and should not be
- # used in production
- with dask_testing_cluster(
- worker_kwargs={'security': tls_security(), "protocol": "tls"},
Review comment:
tls_security in `utils_test.py` part of `distributed` seems to look for the certs in the tests/ path
`path = os.path.join(certs_dir, filename)`
For now commenting out if its OK, the other tests seem to pass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #22027: Fixed dask executor and tests
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #22027:
URL: https://github.com/apache/airflow/pull/22027#issuecomment-1061858270
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org