You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/06/18 21:09:34 UTC
[GitHub] [airflow] pgagnon opened a new pull request #16512: Add extra_headers argument to LivyHook and LivyOperator
pgagnon opened a new pull request #16512:
URL: https://github.com/apache/airflow/pull/16512
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed pull request #16512: Add extra_headers argument to LivyHook and LivyOperator
Posted by GitBox <gi...@apache.org>.
kaxil closed pull request #16512:
URL: https://github.com/apache/airflow/pull/16512
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil merged pull request #16512: Add extra_headers argument to LivyHook and LivyOperator
Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #16512:
URL: https://github.com/apache/airflow/pull/16512
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pgagnon commented on pull request #16512: Add extra_headers argument to LivyHook and LivyOperator
Posted by GitBox <gi...@apache.org>.
pgagnon commented on pull request #16512:
URL: https://github.com/apache/airflow/pull/16512#issuecomment-863633922
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #16512: Add extra_headers argument to LivyHook and LivyOperator
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #16512:
URL: https://github.com/apache/airflow/pull/16512#discussion_r653928131
##########
File path: tests/providers/apache/livy/hooks/test_livy.py
##########
@@ -316,6 +316,20 @@ def test_invalid_uri(self):
with pytest.raises(RequestException):
hook.post_batch(file='sparkapp')
+ @requests_mock.mock()
+ def test_extra_headers(self, mock):
+ mock.register_uri(
+ 'POST',
+ '//livy:8998/batches',
+ json={'id': BATCH_ID, 'state': BatchState.STARTING.value, 'log': []},
+ status_code=201,
+ request_headers={'X-Requested-By': 'user'}
+ )
+
+ hook = LivyHook(extra_headers={'X-Requested-By': 'user'})
+ hook.post_batch(file='sparkapp')
+
Review comment:
What does this test check?
##########
File path: tests/providers/apache/livy/hooks/test_livy.py
##########
@@ -316,6 +316,20 @@ def test_invalid_uri(self):
with pytest.raises(RequestException):
hook.post_batch(file='sparkapp')
+ @requests_mock.mock()
+ def test_extra_headers(self, mock):
+ mock.register_uri(
+ 'POST',
+ '//livy:8998/batches',
+ json={'id': BATCH_ID, 'state': BatchState.STARTING.value, 'log': []},
+ status_code=201,
+ request_headers={'X-Requested-By': 'user'}
+ )
+
+ hook = LivyHook(extra_headers={'X-Requested-By': 'user'})
+ hook.post_batch(file='sparkapp')
+
Review comment:
Tested it locally, works fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pgagnon closed pull request #16512: Add extra_headers argument to LivyHook and LivyOperator
Posted by GitBox <gi...@apache.org>.
pgagnon closed pull request #16512:
URL: https://github.com/apache/airflow/pull/16512
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #16512: Add extra_headers argument to LivyHook and LivyOperator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #16512:
URL: https://github.com/apache/airflow/pull/16512#issuecomment-863577153
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pgagnon commented on a change in pull request #16512: Add extra_headers argument to LivyHook and LivyOperator
Posted by GitBox <gi...@apache.org>.
pgagnon commented on a change in pull request #16512:
URL: https://github.com/apache/airflow/pull/16512#discussion_r653938595
##########
File path: tests/providers/apache/livy/hooks/test_livy.py
##########
@@ -316,6 +316,20 @@ def test_invalid_uri(self):
with pytest.raises(RequestException):
hook.post_batch(file='sparkapp')
+ @requests_mock.mock()
+ def test_extra_headers(self, mock):
+ mock.register_uri(
+ 'POST',
+ '//livy:8998/batches',
+ json={'id': BATCH_ID, 'state': BatchState.STARTING.value, 'log': []},
+ status_code=201,
+ request_headers={'X-Requested-By': 'user'}
+ )
+
+ hook = LivyHook(extra_headers={'X-Requested-By': 'user'})
+ hook.post_batch(file='sparkapp')
+
Review comment:
It checks that `X-Requested-By: user` is present in the request headers.
##########
File path: tests/providers/apache/livy/hooks/test_livy.py
##########
@@ -316,6 +316,20 @@ def test_invalid_uri(self):
with pytest.raises(RequestException):
hook.post_batch(file='sparkapp')
+ @requests_mock.mock()
+ def test_extra_headers(self, mock):
+ mock.register_uri(
+ 'POST',
+ '//livy:8998/batches',
+ json={'id': BATCH_ID, 'state': BatchState.STARTING.value, 'log': []},
+ status_code=201,
+ request_headers={'X-Requested-By': 'user'}
+ )
+
+ hook = LivyHook(extra_headers={'X-Requested-By': 'user'})
+ hook.post_batch(file='sparkapp')
+
Review comment:
It checks that `X-Requested-By: user` is present in the request headers.
edit: ref: https://requests-mock.readthedocs.io/en/latest/matching.html#request-headers
##########
File path: tests/providers/apache/livy/hooks/test_livy.py
##########
@@ -255,7 +255,7 @@ def test_post_batch_arguments(self, mock_request):
resp = hook.post_batch(file='sparkapp')
mock_request.assert_called_once_with(
- method='POST', endpoint='/batches', data=json.dumps({'file': 'sparkapp'})
+ method='POST', endpoint='/batches', data=json.dumps({'file': 'sparkapp'}), headers={}
Review comment:
@kaxil I modified this test slightly. Just wanted to explicitely point it out.
##########
File path: tests/providers/apache/livy/hooks/test_livy.py
##########
@@ -255,7 +255,7 @@ def test_post_batch_arguments(self, mock_request):
resp = hook.post_batch(file='sparkapp')
mock_request.assert_called_once_with(
- method='POST', endpoint='/batches', data=json.dumps({'file': 'sparkapp'})
+ method='POST', endpoint='/batches', data=json.dumps({'file': 'sparkapp'}), headers={}
Review comment:
@kaxil I modified this test slightly. Just wanted to explicitly point it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #16512: Add extra_headers argument to LivyHook and LivyOperator
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #16512:
URL: https://github.com/apache/airflow/pull/16512#issuecomment-863598316
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org