You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/08/16 17:30:39 UTC
[GitHub] [airflow] Aakcht opened a new pull request #17637: hdfs provider: allow SSL webhdfs connections
Aakcht opened a new pull request #17637:
URL: https://github.com/apache/airflow/pull/17637
Allows webhdfs hook to work with SSL enabled HDFS - I added two parameters in hdfs connection `extra` field for configuring SSL. If this solution looks ok, I'll try adding the tests.
closes: #16651
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Aakcht commented on a change in pull request #17637: hdfs provider: allow SSL webhdfs connections
Posted by GitBox <gi...@apache.org>.
Aakcht commented on a change in pull request #17637:
URL: https://github.com/apache/airflow/pull/17637#discussion_r690381098
##########
File path: tests/providers/apache/hdfs/hooks/test_webhdfs.py
##########
@@ -67,7 +68,7 @@ def test_get_conn_kerberos_security_mode(
conn = self.webhdfs_hook.get_conn()
connection = mock_get_connections.return_value[0]
- mock_kerberos_client.assert_called_once_with(f'http://{connection.host}:{connection.port}')
+ mock_kerberos_client.assert_called_once_with(f'http://{connection.host}:{connection.port}', session=None)
Review comment:
Added some tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #17637: hdfs provider: allow SSL webhdfs connections
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #17637:
URL: https://github.com/apache/airflow/pull/17637#discussion_r689755902
##########
File path: tests/providers/apache/hdfs/hooks/test_webhdfs.py
##########
@@ -67,7 +68,7 @@ def test_get_conn_kerberos_security_mode(
conn = self.webhdfs_hook.get_conn()
connection = mock_get_connections.return_value[0]
- mock_kerberos_client.assert_called_once_with(f'http://{connection.host}:{connection.port}')
+ mock_kerberos_client.assert_called_once_with(f'http://{connection.host}:{connection.port}', session=None)
Review comment:
Could you please add test cases with "use_ssl" and `verify`? (Maybe a @parameterized) test even
##########
File path: airflow/providers/apache/hdfs/hooks/webhdfs.py
##########
@@ -93,12 +93,19 @@ def _find_valid_server(self) -> Any:
def _get_client(self, connection: Connection) -> Any:
connection_str = f'http://{connection.host}:{connection.port}'
+ session = None
+
+ if connection.extra_dejson.get('SSL', False):
Review comment:
small NIT. Can we please change it to "use_ssl" instead of 'SSL' ? We are mostly using lowercase for extra names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Aakcht commented on pull request #17637: hdfs provider: allow SSL webhdfs connections
Posted by GitBox <gi...@apache.org>.
Aakcht commented on pull request #17637:
URL: https://github.com/apache/airflow/pull/17637#issuecomment-900867939
Hello, @potiuk ,any updates on this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Aakcht commented on pull request #17637: hdfs provider: allow SSL webhdfs connections
Posted by GitBox <gi...@apache.org>.
Aakcht commented on pull request #17637:
URL: https://github.com/apache/airflow/pull/17637#issuecomment-900867939
Hello, @potiuk ,any updates on this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Aakcht commented on a change in pull request #17637: hdfs provider: allow SSL webhdfs connections
Posted by GitBox <gi...@apache.org>.
Aakcht commented on a change in pull request #17637:
URL: https://github.com/apache/airflow/pull/17637#discussion_r690380865
##########
File path: airflow/providers/apache/hdfs/hooks/webhdfs.py
##########
@@ -93,12 +93,19 @@ def _find_valid_server(self) -> Any:
def _get_client(self, connection: Connection) -> Any:
connection_str = f'http://{connection.host}:{connection.port}'
+ session = None
+
+ if connection.extra_dejson.get('SSL', False):
Review comment:
changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #17637: hdfs provider: allow SSL webhdfs connections
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #17637:
URL: https://github.com/apache/airflow/pull/17637#discussion_r689755902
##########
File path: tests/providers/apache/hdfs/hooks/test_webhdfs.py
##########
@@ -67,7 +68,7 @@ def test_get_conn_kerberos_security_mode(
conn = self.webhdfs_hook.get_conn()
connection = mock_get_connections.return_value[0]
- mock_kerberos_client.assert_called_once_with(f'http://{connection.host}:{connection.port}')
+ mock_kerberos_client.assert_called_once_with(f'http://{connection.host}:{connection.port}', session=None)
Review comment:
Could you please add test cases with "use_ssl" and `verify`? (Maybe a `@parameterized`) test even
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #17637: hdfs provider: allow SSL webhdfs connections
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #17637:
URL: https://github.com/apache/airflow/pull/17637
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #17637: hdfs provider: allow SSL webhdfs connections
Posted by GitBox <gi...@apache.org>.
uranusjr commented on pull request #17637:
URL: https://github.com/apache/airflow/pull/17637#issuecomment-900035976
I think it’d be more straightforward to unconditionally create a `requests.Session` instead. This is the default behaviour if you don’t explicitly pass a session object to the client anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org