You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/19 15:02:34 UTC
[GitHub] [airflow] Aakcht opened a new pull request #19711: hdfs provider: fix HA support for webhdfs
Aakcht opened a new pull request #19711:
URL: https://github.com/apache/airflow/pull/19711
HA support for webHDFS was actually added in #7454 , but stopped working when conn_id became unique in #8608
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #19711: hdfs provider: fix HA support for webhdfs
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #19711:
URL: https://github.com/apache/airflow/pull/19711#discussion_r755367602
##########
File path: airflow/providers/apache/hdfs/hooks/webhdfs.py
##########
@@ -71,30 +71,31 @@ def get_conn(self) -> Any:
def _find_valid_server(self) -> Any:
connection = self.get_connection(self.webhdfs_conn_id)
- host_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
- self.log.info("Trying to connect to %s:%s", connection.host, connection.port)
- try:
- conn_check = host_socket.connect_ex((connection.host, connection.port))
- if conn_check == 0:
- self.log.info('Trying namenode %s', connection.host)
- client = self._get_client(connection)
- client.status('/')
- self.log.info('Using namenode %s for hook', connection.host)
- host_socket.close()
- return client
- else:
- self.log.error("Could not connect to %s:%s", connection.host, connection.port)
- host_socket.close()
- except HdfsError as hdfs_error:
- self.log.error('Read operation on namenode %s failed with error: %s', connection.host, hdfs_error)
+ namenodes = connection.host.split(',')
+ for namenode in namenodes:
+ host_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+ self.log.info("Trying to connect to %s:%s", namenode, connection.port)
+ try:
+ conn_check = host_socket.connect_ex((namenode, connection.port))
+ if conn_check == 0:
+ self.log.info('Trying namenode %s', namenode)
+ client = self._get_client(connection, namenode)
+ client.status('/')
+ self.log.info('Using namenode %s for hook', namenode)
+ host_socket.close()
+ return client
+ else:
+ self.log.error("Could not connect to %s:%s", namenode, connection.port)
Review comment:
If only one connection is needed, would it be better to have this as a `info` log instead, or even lower?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #19711: hdfs provider: fix HA support for webhdfs
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #19711:
URL: https://github.com/apache/airflow/pull/19711#discussion_r755368679
##########
File path: airflow/providers/apache/hdfs/hooks/webhdfs.py
##########
@@ -71,30 +71,31 @@ def get_conn(self) -> Any:
def _find_valid_server(self) -> Any:
connection = self.get_connection(self.webhdfs_conn_id)
- host_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
- self.log.info("Trying to connect to %s:%s", connection.host, connection.port)
- try:
- conn_check = host_socket.connect_ex((connection.host, connection.port))
- if conn_check == 0:
- self.log.info('Trying namenode %s', connection.host)
- client = self._get_client(connection)
- client.status('/')
- self.log.info('Using namenode %s for hook', connection.host)
- host_socket.close()
- return client
- else:
- self.log.error("Could not connect to %s:%s", connection.host, connection.port)
- host_socket.close()
- except HdfsError as hdfs_error:
- self.log.error('Read operation on namenode %s failed with error: %s', connection.host, hdfs_error)
+ namenodes = connection.host.split(',')
+ for namenode in namenodes:
+ host_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+ self.log.info("Trying to connect to %s:%s", namenode, connection.port)
+ try:
+ conn_check = host_socket.connect_ex((namenode, connection.port))
+ if conn_check == 0:
+ self.log.info('Trying namenode %s', namenode)
+ client = self._get_client(connection, namenode)
+ client.status('/')
+ self.log.info('Using namenode %s for hook', namenode)
+ host_socket.close()
+ return client
+ else:
+ self.log.error("Could not connect to %s:%s", namenode, connection.port)
+ except HdfsError as hdfs_error:
+ self.log.error('Read operation on namenode %s failed with error: %s', namenode, hdfs_error)
return None
- def _get_client(self, connection: Connection) -> Any:
- connection_str = f'http://{connection.host}:{connection.port}'
+ def _get_client(self, connection: Connection, namenode: str) -> Any:
Review comment:
It may be better to just pass in `namenode`, `port`, and `extras` separately instead, to avoid `connection.host` being incorrectly used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19711: hdfs provider: fix HA support for webhdfs
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #19711:
URL: https://github.com/apache/airflow/pull/19711#issuecomment-977024904
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Aakcht commented on pull request #19711: hdfs provider: fix HA support for webhdfs
Posted by GitBox <gi...@apache.org>.
Aakcht commented on pull request #19711:
URL: https://github.com/apache/airflow/pull/19711#issuecomment-977079387
I've made the suggested changes and all the checks are now passing :smile: Please tell me if anything else is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Aakcht commented on pull request #19711: hdfs provider: fix HA support for webhdfs
Posted by GitBox <gi...@apache.org>.
Aakcht commented on pull request #19711:
URL: https://github.com/apache/airflow/pull/19711#issuecomment-976886892
If needed, I can add a new test specifically for HA case, but I think it should be enough that I've modified one of the old ones for multiple hosts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #19711: hdfs provider: fix HA support for webhdfs
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #19711:
URL: https://github.com/apache/airflow/pull/19711
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org