You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/06/03 08:44:37 UTC
[GitHub] [airflow] ashb opened a new issue #9118: Improve/surface errors when attempting to read S3 logs
ashb opened a new issue #9118:
URL: https://github.com/apache/airflow/issues/9118
**Description**
As mentioned in #8212 if you have configured S3 logs, but there is a problem then this is never surfaced to the UI (nor the webserver logs) making this very hard to debug.
All you see in the UI is this:
```
*** Log file does not exist: /usr/local/airflow/logs/MY_DAG_NAME/MY_TASK_NAME/2020-04-07T20:59:19.312402+00:00/6.log
*** Fetching from: http://MY_DAG_NAME-0dde5ff5a786437cb14234:8793/log/MY_DAG_NAME/MY_TASK_NAME/2020-04-07T20:59:19.312402+00:00/6.log
*** Failed to fetch log file from worker. HTTPConnectionPool(host='MY_DAG_NAME-0dde5ff5a786437cb14234', port=8793): Max retries exceeded with url: /log/MY_DAG_NAME/MY_TASK_NAME/2020-04-07T20:59:19.312402+00:00/6.log (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f708332fc90>: Failed to establish a new connection: [Errno -2] Name or service not known'))
```
In one such case I was debugging, I found this error when attempting to communicate with S3:
```python console
>>> c.list_buckets()
[2020-06-03 08:27:24,662] {connectionpool.py:735} INFO - Starting new HTTPS connection (1): s3.amazonaws.com
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.7/site-packages/botocore/client.py", line 314, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/lib/python3.7/site-packages/botocore/client.py", line 612, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation: The AWS Access Key Id you provided does not exist in our records.
```
We should add a `***` line showing we attempted to fetch the logs form S3, and the error _that it failed at least_. Right now on error the S3TaskHandler is totally silent in case of error. This is bad.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Siddharthk commented on issue #9118: Improve/surface errors when attempting to read S3 logs
Posted by GitBox <gi...@apache.org>.
Siddharthk commented on issue #9118:
URL: https://github.com/apache/airflow/issues/9118#issuecomment-664914574
@ashb @JPonte I am getting below error. Looks like a bug:
```
>>> from airflow.configuration import conf
>>> from airflow.hooks.S3_hook import S3Hook
>>> h = S3Hook(aws_conn_id=conf.get('core', 'remote_log_conn_id'))
>>> c = h.get_conn()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/hooks/S3_hook.py", line 44, in get_conn
return self.get_client_type('s3')
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/contrib/hooks/aws_hook.py", line 176, in get_client_type
session, endpoint_url = self._get_credentials(region_name)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/contrib/hooks/aws_hook.py", line 102, in _get_credentials
connection_object = self.get_connection(self.aws_conn_id)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/hooks/base_hook.py", line 84, in get_connection
conn = random.choice(list(cls.get_connections(conn_id)))
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/hooks/base_hook.py", line 80, in get_connections
return secrets.get_connections(conn_id)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/secrets/__init__.py", line 52, in get_connections
conn_list = secrets_backend.get_connections(conn_id=conn_id)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/secrets/base_secrets.py", line 69, in get_connections
conn = Connection(conn_id=conn_id, uri=conn_uri)
File "<string>", line 4, in __init__
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/state.py", line 433, in _initialize_instance
manager.dispatch.init_failure(self, args, kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
exc_value, with_traceback=exc_tb,
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/state.py", line 430, in _initialize_instance
return manager.original_init(*mixed[1:], **kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/connection.py", line 119, in __init__
self.parse_from_uri(uri)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/connection.py", line 144, in parse_from_uri
self.port = uri_parts.port
File "/usr/local/lib/python3.6/urllib/parse.py", line 169, in port
port = int(port, 10)
ValueError: invalid literal for int() with base 10: 'abcd12'
```
My aws secret key is(dummy): abcd12/ef34578fgt
Looks like when '/' is coming in the secret key, the connection is not getting created.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimon222 commented on issue #9118: Improve/surface errors when attempting to read S3 logs
Posted by GitBox <gi...@apache.org>.
dimon222 commented on issue #9118:
URL: https://github.com/apache/airflow/issues/9118#issuecomment-641416176
Any clue what component can be touched to impact on this? I'm having trouble to find what code swallows exceptions. Really looking to investigate and resolve S3 logs issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #9118: Improve/surface errors when attempting to read S3 logs
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #9118:
URL: https://github.com/apache/airflow/issues/9118#issuecomment-667719606
@Siddharthk you probably need to URL encode the secret key - `/` to `%2f`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb closed issue #9118: Improve/surface errors when attempting to read S3 logs
Posted by GitBox <gi...@apache.org>.
ashb closed issue #9118:
URL: https://github.com/apache/airflow/issues/9118
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #9118: Improve/surface errors when attempting to read S3 logs
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #9118:
URL: https://github.com/apache/airflow/issues/9118#issuecomment-726753985
Closed by #9908
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] YevhenKv commented on issue #9118: Improve/surface errors when attempting to read S3 logs
Posted by GitBox <gi...@apache.org>.
YevhenKv commented on issue #9118:
URL: https://github.com/apache/airflow/issues/9118#issuecomment-643317328
I used code from the example above and I was able to list log files in my s3 bucket written by Airflow, no permission issues, no errors. However, Airflow seams to be ignoring remote logging configuration when reads logs. Also, used for testing airflow_local_settings.py and set a logging class path in config file, but no luck, S3TaskHandler seams to be ignored by Airflow.
OS: fedora26
python: 3.7.5
Airflow: 1.10.10
AWS EC2 instance with proper role permissions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org