You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/05/11 08:37:29 UTC
[GitHub] [airflow] waiyan1612 opened a new issue #8810: GoogleCloudBaseHook does not exclude 308 response code
waiyan1612 opened a new issue #8810:
URL: https://github.com/apache/airflow/issues/8810
**Apache Airflow version**: 1.10.10
**Environment**:
- **Cloud provider or hardware configuration**: AWS EKS
- **OS** (e.g. from /etc/os-release): `python:3.7-stretch` docker image
- **Install tools**: `pip install --constraint https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt
apache-airflow[gcp, google_auth]==1.10.10`
**What happened**:
httplib2 throws an exception when extending [GoogleDriveHook](https://github.com/apache/airflow/blob/master/airflow/providers/google/suite/hooks/drive.py) to create a resumable upload. The libraries dependencies are based on [requirements-python3.7.txt](https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt).
**What you expected to happen**:
[GoogleCloudBaseHook](https://github.com/apache/airflow/blob/master/airflow/providers/google/common/hooks/base_google.py#L216) should exclude `308` response code from httplib2's list of redirect status codes. I believe this is the same issue reported by https://github.com/googleapis/google-api-python-client/issues/803 and fixed in https://github.com/googleapis/google-api-python-client/pull/813.
Since we are already using `set_user_agent` from `googleapiclient.http`, is it possible to use [build_http](https://github.com/googleapis/google-api-python-client/blob/v1.8.0/googleapiclient/http.py#L1874) instead of [directly creating](https://github.com/apache/airflow/blob/master/airflow/providers/google/common/hooks/base_google.py#L216) from httplib2?
**How to reproduce it**:
Running this Code will give us
```py
from airflow.contrib.hooks.gdrive_hook import GoogleDriveHook
hook = GoogleDriveHook(gcp_conn_id="google_cloud_ecircon")
service = hook.get_conn()
# Set chunksize to 1MB to test resumable upload.
media = MediaFileUpload('/tmp/file_larger_than_1mb', chunksize=1024*1024, resumable=True)
file_metadata = {"name": "file_larger_than_1mb"}
service.files().create(body=file_metadata, media_body=media, fields="id").execute()
```
the following exception:
```
[2020-05-11 15:54:43,012] {taskinstance.py:1145} ERROR - Redirected but the response is missing a Location: header.
Traceback (most recent call last):
File "/anaconda3/envs/airflow_1_10_10/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 983, in _run_raw_task
result = task_copy.execute(context=context)
File "/anaconda3/envs/airflow_1_10_10/lib/python3.7/site-packages/airflow/operators/python_operator.py", line 113, in execute
return_value = self.execute_callable()
File "/anaconda3/envs/airflow_1_10_10/lib/python3.7/site-packages/airflow/operators/python_operator.py", line 118, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/Users/waiyan/airflow/dags/repo/ecircon/utils/dag_factory.py", line 166, in upload_gdrive_and_count
s3_object, gdrive_file_request
File "/Users/waiyan/airflow/dags/repo/common/utils/s3_to_gdrive.py", line 49, in upload_from_s3_to_gdrive_and_count
google_drive_file_response = gdrive_hook.upload(local_file, google_drive_file)
File "/Users/waiyan/airflow/dags/repo/common/hooks/gcp_gdrive_hook.py", line 180, in upload
.execute(num_retries=self.num_retries)
File "/anaconda3/envs/airflow_1_10_10/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
return wrapped(*args, **kwargs)
File "/anaconda3/envs/airflow_1_10_10/lib/python3.7/site-packages/googleapiclient/http.py", line 862, in execute
_, body = self.next_chunk(http=http, num_retries=num_retries)
File "/anaconda3/envs/airflow_1_10_10/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
return wrapped(*args, **kwargs)
File "/anaconda3/envs/airflow_1_10_10/lib/python3.7/site-packages/googleapiclient/http.py", line 1037, in next_chunk
self.resumable_uri, method="PUT", body=data, headers=headers
File "/anaconda3/envs/airflow_1_10_10/lib/python3.7/site-packages/google_auth_httplib2.py", line 198, in request
uri, method, body=body, headers=request_headers, **kwargs)
File "/anaconda3/envs/airflow_1_10_10/lib/python3.7/site-packages/googleapiclient/http.py", line 1813, in new_request
connection_type=connection_type,
File "/anaconda3/envs/airflow_1_10_10/lib/python3.7/site-packages/httplib2/__init__.py", line 1991, in request
cachekey,
File "/anaconda3/envs/airflow_1_10_10/lib/python3.7/site-packages/httplib2/__init__.py", line 1690, in _request
content,
httplib2.RedirectMissingLocation: Redirected but the response is missing a Location: header.
```
**Temporary workaround**:
Downgrading `httplib2==0.15.0` works as a temporary workaround.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #8810: GoogleCloudBaseHook does not exclude 308 response code
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #8810:
URL: https://github.com/apache/airflow/issues/8810#issuecomment-626562158
Hi @waiyan1612 thanks for reporting this issue! Would you like to work on a fix for it as you already figured out what can be improved? Happy to help :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #8810: GoogleCloudBaseHook does not exclude 308 response code
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #8810:
URL: https://github.com/apache/airflow/issues/8810#issuecomment-626558394
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org