Re: Dataflow throwing backend error

[Ahmet Altay <al...@google.com>]

Hi Rajesh,

This looks like a transient error from GCS. Beam SDK will retry tasks in
the face of such errors and those typically do not make your pipeline fail.
If you have additional questions please reach out to Dataflow support (
https://cloud.google.com/dataflow/support).

Thank you,
Ahmet

On Tue, Mar 27, 2018 at 3:58 AM, Rajesh Hegde <rh...@datalicious.com>
wrote:

> Hi,
> We are getting backend error with Google Cloud Storage service, any idea
> why it happens and how to fix it? Error traceback is pasted below.
>
> *Traceback (most recent call last):*
> *  File
> "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py",
> line 609, in do_work*
> *    work_executor.execute()*
> *  File
> "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py", line
> 170, in execute*
> *    op.finish()*
> *  File "dataflow_worker/native_operations.py", line 93, in
> dataflow_worker.native_operations.NativeWriteOperation.finish*
> *    def finish(self):*
> *  File "dataflow_worker/native_operations.py", line 94, in
> dataflow_worker.native_operations.NativeWriteOperation.finish*
> *    with self.scoped_finish_state:*
> *  File "dataflow_worker/native_operations.py", line 95, in
> dataflow_worker.native_operations.NativeWriteOperation.finish*
> *    self.writer.__exit__(None, None, None)*
> *  File
> "/usr/local/lib/python2.7/dist-packages/dataflow_worker/nativefileio.py",
> line 459, in __exit__*
> *    self.file.close()*
> *  File
> "/usr/local/lib/python2.7/dist-packages/apache_beam/io/filesystemio.py",
> line 201, in close*
> *    self._uploader.finish()*
> *  File
> "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/gcsio.py", line
> 575, in finish*
> *    raise self._upload_thread.last_error  # pylint:
> disable=raising-bad-type*
> *HttpError: HttpError accessing
> <https://www.googleapis.com/resumable/upload/storage/v1/b/
> <https://www.googleapis.com/resumable/upload/storage/v1/b/><bucketname>/o?uploadType=resumable&alt=json&upload_id=AEnB2UoK85wrqW1nWJ2cN7DQ5JKdQtTyDX-LwRwgIlIHgVL0KWR8JlUcLOJYFWmv9_YfpVhlKsooB4tHL2cxXch9hVls4nxFnw&name=temp%2Fbeamapp-bzftmxc-0327102557-645475.1522146357.645609%2F11032707444842841251%2Fdax-tmp-2018-03-27_03_27_30-7388048597187286914-S05-0-adbd1398680cf5c7%2F-shard--try-0c9aa3475bd67907-endshard.json>:
> response: <{'status': '410', 'content-length': '177', 'vary': 'Origin,
> X-Origin', 'server': 'UploadServer', 'x-guploader-uploadid':
> 'AEnB2UoK85wrqW1nWJ2cN7DQ5JKdQtTyDX-LwRwgIlIHgVL0KWR8JlUcLOJYFWmv9_YfpVhlKsooB4tHL2cxXch9hVls4nxFnw',
> 'date': 'Tue, 27 Mar 2018 10:41:17 GMT', 'content-type': 'application/json;
> charset=UTF-8'}>, content <{*
> * "error": {*
> *  "errors": [*
> *   {*
> *    "domain": "global",*
> *    "reason": "backendError",*
> *    "message": "Backend Error"*
> *   }*
> *  ],*
> *  "code": 500,*
> *  "message": "Backend Error"*
> * }*
> *}*
> *>*
>
> --
>
> *Rajesh Hegde | Lead Product Developer | Datalicious*
> *e*: rhegde@datalicious.com | *m*: +919167571827 <+91%2091675%2071827>
> *a*: L-77, 15th Cross Rd, Sector 6, HSR Layout,
> <https://maps.google.com/?q=77,+15th+Cross+Rd,+Sector+6,+HSR+Layout,++Bangalore+Karnataka-+560102&entry=gmail&source=g>
> Bangalore Karnataka- 560102
> <https://maps.google.com/?q=77,+15th+Cross+Rd,+Sector+6,+HSR+Layout,++Bangalore+Karnataka-+560102&entry=gmail&source=g>
> *w*: www.datalicious.com
> <http://www.datalicious.com/?utm_source=signaturesatori&utm_medium=email&utm_campaign=signaturesatori>
>
> *Contact support@datalicious.com <su...@datalicious.com> anytime, we're
> keen to help!*
>
> <https://www.linkedin.com/company/datalicious-pty-ltd>
> <https://twitter.com/datalicious>   <https://www.facebook.com/Datalicious>
>    <https://plus.google.com/+Datalicious1>
>
>
> <https://www.datalicious.com/resources/facebook-people-based-measurement-attribution/?utm_source=signaturesatori&utm_medium=email&utm_campaign=signaturesatori>
>
>