[jira] [Commented] (BEAM-13602) Metrics reporting failed bigtable io

["Ning (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17470254#comment-17470254 ] 

Ning commented on BEAM-13602:
-----------------------------

The expected behavior is a single response (based on the [REST API|https://cloud.google.com/bigtable/docs/reference/data/rpc/google.bigtable.v2#mutaterowsresponse]) with multiple {{Status}} entries. In the Python client implementation of bigtable, the returned response is a {{List[Optional[Status]]}} (or equivalently {{{}List[Union[None, Status]]{}}})). Details: default [value|https://github.com/googleapis/python-bigtable/blob/fec06fcd28c36d0d3b347b43d1f3d264e5f5aa39/google/cloud/bigtable/table.py#L1058] is None, retryable errors will [return default value|https://github.com/googleapis/python-bigtable/blob/fec06fcd28c36d0d3b347b43d1f3d264e5f5aa39/google/cloud/bigtable/table.py#L1078] instead of a {{Status}} instance)

I'll send a PR to do the following 2 things:
 # Prevent the metrics gathering from failing the job;
 # Handle None cases and treat them as ['DEADLINE_EXCEEDED'|https://grpc.github.io/grpc/core/md_doc_statuscodes.html].

I fear that there is data-loss throughout the process. If a row mutation fails but no error is raised, DF nor bigtable no longer retries.

> Metrics reporting failed bigtable io
> ------------------------------------
>
>                 Key: BEAM-13602
>                 URL: https://issues.apache.org/jira/browse/BEAM-13602
>             Project: Beam
>          Issue Type: Bug
>          Components: io-py-gcp
>            Reporter: Ning
>            Assignee: Ning
>            Priority: P2
>
> Details see comment in [https://github.com/apache/beam/pull/15562].
> {{{}We are seeing errors which possibly originate from this code:{}}}{{{}def write_mutate_metrics(self, rows): for status in rows: grpc_status_string = ( ServiceCallMetric.bigtable_error_code_to_grpc_status_string( status.code)) self.service_call_metric.call(grpc_status_string){}}}{{{}Error message:{}}}
> {{{}File "/usr/local/lib/python3.6/site-packages/apache_beam/io/gcp/bigtableio.py", line 123, in write_mutate_metrics status.code)) AttributeError: 'NoneType' object has no attribute 'code' [while running 'Write event Bigtable row/ParDo(_BigTableWriteFn)-ptransform-58134']{}}}{{{}Would you have any idea about this issue?{}}}{{{}We are running{}}}{{{}Python 3.6{}}}
> {{Beam 2.35}}
> {{GCP DataflowRunner}}
> {{In addition, the choice of parameter name (rows) is confusing. Perhaps something like responses or response_list ?}}
> {{[https://cloud.google.com/bigtable/docs/writing-data#batch]}}
> {{[https://googleapis.dev/python/bigtable/1.3.0/_modules/google/cloud/bigtable/table.html#Table.mutate_rows]}}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)