[jira] [Created] (BEAM-13162) BigQueryIO Storage Write API method - suppress logging API errors for known use cases.

["Sergei Lilichenko (Jira)" <ji...@apache.org>]

Sergei Lilichenko created BEAM-13162:
----------------------------------------

             Summary:  BigQueryIO Storage Write API method - suppress logging API errors for known use cases.
                 Key: BEAM-13162
                 URL: https://issues.apache.org/jira/browse/BEAM-13162
             Project: Beam
          Issue Type: New Feature
          Components: io-java-gcp
    Affects Versions: 2.33.0
            Reporter: Sergei Lilichenko


Many BigQuery Storage API errors get logged at ERROR level even though they are expected and handled by the Beam's transform. They should be suppresses, or output at DEBUG/INFO level to differentiate from abnormal behavior.

Example:

{
 "jsonPayload": {
 "message": "Got error io.grpc.StatusRuntimeException: ALREADY_EXISTS: The offset is within stream, expected offset 52125, received 51264 Entity: projects/event-processing-demo/datasets/bigquery_io/tables/events/streams/CiQ2MmZlOTFjNS0wMDAwLTIzNTItOWMxYS01ODI0MjlhOWRiOGM closing projects/event-processing-demo/datasets/bigquery_io/tables/events/streams/CiQ2MmZlOTFjNS0wMDAwLTIzNTItOWMxYS01ODI0MjlhOWRiOGM",
 "step": "Save Rows to BigQuery/StorageApiLoads/StorageApiWriteSharded/Write Records",
 "worker": "data-processing-streaming-10250846-kps9-harness-hnj3",
 "stage": "P6",
 "job": "2021-10-25_08_46_20-4863073108555028756",
 "thread": "258",
 "work": "3800026c09938fac-5cf2f68daacdd",
 "logger": "org.apache.beam.sdk.io.gcp.bigquery.StorageApiWritesShardedRecords"
 },
 "resource": {
 "type": "dataflow_step",
 "labels": {
 "project_id": "event-processing-demo",
 "step_id": "Save Rows to BigQuery/StorageApiLoads/StorageApiWriteSharded/Write Records",
 "job_id": "2021-10-25_08_46_20-4863073108555028756",
 "region": "us-central1",
 "job_name": "data-processing-streaming-storage-write-api-200-2"
 }
 },
 "timestamp": "2021-10-25T15:57:43.617Z",
 "severity": "ERROR",
 "labels": {
 "compute.googleapis.com/resource_type": "instance",
 "dataflow.googleapis.com/job_name": "data-processing-streaming-storage-write-api-200-2",
 "dataflow.googleapis.com/log_type": "supportability",
 "compute.googleapis.com/resource_id": "6365565973148954518",
 "dataflow.googleapis.com/job_id": "2021-10-25_08_46_20-4863073108555028756",
 "dataflow.googleapis.com/region": "us-central1",
 "compute.googleapis.com/resource_name": "data-processing-streaming-10250846-kps9-harness-hnj3"
 },
 "logName": "projects/event-processing-demo/logs/dataflow.googleapis.com%2Fworker",
 "receiveTimestamp": "2021-10-25T15:57:47.557344849Z"
}

Additional exceptions that fit this category:
 * Got error io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Stream is closed due to com.google.api.gax.rpc.UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE: Connection closed after GOAWAY. HTTP/2 error code: NO_ERROR, debug data: server_shutting_down closing
 * Got error com.google.api.gax.rpc.AbortedException: io.grpc.StatusRuntimeException: ABORTED: Closing the stream because server is restarted. This is expected and client is advised to reconnect.
 * Got error io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Stream is closed due to com.google.api.gax.rpc.AbortedException: io.grpc.StatusRuntimeException: ABORTED: Closing the stream because server is restarted. This is expected and client is advised to reconnect.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)