[jira] [Created] (NIFI-14453) PutBigQuery Creates too many active streams

["Satya Vadapalli (Jira)" <ji...@apache.org>]

Satya Vadapalli created NIFI-14453:
--------------------------------------

             Summary: PutBigQuery Creates too many active streams
                 Key: NIFI-14453
                 URL: https://issues.apache.org/jira/browse/NIFI-14453
             Project: Apache NiFi
          Issue Type: Bug
          Components: Core Framework
    Affects Versions: 2.3.0
            Reporter: Satya Vadapalli


*The Issue:*
We recently migrated from Nifi 1.x to 2.x. We've been having an issue with PutBigQuery processor. It used to work well on the older version(PutBigQueryStreaming), because I believe it used the BigQuery REST API vs the new one uses the Storage API. I'm running into an issue where the processor is opening over 10k streams, instead of reusing the existing stream. Here's the error message. 
 
{code:java}
PutBigQuery[id=01bc2d0f-0196-1000-0000-0000541e88b5] Processing halted: yielding [1 sec]: com.google.api.gax.rpc.FailedPreconditionException: io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Table has too many active streams with too little traffic. Please send more traffic through existing streams or finalize unused streams. Table=1087270590813:xxxxxx.xxxxxxxx. ActiveStreamCount=10139. ActualPerStreamBytesPerSec=0.0333333. RequiredPerStreamBytesPerSecForMoreStreams=300000. If you have already terminated all the traffic, the error will go away in two hours. To avoid this problem in the long term, please use less streams for the same amount of data Entity: projects/xxx/datasets/xxx/tables/xxx - Caused by: io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Table has too many active streams with too little traffic. Please send more traffic through existing streams or finalize unused streams. Table=1087270590813:xxxxxxx.xxxxxxxxxx. ActiveStreamCount=10139. ActualPerStreamBytesPerSec=0.0333333. RequiredPerStreamBytesPerSecForMoreStreams=300000. If you have already terminated all the traffic, the error will go away in two hours. To avoid this problem in the long term, please use less streams for the same amount of data Entity: projects/xxx/datasets/xxx/tables/xxx
{code}
*What is Expected:*
The PutBigQuery should be able to stream data into Bigquery, with a single stream, using the Storage Write API as described in the document - [https://cloud.google.com/bigquery/docs/write-api-streaming]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)