You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Matthew Ouyang <ma...@gmail.com> on 2023/03/02 21:44:04 UTC
Successful Inserts for Storage Write API?
Thank you to Ahmed and Reuven for the tip on WriteResult::
getFailedStorageApiInserts.
When I tried to get the successful inserts through the Storage Write API, I
received an error message saying that "Retrieving successful inserts is
only supported for streaming inserts. Make sure
withSuccessfulInsertsPropagation is correctly configured for
BigQueryIO.Write object." Did I make a mistake, or is there a reason why
this is not possible? I tried setting triggeringFrequency +
numStorageWriteApiStreams as required by Storage Write, and I tried to set
successfulInsertsPropagation as directed in the error message.
Re: Successful Inserts for Storage Write API?
Posted by Reuven Lax via user <us...@beam.apache.org>.
FYI, I just checked in WriteResult.getSuccessfulStorageApiInserts, which
should give you what you need. This is now checked into Beam HEAD, and
should be included in the next Beam release.
On Fri, Mar 3, 2023 at 12:51 PM Matthew Ouyang <ma...@gmail.com>
wrote:
> I'm currently not using Wait.on. I have a pipeline that uses BigQuery for
> checkpointing purposes. I only want records that are
> successfully checkpointed in BigQuery to be eligible for the next stage in
> my pipeline. With streaming inserts, I can use getSuccessfulInserts to
> achieve this and I was looking for something similar with Storage Write.
>
> On Thu, Mar 2, 2023 at 4:48 PM Reuven Lax via user <us...@beam.apache.org>
> wrote:
>
>> Are you trying to do this in order to use Wait.on? getSuccessfulInserts
>> is not currently supported for Storage Write API.
>>
>> On Thu, Mar 2, 2023 at 1:44 PM Matthew Ouyang <ma...@gmail.com>
>> wrote:
>>
>>> Thank you to Ahmed and Reuven for the tip on
>>> WriteResult::getFailedStorageApiInserts.
>>>
>>> When I tried to get the successful inserts through the Storage Write
>>> API, I received an error message saying that "Retrieving successful inserts
>>> is only supported for streaming inserts. Make sure
>>> withSuccessfulInsertsPropagation is correctly configured for
>>> BigQueryIO.Write object." Did I make a mistake, or is there a reason why
>>> this is not possible? I tried setting triggeringFrequency +
>>> numStorageWriteApiStreams as required by Storage Write, and I tried to set
>>> successfulInsertsPropagation as directed in the error message.
>>>
>>
Re: Successful Inserts for Storage Write API?
Posted by Matthew Ouyang <ma...@gmail.com>.
I'm currently not using Wait.on. I have a pipeline that uses BigQuery for
checkpointing purposes. I only want records that are
successfully checkpointed in BigQuery to be eligible for the next stage in
my pipeline. With streaming inserts, I can use getSuccessfulInserts to
achieve this and I was looking for something similar with Storage Write.
On Thu, Mar 2, 2023 at 4:48 PM Reuven Lax via user <us...@beam.apache.org>
wrote:
> Are you trying to do this in order to use Wait.on? getSuccessfulInserts is
> not currently supported for Storage Write API.
>
> On Thu, Mar 2, 2023 at 1:44 PM Matthew Ouyang <ma...@gmail.com>
> wrote:
>
>> Thank you to Ahmed and Reuven for the tip on
>> WriteResult::getFailedStorageApiInserts.
>>
>> When I tried to get the successful inserts through the Storage Write API,
>> I received an error message saying that "Retrieving successful inserts is
>> only supported for streaming inserts. Make sure
>> withSuccessfulInsertsPropagation is correctly configured for
>> BigQueryIO.Write object." Did I make a mistake, or is there a reason why
>> this is not possible? I tried setting triggeringFrequency +
>> numStorageWriteApiStreams as required by Storage Write, and I tried to set
>> successfulInsertsPropagation as directed in the error message.
>>
>
Re: Successful Inserts for Storage Write API?
Posted by Reuven Lax via user <us...@beam.apache.org>.
Are you trying to do this in order to use Wait.on? getSuccessfulInserts is
not currently supported for Storage Write API.
On Thu, Mar 2, 2023 at 1:44 PM Matthew Ouyang <ma...@gmail.com>
wrote:
> Thank you to Ahmed and Reuven for the tip on
> WriteResult::getFailedStorageApiInserts.
>
> When I tried to get the successful inserts through the Storage Write API,
> I received an error message saying that "Retrieving successful inserts is
> only supported for streaming inserts. Make sure
> withSuccessfulInsertsPropagation is correctly configured for
> BigQueryIO.Write object." Did I make a mistake, or is there a reason why
> this is not possible? I tried setting triggeringFrequency +
> numStorageWriteApiStreams as required by Storage Write, and I tried to set
> successfulInsertsPropagation as directed in the error message.
>