Capturing Statement Executions within JdbcSink

[Rion Williams <ri...@gmail.com>]

Hey all,

Is there any mechanism in place (or out of the box) to support reading the
results of statements executed by the JdbcSink or would I need to implement
my own to support it?

The problem that I'm trying to solve relates to observability (i.e.
metrics) and incrementing specific counters based on the response from a
given statement executing. One example might be if I need to upsert 40
widgets that are coming in, although some may be the same widget, I only
want to increment my metric if the widget didn't already exist.

Thanks,

Rion

Re: Capturing Statement Executions within JdbcSink

[Alexey Romanenko <ar...@gmail.com>]

Hi Rion,

For now, it’s only possible to have a PCollection of Void as a result of JdbcIO's write using “withResults()” [1]. This is mostly used along with “Wait” PTransform in case if you need to continue to execute downstream the transforms only once Jdbc writes are finished (see an example in Javadoc [1]).

I’m not sure if it will work for your case, since you need to check the results of writes. Though, as a workaround, I guess you can do write, wait for finishing and then read it back to check the results.

In the same time, since we already had the similar requests in the past [3], I think we can have an extended version of “withResults()", like “withWriteResults()” that will return a PCollection of statements results (like we do, for example, for BigtableIO [2]).

I’d be happy to guide/assist you (if needed) if you decide to work on this feature [3].

Regards,
Alexey


[1] https://beam.apache.org/releases/javadoc/2.28.0/org/apache/beam/sdk/io/jdbc/JdbcIO.Write.html#withResults--
[2] https://beam.apache.org/releases/javadoc/2.28.0/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.Write.html#withWriteResults--
[3] https://issues.apache.org/jira/browse/BEAM-11873

> On 18 Mar 2021, at 03:26, Rion Williams <ri...@gmail.com> wrote:
> 
> Hey all,
> 
> Is there any mechanism in place (or out of the box) to support reading the results of statements executed by the JdbcSink or would I need to implement my own to support it?
> 
> The problem that I'm trying to solve relates to observability (i.e. metrics) and incrementing specific counters based on the response from a given statement executing. One example might be if I need to upsert 40 widgets that are coming in, although some may be the same widget, I only want to increment my metric if the widget didn't already exist.
> 
> Thanks,
> 
> Rion