You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Nicholas Walton <nw...@me.com> on 2019/11/26 13:20:14 UTC

SQL Performance

I’m streaming records down to an Embedded Derby database, at a rate of around 200 records per second. I’m certain Derby can sustain a higher throughput than that, if I could buffer the records but it seems that I’m writing each record as soon as it arrives and as a single transaction which is inefficient. 

Is there a way to to improve my throughput rate? 

Nick Walton

Re: SQL Performance

Posted by Jingsong Li <ji...@gmail.com>.

Hi Nicholas,

You can take a look to JDBCSinkFunction and JDBCUpsertSinkFunction.
They can set something like flush batch max size just like vino mentioned.
JDBCUpsertOutputFormat also has a async thread to flush batch to JDBC
 to avoid too high delay. In this way, the throughput can be improved while
 the delay is low.

Best,
Jingsong Lee

On Wed, Nov 27, 2019 at 10:16 AM vino yang <ya...@gmail.com> wrote:

> Hi Nick,
>
> Can you provide more details? Are you using JDBCOutputFormat? If yes, can
> `JDBCOutputFormatBuilder#setBatchInterval` help you?
>
> Best,
> Vino
>
> Nicholas Walton <nw...@me.com> 于2019年11月26日周二 下午9:20写道：
>
>> I’m streaming records down to an Embedded Derby database, at a rate of
>> around 200 records per second. I’m certain Derby can sustain a higher
>> throughput than that, if I could buffer the records but it seems that I’m
>> writing each record as soon as it arrives and as a single transaction which
>> is inefficient.
>>
>> Is there a way to to improve my throughput rate?
>>
>> Nick Walton
>
>

-- 
Best, Jingsong Lee

Re: SQL Performance

Posted by vino yang <ya...@gmail.com>.

Hi Nick,

Can you provide more details? Are you using JDBCOutputFormat? If yes, can
`JDBCOutputFormatBuilder#setBatchInterval` help you?

Best,
Vino

Nicholas Walton <nw...@me.com> 于2019年11月26日周二 下午9:20写道：

> I’m streaming records down to an Embedded Derby database, at a rate of
> around 200 records per second. I’m certain Derby can sustain a higher
> throughput than that, if I could buffer the records but it seems that I’m
> writing each record as soon as it arrives and as a single transaction which
> is inefficient.
>
> Is there a way to to improve my throughput rate?
>
> Nick Walton