You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by priyanka B <pr...@gmail.com> on 2019/08/30 05:50:00 UTC
Fetch size does not work on Postgres JDBC plugin
Hi,
To prevent the drill from going out of memory, it is suggested to set the
fetch size in Jdbc URL to enable data streaming(DRILL-6794 discusses this).
This does not work on Postgres for the following reason:
For fetchSize size to work on Postgres, the connection must not be in
autocommit mode. There is no parameter to set autocommit to false in
connection string other than programmatically calling
conn.setAutoCommit(false).
See
https://jdbc.postgresql.org/documentation/93/query.html#fetchsize-example
See https://jdbc.postgresql.org/documentation/head/connect.html for the
list of all connection string properties.
Fix is to add a property 'defaultAutoCommit' to JdbcStorageConfig and call
BasicDataSource#setDefaultAutoCommit in JdbcStoragePlugin.
Let us know your thoughts, we will create an issue and provide a fix.
Thank you,
Priyanka Bhoir
Re: Fetch size does not work on Postgres JDBC plugin
Posted by priyanka B <pr...@gmail.com>.
Sure I will do it.
On Sun, Oct 20, 2019 at 8:51 PM Charles Givre <cg...@gmail.com> wrote:
> Hi Priyanka,
> Did anyone ever respond to this? If not , would you mind please creating
> a JIRA for this?
> Thanks,
> -- C
>
>
> > On Aug 30, 2019, at 1:50 AM, priyanka B <pr...@gmail.com>
> wrote:
> >
> > Hi,
> >
> > To prevent the drill from going out of memory, it is suggested to set the
> > fetch size in Jdbc URL to enable data streaming(DRILL-6794 discusses
> this).
> > This does not work on Postgres for the following reason:
> >
> > For fetchSize size to work on Postgres, the connection must not be in
> > autocommit mode. There is no parameter to set autocommit to false in
> > connection string other than programmatically calling
> > conn.setAutoCommit(false).
> >
> > See
> >
> https://jdbc.postgresql.org/documentation/93/query.html#fetchsize-example
> > See https://jdbc.postgresql.org/documentation/head/connect.html for the
> > list of all connection string properties.
> >
> > Fix is to add a property 'defaultAutoCommit' to JdbcStorageConfig and
> call
> > BasicDataSource#setDefaultAutoCommit in JdbcStoragePlugin.
> >
> > Let us know your thoughts, we will create an issue and provide a fix.
> >
> > Thank you,
> > Priyanka Bhoir
>
>
Re: Fetch size does not work on Postgres JDBC plugin
Posted by Charles Givre <cg...@gmail.com>.
Hi Priyanka,
Did anyone ever respond to this? If not , would you mind please creating a JIRA for this?
Thanks,
-- C
> On Aug 30, 2019, at 1:50 AM, priyanka B <pr...@gmail.com> wrote:
>
> Hi,
>
> To prevent the drill from going out of memory, it is suggested to set the
> fetch size in Jdbc URL to enable data streaming(DRILL-6794 discusses this).
> This does not work on Postgres for the following reason:
>
> For fetchSize size to work on Postgres, the connection must not be in
> autocommit mode. There is no parameter to set autocommit to false in
> connection string other than programmatically calling
> conn.setAutoCommit(false).
>
> See
> https://jdbc.postgresql.org/documentation/93/query.html#fetchsize-example
> See https://jdbc.postgresql.org/documentation/head/connect.html for the
> list of all connection string properties.
>
> Fix is to add a property 'defaultAutoCommit' to JdbcStorageConfig and call
> BasicDataSource#setDefaultAutoCommit in JdbcStoragePlugin.
>
> Let us know your thoughts, we will create an issue and provide a fix.
>
> Thank you,
> Priyanka Bhoir