You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Andrei Sereda <an...@sereda.cc> on 2019/06/14 18:05:39 UTC
calcite connection and thread safety
Hello,
I couldn't find any recommendations on using calcite connection
(JdbcSchema) in a multi-threaded environment.
Probably throwing DataSource on the top of calcite connection is a wrong
thing to do since it doesn't create connections by itself rather delegates
to existing pools.
After creating a test [1], which simply executes a query from different
threads on the same connection, I run into some weird issue:
> Caused by: org.apache.calcite.avatica.NoSuchStatementException
> at
org.apache.calcite.jdbc.CalciteConnectionImpl$CalciteServerImpl.getStatement(CalciteConnectionImpl.java:371)
> at
org.apache.calcite.jdbc.CalciteConnectionImpl.getCancelFlag(CalciteConnectionImpl.java:238)
> at
org.apache.calcite.avatica.AvaticaStatement.<init>(AvaticaStatement.java:121)
> ... 17 more
It seems that non-atomic id increment is happening inside
org.apache.calcite.avatica.MetaImpl:213
```java
public StatementHandle createStatement(ConnectionHandle ch) {
// non-atomic connection.statementCount++
return new StatementHandle(ch.id, connection.statementCount++, null);
}
```
Can you confirm that Calcite connection (or more correctly
AvaticaConnection) is thread-safe by design ? If so, can this be considered
a bug ?
Regards,
Andrei
[1] https://gist.github.com/asereda-gs/cf0e1c0b8fa84a4cf3ceb6a21c9b11a8
Re: calcite connection and thread safety
Posted by Stamatis Zampetakis <za...@gmail.com>.
Typical JDBC does not allow Statements created by the same connection to be
processed in parallel.
Given that other JDBC providers are behaving the same way this may not be a
big issue since users are already accustomed to such limitations.
On Fri, Jun 14, 2019 at 8:47 PM Andrei Sereda <an...@sereda.cc> wrote:
> > Only one thread should access a connection (or objects created from that
> connection, such as statements and result sets) at a time.
> Does it mean that two statements created by same connection can't be
> processed in parallel by different threads ? For example if one
> synchronizes just on statement creation.
>
> In this case connection pool is the only viable option. However having
> multiple connections implies duplicating schemas, metadata, cache etc.
>
> On Fri, Jun 14, 2019 at 2:10 PM Julian Hyde <jh...@apache.org> wrote:
>
> > The general JDBC specification is that connections are not thread-safe.
> > Only one thread should access a connection (or objects created from that
> > connection, such as statements and result sets) at a time. The exception
> is
> > the Statement.cancel() method, which can safely be called from another
> > thread.
> >
> > Calcite abides by that specification. (If we promised more than that we
> > would probably have to reduce performance, and we would definitely have
> to
> > increase testing.)
> >
> > In a multi-threaded environment the safest thing is for each thread to
> > have its own connection.
> >
> > Julian
> >
> >
> >
> > > On Jun 14, 2019, at 11:05 AM, Andrei Sereda <an...@sereda.cc> wrote:
> > >
> > > Hello,
> > >
> > > I couldn't find any recommendations on using calcite connection
> > > (JdbcSchema) in a multi-threaded environment.
> > >
> > > Probably throwing DataSource on the top of calcite connection is a
> wrong
> > > thing to do since it doesn't create connections by itself rather
> > delegates
> > > to existing pools.
> > >
> > > After creating a test [1], which simply executes a query from different
> > > threads on the same connection, I run into some weird issue:
> > >
> > >> Caused by: org.apache.calcite.avatica.NoSuchStatementException
> > >> at
> > >
> >
> org.apache.calcite.jdbc.CalciteConnectionImpl$CalciteServerImpl.getStatement(CalciteConnectionImpl.java:371)
> > >> at
> > >
> >
> org.apache.calcite.jdbc.CalciteConnectionImpl.getCancelFlag(CalciteConnectionImpl.java:238)
> > >> at
> > >
> >
> org.apache.calcite.avatica.AvaticaStatement.<init>(AvaticaStatement.java:121)
> > >> ... 17 more
> > >
> > > It seems that non-atomic id increment is happening inside
> > > org.apache.calcite.avatica.MetaImpl:213
> > >
> > > ```java
> > > public StatementHandle createStatement(ConnectionHandle ch) {
> > > // non-atomic connection.statementCount++
> > > return new StatementHandle(ch.id, connection.statementCount++, null);
> > > }
> > > ```
> > >
> > > Can you confirm that Calcite connection (or more correctly
> > > AvaticaConnection) is thread-safe by design ? If so, can this be
> > considered
> > > a bug ?
> > >
> > > Regards,
> > > Andrei
> > >
> > > [1]
> https://gist.github.com/asereda-gs/cf0e1c0b8fa84a4cf3ceb6a21c9b11a8
> >
> >
>
Re: calcite connection and thread safety
Posted by Andrei Sereda <an...@sereda.cc>.
> Only one thread should access a connection (or objects created from that
connection, such as statements and result sets) at a time.
Does it mean that two statements created by same connection can't be
processed in parallel by different threads ? For example if one
synchronizes just on statement creation.
In this case connection pool is the only viable option. However having
multiple connections implies duplicating schemas, metadata, cache etc.
On Fri, Jun 14, 2019 at 2:10 PM Julian Hyde <jh...@apache.org> wrote:
> The general JDBC specification is that connections are not thread-safe.
> Only one thread should access a connection (or objects created from that
> connection, such as statements and result sets) at a time. The exception is
> the Statement.cancel() method, which can safely be called from another
> thread.
>
> Calcite abides by that specification. (If we promised more than that we
> would probably have to reduce performance, and we would definitely have to
> increase testing.)
>
> In a multi-threaded environment the safest thing is for each thread to
> have its own connection.
>
> Julian
>
>
>
> > On Jun 14, 2019, at 11:05 AM, Andrei Sereda <an...@sereda.cc> wrote:
> >
> > Hello,
> >
> > I couldn't find any recommendations on using calcite connection
> > (JdbcSchema) in a multi-threaded environment.
> >
> > Probably throwing DataSource on the top of calcite connection is a wrong
> > thing to do since it doesn't create connections by itself rather
> delegates
> > to existing pools.
> >
> > After creating a test [1], which simply executes a query from different
> > threads on the same connection, I run into some weird issue:
> >
> >> Caused by: org.apache.calcite.avatica.NoSuchStatementException
> >> at
> >
> org.apache.calcite.jdbc.CalciteConnectionImpl$CalciteServerImpl.getStatement(CalciteConnectionImpl.java:371)
> >> at
> >
> org.apache.calcite.jdbc.CalciteConnectionImpl.getCancelFlag(CalciteConnectionImpl.java:238)
> >> at
> >
> org.apache.calcite.avatica.AvaticaStatement.<init>(AvaticaStatement.java:121)
> >> ... 17 more
> >
> > It seems that non-atomic id increment is happening inside
> > org.apache.calcite.avatica.MetaImpl:213
> >
> > ```java
> > public StatementHandle createStatement(ConnectionHandle ch) {
> > // non-atomic connection.statementCount++
> > return new StatementHandle(ch.id, connection.statementCount++, null);
> > }
> > ```
> >
> > Can you confirm that Calcite connection (or more correctly
> > AvaticaConnection) is thread-safe by design ? If so, can this be
> considered
> > a bug ?
> >
> > Regards,
> > Andrei
> >
> > [1] https://gist.github.com/asereda-gs/cf0e1c0b8fa84a4cf3ceb6a21c9b11a8
>
>
Re: calcite connection and thread safety
Posted by Julian Hyde <jh...@apache.org>.
The general JDBC specification is that connections are not thread-safe. Only one thread should access a connection (or objects created from that connection, such as statements and result sets) at a time. The exception is the Statement.cancel() method, which can safely be called from another thread.
Calcite abides by that specification. (If we promised more than that we would probably have to reduce performance, and we would definitely have to increase testing.)
In a multi-threaded environment the safest thing is for each thread to have its own connection.
Julian
> On Jun 14, 2019, at 11:05 AM, Andrei Sereda <an...@sereda.cc> wrote:
>
> Hello,
>
> I couldn't find any recommendations on using calcite connection
> (JdbcSchema) in a multi-threaded environment.
>
> Probably throwing DataSource on the top of calcite connection is a wrong
> thing to do since it doesn't create connections by itself rather delegates
> to existing pools.
>
> After creating a test [1], which simply executes a query from different
> threads on the same connection, I run into some weird issue:
>
>> Caused by: org.apache.calcite.avatica.NoSuchStatementException
>> at
> org.apache.calcite.jdbc.CalciteConnectionImpl$CalciteServerImpl.getStatement(CalciteConnectionImpl.java:371)
>> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.getCancelFlag(CalciteConnectionImpl.java:238)
>> at
> org.apache.calcite.avatica.AvaticaStatement.<init>(AvaticaStatement.java:121)
>> ... 17 more
>
> It seems that non-atomic id increment is happening inside
> org.apache.calcite.avatica.MetaImpl:213
>
> ```java
> public StatementHandle createStatement(ConnectionHandle ch) {
> // non-atomic connection.statementCount++
> return new StatementHandle(ch.id, connection.statementCount++, null);
> }
> ```
>
> Can you confirm that Calcite connection (or more correctly
> AvaticaConnection) is thread-safe by design ? If so, can this be considered
> a bug ?
>
> Regards,
> Andrei
>
> [1] https://gist.github.com/asereda-gs/cf0e1c0b8fa84a4cf3ceb6a21c9b11a8