You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by "Ulf Andreasson @ MapR" <ua...@maprtech.com> on 2015/10/05 14:14:29 UTC
JDBC driver for MySQL - storage plugin config and push down ?
Got some questions wrt the JDBC mySQL storage plugin
1) Given that we are getting close to 1.2 and full release of the plugin
what is the status of the push down function, which clauses have been
supported by push down?
2) Creation of my MySQLDB storage plugin gives me Error "(invalid JSON
mapping)"
When trying to connect I get the error whatever I define .... even if I
insert {type:jdbc} or a longer version like below. From seeing the code I
seem to have the right properties. I understand the error will also be
given if I cant connect with the MySQL db at the creation of the storage
plugin, correct ? Any hints of what I shall use ?
{
"type":"jdbc",
"driver":"com.mysql.jdbc.Driver",
"url":"jdbc:mysql://node03:3306/test",
"username":“mapr",
"password":“mapr",
"enabled”:true
}
3) In another system we could use the storage plugin although it showed
"null" in the drill web-ui ... the configuration could be seen in
zookeeper.
4) When canceling a querying while its running against a big table (~8M
records) drill crashes and we need to restart it.
5) When running “SELECT COUNT(*) from mysql.test;” it is very slow / no
response for very long time (more than 45 seconds).
reg//ulf
Re: JDBC driver for MySQL - storage plugin config and push down ?
Posted by Neeraja Rentachintala <nr...@maprtech.com>.
Andrew
thanks for the update. Also is it possible for you to share a brief set of
instructions on how to configure/use this plugin with a database , lets say
mysql (or postgres or ..).
There have been few question on the threads around this and could benefit
from a quick summary.
-Neeraja
On Wed, Oct 7, 2015 at 10:42 AM, andrew <an...@primer.org> wrote:
> Hi Ulf,
>
> There are 2 issues with the latest build that make the JDBC storage plugin
> unusable. I will be publishing a fix for both later today.
>
> The first issue is that the assembly phase wasn’t including the
> drill-module.conf for the JDBC plugin, so you could never actually register
> a JDBC source.
>
> The second issue is that the bootstrap-storage-plugins.json provided is
> wrong.
>
> I’ll send a note to the list once I’ve published a PR.
>
> - Andrew
>
> > On Oct 6, 2015, at 9:09 AM, Ulf Andreasson @ MapR <
> uandreasson@maprtech.com> wrote:
> >
> > Jacques et al,
> >
> > CP includes /usr/share/java/mysql-connector-java.jar but no diff.
> >
> > Are now running on a single node and have been monitoring /opt/drill/log
> > but the sqlline.log doesn't let away anything when creating a new plugin.
> > The web-ui only says "error (invalid JSON mapping)". Have also tried
> using
> > only {"type": "jdbc"} but the same which surprises me ...
> >
> > I checked
> > /drill/contrib/storage-jdbc/target/classes/bootstrap-storage-plugins.json
> > as for guidance (below) and the only diff is username/password
> >
> > "jdbc" : {
> > type:"jdbc",
> > enabled: false,
> > driver:"org.apache.derby.jdbc.ClientDriver",
> > url:"jdbc:derby://localhost:20000/memory:testDB;"
> > }
> >
> > I am curious for the exact syntax when connecting to mysql and does a
> > failed connection attempt result in a failure in creating/updating the
> > plugin ?
> >
> > reg//ulf
> >
> >
> > reg//ulf
> >
> > --------
> > Ulf Andreasson | Ericsson Global Alliance Solution Engineer, MapR.com |
> +46
> > 72 700 2295
> >
> >
> > On Mon, Oct 5, 2015 at 5:10 PM, Jacques Nadeau <ja...@dremio.com>
> wrote:
> >
> >> Have you added the MySQL jdbc driver to the Drill classpath?
> >>
> >> For better debugging of the issue: drop down to one node and then
> provide
> >> the server side log when you're trying to save/update the jdbc plugin.
> >>
> >> --
> >> Jacques Nadeau
> >> CTO and Co-Founder, Dremio
> >>
> >> On Mon, Oct 5, 2015 at 5:14 AM, Ulf Andreasson @ MapR <
> >> uandreasson@maprtech.com> wrote:
> >>
> >>> Got some questions wrt the JDBC mySQL storage plugin
> >>>
> >>> 1) Given that we are getting close to 1.2 and full release of the
> plugin
> >>> what is the status of the push down function, which clauses have been
> >>> supported by push down?
> >>>
> >>> 2) Creation of my MySQLDB storage plugin gives me Error "(invalid JSON
> >>> mapping)"
> >>> When trying to connect I get the error whatever I define .... even if I
> >>> insert {type:jdbc} or a longer version like below. From seeing the
> code I
> >>> seem to have the right properties. I understand the error will also be
> >>> given if I cant connect with the MySQL db at the creation of the
> storage
> >>> plugin, correct ? Any hints of what I shall use ?
> >>>
> >>> {
> >>> "type":"jdbc",
> >>> "driver":"com.mysql.jdbc.Driver",
> >>> "url":"jdbc:mysql://node03:3306/test",
> >>> "username":“mapr",
> >>> "password":“mapr",
> >>> "enabled”:true
> >>> }
> >>>
> >>> 3) In another system we could use the storage plugin although it showed
> >>> "null" in the drill web-ui ... the configuration could be seen in
> >>> zookeeper.
> >>>
> >>> 4) When canceling a querying while its running against a big table
> (~8M
> >>> records) drill crashes and we need to restart it.
> >>>
> >>> 5) When running “SELECT COUNT(*) from mysql.test;” it is very slow / no
> >>> response for very long time (more than 45 seconds).
> >>>
> >>>
> >>> reg//ulf
> >>>
> >>
>
>
Re: JDBC driver for MySQL - storage plugin config and push down ?
Posted by andrew <an...@primer.org>.
Hi Ulf,
There are 2 issues with the latest build that make the JDBC storage plugin unusable. I will be publishing a fix for both later today.
The first issue is that the assembly phase wasn’t including the drill-module.conf for the JDBC plugin, so you could never actually register a JDBC source.
The second issue is that the bootstrap-storage-plugins.json provided is wrong.
I’ll send a note to the list once I’ve published a PR.
- Andrew
> On Oct 6, 2015, at 9:09 AM, Ulf Andreasson @ MapR <ua...@maprtech.com> wrote:
>
> Jacques et al,
>
> CP includes /usr/share/java/mysql-connector-java.jar but no diff.
>
> Are now running on a single node and have been monitoring /opt/drill/log
> but the sqlline.log doesn't let away anything when creating a new plugin.
> The web-ui only says "error (invalid JSON mapping)". Have also tried using
> only {"type": "jdbc"} but the same which surprises me ...
>
> I checked
> /drill/contrib/storage-jdbc/target/classes/bootstrap-storage-plugins.json
> as for guidance (below) and the only diff is username/password
>
> "jdbc" : {
> type:"jdbc",
> enabled: false,
> driver:"org.apache.derby.jdbc.ClientDriver",
> url:"jdbc:derby://localhost:20000/memory:testDB;"
> }
>
> I am curious for the exact syntax when connecting to mysql and does a
> failed connection attempt result in a failure in creating/updating the
> plugin ?
>
> reg//ulf
>
>
> reg//ulf
>
> --------
> Ulf Andreasson | Ericsson Global Alliance Solution Engineer, MapR.com | +46
> 72 700 2295
>
>
> On Mon, Oct 5, 2015 at 5:10 PM, Jacques Nadeau <ja...@dremio.com> wrote:
>
>> Have you added the MySQL jdbc driver to the Drill classpath?
>>
>> For better debugging of the issue: drop down to one node and then provide
>> the server side log when you're trying to save/update the jdbc plugin.
>>
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>> On Mon, Oct 5, 2015 at 5:14 AM, Ulf Andreasson @ MapR <
>> uandreasson@maprtech.com> wrote:
>>
>>> Got some questions wrt the JDBC mySQL storage plugin
>>>
>>> 1) Given that we are getting close to 1.2 and full release of the plugin
>>> what is the status of the push down function, which clauses have been
>>> supported by push down?
>>>
>>> 2) Creation of my MySQLDB storage plugin gives me Error "(invalid JSON
>>> mapping)"
>>> When trying to connect I get the error whatever I define .... even if I
>>> insert {type:jdbc} or a longer version like below. From seeing the code I
>>> seem to have the right properties. I understand the error will also be
>>> given if I cant connect with the MySQL db at the creation of the storage
>>> plugin, correct ? Any hints of what I shall use ?
>>>
>>> {
>>> "type":"jdbc",
>>> "driver":"com.mysql.jdbc.Driver",
>>> "url":"jdbc:mysql://node03:3306/test",
>>> "username":“mapr",
>>> "password":“mapr",
>>> "enabled”:true
>>> }
>>>
>>> 3) In another system we could use the storage plugin although it showed
>>> "null" in the drill web-ui ... the configuration could be seen in
>>> zookeeper.
>>>
>>> 4) When canceling a querying while its running against a big table (~8M
>>> records) drill crashes and we need to restart it.
>>>
>>> 5) When running “SELECT COUNT(*) from mysql.test;” it is very slow / no
>>> response for very long time (more than 45 seconds).
>>>
>>>
>>> reg//ulf
>>>
>>
Re: JDBC driver for MySQL - storage plugin config and push down ?
Posted by Andrew Selden <as...@dremio.com>.
There are 2 issues with the latest build that make the JDBC storage plugin unusable. I will be publishing a fix for both later today.
The first issue is that the assembly phase wasn’t including the drill-module.conf for the JDBC plugin, so you could never actually register a JDBC source.
The second issue is that the bootstrap-storage-plugins.json provided is wrong.
I’ll send a note to the list once I’ve published a PR.
> On Oct 6, 2015, at 9:09 AM, Ulf Andreasson @ MapR <ua...@maprtech.com> wrote:
>
> Jacques et al,
>
> CP includes /usr/share/java/mysql-connector-java.jar but no diff.
>
> Are now running on a single node and have been monitoring /opt/drill/log
> but the sqlline.log doesn't let away anything when creating a new plugin.
> The web-ui only says "error (invalid JSON mapping)". Have also tried using
> only {"type": "jdbc"} but the same which surprises me ...
>
> I checked
> /drill/contrib/storage-jdbc/target/classes/bootstrap-storage-plugins.json
> as for guidance (below) and the only diff is username/password
>
> "jdbc" : {
> type:"jdbc",
> enabled: false,
> driver:"org.apache.derby.jdbc.ClientDriver",
> url:"jdbc:derby://localhost:20000/memory:testDB;"
> }
>
> I am curious for the exact syntax when connecting to mysql and does a
> failed connection attempt result in a failure in creating/updating the
> plugin ?
>
> reg//ulf
>
>
> reg//ulf
>
> --------
> Ulf Andreasson | Ericsson Global Alliance Solution Engineer, MapR.com | +46
> 72 700 2295
>
>
> On Mon, Oct 5, 2015 at 5:10 PM, Jacques Nadeau <ja...@dremio.com> wrote:
>
>> Have you added the MySQL jdbc driver to the Drill classpath?
>>
>> For better debugging of the issue: drop down to one node and then provide
>> the server side log when you're trying to save/update the jdbc plugin.
>>
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>> On Mon, Oct 5, 2015 at 5:14 AM, Ulf Andreasson @ MapR <
>> uandreasson@maprtech.com> wrote:
>>
>>> Got some questions wrt the JDBC mySQL storage plugin
>>>
>>> 1) Given that we are getting close to 1.2 and full release of the plugin
>>> what is the status of the push down function, which clauses have been
>>> supported by push down?
>>>
>>> 2) Creation of my MySQLDB storage plugin gives me Error "(invalid JSON
>>> mapping)"
>>> When trying to connect I get the error whatever I define .... even if I
>>> insert {type:jdbc} or a longer version like below. From seeing the code I
>>> seem to have the right properties. I understand the error will also be
>>> given if I cant connect with the MySQL db at the creation of the storage
>>> plugin, correct ? Any hints of what I shall use ?
>>>
>>> {
>>> "type":"jdbc",
>>> "driver":"com.mysql.jdbc.Driver",
>>> "url":"jdbc:mysql://node03:3306/test",
>>> "username":“mapr",
>>> "password":“mapr",
>>> "enabled”:true
>>> }
>>>
>>> 3) In another system we could use the storage plugin although it showed
>>> "null" in the drill web-ui ... the configuration could be seen in
>>> zookeeper.
>>>
>>> 4) When canceling a querying while its running against a big table (~8M
>>> records) drill crashes and we need to restart it.
>>>
>>> 5) When running “SELECT COUNT(*) from mysql.test;” it is very slow / no
>>> response for very long time (more than 45 seconds).
>>>
>>>
>>> reg//ulf
>>>
>>
Re: JDBC driver for MySQL - storage plugin config and push down ?
Posted by "Ulf Andreasson @ MapR" <ua...@maprtech.com>.
Jacques et al,
CP includes /usr/share/java/mysql-connector-java.jar but no diff.
Are now running on a single node and have been monitoring /opt/drill/log
but the sqlline.log doesn't let away anything when creating a new plugin.
The web-ui only says "error (invalid JSON mapping)". Have also tried using
only {"type": "jdbc"} but the same which surprises me ...
I checked
/drill/contrib/storage-jdbc/target/classes/bootstrap-storage-plugins.json
as for guidance (below) and the only diff is username/password
"jdbc" : {
type:"jdbc",
enabled: false,
driver:"org.apache.derby.jdbc.ClientDriver",
url:"jdbc:derby://localhost:20000/memory:testDB;"
}
I am curious for the exact syntax when connecting to mysql and does a
failed connection attempt result in a failure in creating/updating the
plugin ?
reg//ulf
reg//ulf
--------
Ulf Andreasson | Ericsson Global Alliance Solution Engineer, MapR.com | +46
72 700 2295
On Mon, Oct 5, 2015 at 5:10 PM, Jacques Nadeau <ja...@dremio.com> wrote:
> Have you added the MySQL jdbc driver to the Drill classpath?
>
> For better debugging of the issue: drop down to one node and then provide
> the server side log when you're trying to save/update the jdbc plugin.
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Mon, Oct 5, 2015 at 5:14 AM, Ulf Andreasson @ MapR <
> uandreasson@maprtech.com> wrote:
>
> > Got some questions wrt the JDBC mySQL storage plugin
> >
> > 1) Given that we are getting close to 1.2 and full release of the plugin
> > what is the status of the push down function, which clauses have been
> > supported by push down?
> >
> > 2) Creation of my MySQLDB storage plugin gives me Error "(invalid JSON
> > mapping)"
> > When trying to connect I get the error whatever I define .... even if I
> > insert {type:jdbc} or a longer version like below. From seeing the code I
> > seem to have the right properties. I understand the error will also be
> > given if I cant connect with the MySQL db at the creation of the storage
> > plugin, correct ? Any hints of what I shall use ?
> >
> > {
> > "type":"jdbc",
> > "driver":"com.mysql.jdbc.Driver",
> > "url":"jdbc:mysql://node03:3306/test",
> > "username":“mapr",
> > "password":“mapr",
> > "enabled”:true
> > }
> >
> > 3) In another system we could use the storage plugin although it showed
> > "null" in the drill web-ui ... the configuration could be seen in
> > zookeeper.
> >
> > 4) When canceling a querying while its running against a big table (~8M
> > records) drill crashes and we need to restart it.
> >
> > 5) When running “SELECT COUNT(*) from mysql.test;” it is very slow / no
> > response for very long time (more than 45 seconds).
> >
> >
> > reg//ulf
> >
>
Re: JDBC driver for MySQL - storage plugin config and push down ?
Posted by Jacques Nadeau <ja...@dremio.com>.
Have you added the MySQL jdbc driver to the Drill classpath?
For better debugging of the issue: drop down to one node and then provide
the server side log when you're trying to save/update the jdbc plugin.
--
Jacques Nadeau
CTO and Co-Founder, Dremio
On Mon, Oct 5, 2015 at 5:14 AM, Ulf Andreasson @ MapR <
uandreasson@maprtech.com> wrote:
> Got some questions wrt the JDBC mySQL storage plugin
>
> 1) Given that we are getting close to 1.2 and full release of the plugin
> what is the status of the push down function, which clauses have been
> supported by push down?
>
> 2) Creation of my MySQLDB storage plugin gives me Error "(invalid JSON
> mapping)"
> When trying to connect I get the error whatever I define .... even if I
> insert {type:jdbc} or a longer version like below. From seeing the code I
> seem to have the right properties. I understand the error will also be
> given if I cant connect with the MySQL db at the creation of the storage
> plugin, correct ? Any hints of what I shall use ?
>
> {
> "type":"jdbc",
> "driver":"com.mysql.jdbc.Driver",
> "url":"jdbc:mysql://node03:3306/test",
> "username":“mapr",
> "password":“mapr",
> "enabled”:true
> }
>
> 3) In another system we could use the storage plugin although it showed
> "null" in the drill web-ui ... the configuration could be seen in
> zookeeper.
>
> 4) When canceling a querying while its running against a big table (~8M
> records) drill crashes and we need to restart it.
>
> 5) When running “SELECT COUNT(*) from mysql.test;” it is very slow / no
> response for very long time (more than 45 seconds).
>
>
> reg//ulf
>