You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Xiao Meng <xi...@simba.com> on 2014/04/09 05:05:23 UTC
RE: Setting up Hive...
Hi,
We finally get the hive storage engine up (the previous problem is mainly due to some environmental variable settings)
However, it has problem when we try to query a specific hive table.
For example, we have the following table in the INFROMATION_SCHEMA.TABLES:
| DRILL | hivestg | store_sales_fact_denorm | TABLE |
When we query:
> select * from hivestg.`store_sales_fact_denorm` limit 1;
It shows the following exception:
Query failed: org.apache.drill.exec.rpc.RpcException: Remote failure while running query.[error_id: "8bcf902e-a0c6-461c-94db-916d2188d1b2"
endpoint {
address: <my host name>
user_port: 31016
control_port: 31017
data_port: 31018
}
error_type: 0
message: "Failure while parsing sql. < OptimizerException:[ Failure while attempting to retrieve storage engine. ] < IOException:[ org.apache.drill.common.exceptions.ExecutionSetupException: org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4 ] < ExecutionSetupException:[ org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4 ] < RemoteException:[ Server IPC version 9 cannot communicate with client version 4 ]"
]
It seems a version problem. Any suggestion on this?
BTW, we use the following storage-engines settings:
{
...
hivestg : {
type:"hive",
config :
{
"hive.metastore.uris" : "thrift://<hostname>:9083",
"hive.metastore.sasl.enabled" : "false"
}
}
}
}
Thanks,
Xiao
-----Original Message-----
From: George Chow [mailto:george@overcoil.com]
Sent: March-27-14 5:12 PM
To: drill-dev@incubator.apache.org
Subject: Re: Setting up Hive...
Hi Venki,
Thanks for the tip. I think I may be confused about my setup.
I have a remote metastore running alongside my HiveServer1. Thus, my HiveServer1's hive-site.xml has the line ConnectionURL=jdbc:mysql://localhost:3306/metastore.
I have tried configuring my Drillbit by pointing hive.metastore.uris to this same URL (substituting the IP for localhost). But I'm not seeing anything.
Looking thru the various logs doesn't turn up anything.
I have already tried starting up the metastore in addition to the
HiveServer1 to no avail.
What am I doing wrong?
George
On Tue, Mar 25, 2014 at 7:07 PM, Venki Korukanti
<ve...@gmail.com>wrote:
> On Tue, Mar 25, 2014 at 6:56 PM, George Chow <ge...@overcoil.com> wrote:
>
> > Hi Venki,
> >
> > I made a mistake initially and thought I was embedded. But I'm
> > actually
> in
> > scenario 1 (remote metastore).
>
>
> > I am setting hive.metastore.uris to my HiveServer's address and
> > port. Is that correct?
> >
> hive.metastore.uris should point to metastore server which is
> different from HiveServer1/2. Drill uses Metastore thrift interface to
> fetch the table/db info. From your first email, it looks like you are
> embedding the metastore within HiveServer1. You can copy the following
> settings from you current hiveserver's hive-site.xml to
> storage-engines.json to embed metastore in Drillbit or start the
> metastore as separate service using 'hive --service metastore' and point hive.metastore.uris to the address.
>
> "javax.jdo.option.ConnectionURL" :
> "jdbc:mysql://hostname/hivedrill?createDatabaseIfNotExist=true",
>
> "javax.jdo.option.ConnectionDriverName" : "com.mysql.jdbc.Driver",
>
>
> "javax.jdo.option.ConnectionUserName" : "user",
>
> "javax.jdo.option.ConnectionPassword" : "passwd"
>
>
> Thanks
>
> Venki
>
>
>
>
> > George
> >
> >
> > On Tue, Mar 25, 2014 at 6:48 PM, Venki Korukanti
> > <ve...@gmail.com>wrote:
> >
> > > Hi George,
> > >
> > > Are you embedding the metastore in drillbit
> > > (hive.metastore.uris="") or drillbit is connecting to remote
> > > metastore (hive.metastore.uris=valid address)?
> > >
> > > If you are embedding the metastore, make sure you have the
> > > appropriate driver (if MySql backend then mysql jdbc driver) in
> > > classpath or
> > drill/lib
> > > folder.
> > >
> > > Thanks
> > > Venki
> > >
> > >
> > > On Tue, Mar 25, 2014 at 6:30 PM, George Chow <ge...@overcoil.com>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > Thanks Tim for the tip. It helps... so I copied the settings
> > > > from my hive-site.xml over to storage-engines.json.
> > > >
> > > > I restart my drillbit but I still don't see the contents from my
> > > > Hive metastore. The log file
> > > > (/var/log/drill/node11/drillbits.out) doesn't contain anything about the inclusion of Hive.
> > > >
> > > > (I'm assuming that SELECT * FROM INFORMATION_SCHEMA.TABLES will
> return
> > > more
> > > > than the usual 5 tables (VIEW/COLUMNS/TABLES/CATALOGS/SCHEMATA.)
> > > >
> > > > George
> > > >
> > > >
> > > > On Tue, Mar 25, 2014 at 6:12 PM, Timothy Chen
> > > > <tn...@gmail.com>
> > wrote:
> > > >
> > > > > Hi George,
> > > > >
> > > > > I believe we have a wiki section on setting up hive + drill :
> > > > >
> > > > >
> > https://cwiki.apache.org/confluence/display/DRILL/Hive+Storage+Engin
> > e
> > > > >
> > > > > I haven't personally tried it yet but see if this helps.
> > > > >
> > > > > Tim
> > > > >
> > > > >
> > > > > On Tue, Mar 25, 2014 at 6:10 PM, George Chow
> > > > > <ge...@overcoil.com>
> > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > So I'm trying to setup a drillbit to reach across to my Hive
> > > > metastore. I
> > > > > > dug into JIRA and see that DRILL-354/357 are both resolved.
> > > > > > I've
> > also
> > > > > > looked into my drillbit and found .../conf/storage-engines.json.
> > > > > >
> > > > > > Inside, I see an entry for Hive that's commented out.
> > > > > > However, it
> > is
> > > > for
> > > > > an
> > > > > > instance Hive with a Derby metastore. My instance is a
> HiveServer1
> > > > with a
> > > > > > MySQL metastore. So both javax.jdo.option.ConnectionURL and
> > > > > > hive.metastore.sasl.enabled are clear to me. But what about
> > > > > > the
> > > > > remaining:
> > > > > >
> > > > > > hive.metastore.uris
> > > > > > hive.metastore.warehouse.dir
> > > > > > fs.default.name
> > > > > >
> > > > > > George
> > > > > >
> > > > > > --
> > > > > > --
> > > > > > "Not everything that can be counted counts, and not
> > > > > > everything
> that
> > > > > counts
> > > > > > can be counted." Albert Einstein
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > --
> > > > "Not everything that can be counted counts, and not everything
> > > > that
> > > counts
> > > > can be counted." Albert Einstein
> > > >
> > >
> >
> >
> >
> > --
> > --
> > "Not everything that can be counted counts, and not everything that
> counts
> > can be counted." Albert Einstein
> >
>
--
--
"Not everything that can be counted counts, and not everything that counts can be counted." Albert Einstein
RE: Setting up Hive...
Posted by Xiao Meng <mx...@gmail.com>.
Best,
Xiao
On Tue, Apr 8, 2014 at 8:07 PM, Xiao Meng <xi...@simba.com> wrote:
> Hi,
> We finally get the hive storage engine up (the previous problem is mainly due to some environmental variable settings)
> However, it has problem when we try to query a specific hive table.
> For example, we have the following table in the INFROMATION_SCHEMA.TABLES:
> | DRILL | hivestg | store_sales_fact_denorm | TABLE |
> When we query:
>> select * from hivestg.`store_sales_fact_denorm` limit 1;
> It shows the following exception:
> Query failed: org.apache.drill.exec.rpc.RpcException: Remote failure while running query.[error_id: "8bcf902e-a0c6-461c-94db-916d2188d1b2"
> endpoint {
> address: <my host name>
> user_port: 31016
> control_port: 31017
> data_port: 31018
> }
> error_type: 0
> message: "Failure while parsing sql. < OptimizerException:[ Failure while attempting to retrieve storage engine. ] < IOException:[ org.apache.drill.common.exceptions.ExecutionSetupException: org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4 ] < ExecutionSetupException:[ org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4 ] < RemoteException:[ Server IPC version 9 cannot communicate with client version 4 ]"
> ]
> It seems a version problem. Any suggestion on this?
> BTW, we use the following storage-engines settings:
> {
> ...
> hivestg : {
> type:"hive",
> config :
> {
> "hive.metastore.uris" : "thrift://<hostname>:9083",
> "hive.metastore.sasl.enabled" : "false"
> }
> }
> }
> }
> Thanks,
> Xiao
> -----Original Message-----
> From: George Chow [mailto:george@overcoil.com]
> Sent: March-27-14 5:12 PM
> To: drill-dev@incubator.apache.org
> Subject: Re: Setting up Hive...
> Hi Venki,
> Thanks for the tip. I think I may be confused about my setup.
> I have a remote metastore running alongside my HiveServer1. Thus, my HiveServer1's hive-site.xml has the line ConnectionURL=jdbc:mysql://localhost:3306/metastore.
> I have tried configuring my Drillbit by pointing hive.metastore.uris to this same URL (substituting the IP for localhost). But I'm not seeing anything.
> Looking thru the various logs doesn't turn up anything.
> I have already tried starting up the metastore in addition to the
> HiveServer1 to no avail.
> What am I doing wrong?
> George
> On Tue, Mar 25, 2014 at 7:07 PM, Venki Korukanti
> <ve...@gmail.com>wrote:
>> On Tue, Mar 25, 2014 at 6:56 PM, George Chow <ge...@overcoil.com> wrote:
>>
>> > Hi Venki,
>> >
>> > I made a mistake initially and thought I was embedded. But I'm
>> > actually
>> in
>> > scenario 1 (remote metastore).
>>
>>
>> > I am setting hive.metastore.uris to my HiveServer's address and
>> > port. Is that correct?
>> >
>> hive.metastore.uris should point to metastore server which is
>> different from HiveServer1/2. Drill uses Metastore thrift interface to
>> fetch the table/db info. From your first email, it looks like you are
>> embedding the metastore within HiveServer1. You can copy the following
>> settings from you current hiveserver's hive-site.xml to
>> storage-engines.json to embed metastore in Drillbit or start the
>> metastore as separate service using 'hive --service metastore' and point hive.metastore.uris to the address.
>>
>> "javax.jdo.option.ConnectionURL" :
>> "jdbc:mysql://hostname/hivedrill?createDatabaseIfNotExist=true",
>>
>> "javax.jdo.option.ConnectionDriverName" : "com.mysql.jdbc.Driver",
>>
>>
>> "javax.jdo.option.ConnectionUserName" : "user",
>>
>> "javax.jdo.option.ConnectionPassword" : "passwd"
>>
>>
>> Thanks
>>
>> Venki
>>
>>
>>
>>
>> > George
>> >
>> >
>> > On Tue, Mar 25, 2014 at 6:48 PM, Venki Korukanti
>> > <ve...@gmail.com>wrote:
>> >
>> > > Hi George,
>> > >
>> > > Are you embedding the metastore in drillbit
>> > > (hive.metastore.uris="") or drillbit is connecting to remote
>> > > metastore (hive.metastore.uris=valid address)?
>> > >
>> > > If you are embedding the metastore, make sure you have the
>> > > appropriate driver (if MySql backend then mysql jdbc driver) in
>> > > classpath or
>> > drill/lib
>> > > folder.
>> > >
>> > > Thanks
>> > > Venki
>> > >
>> > >
>> > > On Tue, Mar 25, 2014 at 6:30 PM, George Chow <ge...@overcoil.com>
>> > wrote:
>> > >
>> > > > Hi,
>> > > >
>> > > > Thanks Tim for the tip. It helps... so I copied the settings
>> > > > from my hive-site.xml over to storage-engines.json.
>> > > >
>> > > > I restart my drillbit but I still don't see the contents from my
>> > > > Hive metastore. The log file
>> > > > (/var/log/drill/node11/drillbits.out) doesn't contain anything about the inclusion of Hive.
>> > > >
>> > > > (I'm assuming that SELECT * FROM INFORMATION_SCHEMA.TABLES will
>> return
>> > > more
>> > > > than the usual 5 tables (VIEW/COLUMNS/TABLES/CATALOGS/SCHEMATA.)
>> > > >
>> > > > George
>> > > >
>> > > >
>> > > > On Tue, Mar 25, 2014 at 6:12 PM, Timothy Chen
>> > > > <tn...@gmail.com>
>> > wrote:
>> > > >
>> > > > > Hi George,
>> > > > >
>> > > > > I believe we have a wiki section on setting up hive + drill :
>> > > > >
>> > > > >
>> > https://cwiki.apache.org/confluence/display/DRILL/Hive+Storage+Engin
>> > e
>> > > > >
>> > > > > I haven't personally tried it yet but see if this helps.
>> > > > >
>> > > > > Tim
>> > > > >
>> > > > >
>> > > > > On Tue, Mar 25, 2014 at 6:10 PM, George Chow
>> > > > > <ge...@overcoil.com>
>> > > > wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > So I'm trying to setup a drillbit to reach across to my Hive
>> > > > metastore. I
>> > > > > > dug into JIRA and see that DRILL-354/357 are both resolved.
>> > > > > > I've
>> > also
>> > > > > > looked into my drillbit and found .../conf/storage-engines.json.
>> > > > > >
>> > > > > > Inside, I see an entry for Hive that's commented out.
>> > > > > > However, it
>> > is
>> > > > for
>> > > > > an
>> > > > > > instance Hive with a Derby metastore. My instance is a
>> HiveServer1
>> > > > with a
>> > > > > > MySQL metastore. So both javax.jdo.option.ConnectionURL and
>> > > > > > hive.metastore.sasl.enabled are clear to me. But what about
>> > > > > > the
>> > > > > remaining:
>> > > > > >
>> > > > > > hive.metastore.uris
>> > > > > > hive.metastore.warehouse.dir
>> > > > > > fs.default.name
>> > > > > >
>> > > > > > George
>> > > > > >
>> > > > > > --
>> > > > > > --
>> > > > > > "Not everything that can be counted counts, and not
>> > > > > > everything
>> that
>> > > > > counts
>> > > > > > can be counted." Albert Einstein
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > --
>> > > > "Not everything that can be counted counts, and not everything
>> > > > that
>> > > counts
>> > > > can be counted." Albert Einstein
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > --
>> > "Not everything that can be counted counts, and not everything that
>> counts
>> > can be counted." Albert Einstein
>> >
>>
> --
> --
> "Not everything that can be counted counts, and not everything that counts can be counted." Albert Einstein
RE: Setting up Hive...
Posted by Xiao Meng <xi...@simba.com>.
It is the problem of hadoop/hdfs version. We are running hive 0.12 on top of hadoop 2.x.
But it seems Drill use hadoop 1.x currently.
Best,
Xiao
-----Original Message-----
From: George Chow [mailto:george@overcoil.com]
Sent: April-08-14 10:29 PM
To: Neeraja Rentachintala
Cc: drill-dev@incubator.apache.org; venki.korukanti@gmail.com; Parth Chandra
Subject: Re: Setting up Hive...
This is Hive 0.12 now.
George
On Tue, Apr 8, 2014 at 10:21 PM, Neeraja Rentachintala < nrentachintala@maprtech.com> wrote:
> Which version of Hive are you using.
>
>
> On Tue, Apr 8, 2014 at 8:05 PM, Xiao Meng <xi...@simba.com> wrote:
>
>> Hi,
>>
>> We finally get the hive storage engine up (the previous problem is
>> mainly due to some environmental variable settings)
>>
>> However, it has problem when we try to query a specific hive table.
>>
>> For example, we have the following table in the INFROMATION_SCHEMA.TABLES:
>>
>> | DRILL | hivestg | store_sales_fact_denorm | TABLE |
>>
>> When we query:
>>
>> > select * from hivestg.`store_sales_fact_denorm` limit 1;
>>
>>
>> It shows the following exception:
>>
>>
>> Query failed: org.apache.drill.exec.rpc.RpcException: Remote failure
>> while running query.[error_id: "8bcf902e-a0c6-461c-94db-916d2188d1b2"
>> endpoint {
>> address: <my host name>
>> user_port: 31016
>> control_port: 31017
>> data_port: 31018
>> }
>> error_type: 0
>> message: "Failure while parsing sql. < OptimizerException:[ Failure
>> while attempting to retrieve storage engine. ] < IOException:[
>> org.apache.drill.common.exceptions.ExecutionSetupException:
>> org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot
>> communicate with client version 4 ] < ExecutionSetupException:[
>> org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot
>> communicate with client version 4 ] < RemoteException:[ Server IPC
>> version
>> 9 cannot communicate with client version 4 ]"
>> ]
>>
>> It seems a version problem. Any suggestion on this?
>>
>>
>> BTW, we use the following storage-engines settings:
>>
>> {
>> ...
>> hivestg : {
>> type:"hive",
>> config :
>> {
>> "hive.metastore.uris" : "thrift://<hostname>:9083",
>> "hive.metastore.sasl.enabled" : "false"
>> }
>> }
>> }
>> }
>>
>>
>> Thanks,
>>
>> Xiao
>>
>> -----Original Message-----
>> From: George Chow [mailto:george@overcoil.com]
>> Sent: March-27-14 5:12 PM
>> To: drill-dev@incubator.apache.org
>> Subject: Re: Setting up Hive...
>>
>> Hi Venki,
>>
>> Thanks for the tip. I think I may be confused about my setup.
>>
>> I have a remote metastore running alongside my HiveServer1. Thus, my
>> HiveServer1's hive-site.xml has the line
>> ConnectionURL=jdbc:mysql://localhost:3306/metastore.
>>
>> I have tried configuring my Drillbit by pointing hive.metastore.uris
>> to this same URL (substituting the IP for localhost). But I'm not
>> seeing anything.
>>
>> Looking thru the various logs doesn't turn up anything.
>>
>> I have already tried starting up the metastore in addition to the
>> HiveServer1 to no avail.
>>
>> What am I doing wrong?
>>
>> George
>>
>>
>>
>> On Tue, Mar 25, 2014 at 7:07 PM, Venki Korukanti
>> <ve...@gmail.com>wrote:
>>
>> > On Tue, Mar 25, 2014 at 6:56 PM, George Chow <ge...@overcoil.com>
>> wrote:
>> >
>> > > Hi Venki,
>> > >
>> > > I made a mistake initially and thought I was embedded. But I'm
>> > > actually
>> > in
>> > > scenario 1 (remote metastore).
>> >
>> >
>> > > I am setting hive.metastore.uris to my HiveServer's address and
>> > > port. Is that correct?
>> > >
>> > hive.metastore.uris should point to metastore server which is
>> > different from HiveServer1/2. Drill uses Metastore thrift interface
>> > to fetch the table/db info. From your first email, it looks like
>> > you are embedding the metastore within HiveServer1. You can copy
>> > the following settings from you current hiveserver's hive-site.xml
>> > to storage-engines.json to embed metastore in Drillbit or start the
>> > metastore as separate service using 'hive --service metastore' and
>> point hive.metastore.uris to the address.
>> >
>> > "javax.jdo.option.ConnectionURL" :
>> > "jdbc:mysql://hostname/hivedrill?createDatabaseIfNotExist=true",
>> >
>> > "javax.jdo.option.ConnectionDriverName" :
>> > "com.mysql.jdbc.Driver",
>> >
>> >
>> > "javax.jdo.option.ConnectionUserName" : "user",
>> >
>> > "javax.jdo.option.ConnectionPassword" : "passwd"
>> >
>> >
>> > Thanks
>> >
>> > Venki
>> >
>> >
>> >
>> >
>> > > George
>> > >
>> > >
>> > > On Tue, Mar 25, 2014 at 6:48 PM, Venki Korukanti
>> > > <ve...@gmail.com>wrote:
>> > >
>> > > > Hi George,
>> > > >
>> > > > Are you embedding the metastore in drillbit
>> > > > (hive.metastore.uris="") or drillbit is connecting to remote
>> > > > metastore (hive.metastore.uris=valid address)?
>> > > >
>> > > > If you are embedding the metastore, make sure you have the
>> > > > appropriate driver (if MySql backend then mysql jdbc driver) in
>> > > > classpath or
>> > > drill/lib
>> > > > folder.
>> > > >
>> > > > Thanks
>> > > > Venki
>> > > >
>> > > >
>> > > > On Tue, Mar 25, 2014 at 6:30 PM, George Chow
>> > > > <ge...@overcoil.com>
>> > > wrote:
>> > > >
>> > > > > Hi,
>> > > > >
>> > > > > Thanks Tim for the tip. It helps... so I copied the settings
>> > > > > from my hive-site.xml over to storage-engines.json.
>> > > > >
>> > > > > I restart my drillbit but I still don't see the contents from
>> > > > > my Hive metastore. The log file
>> > > > > (/var/log/drill/node11/drillbits.out) doesn't contain
>> > > > > anything
>> about the inclusion of Hive.
>> > > > >
>> > > > > (I'm assuming that SELECT * FROM INFORMATION_SCHEMA.TABLES
>> > > > > will
>> > return
>> > > > more
>> > > > > than the usual 5 tables
>> > > > > (VIEW/COLUMNS/TABLES/CATALOGS/SCHEMATA.)
>> > > > >
>> > > > > George
>> > > > >
>> > > > >
>> > > > > On Tue, Mar 25, 2014 at 6:12 PM, Timothy Chen
>> > > > > <tn...@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > > Hi George,
>> > > > > >
>> > > > > > I believe we have a wiki section on setting up hive + drill :
>> > > > > >
>> > > > > >
>> > > https://cwiki.apache.org/confluence/display/DRILL/Hive+Storage+En
>> > > gin
>> > > e
>> > > > > >
>> > > > > > I haven't personally tried it yet but see if this helps.
>> > > > > >
>> > > > > > Tim
>> > > > > >
>> > > > > >
>> > > > > > On Tue, Mar 25, 2014 at 6:10 PM, George Chow
>> > > > > > <ge...@overcoil.com>
>> > > > > wrote:
>> > > > > >
>> > > > > > > Hi,
>> > > > > > >
>> > > > > > > So I'm trying to setup a drillbit to reach across to my
>> > > > > > > Hive
>> > > > > metastore. I
>> > > > > > > dug into JIRA and see that DRILL-354/357 are both resolved.
>> > > > > > > I've
>> > > also
>> > > > > > > looked into my drillbit and found
>> .../conf/storage-engines.json.
>> > > > > > >
>> > > > > > > Inside, I see an entry for Hive that's commented out.
>> > > > > > > However, it
>> > > is
>> > > > > for
>> > > > > > an
>> > > > > > > instance Hive with a Derby metastore. My instance is a
>> > HiveServer1
>> > > > > with a
>> > > > > > > MySQL metastore. So both javax.jdo.option.ConnectionURL
>> > > > > > > and hive.metastore.sasl.enabled are clear to me. But what
>> > > > > > > about the
>> > > > > > remaining:
>> > > > > > >
>> > > > > > > hive.metastore.uris
>> > > > > > > hive.metastore.warehouse.dir
>> > > > > > > fs.default.name
>> > > > > > >
>> > > > > > > George
>> > > > > > >
>> > > > > > > --
>> > > > > > > --
>> > > > > > > "Not everything that can be counted counts, and not
>> > > > > > > everything
>> > that
>> > > > > > counts
>> > > > > > > can be counted." Albert Einstein
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > --
>> > > > > "Not everything that can be counted counts, and not
>> > > > > everything that
>> > > > counts
>> > > > > can be counted." Albert Einstein
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > --
>> > > "Not everything that can be counted counts, and not everything
>> > > that
>> > counts
>> > > can be counted." Albert Einstein
>> > >
>> >
>>
>>
>>
>> --
>> --
>> "Not everything that can be counted counts, and not everything that
>> counts can be counted." Albert Einstein
>>
>
>
--
--
"Not everything that can be counted counts, and not everything that counts can be counted." Albert Einstein
Re: Setting up Hive...
Posted by George Chow <ge...@overcoil.com>.
This is Hive 0.12 now.
George
On Tue, Apr 8, 2014 at 10:21 PM, Neeraja Rentachintala <
nrentachintala@maprtech.com> wrote:
> Which version of Hive are you using.
>
>
> On Tue, Apr 8, 2014 at 8:05 PM, Xiao Meng <xi...@simba.com> wrote:
>
>> Hi,
>>
>> We finally get the hive storage engine up (the previous problem is mainly
>> due to some environmental variable settings)
>>
>> However, it has problem when we try to query a specific hive table.
>>
>> For example, we have the following table in the INFROMATION_SCHEMA.TABLES:
>>
>> | DRILL | hivestg | store_sales_fact_denorm | TABLE |
>>
>> When we query:
>>
>> > select * from hivestg.`store_sales_fact_denorm` limit 1;
>>
>>
>> It shows the following exception:
>>
>>
>> Query failed: org.apache.drill.exec.rpc.RpcException: Remote failure
>> while running query.[error_id: "8bcf902e-a0c6-461c-94db-916d2188d1b2"
>> endpoint {
>> address: <my host name>
>> user_port: 31016
>> control_port: 31017
>> data_port: 31018
>> }
>> error_type: 0
>> message: "Failure while parsing sql. < OptimizerException:[ Failure while
>> attempting to retrieve storage engine. ] < IOException:[
>> org.apache.drill.common.exceptions.ExecutionSetupException:
>> org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot
>> communicate with client version 4 ] < ExecutionSetupException:[
>> org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot
>> communicate with client version 4 ] < RemoteException:[ Server IPC version
>> 9 cannot communicate with client version 4 ]"
>> ]
>>
>> It seems a version problem. Any suggestion on this?
>>
>>
>> BTW, we use the following storage-engines settings:
>>
>> {
>> ...
>> hivestg : {
>> type:"hive",
>> config :
>> {
>> "hive.metastore.uris" : "thrift://<hostname>:9083",
>> "hive.metastore.sasl.enabled" : "false"
>> }
>> }
>> }
>> }
>>
>>
>> Thanks,
>>
>> Xiao
>>
>> -----Original Message-----
>> From: George Chow [mailto:george@overcoil.com]
>> Sent: March-27-14 5:12 PM
>> To: drill-dev@incubator.apache.org
>> Subject: Re: Setting up Hive...
>>
>> Hi Venki,
>>
>> Thanks for the tip. I think I may be confused about my setup.
>>
>> I have a remote metastore running alongside my HiveServer1. Thus, my
>> HiveServer1's hive-site.xml has the line
>> ConnectionURL=jdbc:mysql://localhost:3306/metastore.
>>
>> I have tried configuring my Drillbit by pointing hive.metastore.uris to
>> this same URL (substituting the IP for localhost). But I'm not seeing
>> anything.
>>
>> Looking thru the various logs doesn't turn up anything.
>>
>> I have already tried starting up the metastore in addition to the
>> HiveServer1 to no avail.
>>
>> What am I doing wrong?
>>
>> George
>>
>>
>>
>> On Tue, Mar 25, 2014 at 7:07 PM, Venki Korukanti
>> <ve...@gmail.com>wrote:
>>
>> > On Tue, Mar 25, 2014 at 6:56 PM, George Chow <ge...@overcoil.com>
>> wrote:
>> >
>> > > Hi Venki,
>> > >
>> > > I made a mistake initially and thought I was embedded. But I'm
>> > > actually
>> > in
>> > > scenario 1 (remote metastore).
>> >
>> >
>> > > I am setting hive.metastore.uris to my HiveServer's address and
>> > > port. Is that correct?
>> > >
>> > hive.metastore.uris should point to metastore server which is
>> > different from HiveServer1/2. Drill uses Metastore thrift interface to
>> > fetch the table/db info. From your first email, it looks like you are
>> > embedding the metastore within HiveServer1. You can copy the following
>> > settings from you current hiveserver's hive-site.xml to
>> > storage-engines.json to embed metastore in Drillbit or start the
>> > metastore as separate service using 'hive --service metastore' and
>> point hive.metastore.uris to the address.
>> >
>> > "javax.jdo.option.ConnectionURL" :
>> > "jdbc:mysql://hostname/hivedrill?createDatabaseIfNotExist=true",
>> >
>> > "javax.jdo.option.ConnectionDriverName" : "com.mysql.jdbc.Driver",
>> >
>> >
>> > "javax.jdo.option.ConnectionUserName" : "user",
>> >
>> > "javax.jdo.option.ConnectionPassword" : "passwd"
>> >
>> >
>> > Thanks
>> >
>> > Venki
>> >
>> >
>> >
>> >
>> > > George
>> > >
>> > >
>> > > On Tue, Mar 25, 2014 at 6:48 PM, Venki Korukanti
>> > > <ve...@gmail.com>wrote:
>> > >
>> > > > Hi George,
>> > > >
>> > > > Are you embedding the metastore in drillbit
>> > > > (hive.metastore.uris="") or drillbit is connecting to remote
>> > > > metastore (hive.metastore.uris=valid address)?
>> > > >
>> > > > If you are embedding the metastore, make sure you have the
>> > > > appropriate driver (if MySql backend then mysql jdbc driver) in
>> > > > classpath or
>> > > drill/lib
>> > > > folder.
>> > > >
>> > > > Thanks
>> > > > Venki
>> > > >
>> > > >
>> > > > On Tue, Mar 25, 2014 at 6:30 PM, George Chow <ge...@overcoil.com>
>> > > wrote:
>> > > >
>> > > > > Hi,
>> > > > >
>> > > > > Thanks Tim for the tip. It helps... so I copied the settings
>> > > > > from my hive-site.xml over to storage-engines.json.
>> > > > >
>> > > > > I restart my drillbit but I still don't see the contents from my
>> > > > > Hive metastore. The log file
>> > > > > (/var/log/drill/node11/drillbits.out) doesn't contain anything
>> about the inclusion of Hive.
>> > > > >
>> > > > > (I'm assuming that SELECT * FROM INFORMATION_SCHEMA.TABLES will
>> > return
>> > > > more
>> > > > > than the usual 5 tables (VIEW/COLUMNS/TABLES/CATALOGS/SCHEMATA.)
>> > > > >
>> > > > > George
>> > > > >
>> > > > >
>> > > > > On Tue, Mar 25, 2014 at 6:12 PM, Timothy Chen
>> > > > > <tn...@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > > Hi George,
>> > > > > >
>> > > > > > I believe we have a wiki section on setting up hive + drill :
>> > > > > >
>> > > > > >
>> > > https://cwiki.apache.org/confluence/display/DRILL/Hive+Storage+Engin
>> > > e
>> > > > > >
>> > > > > > I haven't personally tried it yet but see if this helps.
>> > > > > >
>> > > > > > Tim
>> > > > > >
>> > > > > >
>> > > > > > On Tue, Mar 25, 2014 at 6:10 PM, George Chow
>> > > > > > <ge...@overcoil.com>
>> > > > > wrote:
>> > > > > >
>> > > > > > > Hi,
>> > > > > > >
>> > > > > > > So I'm trying to setup a drillbit to reach across to my Hive
>> > > > > metastore. I
>> > > > > > > dug into JIRA and see that DRILL-354/357 are both resolved.
>> > > > > > > I've
>> > > also
>> > > > > > > looked into my drillbit and found
>> .../conf/storage-engines.json.
>> > > > > > >
>> > > > > > > Inside, I see an entry for Hive that's commented out.
>> > > > > > > However, it
>> > > is
>> > > > > for
>> > > > > > an
>> > > > > > > instance Hive with a Derby metastore. My instance is a
>> > HiveServer1
>> > > > > with a
>> > > > > > > MySQL metastore. So both javax.jdo.option.ConnectionURL and
>> > > > > > > hive.metastore.sasl.enabled are clear to me. But what about
>> > > > > > > the
>> > > > > > remaining:
>> > > > > > >
>> > > > > > > hive.metastore.uris
>> > > > > > > hive.metastore.warehouse.dir
>> > > > > > > fs.default.name
>> > > > > > >
>> > > > > > > George
>> > > > > > >
>> > > > > > > --
>> > > > > > > --
>> > > > > > > "Not everything that can be counted counts, and not
>> > > > > > > everything
>> > that
>> > > > > > counts
>> > > > > > > can be counted." Albert Einstein
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > --
>> > > > > "Not everything that can be counted counts, and not everything
>> > > > > that
>> > > > counts
>> > > > > can be counted." Albert Einstein
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > --
>> > > "Not everything that can be counted counts, and not everything that
>> > counts
>> > > can be counted." Albert Einstein
>> > >
>> >
>>
>>
>>
>> --
>> --
>> "Not everything that can be counted counts, and not everything that
>> counts can be counted." Albert Einstein
>>
>
>
--
--
"Not everything that can be counted counts, and not everything that counts
can be counted." Albert Einstein
Re: Setting up Hive...
Posted by Neeraja Rentachintala <nr...@maprtech.com>.
Which version of Hive are you using.
On Tue, Apr 8, 2014 at 8:05 PM, Xiao Meng <xi...@simba.com> wrote:
> Hi,
>
> We finally get the hive storage engine up (the previous problem is mainly
> due to some environmental variable settings)
>
> However, it has problem when we try to query a specific hive table.
>
> For example, we have the following table in the INFROMATION_SCHEMA.TABLES:
>
> | DRILL | hivestg | store_sales_fact_denorm | TABLE |
>
> When we query:
>
> > select * from hivestg.`store_sales_fact_denorm` limit 1;
>
>
> It shows the following exception:
>
>
> Query failed: org.apache.drill.exec.rpc.RpcException: Remote failure while
> running query.[error_id: "8bcf902e-a0c6-461c-94db-916d2188d1b2"
> endpoint {
> address: <my host name>
> user_port: 31016
> control_port: 31017
> data_port: 31018
> }
> error_type: 0
> message: "Failure while parsing sql. < OptimizerException:[ Failure while
> attempting to retrieve storage engine. ] < IOException:[
> org.apache.drill.common.exceptions.ExecutionSetupException:
> org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot
> communicate with client version 4 ] < ExecutionSetupException:[
> org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot
> communicate with client version 4 ] < RemoteException:[ Server IPC version
> 9 cannot communicate with client version 4 ]"
> ]
>
> It seems a version problem. Any suggestion on this?
>
>
> BTW, we use the following storage-engines settings:
>
> {
> ...
> hivestg : {
> type:"hive",
> config :
> {
> "hive.metastore.uris" : "thrift://<hostname>:9083",
> "hive.metastore.sasl.enabled" : "false"
> }
> }
> }
> }
>
>
> Thanks,
>
> Xiao
>
> -----Original Message-----
> From: George Chow [mailto:george@overcoil.com]
> Sent: March-27-14 5:12 PM
> To: drill-dev@incubator.apache.org
> Subject: Re: Setting up Hive...
>
> Hi Venki,
>
> Thanks for the tip. I think I may be confused about my setup.
>
> I have a remote metastore running alongside my HiveServer1. Thus, my
> HiveServer1's hive-site.xml has the line
> ConnectionURL=jdbc:mysql://localhost:3306/metastore.
>
> I have tried configuring my Drillbit by pointing hive.metastore.uris to
> this same URL (substituting the IP for localhost). But I'm not seeing
> anything.
>
> Looking thru the various logs doesn't turn up anything.
>
> I have already tried starting up the metastore in addition to the
> HiveServer1 to no avail.
>
> What am I doing wrong?
>
> George
>
>
>
> On Tue, Mar 25, 2014 at 7:07 PM, Venki Korukanti
> <ve...@gmail.com>wrote:
>
> > On Tue, Mar 25, 2014 at 6:56 PM, George Chow <ge...@overcoil.com>
> wrote:
> >
> > > Hi Venki,
> > >
> > > I made a mistake initially and thought I was embedded. But I'm
> > > actually
> > in
> > > scenario 1 (remote metastore).
> >
> >
> > > I am setting hive.metastore.uris to my HiveServer's address and
> > > port. Is that correct?
> > >
> > hive.metastore.uris should point to metastore server which is
> > different from HiveServer1/2. Drill uses Metastore thrift interface to
> > fetch the table/db info. From your first email, it looks like you are
> > embedding the metastore within HiveServer1. You can copy the following
> > settings from you current hiveserver's hive-site.xml to
> > storage-engines.json to embed metastore in Drillbit or start the
> > metastore as separate service using 'hive --service metastore' and point
> hive.metastore.uris to the address.
> >
> > "javax.jdo.option.ConnectionURL" :
> > "jdbc:mysql://hostname/hivedrill?createDatabaseIfNotExist=true",
> >
> > "javax.jdo.option.ConnectionDriverName" : "com.mysql.jdbc.Driver",
> >
> >
> > "javax.jdo.option.ConnectionUserName" : "user",
> >
> > "javax.jdo.option.ConnectionPassword" : "passwd"
> >
> >
> > Thanks
> >
> > Venki
> >
> >
> >
> >
> > > George
> > >
> > >
> > > On Tue, Mar 25, 2014 at 6:48 PM, Venki Korukanti
> > > <ve...@gmail.com>wrote:
> > >
> > > > Hi George,
> > > >
> > > > Are you embedding the metastore in drillbit
> > > > (hive.metastore.uris="") or drillbit is connecting to remote
> > > > metastore (hive.metastore.uris=valid address)?
> > > >
> > > > If you are embedding the metastore, make sure you have the
> > > > appropriate driver (if MySql backend then mysql jdbc driver) in
> > > > classpath or
> > > drill/lib
> > > > folder.
> > > >
> > > > Thanks
> > > > Venki
> > > >
> > > >
> > > > On Tue, Mar 25, 2014 at 6:30 PM, George Chow <ge...@overcoil.com>
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Thanks Tim for the tip. It helps... so I copied the settings
> > > > > from my hive-site.xml over to storage-engines.json.
> > > > >
> > > > > I restart my drillbit but I still don't see the contents from my
> > > > > Hive metastore. The log file
> > > > > (/var/log/drill/node11/drillbits.out) doesn't contain anything
> about the inclusion of Hive.
> > > > >
> > > > > (I'm assuming that SELECT * FROM INFORMATION_SCHEMA.TABLES will
> > return
> > > > more
> > > > > than the usual 5 tables (VIEW/COLUMNS/TABLES/CATALOGS/SCHEMATA.)
> > > > >
> > > > > George
> > > > >
> > > > >
> > > > > On Tue, Mar 25, 2014 at 6:12 PM, Timothy Chen
> > > > > <tn...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi George,
> > > > > >
> > > > > > I believe we have a wiki section on setting up hive + drill :
> > > > > >
> > > > > >
> > > https://cwiki.apache.org/confluence/display/DRILL/Hive+Storage+Engin
> > > e
> > > > > >
> > > > > > I haven't personally tried it yet but see if this helps.
> > > > > >
> > > > > > Tim
> > > > > >
> > > > > >
> > > > > > On Tue, Mar 25, 2014 at 6:10 PM, George Chow
> > > > > > <ge...@overcoil.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > So I'm trying to setup a drillbit to reach across to my Hive
> > > > > metastore. I
> > > > > > > dug into JIRA and see that DRILL-354/357 are both resolved.
> > > > > > > I've
> > > also
> > > > > > > looked into my drillbit and found
> .../conf/storage-engines.json.
> > > > > > >
> > > > > > > Inside, I see an entry for Hive that's commented out.
> > > > > > > However, it
> > > is
> > > > > for
> > > > > > an
> > > > > > > instance Hive with a Derby metastore. My instance is a
> > HiveServer1
> > > > > with a
> > > > > > > MySQL metastore. So both javax.jdo.option.ConnectionURL and
> > > > > > > hive.metastore.sasl.enabled are clear to me. But what about
> > > > > > > the
> > > > > > remaining:
> > > > > > >
> > > > > > > hive.metastore.uris
> > > > > > > hive.metastore.warehouse.dir
> > > > > > > fs.default.name
> > > > > > >
> > > > > > > George
> > > > > > >
> > > > > > > --
> > > > > > > --
> > > > > > > "Not everything that can be counted counts, and not
> > > > > > > everything
> > that
> > > > > > counts
> > > > > > > can be counted." Albert Einstein
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > --
> > > > > "Not everything that can be counted counts, and not everything
> > > > > that
> > > > counts
> > > > > can be counted." Albert Einstein
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > --
> > > "Not everything that can be counted counts, and not everything that
> > counts
> > > can be counted." Albert Einstein
> > >
> >
>
>
>
> --
> --
> "Not everything that can be counted counts, and not everything that counts
> can be counted." Albert Einstein
>