You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kudu.apache.org by Cliff Resnick <cr...@gmail.com> on 2020/01/14 00:23:48 UTC
impala 3.3 and hive metastore
I just built Impala 3.3 from source with Kudu 1.11. Unlike previous
versions, in 3.3 impalad would not start unless installed a local hive
metastore db on the impalad instance, not just the catalogd instance. I'm
now getting a strange "IllegalArgumentException:null" error when creating
kudu tables from impala, where the kudu table is created but impala does
not create metadata. Logs show nothing but I'm guessing this has something
to do with metastore integration. Can I somehow disable metastore
integration from impalad?
x-posting with kudu in case there might be any insight from that side.
Thanks,
Cliff
Re: impala 3.3 and hive metastore
Posted by Cliff Resnick <cr...@gmail.com>.
Hi Jeszy,
Thanks for looking into this. I think we were configuring impalad how you
suggest, but just to be sure, please see my questions inline below:
- impalad should be configured to have a catalog and statestore
> service through its startup flags
>
This is how we've always configured impalad, with remote hosts:
-catalog_service_host=${CATALOG_HOST} -state_store_host=${STATE_STORE_HOST}
> - you should have a hive-site.xml pointing to the metastore URIs on
> the classpath for all deamons (I'm not sure if it's strictly required
> for impalad - maybe others can clear that up -, but assuming yes for
> now).
>
- after this catalogd will manage HMS interactions and propagate
> metadata to the impalad. Local catalog alters the workings of the
> impalad-internal catalog cache and the interaction between catalogd,
> statestored, and impalad, but catalogd remains the source of truth.
We've never needed hive_site on impalad before. In 3.3 impalad will start
with a hive-site.xml referencing *localhost* catalogs, and everything works
except for when we try to create kudu tables (simple CSV tables do not
fail). Are you suggesting we do this with the *remote *metastore host URIs?
> The logs tell that your impalad doesn't find a hive-site.xml, and the
> weird errors could be the result of misconfiguration regarding catalog
> and statestore.
>
> HTH!
>
> On Tue, Jan 14, 2020 at 5:41 PM Cliff Resnick <cr...@gmail.com> wrote:
> >
> > The failed attempt to initialize the metastore was near the very start
> of the impalad log
> > I0114 16:31:24.238524 11902 HiveConf.java:188] Found configuration file
> null
> > I0114 16:31:24.498625 11902 QueryEventHookManager.java:92]
> QueryEventHook config:
> > I0114 16:31:24.498726 11902 QueryEventHookManager.java:93] -
> query_event_hook_nthreads=1
> > I0114 16:31:24.498816 11902 QueryEventHookManager.java:94] -
> query_event_hook_classes=
> > I0114 16:31:24.553463 11902 HiveMetaStoreClient.java:308] HMS client
> filtering is enabled.
> > I0114 16:31:24.618201 11902 HiveMetaStore.java:674] 0: Opening raw store
> with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
> >
> > After this was the usual metastore tables not found errors that you get
> if you have an empty DB.
> >
> > I tried the flag --use_local_catalog=false, even though it's the default.
> >
> > Setting up a metastore with a proper hive-site.xml will get impalad
> running but weird errors ensue on create table, as you would expect. I'm
> doing a straightforward build and deploy, never seen anything like this
> before 3.3.
> >
> >
> >
> > On Tue, Jan 14, 2020, 5:12 AM Jeszy <je...@gmail.com> wrote:
> >>
> >> 'Unlike previous versions, in 3.3 impalad would not start unless
> >> installed a local hive metastore db on the impalad instance, not just
> >> the catalogd instance.' - this sounds weird and is unexpected, and
> >> multiple metastores can easily lead to issues further down the line.
> >> What was the end of the log when the impalad failed to start?
> >>
> >> On Tue, Jan 14, 2020 at 1:24 AM Cliff Resnick <cr...@gmail.com> wrote:
> >> >
> >> > I just built Impala 3.3 from source with Kudu 1.11. Unlike previous
> versions, in 3.3 impalad would not start unless installed a local hive
> metastore db on the impalad instance, not just the catalogd instance. I'm
> now getting a strange "IllegalArgumentException:null" error when creating
> kudu tables from impala, where the kudu table is created but impala does
> not create metadata. Logs show nothing but I'm guessing this has something
> to do with metastore integration. Can I somehow disable metastore
> integration from impalad?
> >> >
> >> > x-posting with kudu in case there might be any insight from that side.
> >> >
> >> > Thanks,
> >> > Cliff
>
Re: impala 3.3 and hive metastore
Posted by Jeszy <je...@gmail.com>.
Hey Cliff,
Thanks.
In general:
- you should have a hive-site.xml pointing to the metastore URIs on
the classpath for all deamons (I'm not sure if it's strictly required
for impalad - maybe others can clear that up -, but assuming yes for
now).
- impalad should be configured to have a catalog and statestore
service through its startup flags
- after this catalogd will manage HMS interactions and propagate
metadata to the impalad. Local catalog alters the workings of the
impalad-internal catalog cache and the interaction between catalogd,
statestored, and impalad, but catalogd remains the source of truth.
The logs tell that your impalad doesn't find a hive-site.xml, and the
weird errors could be the result of misconfiguration regarding catalog
and statestore.
HTH!
On Tue, Jan 14, 2020 at 5:41 PM Cliff Resnick <cr...@gmail.com> wrote:
>
> The failed attempt to initialize the metastore was near the very start of the impalad log
> I0114 16:31:24.238524 11902 HiveConf.java:188] Found configuration file null
> I0114 16:31:24.498625 11902 QueryEventHookManager.java:92] QueryEventHook config:
> I0114 16:31:24.498726 11902 QueryEventHookManager.java:93] - query_event_hook_nthreads=1
> I0114 16:31:24.498816 11902 QueryEventHookManager.java:94] - query_event_hook_classes=
> I0114 16:31:24.553463 11902 HiveMetaStoreClient.java:308] HMS client filtering is enabled.
> I0114 16:31:24.618201 11902 HiveMetaStore.java:674] 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
>
> After this was the usual metastore tables not found errors that you get if you have an empty DB.
>
> I tried the flag --use_local_catalog=false, even though it's the default.
>
> Setting up a metastore with a proper hive-site.xml will get impalad running but weird errors ensue on create table, as you would expect. I'm doing a straightforward build and deploy, never seen anything like this before 3.3.
>
>
>
> On Tue, Jan 14, 2020, 5:12 AM Jeszy <je...@gmail.com> wrote:
>>
>> 'Unlike previous versions, in 3.3 impalad would not start unless
>> installed a local hive metastore db on the impalad instance, not just
>> the catalogd instance.' - this sounds weird and is unexpected, and
>> multiple metastores can easily lead to issues further down the line.
>> What was the end of the log when the impalad failed to start?
>>
>> On Tue, Jan 14, 2020 at 1:24 AM Cliff Resnick <cr...@gmail.com> wrote:
>> >
>> > I just built Impala 3.3 from source with Kudu 1.11. Unlike previous versions, in 3.3 impalad would not start unless installed a local hive metastore db on the impalad instance, not just the catalogd instance. I'm now getting a strange "IllegalArgumentException:null" error when creating kudu tables from impala, where the kudu table is created but impala does not create metadata. Logs show nothing but I'm guessing this has something to do with metastore integration. Can I somehow disable metastore integration from impalad?
>> >
>> > x-posting with kudu in case there might be any insight from that side.
>> >
>> > Thanks,
>> > Cliff
Re: impala 3.3 and hive metastore
Posted by Cliff Resnick <cr...@gmail.com>.
The failed attempt to initialize the metastore was near the very start of
the impalad log
I0114 16:31:24.238524 11902 HiveConf.java:188] Found configuration file null
I0114 16:31:24.498625 11902 QueryEventHookManager.java:92] QueryEventHook
config:
I0114 16:31:24.498726 11902 QueryEventHookManager.java:93] -
query_event_hook_nthreads=1
I0114 16:31:24.498816 11902 QueryEventHookManager.java:94] -
query_event_hook_classes=
I0114 16:31:24.553463 11902 HiveMetaStoreClient.java:308] HMS client
filtering is enabled.
I0114 16:31:24.618201 11902 HiveMetaStore.java:674] 0: Opening raw store
with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
After this was the usual metastore tables not found errors that you get if
you have an empty DB.
I tried the flag --use_local_catalog=false, even though it's the default.
Setting up a metastore with a proper hive-site.xml will get impalad running
but weird errors ensue on create table, as you would expect. I'm doing a
straightforward build and deploy, never seen anything like this before
3.3.
On Tue, Jan 14, 2020, 5:12 AM Jeszy <je...@gmail.com> wrote:
> 'Unlike previous versions, in 3.3 impalad would not start unless
> installed a local hive metastore db on the impalad instance, not just
> the catalogd instance.' - this sounds weird and is unexpected, and
> multiple metastores can easily lead to issues further down the line.
> What was the end of the log when the impalad failed to start?
>
> On Tue, Jan 14, 2020 at 1:24 AM Cliff Resnick <cr...@gmail.com> wrote:
> >
> > I just built Impala 3.3 from source with Kudu 1.11. Unlike previous
> versions, in 3.3 impalad would not start unless installed a local hive
> metastore db on the impalad instance, not just the catalogd instance. I'm
> now getting a strange "IllegalArgumentException:null" error when creating
> kudu tables from impala, where the kudu table is created but impala does
> not create metadata. Logs show nothing but I'm guessing this has something
> to do with metastore integration. Can I somehow disable metastore
> integration from impalad?
> >
> > x-posting with kudu in case there might be any insight from that side.
> >
> > Thanks,
> > Cliff
>
Re: impala 3.3 and hive metastore
Posted by Jeszy <je...@gmail.com>.
'Unlike previous versions, in 3.3 impalad would not start unless
installed a local hive metastore db on the impalad instance, not just
the catalogd instance.' - this sounds weird and is unexpected, and
multiple metastores can easily lead to issues further down the line.
What was the end of the log when the impalad failed to start?
On Tue, Jan 14, 2020 at 1:24 AM Cliff Resnick <cr...@gmail.com> wrote:
>
> I just built Impala 3.3 from source with Kudu 1.11. Unlike previous versions, in 3.3 impalad would not start unless installed a local hive metastore db on the impalad instance, not just the catalogd instance. I'm now getting a strange "IllegalArgumentException:null" error when creating kudu tables from impala, where the kudu table is created but impala does not create metadata. Logs show nothing but I'm guessing this has something to do with metastore integration. Can I somehow disable metastore integration from impalad?
>
> x-posting with kudu in case there might be any insight from that side.
>
> Thanks,
> Cliff
Re: impala 3.3 and hive metastore
Posted by Jeszy <je...@gmail.com>.
'Unlike previous versions, in 3.3 impalad would not start unless
installed a local hive metastore db on the impalad instance, not just
the catalogd instance.' - this sounds weird and is unexpected, and
multiple metastores can easily lead to issues further down the line.
What was the end of the log when the impalad failed to start?
On Tue, Jan 14, 2020 at 1:24 AM Cliff Resnick <cr...@gmail.com> wrote:
>
> I just built Impala 3.3 from source with Kudu 1.11. Unlike previous versions, in 3.3 impalad would not start unless installed a local hive metastore db on the impalad instance, not just the catalogd instance. I'm now getting a strange "IllegalArgumentException:null" error when creating kudu tables from impala, where the kudu table is created but impala does not create metadata. Logs show nothing but I'm guessing this has something to do with metastore integration. Can I somehow disable metastore integration from impalad?
>
> x-posting with kudu in case there might be any insight from that side.
>
> Thanks,
> Cliff