You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by Valencia Serrao <vs...@us.ibm.com> on 2016/12/30 04:17:27 UTC
Issues while creating hbase data tables
Hi All,
While working to get impala 2.6 setup on a Ubuntu16.04 on ppc64le we are
facing following issue while testdata loading:
..................................
16/12/28 01:56:25 ERROR client.AsyncProcess: Failed to get region location
org.apache.hadoop.hbase.TableNotFoundException:
functional_hbase.hbasealltypeserror
.....................................
Here's the complete log. (See attached file: HBaseLog.zip)
Environment info:
HBase: hbase-1.2.0-cdh5.7.0
Hadoop: hadoop-2.6.0-cdh5.7.0
Command used to load data: ${IMPALA_HOME}/buildall.sh -noclean -notests
-testdata
I had also tried using default hbase-1.2.0-cdh5.8.0, however, it throws a
InvalidTargetException and also the HRegionServer service would not start.
Therefore, I switched back to hbase5.7.0.
Checking into the create_hbase.sh file, I executed the following query on
hbase shell manually and got the same error as above:
Query: put 'functional_hbase.hbasealltypeserror', '20', 'd:bigint_col', '0'
IMO the required functional_hbase.hbasealltypeserror table is not getting
created(?). I am further tracing the code to get to the point where the
hbase tables are created i.e. in generate_schema script.
When tried earlier on Ubuntu15.10, data loading was working fine with the
same environment.
Please guide me on this issue.
Regards,
Valencia
Re: Issues while creating hbase data tables
Posted by Valencia Serrao <vs...@us.ibm.com>.
Thanks for your inputs, Jim!. The steps you have outlined in previous
email, match with my findings too. I continued working further on the
issue. Finally, I was able to resolve it by rebuilding hadoop with
snappy-java support inclusion (Dsnappy.prefix) in the mvn command.
From: Jim Apple <jb...@cloudera.com>
To: Valencia Serrao/Austin/Contr/IBM@IBMUS
Cc: "dev@impala" <de...@impala.incubator.apache.org>, Manish
Patil/Austin/Contr/IBM@IBMUS, Nishidha
Panpaliya/Austin/Contr/IBM@IBMUS, Sudarshan
Jagadale/Austin/Contr/IBM@IBMUS
Date: 01/27/2017 03:28 AM
Subject: Re: Issues while creating hbase data tables
It looks to me like functional_schema_template.sql is called from
testdata/bin/generate-schema-statements.py, which is called from
bin/load-data.py, which is called from testdata/bin/create-load-data.sh,
which is called from buildall.sh
$ git grep schema_tem
...
testdata/bin/generate-schema-statements.py:def parse_schema_template_file
(file_name):
testdata/bin/generate-schema-statements.py: schema_template_file =
os.path.join(DATASET_DIR, target_dataset,
testdata/bin/generate-schema-statements.py:
'%s_schema_template.sql' % target_dataset)
...
$ git grep generate-schema-statements
bin/load-data.py:GENERATE_SCHEMA_CMD = "generate-schema-statements.py
--exploration_strategy=%s "\
$ git grep load-data
...
buildall.sh: "${IMPALA_HOME}/testdata/bin/create-load-data.sh" $
{CREATE_LOAD_DATA_ARGS} <<< Y
...
testdata/bin/create-load-data.sh: if ! impala-python -u $
{IMPALA_HOME}/bin/load-data.py ${ARGS[@]} &> ${LOG_FILE}; then
testdata/bin/create-load-data.sh: if ! impala-python -u $
{IMPALA_HOME}/bin/load-data.py --workloads all\
On Wed, Jan 4, 2017 at 10:32 PM, Valencia Serrao <vs...@us.ibm.com>
wrote:
Hi Jim,
Thanks for the inputs. But our requirement is to get Impala 2.6 setup on
Ubunutu16.04.
I continued with debugging this issue, and i can confirm that the error
is due to the table 'functional_hbase.hbasealltypeserror' not getting
created.
To ascertain this , I manually ran the hbase table creation query "create
'functional_hbase.hbasealltypeserror', 'd' " in hbase shell and then
tried the 'put' statements and it worked fine in hbase shell. Later, I
tried to inject the same create query in '$
{IMPALA_HOME}/testdata/HBaseAllTypesError/functional_hbase.hbasealltypeserror.hbase',
used in dataloading sequence and that worked fine too.
I now require to find where the hbase table creation is triggered. For
this, I've tried to trace the script execution flow. However, I'm not
able to clearly reach at the table creation point. Below is the script
execution flow i was tracing.
buildall.sh - > create-load-data.sh -> create-hbase.sh
In create-hbase.sh, I see only the 'put' statements for the hbase table.
I also found a table creation query for '
functional_hbase.hbasealltypeserror' in '$
{IMPALA_HOME}/testdata/datasets/functional/functional_schema_template.sql,
but I could not clearly find where it is triggered.
Am I missing out on something ?
It would be great if you could guide me to find the point where the table
'functional_hbase.hbasealltypeserror' is created.
Regards,
Valencia
Inactive hide details for Jim Apple ---01/03/2017 09:06:00 PM---I think
16.04 has not always worked well. IIRC, Impala 2.6 was Jim Apple
---01/03/2017 09:06:00 PM---I think 16.04 has not always worked well.
IIRC, Impala 2.6 was released only a month or two after Ub
From: Jim Apple <jb...@cloudera.com>
To: "dev@impala" <de...@impala.incubator.apache.org>
Cc: Valencia Serrao/Austin/Contr/IBM@IBMUS, Nishidha
Panpaliya/Austin/Contr/IBM@IBMUS, Sudarshan
Jagadale/Austin/Contr/IBM@IBMUS, Manish Patil/Austin/Contr/IBM@IBMUS
Date: 01/03/2017 09:06 PM
Subject: Re: Issues while creating hbase data tables
I think 16.04 has not always worked well. IIRC, Impala 2.6 was
released only a month or two after Ubuntu 16.04. Our pre-merge testing
and some of our developers use 14.04 with success.
On Thu, Dec 29, 2016 at 8:17 PM, Valencia Serrao <vs...@us.ibm.com>
wrote:
> Hi All,
>
> While working to get impala 2.6 setup on a Ubuntu16.04 on ppc64le we
are
> facing following issue while testdata loading:
>
> ..................................
> 16/12/28 01:56:25 ERROR client.AsyncProcess: Failed to get region
location
> org.apache.hadoop.hbase.TableNotFoundException:
> functional_hbase.hbasealltypeserror
> .....................................
>
> Here's the complete log. (See attached file: HBaseLog.zip)
>
> Environment info:
> HBase: hbase-1.2.0-cdh5.7.0
> Hadoop: hadoop-2.6.0-cdh5.7.0
> Command used to load data: ${IMPALA_HOME}/buildall.sh -noclean -notests
> -testdata
>
> I had also tried using default hbase-1.2.0-cdh5.8.0, however, it throws
a
> InvalidTargetException and also the HRegionServer service would not
start.
> Therefore, I switched back to hbase5.7.0.
>
> Checking into the create_hbase.sh file, I executed the following query
on
> hbase shell manually and got the same error as above:
> Query: put 'functional_hbase.hbasealltypeserror', '20', 'd:bigint_col',
'0'
>
> IMO the required functional_hbase.hbasealltypeserror table is not
getting
> created(?). I am further tracing the code to get to the point where the
> hbase tables are created i.e. in generate_schema script.
>
> When tried earlier on Ubuntu15.10, data loading was working fine with
the
> same environment.
>
> Please guide me on this issue.
>
> Regards,
> Valencia
Re: Issues while creating hbase data tables
Posted by Jim Apple <jb...@cloudera.com>.
It looks to me like functional_schema_template.sql is called from
testdata/bin/generate-schema-statements.py, which is called from
bin/load-data.py, which is called from testdata/bin/create-load-data.sh,
which is called from buildall.sh
$ git grep schema_tem
...
testdata/bin/generate-schema-statements.py:def
parse_schema_template_file(file_name):
testdata/bin/generate-schema-statements.py: schema_template_file =
os.path.join(DATASET_DIR, target_dataset,
testdata/bin/generate-schema-statements.py:
'%s_schema_template.sql' % target_dataset)
...
$ git grep generate-schema-statements
bin/load-data.py:GENERATE_SCHEMA_CMD = "generate-schema-statements.py
--exploration_strategy=%s "\
$ git grep load-data
...
buildall.sh: "${IMPALA_HOME}/testdata/bin/create-load-data.sh"
${CREATE_LOAD_DATA_ARGS} <<< Y
...
testdata/bin/create-load-data.sh: if ! impala-python -u
${IMPALA_HOME}/bin/load-data.py ${ARGS[@]} &> ${LOG_FILE}; then
testdata/bin/create-load-data.sh: if ! impala-python -u
${IMPALA_HOME}/bin/load-data.py --workloads all\
On Wed, Jan 4, 2017 at 10:32 PM, Valencia Serrao <vs...@us.ibm.com> wrote:
> Hi Jim,
>
> Thanks for the inputs. But our requirement is to get Impala 2.6 setup on
> Ubunutu16.04.
>
> I continued with debugging this issue, and i can confirm that the error is
> due to the table 'functional_hbase.hbasealltypeserror' not getting
> created.
> To ascertain this , I manually ran the hbase table creation query *"create
> '**functional_hbase.hbasealltypeserror**', 'd' "* in hbase shell and then
> tried the 'put' statements and it worked fine in hbase shell. Later, I
> tried to inject the same create query in '${IMPALA_HOME}/testdata/
> HBaseAllTypesError/functional_hbase.hbasealltypeserror.hbase', used in
> dataloading sequence and that worked fine too.
>
> I now require to find where the hbase table creation is triggered. For
> this, I've tried to trace the script execution flow. However, I'm not able
> to clearly reach at the table creation point. Below is the script execution
> flow i was tracing.
> *buildall.sh - > create-load-data.sh -> create-hbase.sh*
> In create-hbase.sh, I see only the 'put' statements for the hbase table. I
> also found a table creation query for 'functional_hbase.hbasealltypeserror'
> in '${IMPALA_HOME}/testdata/datasets/functional/
> functional_schema_template.sql, but I could not clearly find where it is
> triggered.
> Am I missing out on something ?
>
> It would be great if you could guide me to find the point where the table '
> functional_hbase.hbasealltypeserror' is created.
>
> Regards,
> Valencia
>
> [image: Inactive hide details for Jim Apple ---01/03/2017 09:06:00 PM---I
> think 16.04 has not always worked well. IIRC, Impala 2.6 was]Jim Apple
> ---01/03/2017 09:06:00 PM---I think 16.04 has not always worked well. IIRC,
> Impala 2.6 was released only a month or two after Ub
>
> From: Jim Apple <jb...@cloudera.com>
> To: "dev@impala" <de...@impala.incubator.apache.org>
> Cc: Valencia Serrao/Austin/Contr/IBM@IBMUS, Nishidha
> Panpaliya/Austin/Contr/IBM@IBMUS, Sudarshan Jagadale/Austin/Contr/IBM@IBMUS,
> Manish Patil/Austin/Contr/IBM@IBMUS
> Date: 01/03/2017 09:06 PM
> Subject: Re: Issues while creating hbase data tables
> ------------------------------
>
>
>
> I think 16.04 has not always worked well. IIRC, Impala 2.6 was
> released only a month or two after Ubuntu 16.04. Our pre-merge testing
> and some of our developers use 14.04 with success.
>
> On Thu, Dec 29, 2016 at 8:17 PM, Valencia Serrao <vs...@us.ibm.com>
> wrote:
> > Hi All,
> >
> > While working to get impala 2.6 setup on a Ubuntu16.04 on ppc64le we are
> > facing following issue while testdata loading:
> >
> > ..................................
> > 16/12/28 01:56:25 ERROR client.AsyncProcess: Failed to get region
> location
> > org.apache.hadoop.hbase.TableNotFoundException:
> > functional_hbase.hbasealltypeserror
> > .....................................
> >
> > Here's the complete log. (See attached file: HBaseLog.zip)
> >
> > Environment info:
> > HBase: hbase-1.2.0-cdh5.7.0
> > Hadoop: hadoop-2.6.0-cdh5.7.0
> > Command used to load data: ${IMPALA_HOME}/buildall.sh -noclean -notests
> > -testdata
> >
> > I had also tried using default hbase-1.2.0-cdh5.8.0, however, it throws a
> > InvalidTargetException and also the HRegionServer service would not
> start.
> > Therefore, I switched back to hbase5.7.0.
> >
> > Checking into the create_hbase.sh file, I executed the following query on
> > hbase shell manually and got the same error as above:
> > Query: put 'functional_hbase.hbasealltypeserror', '20', 'd:bigint_col',
> '0'
> >
> > IMO the required functional_hbase.hbasealltypeserror table is not
> getting
> > created(?). I am further tracing the code to get to the point where the
> > hbase tables are created i.e. in generate_schema script.
> >
> > When tried earlier on Ubuntu15.10, data loading was working fine with the
> > same environment.
> >
> > Please guide me on this issue.
> >
> > Regards,
> > Valencia
>
>
>
>
>
Re: Issues while creating hbase data tables
Posted by Valencia Serrao <vs...@us.ibm.com>.
Hi Jim,
Thanks for the inputs. But our requirement is to get Impala 2.6 setup on
Ubunutu16.04.
I continued with debugging this issue, and i can confirm that the error is
due to the table 'functional_hbase.hbasealltypeserror' not getting created.
To ascertain this , I manually ran the hbase table creation query "create '
functional_hbase.hbasealltypeserror', 'd' " in hbase shell and then tried
the 'put' statements and it worked fine in hbase shell. Later, I tried to
inject the same create query in '$
{IMPALA_HOME}/testdata/HBaseAllTypesError/functional_hbase.hbasealltypeserror.hbase',
used in dataloading sequence and that worked fine too.
I now require to find where the hbase table creation is triggered. For
this, I've tried to trace the script execution flow. However, I'm not able
to clearly reach at the table creation point. Below is the script execution
flow i was tracing.
buildall.sh - > create-load-data.sh -> create-hbase.sh
In create-hbase.sh, I see only the 'put' statements for the hbase table. I
also found a table creation query for 'functional_hbase.hbasealltypeserror'
in '$
{IMPALA_HOME}/testdata/datasets/functional/functional_schema_template.sql,
but I could not clearly find where it is triggered.
Am I missing out on something ?
It would be great if you could guide me to find the point where the table
'functional_hbase.hbasealltypeserror' is created.
Regards,
Valencia
From: Jim Apple <jb...@cloudera.com>
To: "dev@impala" <de...@impala.incubator.apache.org>
Cc: Valencia Serrao/Austin/Contr/IBM@IBMUS, Nishidha
Panpaliya/Austin/Contr/IBM@IBMUS, Sudarshan
Jagadale/Austin/Contr/IBM@IBMUS, Manish
Patil/Austin/Contr/IBM@IBMUS
Date: 01/03/2017 09:06 PM
Subject: Re: Issues while creating hbase data tables
I think 16.04 has not always worked well. IIRC, Impala 2.6 was
released only a month or two after Ubuntu 16.04. Our pre-merge testing
and some of our developers use 14.04 with success.
On Thu, Dec 29, 2016 at 8:17 PM, Valencia Serrao <vs...@us.ibm.com>
wrote:
> Hi All,
>
> While working to get impala 2.6 setup on a Ubuntu16.04 on ppc64le we are
> facing following issue while testdata loading:
>
> ..................................
> 16/12/28 01:56:25 ERROR client.AsyncProcess: Failed to get region
location
> org.apache.hadoop.hbase.TableNotFoundException:
> functional_hbase.hbasealltypeserror
> .....................................
>
> Here's the complete log. (See attached file: HBaseLog.zip)
>
> Environment info:
> HBase: hbase-1.2.0-cdh5.7.0
> Hadoop: hadoop-2.6.0-cdh5.7.0
> Command used to load data: ${IMPALA_HOME}/buildall.sh -noclean -notests
> -testdata
>
> I had also tried using default hbase-1.2.0-cdh5.8.0, however, it throws a
> InvalidTargetException and also the HRegionServer service would not
start.
> Therefore, I switched back to hbase5.7.0.
>
> Checking into the create_hbase.sh file, I executed the following query on
> hbase shell manually and got the same error as above:
> Query: put 'functional_hbase.hbasealltypeserror', '20', 'd:bigint_col',
'0'
>
> IMO the required functional_hbase.hbasealltypeserror table is not getting
> created(?). I am further tracing the code to get to the point where the
> hbase tables are created i.e. in generate_schema script.
>
> When tried earlier on Ubuntu15.10, data loading was working fine with the
> same environment.
>
> Please guide me on this issue.
>
> Regards,
> Valencia
Re: Issues while creating hbase data tables
Posted by Jim Apple <jb...@cloudera.com>.
I think 16.04 has not always worked well. IIRC, Impala 2.6 was
released only a month or two after Ubuntu 16.04. Our pre-merge testing
and some of our developers use 14.04 with success.
On Thu, Dec 29, 2016 at 8:17 PM, Valencia Serrao <vs...@us.ibm.com> wrote:
> Hi All,
>
> While working to get impala 2.6 setup on a Ubuntu16.04 on ppc64le we are
> facing following issue while testdata loading:
>
> ..................................
> 16/12/28 01:56:25 ERROR client.AsyncProcess: Failed to get region location
> org.apache.hadoop.hbase.TableNotFoundException:
> functional_hbase.hbasealltypeserror
> .....................................
>
> Here's the complete log. (See attached file: HBaseLog.zip)
>
> Environment info:
> HBase: hbase-1.2.0-cdh5.7.0
> Hadoop: hadoop-2.6.0-cdh5.7.0
> Command used to load data: ${IMPALA_HOME}/buildall.sh -noclean -notests
> -testdata
>
> I had also tried using default hbase-1.2.0-cdh5.8.0, however, it throws a
> InvalidTargetException and also the HRegionServer service would not start.
> Therefore, I switched back to hbase5.7.0.
>
> Checking into the create_hbase.sh file, I executed the following query on
> hbase shell manually and got the same error as above:
> Query: put 'functional_hbase.hbasealltypeserror', '20', 'd:bigint_col', '0'
>
> IMO the required functional_hbase.hbasealltypeserror table is not getting
> created(?). I am further tracing the code to get to the point where the
> hbase tables are created i.e. in generate_schema script.
>
> When tried earlier on Ubuntu15.10, data loading was working fine with the
> same environment.
>
> Please guide me on this issue.
>
> Regards,
> Valencia