You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by Tamas Mate <tm...@cloudera.com> on 2019/07/16 16:02:59 UTC
Failing build during data data_loading
Hi All,
With a recent change that seemingly unrelated to the data loading the
builds are failing for me in 'gerrit-verify-dryrun'. Looks like the
provided {error message} is trimmed. I tried to run it with different bases
later, 3 times, but the builds fail after 1h 38 mins. Also, with the latest
build a 0B artifact was generated.
I created a new build env and the master failed during data load, in that
case there were not enough disk space. Could you help me troubleshoot these
builds? I don't know where could I check the NameNode/Impala logs of the
builds.
Change: https://gerrit.cloudera.org/#/c/13806/
Build with artifact:
https://jenkins.impala.io/job/gerrit-verify-dryrun/4587/
Best regards,
Tamas
{error message}
08:26:27 Error executing impala SQL:
/home/ubuntu/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-seq-gzip-block.sql
See:
/home/ubuntu/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-seq-gzip-block.sql.log
FAILED (Took: 4 min 11 sec)
'load-data functional-query exhaustive' failed. Tail of log:
] 08:29:49 load-functional-query-exhaustive-hive-generated-rc-bzip-block.sql
Re: Failing build during data data_loading
Posted by Tamas Mate <tm...@cloudera.com>.
Ah, yes, it is related.
Thanks for the walk through Tim.
On Tue, Jul 16, 2019 at 6:17 PM Tim Armstrong <ta...@cloudera.com>
wrote:
> You can get the logs by clicking through to look at jobs that failed. I'll
> step through since the jobs are somewhat deeply nested
>
> E.g.
> -> https://jenkins.impala.io/job/parallel-all-tests/5991/
> -> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6580/
> -> *08:09:28* 08:09:28 Error executing impala SQL:
>
> /home/ubuntu/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-orc-def-block.sql
> See:
>
> /home/ubuntu/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-orc-def-block.sql.log
> ->
>
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6580/artifact/Impala/logs_static/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-orc-def-block.sql.log
> (found
> via the job's "Build Artifacts" link
> ->
>
> ALTER TABLE alltypes ADD IF NOT EXISTS PARTITION(year=2010, month=7)
> Summary: Returned 1 rows
> Success: True
> Took: 0.0341980457306(s)
> Data:
> New partition has been added to the table.
>
> ERROR: ALTER TABLE alltypes ADD IF NOT EXISTS PARTITION(year=2010, month=8)
> Traceback (most recent call last):
> File "/home/ubuntu/Impala/bin/load-data.py", line 208, in
> exec_impala_query_from_file
> result = impala_client.execute(query)
> File "/home/ubuntu/Impala/tests/beeswax/impala_beeswax.py", line
> 187, in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> File "/home/ubuntu/Impala/tests/beeswax/impala_beeswax.py", line
> 362, in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> File "/home/ubuntu/Impala/tests/beeswax/impala_beeswax.py", line
> 356, in execute_query_async
> handle = self.__do_rpc(lambda: self.imp_service.query(query,))
> File "/home/ubuntu/Impala/tests/beeswax/impala_beeswax.py", line
> 519, in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> ImpalaBeeswaxException: ImpalaBeeswaxException:
> INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
> MESSAGE: NullPointerException: null
>
> If you dig into the catalogd logs for data loading at
>
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6580/artifact/Impala/logs_static/logs/data_loading/catalogd.INFO/*view*/
> and look for a NPE, the backtrace is visible:
>
> I0709 08:09:28.424849 47599 jni-util.cc:288] java.lang.NullPointerException
> at
> org.apache.impala.catalog.monitor.CatalogOpRequestCounter.decrementDdlTypeCounter(CatalogOpRequestCounter.java:66)
> at
> org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:172)
>
> So it probably is in fact related to the patch
>
>
> On Tue, Jul 16, 2019 at 9:03 AM Tamas Mate <tm...@cloudera.com> wrote:
>
> > Hi All,
> >
> > With a recent change that seemingly unrelated to the data loading the
> > builds are failing for me in 'gerrit-verify-dryrun'. Looks like the
> > provided {error message} is trimmed. I tried to run it with different
> bases
> > later, 3 times, but the builds fail after 1h 38 mins. Also, with the
> latest
> > build a 0B artifact was generated.
> >
> > I created a new build env and the master failed during data load, in that
> > case there were not enough disk space. Could you help me troubleshoot
> these
> > builds? I don't know where could I check the NameNode/Impala logs of the
> > builds.
> >
> > Change: https://gerrit.cloudera.org/#/c/13806/
> > Build with artifact:
> > https://jenkins.impala.io/job/gerrit-verify-dryrun/4587/
> >
> > Best regards,
> > Tamas
> >
> > {error message}
> > 08:26:27 Error executing impala SQL:
> >
> >
> /home/ubuntu/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-seq-gzip-block.sql
> > See:
> >
> >
> /home/ubuntu/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-seq-gzip-block.sql.log
> > FAILED (Took: 4 min 11 sec)
> > 'load-data functional-query exhaustive' failed. Tail of log:
> > ] 08:29:49
> > load-functional-query-exhaustive-hive-generated-rc-bzip-block.sql
> >
>
--
Tamas Mate
Senior Customer Operations Engineer
Cloudera
Re: Failing build during data data_loading
Posted by Tim Armstrong <ta...@cloudera.com>.
You can get the logs by clicking through to look at jobs that failed. I'll
step through since the jobs are somewhat deeply nested
E.g.
-> https://jenkins.impala.io/job/parallel-all-tests/5991/
-> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6580/
-> *08:09:28* 08:09:28 Error executing impala SQL:
/home/ubuntu/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-orc-def-block.sql
See:
/home/ubuntu/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-orc-def-block.sql.log
->
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6580/artifact/Impala/logs_static/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-orc-def-block.sql.log
(found
via the job's "Build Artifacts" link
->
ALTER TABLE alltypes ADD IF NOT EXISTS PARTITION(year=2010, month=7)
Summary: Returned 1 rows
Success: True
Took: 0.0341980457306(s)
Data:
New partition has been added to the table.
ERROR: ALTER TABLE alltypes ADD IF NOT EXISTS PARTITION(year=2010, month=8)
Traceback (most recent call last):
File "/home/ubuntu/Impala/bin/load-data.py", line 208, in
exec_impala_query_from_file
result = impala_client.execute(query)
File "/home/ubuntu/Impala/tests/beeswax/impala_beeswax.py", line
187, in execute
handle = self.__execute_query(query_string.strip(), user=user)
File "/home/ubuntu/Impala/tests/beeswax/impala_beeswax.py", line
362, in __execute_query
handle = self.execute_query_async(query_string, user=user)
File "/home/ubuntu/Impala/tests/beeswax/impala_beeswax.py", line
356, in execute_query_async
handle = self.__do_rpc(lambda: self.imp_service.query(query,))
File "/home/ubuntu/Impala/tests/beeswax/impala_beeswax.py", line
519, in __do_rpc
raise ImpalaBeeswaxException(self.__build_error_message(b), b)
ImpalaBeeswaxException: ImpalaBeeswaxException:
INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
MESSAGE: NullPointerException: null
If you dig into the catalogd logs for data loading at
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6580/artifact/Impala/logs_static/logs/data_loading/catalogd.INFO/*view*/
and look for a NPE, the backtrace is visible:
I0709 08:09:28.424849 47599 jni-util.cc:288] java.lang.NullPointerException
at org.apache.impala.catalog.monitor.CatalogOpRequestCounter.decrementDdlTypeCounter(CatalogOpRequestCounter.java:66)
at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:172)
So it probably is in fact related to the patch
On Tue, Jul 16, 2019 at 9:03 AM Tamas Mate <tm...@cloudera.com> wrote:
> Hi All,
>
> With a recent change that seemingly unrelated to the data loading the
> builds are failing for me in 'gerrit-verify-dryrun'. Looks like the
> provided {error message} is trimmed. I tried to run it with different bases
> later, 3 times, but the builds fail after 1h 38 mins. Also, with the latest
> build a 0B artifact was generated.
>
> I created a new build env and the master failed during data load, in that
> case there were not enough disk space. Could you help me troubleshoot these
> builds? I don't know where could I check the NameNode/Impala logs of the
> builds.
>
> Change: https://gerrit.cloudera.org/#/c/13806/
> Build with artifact:
> https://jenkins.impala.io/job/gerrit-verify-dryrun/4587/
>
> Best regards,
> Tamas
>
> {error message}
> 08:26:27 Error executing impala SQL:
>
> /home/ubuntu/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-seq-gzip-block.sql
> See:
>
> /home/ubuntu/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-seq-gzip-block.sql.log
> FAILED (Took: 4 min 11 sec)
> 'load-data functional-query exhaustive' failed. Tail of log:
> ] 08:29:49
> load-functional-query-exhaustive-hive-generated-rc-bzip-block.sql
>