You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by Valencia Serrao <vs...@us.ibm.com> on 2016/06/20 06:54:43 UTC

Issues with Impala dataloading (power8 gcc4.9.3 Impala2.6)


Hi All,

Having identified that there is a LLVM codegen issue with gcc5.x.x, on
ppc64le, a separate thread has been started with the LLVM community by
Nishidha regarding this issue. LLVM community has said that this issue may
get fixed in LLVM's next release(3.8.1 or 3.9 not sure). By the time they
fix it, we better thought to test it once with gcc 4.9.3 on ppc64le . And
we could do that successfully and got the code gen issue fixed.

On the same setup, we started testing the entire testsuite of Impala(FE,
CustomCluster, JDBC, End-toEnd ). To start with, when we tried generating
and loading the testdata for Impala, we encountered the following issue
while loading the testdata.

java.sql.SQLException
        at org.apache.hive.jdbc.HiveStatement.execute
(HiveStatement.java:279)
        at org.apache.hive.beeline.Commands.executeInternal
(Commands.java:893)
        at org.apache.hive.beeline.Commands.execute(Commands.java:1079)
        at org.apache.hive.beeline.Commands.sql(Commands.java:976)
        at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1085)
        at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:917)
        at org.apache.hive.beeline.BeeLine.executeFile(BeeLine.java:895)
        at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:837)
        at org.apache.hive.beeline.BeeLine.mainWithInputRedirection
(BeeLine.java:482)
        at org.apache.hive.beeline.BeeLine.main(BeeLine.java:465)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Closing: 0: jdbc:hive2://localhost:11050/default;auth=noSasl

Caused by: java.net.SocketException: Broken pipe
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite
(SocketOutputStream.java:113)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
	at java.io.BufferedOutputStream.flushBuffer
(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
	at org.apache.thrift.transport.TIOStreamTransport.flush
(TIOStreamTransport.java:159)
	... 22 more
Error executing file from Hive:
load-functional-query-exhaustive-hive-generated.sql


This occurred around 2-3 times, due to this i had to resume the data
loading from the last successful data load. Although the data loading
completed,  we see issues of some tables not found or not created. (maybe
because the data loading was aborted for couple of times.)

Any insights on this issue will be helpful.


Regards,
Valencia

Re: Issues with Impala dataloading (power8 gcc4.9.3 Impala2.6)

Posted by Valencia Serrao <vs...@us.ibm.com>.

Thanks, Jim!

I'll check the logs you mentioned.



From:	Jim Apple <jb...@cloudera.com>
To:	dev@impala.incubator.apache.org
Cc:	Manish Patil/Austin/Contr/IBM@IBMUS, Sudarshan
            Jagadale/Austin/Contr/IBM@IBMUS, Nishidha
            Panpaliya/Austin/Contr/IBM@IBMUS, Valencia
            Serrao/Austin/Contr/IBM@IBMUS, David Clissold/Austin/IBM@IBMUS,
            Zhi Zhi NY Yang <zh...@cn.ibm.com>
Date:	06/20/2016 09:09 PM
Subject:	Re: Issues with Impala dataloading (power8 gcc4.9.3 Impala2.6)



I haven't seen this before, but data loading is sometimes troublesome.
Other Impala developers, have you seen this before?

This looks to me like a Hive error from our
load-functional-query-exhaustive-hive-generated.sql. You might find
some interesting information in
$IMPALA_HOME/cluster_logs/data_loading/.

We have a JIRA to make dataloading easier:
https://issues.cloudera.org/browse/IMPALA-3227

On Sun, Jun 19, 2016 at 11:54 PM, Valencia Serrao <vs...@us.ibm.com>
wrote:
>
>
> Hi All,
>
> Having identified that there is a LLVM codegen issue with gcc5.x.x, on
> ppc64le, a separate thread has been started with the LLVM community by
> Nishidha regarding this issue. LLVM community has said that this issue
may
> get fixed in LLVM's next release(3.8.1 or 3.9 not sure). By the time they
> fix it, we better thought to test it once with gcc 4.9.3 on ppc64le . And
> we could do that successfully and got the code gen issue fixed.
>
> On the same setup, we started testing the entire testsuite of Impala(FE,
> CustomCluster, JDBC, End-toEnd ). To start with, when we tried generating
> and loading the testdata for Impala, we encountered the following issue
> while loading the testdata.
>
> java.sql.SQLException
>         at org.apache.hive.jdbc.HiveStatement.execute
> (HiveStatement.java:279)
>         at org.apache.hive.beeline.Commands.executeInternal
> (Commands.java:893)
>         at org.apache.hive.beeline.Commands.execute(Commands.java:1079)
>         at org.apache.hive.beeline.Commands.sql(Commands.java:976)
>         at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1085)
>         at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:917)
>         at org.apache.hive.beeline.BeeLine.executeFile(BeeLine.java:895)
>         at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:837)
>         at org.apache.hive.beeline.BeeLine.mainWithInputRedirection
> (BeeLine.java:482)
>         at org.apache.hive.beeline.BeeLine.main(BeeLine.java:465)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke
> (NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke
> (DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>
> Closing: 0: jdbc:hive2://localhost:11050/default;auth=noSasl
>
> Caused by: java.net.SocketException: Broken pipe
>         at java.net.SocketOutputStream.socketWrite0(Native Method)
>         at java.net.SocketOutputStream.socketWrite
> (SocketOutputStream.java:113)
>         at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>         at java.io.BufferedOutputStream.flushBuffer
> (BufferedOutputStream.java:82)
>         at java.io.BufferedOutputStream.flush
(BufferedOutputStream.java:140)
>         at org.apache.thrift.transport.TIOStreamTransport.flush
> (TIOStreamTransport.java:159)
>         ... 22 more
> Error executing file from Hive:
> load-functional-query-exhaustive-hive-generated.sql
>
>
> This occurred around 2-3 times, due to this i had to resume the data
> loading from the last successful data load. Although the data loading
> completed,  we see issues of some tables not found or not created. (maybe
> because the data loading was aborted for couple of times.)
>
> Any insights on this issue will be helpful.
>
>
> Regards,
> Valencia

Re: Issues with Impala dataloading (power8 gcc4.9.3 Impala2.6)

Posted by Jim Apple <jb...@cloudera.com>.
I haven't seen this before, but data loading is sometimes troublesome.
Other Impala developers, have you seen this before?

This looks to me like a Hive error from our
load-functional-query-exhaustive-hive-generated.sql. You might find
some interesting information in
$IMPALA_HOME/cluster_logs/data_loading/.

We have a JIRA to make dataloading easier:
https://issues.cloudera.org/browse/IMPALA-3227

On Sun, Jun 19, 2016 at 11:54 PM, Valencia Serrao <vs...@us.ibm.com> wrote:
>
>
> Hi All,
>
> Having identified that there is a LLVM codegen issue with gcc5.x.x, on
> ppc64le, a separate thread has been started with the LLVM community by
> Nishidha regarding this issue. LLVM community has said that this issue may
> get fixed in LLVM's next release(3.8.1 or 3.9 not sure). By the time they
> fix it, we better thought to test it once with gcc 4.9.3 on ppc64le . And
> we could do that successfully and got the code gen issue fixed.
>
> On the same setup, we started testing the entire testsuite of Impala(FE,
> CustomCluster, JDBC, End-toEnd ). To start with, when we tried generating
> and loading the testdata for Impala, we encountered the following issue
> while loading the testdata.
>
> java.sql.SQLException
>         at org.apache.hive.jdbc.HiveStatement.execute
> (HiveStatement.java:279)
>         at org.apache.hive.beeline.Commands.executeInternal
> (Commands.java:893)
>         at org.apache.hive.beeline.Commands.execute(Commands.java:1079)
>         at org.apache.hive.beeline.Commands.sql(Commands.java:976)
>         at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1085)
>         at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:917)
>         at org.apache.hive.beeline.BeeLine.executeFile(BeeLine.java:895)
>         at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:837)
>         at org.apache.hive.beeline.BeeLine.mainWithInputRedirection
> (BeeLine.java:482)
>         at org.apache.hive.beeline.BeeLine.main(BeeLine.java:465)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke
> (NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke
> (DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>
> Closing: 0: jdbc:hive2://localhost:11050/default;auth=noSasl
>
> Caused by: java.net.SocketException: Broken pipe
>         at java.net.SocketOutputStream.socketWrite0(Native Method)
>         at java.net.SocketOutputStream.socketWrite
> (SocketOutputStream.java:113)
>         at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>         at java.io.BufferedOutputStream.flushBuffer
> (BufferedOutputStream.java:82)
>         at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>         at org.apache.thrift.transport.TIOStreamTransport.flush
> (TIOStreamTransport.java:159)
>         ... 22 more
> Error executing file from Hive:
> load-functional-query-exhaustive-hive-generated.sql
>
>
> This occurred around 2-3 times, due to this i had to resume the data
> loading from the last successful data load. Although the data loading
> completed,  we see issues of some tables not found or not created. (maybe
> because the data loading was aborted for couple of times.)
>
> Any insights on this issue will be helpful.
>
>
> Regards,
> Valencia