Debugging Impala code

[Nishidha Panpaliya <ni...@us.ibm.com>]

Hi All,

I'm able to build Impala on Ubuntu ppc64le but getting crashes while
loading test data.

I wanted to know how do you normally debug Impala code while loading test
data before running unit tests. Other than core dump, what are the other
ways to find out causes of crash in Impala?


Thanks,
Nishidha

Re: Debugging Impala code

[Skye Wanderman-Milne <sk...@cloudera.com>]

One small note re: logs: I always run my impalad
with -redirect_stdout_stderr=0 and the environment variable
GLOG_logtostderr=1. This causes Impala to output all log messages to
stderr, which I then redirect to a log file (e.g.
$IMPALA_HOME/be/build/debug/service/impalad -redirect_stdout_stderr=0 &>
/tmp/impalad.log). I find it's easier to have all log messages in a single
file for development.

On Thu, Mar 17, 2016 at 10:08 AM, Tim Armstrong <ta...@cloudera.com>
wrote:

> Was it the impalad process that crashed? If so, there are a few places you
> can check:
>
>    - Look in /tmp/impalad.ERROR, /tmp/impalad_node1.ERROR and
>    /tmp/impalad_node2.ERROR for error messages. If it hit an assertion, you
>    will get the message in there.
>    - Look in the equivalent INFO logs for other error messages (for some
>    crashes, there is info sent to INFO but not ERROR)
>    - Look for hs_err_pid*.log files in the directory you ran Impala from.
>    These are crash reports from the embedded JVM in the impalad process
>    - Get impala to produce a core dump (make sure you have ulimit -c
>    unlimited set when starting the cluster. I have it set in my .bashrc
> file)
>    then debug with gdb.
>
>
>
> On Thu, Mar 17, 2016 at 8:59 AM, Jim Apple <jb...@cloudera.com> wrote:
>
> > I believe Hive is sometimes used for data loading, though I'm not sure.
> >
> > I haven't debugged impala during data loading, but when I do need to
> debug
> > the backend, I often do
> >
> > sudo gdb -p $(ps -C impalad -o pid | tail -1 | awk '{print $1}')
> >
> >
> > On Thu, Mar 17, 2016 at 8:50 AM, Nishidha Panpaliya <nishidha@us.ibm.com
> >
> > wrote:
> >
> > >
> > > Hi All,
> > >
> > > I'm able to build Impala on Ubuntu ppc64le but getting crashes while
> > > loading test data.
> > >
> > > I wanted to know how do you normally debug Impala code while loading
> test
> > > data before running unit tests. Other than core dump, what are the
> other
> > > ways to find out causes of crash in Impala?
> > >
> > >
> > > Thanks,
> > > Nishidha
> > >
> >
>

Re: Debugging Impala code

[Tim Armstrong <ta...@cloudera.com>]

Was it the impalad process that crashed? If so, there are a few places you
can check:

   - Look in /tmp/impalad.ERROR, /tmp/impalad_node1.ERROR and
   /tmp/impalad_node2.ERROR for error messages. If it hit an assertion, you
   will get the message in there.
   - Look in the equivalent INFO logs for other error messages (for some
   crashes, there is info sent to INFO but not ERROR)
   - Look for hs_err_pid*.log files in the directory you ran Impala from.
   These are crash reports from the embedded JVM in the impalad process
   - Get impala to produce a core dump (make sure you have ulimit -c
   unlimited set when starting the cluster. I have it set in my .bashrc file)
   then debug with gdb.



On Thu, Mar 17, 2016 at 8:59 AM, Jim Apple <jb...@cloudera.com> wrote:

> I believe Hive is sometimes used for data loading, though I'm not sure.
>
> I haven't debugged impala during data loading, but when I do need to debug
> the backend, I often do
>
> sudo gdb -p $(ps -C impalad -o pid | tail -1 | awk '{print $1}')
>
>
> On Thu, Mar 17, 2016 at 8:50 AM, Nishidha Panpaliya <ni...@us.ibm.com>
> wrote:
>
> >
> > Hi All,
> >
> > I'm able to build Impala on Ubuntu ppc64le but getting crashes while
> > loading test data.
> >
> > I wanted to know how do you normally debug Impala code while loading test
> > data before running unit tests. Other than core dump, what are the other
> > ways to find out causes of crash in Impala?
> >
> >
> > Thanks,
> > Nishidha
> >
>

Re: Debugging Impala code

[Jim Apple <jb...@cloudera.com>]

I believe Hive is sometimes used for data loading, though I'm not sure.

I haven't debugged impala during data loading, but when I do need to debug
the backend, I often do

sudo gdb -p $(ps -C impalad -o pid | tail -1 | awk '{print $1}')


On Thu, Mar 17, 2016 at 8:50 AM, Nishidha Panpaliya <ni...@us.ibm.com>
wrote:

>
> Hi All,
>
> I'm able to build Impala on Ubuntu ppc64le but getting crashes while
> loading test data.
>
> I wanted to know how do you normally debug Impala code while loading test
> data before running unit tests. Other than core dump, what are the other
> ways to find out causes of crash in Impala?
>
>
> Thanks,
> Nishidha
>