You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafodion.apache.org by Anoop Sharma <an...@esgyn.com> on 2015/09/11 23:24:15 UTC
RE: core trace: HammerDB issues
hi
could you do:
gdb tdm_arkesp core.36479
and send in the trace?
anoop
-----Original Message-----
From: Radu Marias [mailto:radumarias@gmail.com]
Sent: Friday, September 11, 2015 12:26 PM
To: dev <de...@trafodion.incubator.apache.org>
Subject: Re: HammerDB issues
Dave, Narendra
BTW, Amanda has access to our cluster. Please feel free to take a look if
you think it helps in debugging the issue.
On Fri, Sep 11, 2015 at 10:24 PM, Radu Marias <ra...@gmail.com> wrote:
> $ file core.36479
> core.36479: ELF 64-bit LSB core file x86-64, version 1 (SYSV),
> SVR4-style, from 'tdm_arkesp SQMON1.1 00000 00000 036479 $Z000US9
> <IP>:35596 00004 0000'
>
>
> On Fri, Sep 11, 2015 at 10:22 PM, Narendra Goyal
> <narendra.goyal@esgyn.com
> > wrote:
>
>> Hi Radu,
>>
>> Could you please run 'file core*' in the same directory. At least it
>> will identify the program that core'd.
>>
>> Thanks,
>> -Narendra
>>
>> -----Original Message-----
>> From: Radu Marias [mailto:radumarias@gmail.com]
>> Sent: Friday, September 11, 2015 12:15 PM
>> To: dev <de...@trafodion.incubator.apache.org>
>> Subject: Re: HammerDB issues
>>
>> yes,
>>
>> $ pwd
>> /home/trafodion/trafodion-20150828_0830/sql/scripts
>> $ ls -ltrh | grep core.
>> -rwxr-x--- 1 trafodion trafodion 3.3K Aug 28 08:30 sqcorefile
>> -rw------- 1 trafodion trafodion 152M Sep 11 13:27 core.42133
>> -rw------- 1 trafodion trafodion 162M Sep 11 17:36 core.9560
>> -rw------- 1 trafodion trafodion 162M Sep 11 17:39 core.10408
>> -rw------- 1 trafodion trafodion 152M Sep 11 18:27 core.36479
>>
>> All are about 5.3 MB archived. Do you want to connect to the cluster
>> and debug them? Or should I share over email? Because I don't know if
>> sensitive information is included to post them to mailing list also.
>>
>> A quick look, but don't think it helps without symbol-file:
>>
>> Program terminated with signal 11, Segmentation fault.
>> #0 0x00007f619ae22b56 in ?? ()
>> (gdb) bt
>> #0 0x00007f619ae22b56 in ?? ()
>> #1 0x00007f619ae0d710 in ?? ()
>> #2 0x00007f618ceafb30 in ?? ()
>> #3 0x00007fffe7279c10 in ?? ()
>> #4 0x000000000040bd4d in ?? ()
>> #5 0x00007f618ceafb30 in ?? ()
>> #6 0x0000000000000000 in ?? ()
>>
>> On Fri, Sep 11, 2015 at 9:53 PM, Dave Birdsall
>> <da...@esgyn.com>
>> wrote:
>>
>> > Sorry, I meant core dumps from abending processes.
>> >
>> > See any files with file name "core.<node name>.<pid>.<executable
>> > name>" out there?
>> >
>> > If so might be interesting to use gdb to see the stack trace.
>> >
>> > -----Original Message-----
>> > From: Radu Marias [mailto:radumarias@gmail.com]
>> > Sent: Friday, September 11, 2015 11:44 AM
>> > To: dev <de...@trafodion.incubator.apache.org>
>> > Subject: Re: HammerDB issues
>> >
>> > By "core files" you mean mxssmp processes? If so then this is
>> > present on all
>> > nodes:
>> >
>> > $ ps x | grep mxssmp
>> > 46157 ? SNl 0:02 mxssmp SQMON1.1 00004 00004 046157 $ZSM004
>> > 188.138.61.180:35762 00011 00004 00005 SPARE
>> >
>> > I tried this:
>> >
>> > sqstop (then needed to use ckillall because it was stuck) restarted
>> > hdp sqstart
>> >
>> > Now this works from sqlci:
>> >
>> > CREATE UNIQUE INDEX CUSTOMER_I100 ON CUSTOMER (C_W_ID, C_D_ID,
>> > C_LAST, C_FIRST, C_ID);
>> >
>> > But when I run hammerdb without SPJ it inserts data in some tables
>> > and then it fails again with
>> >
>> > Vuser 1:Creating Index CUSTOMER_I2...
>> > Vuser 1:Failed to Create Index
>> >
>> > And now if I run the create statement in sqlci I get the same error
>> > "Operating system error 201".
>> >
>> > *So it seems that while hammerdb inserts data some trafodion
>> > processes went away unexpectedly?*
>> >
>> > On Fri, Sep 11, 2015 at 8:47 PM, Dave Birdsall
>> > <da...@esgyn.com>
>> > wrote:
>> >
>> > > The first one ("Operating system error 201") means a process went
>> > > away unexpectedly. Do you see any core files?
>> > >
>> > > -----Original Message-----
>> > > From: Radu Marias [mailto:radumarias@gmail.com]
>> > > Sent: Friday, September 11, 2015 10:46 AM
>> > > To: dev <de...@trafodion.incubator.apache.org>
>> > > Subject: Re: HammerDB issues
>> > >
>> > > This is what I get for the DDL you mentioned:
>> > >
>> > > >>CREATE UNIQUE INDEX CUSTOMER_I2 ON CUSTOMER (C_W_ID, C_D_ID,
>> > > >>C_LAST,
>> > > C_FIRST, C_ID);
>> > >
>> > > *** ERROR[2034] $Z00083Z:144: Operating system error 201 while
>> > > communicating with server process $Z0201A7:26.
>> > >
>> > > *** ERROR[1081] Loading of index TRAFODION.TPCC.CUSTOMER_I2
>> > > failed unexpectedly.
>> > >
>> > > --- SQL operation failed with errors.
>> > >
>> > > The one for ORDERS is working:
>> > >
>> > > >>CREATE UNIQUE INDEX ORDERS_I2 ON ORDERS (O_W_ID, O_D_ID,
>> > > >>O_C_ID, O_ID);
>> > >
>> > > --- SQL operation complete.
>> > >
>> > > $ sqcheck
>> > > Checking if processes are up.
>> > > Checking attempt: 1; user specified max: 2. Execution time in
>> > > seconds:
>> > > 0.
>> > >
>> > > The SQ environment is up!
>> > >
>> > >
>> > > Process Configured Actual Down
>> > > ------- ---------- ------ ----
>> > > DTM 5 5
>> > > RMS 10 10
>> > > MXOSRVR 20 20
>> > >
>> > > Don't see any errors in trafodion logs.
>> > >
>> > > On Fri, Sep 11, 2015 at 8:19 PM, Radu Marias
>> > > <ra...@gmail.com>
>> > wrote:
>> > >
>> > > > pdsh is available but how should the jar files be replicated?
>> > > >
>> > > > I also installed HammerDB on the other 4 nodes and setup first
>> > > > node as master and the 4 nodes as slaves as described here
>> > > > http://hammerora.sourceforge.net/hammerdb_remote_modes.pdf
>> > > > I also checked "*Copy Stored Procedures to Remote Nodes*" and
>> > > > added the list of all 5 nodes but I don't see *NEWORDER.jar* on
>> > > > the other
>> > > > 4 nodes in *HammerDB-2.18* folder. I assume all the jars are
>> > > > generated on the first node and copied to the others. But it
>> > > > only generates
>> > > > *NEWORDER.jar* on the first node and it fails and it's not
>> > > > present on the other nodes. Also the rest of the stored
>> > > > procedure jars are not created.
>> > > >
>> > > > For the create index DDL used, I changed config.xml file from
>> > > > hammerdb to enable logs in /tmp and in that file I can only see
>> > > > the trimmed statement.
>> > > > Is there anywhere I can see additional logs? Or can I see in
>> > > > trafodion logs the last statement executed from hammerdb over odbc?
>> > > >
>> > > > On Fri, Sep 11, 2015 at 6:38 PM, Suresh Subbiah <
>> > > > suresh.subbiah60@gmail.com> wrote:
>> > > >
>> > > >> Hi Radu,
>> > > >>
>> > > >> Thanks for trying this out.
>> > > >>
>> > > >> I think what is happening here is that the jar file did not
>> > > >> get pushed to all the nodes. HammerDB has so far been tested
>> > > >> on single node instances or multiple nodes where we did some
>> > > >> intermediate steps manually. I don't think HammerDB uses pdsh
>> > > >> to push the jar file to all nodes, or maybe pdsh is somehow
>> > > >> not available. Could you please check if the jar file is
>> > > >> available from every node in the cluster? If this does turn
>> > > >> out to be the problem, I promise to add more info about
>> > > >> HammerDB in the Trafodion wiki and mention a workaround for these
>> > > >> issues.
>> > > >>
>> > > >> For the Failed To Create Index issue, is there an error message?
>> > > >> IF not we can take the create index statement HammerDB is
>> > > >> using and issue it from sqlci. I think thisis the DDL that is
>> > > >> used.
>> > > >>
>> > > >> CREATE UNIQUE INDEX CUSTOMER_I2 ON CUSTOMER (C_W_ID, C_D_ID,
>> > > >> C_LAST, C_FIRST, C_ID) ; CREATE UNIQUE INDEX ORDERS_I2 ON
>> > > >> ORDERS (O_W_ID, O_D_ID, O_C_ID, O_ID) ;
>> > > >>
>> > > >> Thanks
>> > > >> Suresh
>> > > >>
>> > > >>
>> > > >>
>> > > >> On Fri, Sep 11, 2015 at 9:27 AM, Radu Marias
>> > > >> <ra...@gmail.com>
>> > > >> wrote:
>> > > >>
>> > > >> > Hi,
>> > > >> >
>> > > >> > I'm trying to run HammerDB TPC-C from this tutorial
>> > > >> > http://hammerora.sourceforge.net/hammerdb_quickstart_trafodion.
>> > > >> > pd
>> > > >> > f
>> > > >> >
>> > > >> > I have this environment:
>> > > >> >
>> > > >> >
>> > > >> > *CentOS release 6.7 (Final)* *Ambari 2.1.1* *HDP 2.2*
>> > > >> > *trafodion-20150828_0830*
>> > > >> >
>> > > >> > *HammerDB-2.18*
>> > > >> >
>> > > >> > *java version "1.7.0_79"*
>> > > >> >
>> > > >> > On *Schema Build* I get errors when stored procedures are
>> created.
>> > > >> >
>> > > >> > *This is from the hammerdb logs:*
>> > > >> >
>> > > >> > Vuser 1:CREATING TPCC STORED PROCEDURES Vuser 1:Failed to
>> > > >> > create library
>> > > >> /home/trafodion/HammerDB-2.18/NEWORDER.jar
>> > > >> >
>> > > >> > *This is what I see in the UI (attached is also a
>> > > >> > screenshot):*
>> > > >> >
>> > > >> > loading history file ... 0 events added Main console display
>> > > >> > active
>> > > >> > (Tcl8.6.0 / Tk8.6.0) The xml in config.xml is well-formed,
>> > > >> > applying variables Error in Virtual User 1: [Trafodion ODBC
>> > > >> > Driver][Trafodion Database] SQL
>> > > >> > ERROR:*** ERROR[1382] JAR or DLL file
>> > > >> > /home/trafodion/HammerDB-2.18/NEWORDER.jar not found.
>> > > >> > [2015-09-11
>> > > >> > 07:50:53]À€ç¿™À€éŽˆç»„ç¿™À€éŽ„组翙À€éŽ€ç»„ç¿™À€é
>> > > >> > 组翙À€é 组翙À€éŒ¸ç»„ç¿™À€å ¦è šç¿™À€ìªˆÊŒÀ€À€ì¹°Ê‰À€À€é
>> > > >> >
>> > > >> 组翙
>> > > >> À€Õ
>> > > >> €Ë
>> > > >> ®À
>> > > >> €À€ìŸ
>> > > >> > Ì…À€À€À€À€À€À€À€À€À€À€ï¿¿ï¿¿À€À€ì« ï¿¿
>> > > >> À€À€À€À€À€À€À€À€À€À€À€À€À€À€À€À€À€é
>> > > >> > °À€À€À€ì² ʉÀ€À€îˆ°Ò»À€À€éŸ¼ç« ç¿™À€ë¾°Ç´À€À€å–€Ò À€À€ç½
>> > > >> > Ë—À€À€ì² ʉÀ€À€é’€ç»„ç¿™À€ã µç« ç¿™À€é‘ 组翙À€ìµ ʉÀ€À€é
>> > > >> > °ç»„ (executing the statement)
>> > > >> > (HammerDB-2.18) 1 %
>> > > >> >
>> > > >> > By looking in the tpcc schema I see some tables are created
>> > > >> > and are populated with data:
>> > > >> >
>> > > >> > >>set schema tpcc;
>> > > >> >
>> > > >> > --- SQL operation complete.
>> > > >> > >>get tables;
>> > > >> >
>> > > >> > Tables in Schema TRAFODION.TPCC
>> > > >> > ===============================
>> > > >> >
>> > > >> > CUSTOMER
>> > > >> > DISTRICT
>> > > >> > HISTORY
>> > > >> > ITEM
>> > > >> > NEW_ORDER
>> > > >> > ORDERS
>> > > >> > ORDER_LINE
>> > > >> > STOCK
>> > > >> > WAREHOUSE
>> > > >> >
>> > > >> > The file '*/home/trafodion/HammerDB-2.18/NEWORDER.jar*'
>> > > >> > exists, attached is the NEWORDER.java
>> > > >> >
>> > > >> > *When trying to create the stored procedure from sqlci I get
>> > > >> > this:*
>> > > >> >
>> > > >> > set schema tpcc;
>> > > >> > create library testrs file
>> > > >> > '/home/trafodion/HammerDB-2.18/NEWORDER.jar';
>> > > >> > create procedure NEWORD()
>> > > >> > language java
>> > > >> > parameter style java
>> > > >> > external name 'NEWORDER.NEWORD'
>> > > >> > dynamic result sets 1
>> > > >> > library testrs;
>> > > >> >
>> > > >> > *** ERROR[11239] No compatible Java methods named 'NEWORD'
>> > > >> > were found in Java class 'NEWORDER'.
>> > > >> >
>> > > >> > *** ERROR[1231] User-defined routine TRAFODION.TPCC.NEWORDER
>> > > >> > could not
>> > > >> be
>> > > >> > created.
>> > > >> >
>> > > >> > Is there an API change on methods for stored procedures in
>> > > >> > the latest trafodion and hammerdb uses the older syntax?
>> > > >> > This is the
>> > > >> > method:
>> > > >> >
>> > > >> > public static void NEWORD (int no_w_id, int no_max_w_id, int
>> > > >> > no_d_id,
>> > > >> int
>> > > >> > no_c_id, int no_o_ol_cnt, BigDecimal[] no_c_discount,
>> > > >> > String[]
>> > > >> no_c_last,
>> > > >> > String[] no_c_credit, BigDecimal[] no_d_tax, BigDecimal[]
>> > > >> > no_w_tax,
>> > > >> int[]
>> > > >> > no_d_next_o_id, Timestamp tstamp, ResultSet[] opres) throws
>> > > >> > SQLException
>> > > >> >
>> > > >> > Also If I disable "*Build Java Stored Procedures Locally*"
>> > > >> > from hammerdb I get:
>> > > >> >
>> > > >> > Vuser 1:CREATING TPCC INDEXES Vuser 1:Creating Index
>> > > >> > CUSTOMER_I2...
>> > > >> > Vuser 1:Failed to Create Index
>> > > >> >
>> > > >> > --
>> > > >> > And in the end, it's not the years in your life that count.
>> > > >> > It's the
>> > > >> life
>> > > >> > in your years.
>> > > >> >
>> > > >>
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > And in the end, it's not the years in your life that count.
>> > > > It's the life in your years.
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > And in the end, it's not the years in your life that count. It's
>> > > the life in your years.
>> > >
>> >
>> >
>> >
>> > --
>> > And in the end, it's not the years in your life that count. It's
>> > the life in your years.
>> >
>>
>>
>>
>> --
>> And in the end, it's not the years in your life that count. It's the
>> life in your years.
>>
>
>
>
> --
> And in the end, it's not the years in your life that count. It's the life
> in your years.
>
--
And in the end, it's not the years in your life that count. It's the life
in your years.