You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafodion.apache.org by Selva Govindarajan <se...@esgyn.com> on 2016/04/07 00:56:12 UTC
Change in sqenvcom.sh - Response required
Change in sqenvcom.sh - Response required
Changed the subject line to avoid filtering….
_____________________________________________
*From:* Selva Govindarajan [mailto:selva.govindarajan@esgyn.com
<se...@esgyn.com>]
*Sent:* Wednesday, April 6, 2016 3:25 PM
*To:* 'dev@trafodion.incubator.apache.org' <
dev@trafodion.incubator.apache.org>
*Subject:* RE: Trafodion master Daily Test Result - 167 - Still Failing -
Change in sqenvcom.sh
It looks like color didn’t show up in the message correctly. Line 351 was
added.
_____________________________________________
*From:* Selva Govindarajan [*mailto:selva.govindarajan@esgyn.com*
<se...@esgyn.com>]
*Sent:* Wednesday, April 6, 2016 3:21 PM
*To:* 'dev@trafodion.incubator.apache.org' <
*dev@trafodion.incubator.apache.org* <de...@trafodion.incubator.apache.org>>
*Subject:* RE: Trafodion master Daily Test Result - 167 - Still Failing -
Change in sqenvcom.sh
I added the line in red to sqenvcom.sh and removed ms.env and regenerated
ms.env.
347 # directories with jar files and list of jar files
348 # (could try to reduce the number of jars in the classpath)
349 export HADOOP_JAR_DIRS="/usr/lib/hadoop
350 /usr/lib/hadoop/lib
351 /usr/lib/hadoop-0.20-mapreduce"
This makes hive to work from Trafodion. But, I am not sure if the Jenkins
installation is done differently from other CDH installation. Also, I am
not sure if it is ok to change sqenvcom.sh this way. If so, we can make
this change as a short term solution. So, please provide your feedback on
this.
However, as a long term solution I feel sqenvcom.sh is becoming quite
complex. There is a need to simplify it. One of the suggestion would be to
generate sqenvcom.sh based on the distro rather than making sqenvcom.sh to
cater to different distros. Even if we don’t generate the file fully, we
should consider generating CLASSPATH for different distro and source in to
sqenvcom.sh only the generated installed distro CLASSPATH.
To generate the CLASSPATH, Narendra said we can do the following
hadoop classpath
hbase classpath
And expand all the * in the resulting class path to include all the jar
files in the path (We don’t expand *, the jar files are not seen from
JNI). This would make it better to cater to installation specific
classpath changes too.
Selva
-----Original Message-----
From: Steve Varnau [*mailto:steve.varnau@esgyn.com* <st...@esgyn.com>
]
Sent: Wednesday, April 6, 2016 9:55 AM
To: *dev@trafodion.incubator.apache.org*
<de...@trafodion.incubator.apache.org>
Subject: RE: Trafodion master Daily Test Result - 167 - Still Failing
Yes. Before we hung trying to load the tpc-ds data into hive. That script
seems to be running fine now, and the test suite actually runs, though all
the hive related tests fail. So something else is not right.
Test automation saves the hbase logs, but not hive logs, so I think we will
need to spin up a vm to debug it. Who would like a crack at this one?
--Steve
> -----Original Message-----
> From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com
<se...@esgyn.com>]
> Sent: Wednesday, April 6, 2016 9:19 AM
> To: dev@trafodion.incubator.apache.org
> Subject: RE: Trafodion master Daily Test Result - 167 - Still Failing
>
> I think it looks like a class path issue for hive tests in CDH. Need
> to be analyzed in CDH vm.
>
> Selva
>
> -----Original Message-----
> From: Sandhya Sundaresan [mailto:sandhya.sundaresan@esgyn.com
<sa...@esgyn.com>]
> Sent: Wednesday, April 6, 2016 7:48 AM
> To: dev@trafodion.incubator.apache.org
> Subject: RE: Trafodion master Daily Test Result - 167 - Still Failing
>
> The executor suite passed after weeks but the problematic test
> (TEST106) has been commented out until the new hang for "drop table"
> is fixed just so the rest of the tests in the suite get run. The old
> hang seems to be fixed now.
>
> Steve, looks like the Hive metadata is corrupt or not setup properly
> on the VMs where the core and hive suites are run.
>
> Arvind, I suppose you're following up on the phoenix failures ?
>
> Sandhya
>
> -----Original Message-----
> From: steve.varnau@esgyn.com [mailto:steve.varnau@esgyn.com
<st...@esgyn.com>]
> Sent: Wednesday, April 6, 2016 4:02 AM
> To: dev@trafodion.incubator.apache.org
> Subject: Trafodion master Daily Test Result - 167 - Still Failing
>
> Daily Automated Testing master
>
> Jenkins Job: https://jenkins.esgyn.com/job/Check-Daily-master/167/
> Archived Logs: http://traf-testlogs.esgyn.com/Daily-master/167
> Bld Downloads: http://traf-builds.esgyn.com
>
> Changes since previous daily build:
> [Steve Varnau] [TRAFODION-1816] License info for binary distribution
> of REST bundle
>
> [selvaganesang] [TRAFODION-1918] Remove auto flush concept in
> Trafodion
>
> [selvaganesang] [TRAFODION-1918] Remove disabling auto flush concept
> in Trafodion
>
>
>
> Test Job Results:
>
> FAILURE core-regress-core-cdh (1 hr 34 min) FAILURE
> core-regress-hive-cdh
> (22 min) FAILURE core-regress-privs1-cdh (41 min) FAILURE
> phoenix_part1_T2-cdh (11 min) FAILURE phoenix_part1_T2-hdp (39 min)
> FAILURE phoenix_part2_T2-cdh (11 min) FAILURE phoenix_part2_T2-hdp (38
> min) SUCCESS build-master-debug (25 min) SUCCESS build-master-release
> (26 min) SUCCESS core-regress-charsets-cdh (27 min) SUCCESS
> core-regress-charsets-hdp (1 hr
> 29 min) SUCCESS core-regress-compGeneral-cdh (1 hr 25 min) SUCCESS
> core-regress-compGeneral-hdp (47 min) SUCCESS core-regress-core-hdp (1
> hr
> 1
> min) SUCCESS core-regress-executor-cdh (49 min) SUCCESS
> core-regress-executor-hdp (1 hr 3 min) SUCCESS
> core-regress-fullstack2-cdh
> (1 hr 2 min) SUCCESS core-regress-fullstack2-hdp (14 min) SUCCESS
> core-regress-hive-hdp (41 min) SUCCESS core-regress-privs1-hdp (1 hr 1
> min)
> SUCCESS core-regress-privs2-cdh (35 min) SUCCESS
> core-regress-privs2-hdp
> (43
> min) SUCCESS core-regress-qat-cdh (14 min) SUCCESS
> core-regress-qat-hdp (1 hr 13 min) SUCCESS core-regress-seabase-cdh (1
> hr 45 min) SUCCESS core-regress-seabase-hdp (2 hr 7 min) SUCCESS
> core-regress-udr-cdh (18
> min)
> SUCCESS core-regress-udr-hdp (40 min) SUCCESS jdbc_test-cdh (50 min)
> SUCCESS jdbc_test-hdp (57 min) SUCCESS phoenix_part1_T4-cdh (42 min)
> SUCCESS phoenix_part1_T4-hdp (1 hr 26 min) SUCCESS
> phoenix_part2_T4-cdh (44 min) SUCCESS phoenix_part2_T4-hdp (1 hr 4
> min) SUCCESS pyodbc_test-cdh (10 min) SUCCESS pyodbc_test-hdp (38 min)
RE: Change in sqenvcom.sh - Response required
Posted by Selva Govindarajan <se...@esgyn.com>.
In the install_local_hadoop installation, the class
org/apache/hadoop/mapred/MRVersion is in
hadoop-mapreduce-client-common-2.6.0-cdh5.4.4.jar. But, I think this class
is not found in Jenkins installation in
hadoop-mapreduce-client-common-2.6.0-cdh5.4.4.jar. This jar is part of
class path as seen from the console output in regression logs. But we need
to check if it is part of CLASSPATH in ms.env. We need to consider saving
off etc directory too as part of regression. This class path is definitely
found in the MR1 directory. So there is some installation difference.
Selva
----Original Message-----
From: Steve Varnau [mailto:steve.varnau@esgyn.com]
Sent: Thursday, April 7, 2016 9:50 AM
To: dev@trafodion.incubator.apache.org
Subject: RE: Change in sqenvcom.sh - Response required
I did not intentionally do anything strange with the CDH set-up, just
install CDH5.4.
A little googling tells me that the Hadoop-0.20-mapred package is MR1. It is
optional, but valid to have both MR1 and MR2 installed on a system at the
same time. So I'm not sure if this is a mismatch between hive and trafodion
classpaths.
--Steve
> -----Original Message-----
> From: Dave Birdsall [mailto:dave.birdsall@esgyn.com]
> Sent: Thursday, April 7, 2016 8:32 AM
> To: dev@trafodion.incubator.apache.org
> Subject: RE: Change in sqenvcom.sh - Response required
>
> Hi Selva,
>
> I'm no expert in this, but your short-term change to sqenvcom.sh
> sounds good.
>
> I wonder if Steve could comment on the Jenkins angle?
>
> Dave
>
> -----Original Message-----
> From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
> Sent: Wednesday, April 6, 2016 3:56 PM
> To: dev@trafodion.incubator.apache.org
> Subject: Change in sqenvcom.sh - Response required
>
> Change in sqenvcom.sh - Response required
>
> Changed the subject line to avoid filtering….
>
> _____________________________________________
> *From:* Selva Govindarajan [mailto:selva.govindarajan@esgyn.com
> <se...@esgyn.com>]
> *Sent:* Wednesday, April 6, 2016 3:25 PM
> *To:* 'dev@trafodion.incubator.apache.org' <
> dev@trafodion.incubator.apache.org>
> *Subject:* RE: Trafodion master Daily Test Result - 167 - Still
> Failing - Change in sqenvcom.sh
>
> It looks like color didn’t show up in the message correctly. Line 351
> was added.
>
> _____________________________________________
> *From:* Selva Govindarajan [*mailto:selva.govindarajan@esgyn.com*
> <se...@esgyn.com>]
> *Sent:* Wednesday, April 6, 2016 3:21 PM
> *To:* 'dev@trafodion.incubator.apache.org' <
> *dev@trafodion.incubator.apache.org*
> <de...@trafodion.incubator.apache.org>>
> *Subject:* RE: Trafodion master Daily Test Result - 167 - Still
> Failing - Change in sqenvcom.sh
>
> I added the line in red to sqenvcom.sh and removed ms.env and
> regenerated ms.env.
>
> 347 # directories with jar files and list of jar files
>
> 348 # (could try to reduce the number of jars in the classpath)
>
> 349 export HADOOP_JAR_DIRS="/usr/lib/hadoop
>
> 350 /usr/lib/hadoop/lib
>
> 351 /usr/lib/hadoop-0.20-mapreduce"
>
>
> This makes hive to work from Trafodion. But, I am not sure if the
> Jenkins installation is done differently from other CDH installation.
> Also, I am not sure if it is ok to change sqenvcom.sh this way. If so,
> we can make this change as a short term solution. So, please provide
> your feedback on this.
>
> However, as a long term solution I feel sqenvcom.sh is becoming quite
> complex. There is a need to simplify it. One of the suggestion would
> be to generate sqenvcom.sh based on the distro rather than making
> sqenvcom.sh to cater to different distros. Even if we don’t generate
> the file fully, we should consider generating CLASSPATH for different
> distro and source in to sqenvcom.sh only the generated installed
> distro CLASSPATH.
>
> To generate the CLASSPATH, Narendra said we can do the following
>
> hadoop classpath
>
> hbase classpath
>
> And expand all the * in the resulting class path to include all the
> jar files in the path (We don’t expand *, the jar files are not seen
> from JNI).
> This would make it better to cater to installation specific classpath
> changes too.
>
> Selva
>
> -----Original Message-----
> From: Steve Varnau [*mailto:steve.varnau@esgyn.com*
> <st...@esgyn.com> ]
> Sent: Wednesday, April 6, 2016 9:55 AM
> To: *dev@trafodion.incubator.apache.org*
> <de...@trafodion.incubator.apache.org>
> Subject: RE: Trafodion master Daily Test Result - 167 - Still Failing
>
> Yes. Before we hung trying to load the tpc-ds data into hive. That
> script seems to be running fine now, and the test suite actually runs,
> though all the hive related tests fail. So something else is not right.
>
> Test automation saves the hbase logs, but not hive logs, so I think we
> will need to spin up a vm to debug it. Who would like a crack at this
> one?
>
> --Steve
>
> > -----Original Message-----
>
> > From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com
> <se...@esgyn.com>]
>
> > Sent: Wednesday, April 6, 2016 9:19 AM
>
> > To: dev@trafodion.incubator.apache.org
>
> > Subject: RE: Trafodion master Daily Test Result - 167 - Still
> > Failing
>
> >
>
> > I think it looks like a class path issue for hive tests in CDH.
> > Need
>
> > to be analyzed in CDH vm.
>
> >
>
> > Selva
>
> >
>
> > -----Original Message-----
>
> > From: Sandhya Sundaresan [mailto:sandhya.sundaresan@esgyn.com
> <sa...@esgyn.com>]
>
> > Sent: Wednesday, April 6, 2016 7:48 AM
>
> > To: dev@trafodion.incubator.apache.org
>
> > Subject: RE: Trafodion master Daily Test Result - 167 - Still
> > Failing
>
> >
>
> > The executor suite passed after weeks but the problematic test
>
> > (TEST106) has been commented out until the new hang for "drop table"
>
> > is fixed just so the rest of the tests in the suite get run. The
> > old
>
> > hang seems to be fixed now.
>
> >
>
> > Steve, looks like the Hive metadata is corrupt or not setup properly
>
> > on the VMs where the core and hive suites are run.
>
> >
>
> > Arvind, I suppose you're following up on the phoenix failures ?
>
> >
>
> > Sandhya
>
> >
>
> > -----Original Message-----
>
> > From: steve.varnau@esgyn.com [mailto:steve.varnau@esgyn.com
> <st...@esgyn.com>]
>
> > Sent: Wednesday, April 6, 2016 4:02 AM
>
> > To: dev@trafodion.incubator.apache.org
>
> > Subject: Trafodion master Daily Test Result - 167 - Still Failing
>
> >
>
> > Daily Automated Testing master
>
> >
>
> > Jenkins Job: https://jenkins.esgyn.com/job/Check-Daily-master/167/
>
> > Archived Logs: http://traf-testlogs.esgyn.com/Daily-master/167
>
> > Bld Downloads: http://traf-builds.esgyn.com
>
> >
>
> > Changes since previous daily build:
>
> > [Steve Varnau] [TRAFODION-1816] License info for binary distribution
>
> > of REST bundle
>
> >
>
> > [selvaganesang] [TRAFODION-1918] Remove auto flush concept in
>
> > Trafodion
>
> >
>
> > [selvaganesang] [TRAFODION-1918] Remove disabling auto flush concept
>
> > in Trafodion
>
> >
>
> >
>
> >
>
> > Test Job Results:
>
> >
>
> > FAILURE core-regress-core-cdh (1 hr 34 min) FAILURE
>
> > core-regress-hive-cdh
>
> > (22 min) FAILURE core-regress-privs1-cdh (41 min) FAILURE
>
> > phoenix_part1_T2-cdh (11 min) FAILURE phoenix_part1_T2-hdp (39 min)
>
> > FAILURE phoenix_part2_T2-cdh (11 min) FAILURE phoenix_part2_T2-hdp
> > (38
>
> > min) SUCCESS build-master-debug (25 min) SUCCESS
> > build-master-release
>
> > (26 min) SUCCESS core-regress-charsets-cdh (27 min) SUCCESS
>
> > core-regress-charsets-hdp (1 hr
>
> > 29 min) SUCCESS core-regress-compGeneral-cdh (1 hr 25 min) SUCCESS
>
> > core-regress-compGeneral-hdp (47 min) SUCCESS core-regress-core-hdp
> > (1
>
> > hr
>
> > 1
>
> > min) SUCCESS core-regress-executor-cdh (49 min) SUCCESS
>
> > core-regress-executor-hdp (1 hr 3 min) SUCCESS
>
> > core-regress-fullstack2-cdh
>
> > (1 hr 2 min) SUCCESS core-regress-fullstack2-hdp (14 min) SUCCESS
>
> > core-regress-hive-hdp (41 min) SUCCESS core-regress-privs1-hdp (1 hr
> > 1
>
> > min)
>
> > SUCCESS core-regress-privs2-cdh (35 min) SUCCESS
>
> > core-regress-privs2-hdp
>
> > (43
>
> > min) SUCCESS core-regress-qat-cdh (14 min) SUCCESS
>
> > core-regress-qat-hdp (1 hr 13 min) SUCCESS core-regress-seabase-cdh
> > (1
>
> > hr 45 min) SUCCESS core-regress-seabase-hdp (2 hr 7 min) SUCCESS
>
> > core-regress-udr-cdh (18
>
> > min)
>
> > SUCCESS core-regress-udr-hdp (40 min) SUCCESS jdbc_test-cdh (50 min)
>
> > SUCCESS jdbc_test-hdp (57 min) SUCCESS phoenix_part1_T4-cdh (42 min)
>
> > SUCCESS phoenix_part1_T4-hdp (1 hr 26 min) SUCCESS
>
> > phoenix_part2_T4-cdh (44 min) SUCCESS phoenix_part2_T4-hdp (1 hr 4
>
> > min) SUCCESS pyodbc_test-cdh (10 min) SUCCESS pyodbc_test-hdp (38
> > min)
RE: Change in sqenvcom.sh - Response required
Posted by Steve Varnau <st...@esgyn.com>.
I did not intentionally do anything strange with the CDH set-up, just
install CDH5.4.
A little googling tells me that the Hadoop-0.20-mapred package is MR1. It is
optional, but valid to have both MR1 and MR2 installed on a system at the
same time. So I'm not sure if this is a mismatch between hive and trafodion
classpaths.
--Steve
> -----Original Message-----
> From: Dave Birdsall [mailto:dave.birdsall@esgyn.com]
> Sent: Thursday, April 7, 2016 8:32 AM
> To: dev@trafodion.incubator.apache.org
> Subject: RE: Change in sqenvcom.sh - Response required
>
> Hi Selva,
>
> I'm no expert in this, but your short-term change to sqenvcom.sh sounds
> good.
>
> I wonder if Steve could comment on the Jenkins angle?
>
> Dave
>
> -----Original Message-----
> From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
> Sent: Wednesday, April 6, 2016 3:56 PM
> To: dev@trafodion.incubator.apache.org
> Subject: Change in sqenvcom.sh - Response required
>
> Change in sqenvcom.sh - Response required
>
> Changed the subject line to avoid filtering….
>
> _____________________________________________
> *From:* Selva Govindarajan [mailto:selva.govindarajan@esgyn.com
> <se...@esgyn.com>]
> *Sent:* Wednesday, April 6, 2016 3:25 PM
> *To:* 'dev@trafodion.incubator.apache.org' <
> dev@trafodion.incubator.apache.org>
> *Subject:* RE: Trafodion master Daily Test Result - 167 - Still Failing -
> Change in sqenvcom.sh
>
> It looks like color didn’t show up in the message correctly. Line 351 was
> added.
>
> _____________________________________________
> *From:* Selva Govindarajan [*mailto:selva.govindarajan@esgyn.com*
> <se...@esgyn.com>]
> *Sent:* Wednesday, April 6, 2016 3:21 PM
> *To:* 'dev@trafodion.incubator.apache.org' <
> *dev@trafodion.incubator.apache.org*
> <de...@trafodion.incubator.apache.org>>
> *Subject:* RE: Trafodion master Daily Test Result - 167 - Still Failing -
> Change in sqenvcom.sh
>
> I added the line in red to sqenvcom.sh and removed ms.env and regenerated
> ms.env.
>
> 347 # directories with jar files and list of jar files
>
> 348 # (could try to reduce the number of jars in the classpath)
>
> 349 export HADOOP_JAR_DIRS="/usr/lib/hadoop
>
> 350 /usr/lib/hadoop/lib
>
> 351 /usr/lib/hadoop-0.20-mapreduce"
>
>
> This makes hive to work from Trafodion. But, I am not sure if the Jenkins
> installation is done differently from other CDH installation. Also, I am
> not
> sure if it is ok to change sqenvcom.sh this way. If so, we can make this
> change as a short term solution. So, please provide your feedback on this.
>
> However, as a long term solution I feel sqenvcom.sh is becoming quite
> complex. There is a need to simplify it. One of the suggestion would be to
> generate sqenvcom.sh based on the distro rather than making sqenvcom.sh to
> cater to different distros. Even if we don’t generate the file fully, we
> should consider generating CLASSPATH for different distro and source in
> to
> sqenvcom.sh only the generated installed distro CLASSPATH.
>
> To generate the CLASSPATH, Narendra said we can do the following
>
> hadoop classpath
>
> hbase classpath
>
> And expand all the * in the resulting class path to include all the jar
> files in the path (We don’t expand *, the jar files are not seen from
> JNI).
> This would make it better to cater to installation specific classpath
> changes too.
>
> Selva
>
> -----Original Message-----
> From: Steve Varnau [*mailto:steve.varnau@esgyn.com*
> <st...@esgyn.com> ]
> Sent: Wednesday, April 6, 2016 9:55 AM
> To: *dev@trafodion.incubator.apache.org*
> <de...@trafodion.incubator.apache.org>
> Subject: RE: Trafodion master Daily Test Result - 167 - Still Failing
>
> Yes. Before we hung trying to load the tpc-ds data into hive. That script
> seems to be running fine now, and the test suite actually runs, though all
> the hive related tests fail. So something else is not right.
>
> Test automation saves the hbase logs, but not hive logs, so I think we
> will
> need to spin up a vm to debug it. Who would like a crack at this one?
>
> --Steve
>
> > -----Original Message-----
>
> > From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com
> <se...@esgyn.com>]
>
> > Sent: Wednesday, April 6, 2016 9:19 AM
>
> > To: dev@trafodion.incubator.apache.org
>
> > Subject: RE: Trafodion master Daily Test Result - 167 - Still Failing
>
> >
>
> > I think it looks like a class path issue for hive tests in CDH. Need
>
> > to be analyzed in CDH vm.
>
> >
>
> > Selva
>
> >
>
> > -----Original Message-----
>
> > From: Sandhya Sundaresan [mailto:sandhya.sundaresan@esgyn.com
> <sa...@esgyn.com>]
>
> > Sent: Wednesday, April 6, 2016 7:48 AM
>
> > To: dev@trafodion.incubator.apache.org
>
> > Subject: RE: Trafodion master Daily Test Result - 167 - Still Failing
>
> >
>
> > The executor suite passed after weeks but the problematic test
>
> > (TEST106) has been commented out until the new hang for "drop table"
>
> > is fixed just so the rest of the tests in the suite get run. The old
>
> > hang seems to be fixed now.
>
> >
>
> > Steve, looks like the Hive metadata is corrupt or not setup properly
>
> > on the VMs where the core and hive suites are run.
>
> >
>
> > Arvind, I suppose you're following up on the phoenix failures ?
>
> >
>
> > Sandhya
>
> >
>
> > -----Original Message-----
>
> > From: steve.varnau@esgyn.com [mailto:steve.varnau@esgyn.com
> <st...@esgyn.com>]
>
> > Sent: Wednesday, April 6, 2016 4:02 AM
>
> > To: dev@trafodion.incubator.apache.org
>
> > Subject: Trafodion master Daily Test Result - 167 - Still Failing
>
> >
>
> > Daily Automated Testing master
>
> >
>
> > Jenkins Job: https://jenkins.esgyn.com/job/Check-Daily-master/167/
>
> > Archived Logs: http://traf-testlogs.esgyn.com/Daily-master/167
>
> > Bld Downloads: http://traf-builds.esgyn.com
>
> >
>
> > Changes since previous daily build:
>
> > [Steve Varnau] [TRAFODION-1816] License info for binary distribution
>
> > of REST bundle
>
> >
>
> > [selvaganesang] [TRAFODION-1918] Remove auto flush concept in
>
> > Trafodion
>
> >
>
> > [selvaganesang] [TRAFODION-1918] Remove disabling auto flush concept
>
> > in Trafodion
>
> >
>
> >
>
> >
>
> > Test Job Results:
>
> >
>
> > FAILURE core-regress-core-cdh (1 hr 34 min) FAILURE
>
> > core-regress-hive-cdh
>
> > (22 min) FAILURE core-regress-privs1-cdh (41 min) FAILURE
>
> > phoenix_part1_T2-cdh (11 min) FAILURE phoenix_part1_T2-hdp (39 min)
>
> > FAILURE phoenix_part2_T2-cdh (11 min) FAILURE phoenix_part2_T2-hdp (38
>
> > min) SUCCESS build-master-debug (25 min) SUCCESS build-master-release
>
> > (26 min) SUCCESS core-regress-charsets-cdh (27 min) SUCCESS
>
> > core-regress-charsets-hdp (1 hr
>
> > 29 min) SUCCESS core-regress-compGeneral-cdh (1 hr 25 min) SUCCESS
>
> > core-regress-compGeneral-hdp (47 min) SUCCESS core-regress-core-hdp (1
>
> > hr
>
> > 1
>
> > min) SUCCESS core-regress-executor-cdh (49 min) SUCCESS
>
> > core-regress-executor-hdp (1 hr 3 min) SUCCESS
>
> > core-regress-fullstack2-cdh
>
> > (1 hr 2 min) SUCCESS core-regress-fullstack2-hdp (14 min) SUCCESS
>
> > core-regress-hive-hdp (41 min) SUCCESS core-regress-privs1-hdp (1 hr 1
>
> > min)
>
> > SUCCESS core-regress-privs2-cdh (35 min) SUCCESS
>
> > core-regress-privs2-hdp
>
> > (43
>
> > min) SUCCESS core-regress-qat-cdh (14 min) SUCCESS
>
> > core-regress-qat-hdp (1 hr 13 min) SUCCESS core-regress-seabase-cdh (1
>
> > hr 45 min) SUCCESS core-regress-seabase-hdp (2 hr 7 min) SUCCESS
>
> > core-regress-udr-cdh (18
>
> > min)
>
> > SUCCESS core-regress-udr-hdp (40 min) SUCCESS jdbc_test-cdh (50 min)
>
> > SUCCESS jdbc_test-hdp (57 min) SUCCESS phoenix_part1_T4-cdh (42 min)
>
> > SUCCESS phoenix_part1_T4-hdp (1 hr 26 min) SUCCESS
>
> > phoenix_part2_T4-cdh (44 min) SUCCESS phoenix_part2_T4-hdp (1 hr 4
>
> > min) SUCCESS pyodbc_test-cdh (10 min) SUCCESS pyodbc_test-hdp (38 min)
RE: Change in sqenvcom.sh - Response required
Posted by Dave Birdsall <da...@esgyn.com>.
Hi Selva,
I'm no expert in this, but your short-term change to sqenvcom.sh sounds
good.
I wonder if Steve could comment on the Jenkins angle?
Dave
-----Original Message-----
From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
Sent: Wednesday, April 6, 2016 3:56 PM
To: dev@trafodion.incubator.apache.org
Subject: Change in sqenvcom.sh - Response required
Change in sqenvcom.sh - Response required
Changed the subject line to avoid filtering….
_____________________________________________
*From:* Selva Govindarajan [mailto:selva.govindarajan@esgyn.com
<se...@esgyn.com>]
*Sent:* Wednesday, April 6, 2016 3:25 PM
*To:* 'dev@trafodion.incubator.apache.org' <
dev@trafodion.incubator.apache.org>
*Subject:* RE: Trafodion master Daily Test Result - 167 - Still Failing -
Change in sqenvcom.sh
It looks like color didn’t show up in the message correctly. Line 351 was
added.
_____________________________________________
*From:* Selva Govindarajan [*mailto:selva.govindarajan@esgyn.com*
<se...@esgyn.com>]
*Sent:* Wednesday, April 6, 2016 3:21 PM
*To:* 'dev@trafodion.incubator.apache.org' <
*dev@trafodion.incubator.apache.org* <de...@trafodion.incubator.apache.org>>
*Subject:* RE: Trafodion master Daily Test Result - 167 - Still Failing -
Change in sqenvcom.sh
I added the line in red to sqenvcom.sh and removed ms.env and regenerated
ms.env.
347 # directories with jar files and list of jar files
348 # (could try to reduce the number of jars in the classpath)
349 export HADOOP_JAR_DIRS="/usr/lib/hadoop
350 /usr/lib/hadoop/lib
351 /usr/lib/hadoop-0.20-mapreduce"
This makes hive to work from Trafodion. But, I am not sure if the Jenkins
installation is done differently from other CDH installation. Also, I am not
sure if it is ok to change sqenvcom.sh this way. If so, we can make this
change as a short term solution. So, please provide your feedback on this.
However, as a long term solution I feel sqenvcom.sh is becoming quite
complex. There is a need to simplify it. One of the suggestion would be to
generate sqenvcom.sh based on the distro rather than making sqenvcom.sh to
cater to different distros. Even if we don’t generate the file fully, we
should consider generating CLASSPATH for different distro and source in to
sqenvcom.sh only the generated installed distro CLASSPATH.
To generate the CLASSPATH, Narendra said we can do the following
hadoop classpath
hbase classpath
And expand all the * in the resulting class path to include all the jar
files in the path (We don’t expand *, the jar files are not seen from JNI).
This would make it better to cater to installation specific classpath
changes too.
Selva
-----Original Message-----
From: Steve Varnau [*mailto:steve.varnau@esgyn.com*
<st...@esgyn.com> ]
Sent: Wednesday, April 6, 2016 9:55 AM
To: *dev@trafodion.incubator.apache.org*
<de...@trafodion.incubator.apache.org>
Subject: RE: Trafodion master Daily Test Result - 167 - Still Failing
Yes. Before we hung trying to load the tpc-ds data into hive. That script
seems to be running fine now, and the test suite actually runs, though all
the hive related tests fail. So something else is not right.
Test automation saves the hbase logs, but not hive logs, so I think we will
need to spin up a vm to debug it. Who would like a crack at this one?
--Steve
> -----Original Message-----
> From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com
<se...@esgyn.com>]
> Sent: Wednesday, April 6, 2016 9:19 AM
> To: dev@trafodion.incubator.apache.org
> Subject: RE: Trafodion master Daily Test Result - 167 - Still Failing
>
> I think it looks like a class path issue for hive tests in CDH. Need
> to be analyzed in CDH vm.
>
> Selva
>
> -----Original Message-----
> From: Sandhya Sundaresan [mailto:sandhya.sundaresan@esgyn.com
<sa...@esgyn.com>]
> Sent: Wednesday, April 6, 2016 7:48 AM
> To: dev@trafodion.incubator.apache.org
> Subject: RE: Trafodion master Daily Test Result - 167 - Still Failing
>
> The executor suite passed after weeks but the problematic test
> (TEST106) has been commented out until the new hang for "drop table"
> is fixed just so the rest of the tests in the suite get run. The old
> hang seems to be fixed now.
>
> Steve, looks like the Hive metadata is corrupt or not setup properly
> on the VMs where the core and hive suites are run.
>
> Arvind, I suppose you're following up on the phoenix failures ?
>
> Sandhya
>
> -----Original Message-----
> From: steve.varnau@esgyn.com [mailto:steve.varnau@esgyn.com
<st...@esgyn.com>]
> Sent: Wednesday, April 6, 2016 4:02 AM
> To: dev@trafodion.incubator.apache.org
> Subject: Trafodion master Daily Test Result - 167 - Still Failing
>
> Daily Automated Testing master
>
> Jenkins Job: https://jenkins.esgyn.com/job/Check-Daily-master/167/
> Archived Logs: http://traf-testlogs.esgyn.com/Daily-master/167
> Bld Downloads: http://traf-builds.esgyn.com
>
> Changes since previous daily build:
> [Steve Varnau] [TRAFODION-1816] License info for binary distribution
> of REST bundle
>
> [selvaganesang] [TRAFODION-1918] Remove auto flush concept in
> Trafodion
>
> [selvaganesang] [TRAFODION-1918] Remove disabling auto flush concept
> in Trafodion
>
>
>
> Test Job Results:
>
> FAILURE core-regress-core-cdh (1 hr 34 min) FAILURE
> core-regress-hive-cdh
> (22 min) FAILURE core-regress-privs1-cdh (41 min) FAILURE
> phoenix_part1_T2-cdh (11 min) FAILURE phoenix_part1_T2-hdp (39 min)
> FAILURE phoenix_part2_T2-cdh (11 min) FAILURE phoenix_part2_T2-hdp (38
> min) SUCCESS build-master-debug (25 min) SUCCESS build-master-release
> (26 min) SUCCESS core-regress-charsets-cdh (27 min) SUCCESS
> core-regress-charsets-hdp (1 hr
> 29 min) SUCCESS core-regress-compGeneral-cdh (1 hr 25 min) SUCCESS
> core-regress-compGeneral-hdp (47 min) SUCCESS core-regress-core-hdp (1
> hr
> 1
> min) SUCCESS core-regress-executor-cdh (49 min) SUCCESS
> core-regress-executor-hdp (1 hr 3 min) SUCCESS
> core-regress-fullstack2-cdh
> (1 hr 2 min) SUCCESS core-regress-fullstack2-hdp (14 min) SUCCESS
> core-regress-hive-hdp (41 min) SUCCESS core-regress-privs1-hdp (1 hr 1
> min)
> SUCCESS core-regress-privs2-cdh (35 min) SUCCESS
> core-regress-privs2-hdp
> (43
> min) SUCCESS core-regress-qat-cdh (14 min) SUCCESS
> core-regress-qat-hdp (1 hr 13 min) SUCCESS core-regress-seabase-cdh (1
> hr 45 min) SUCCESS core-regress-seabase-hdp (2 hr 7 min) SUCCESS
> core-regress-udr-cdh (18
> min)
> SUCCESS core-regress-udr-hdp (40 min) SUCCESS jdbc_test-cdh (50 min)
> SUCCESS jdbc_test-hdp (57 min) SUCCESS phoenix_part1_T4-cdh (42 min)
> SUCCESS phoenix_part1_T4-hdp (1 hr 26 min) SUCCESS
> phoenix_part2_T4-cdh (44 min) SUCCESS phoenix_part2_T4-hdp (1 hr 4
> min) SUCCESS pyodbc_test-cdh (10 min) SUCCESS pyodbc_test-hdp (38 min)