You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Sean Owen <sr...@gmail.com> on 2019/12/04 14:28:41 UTC
SQL test failures in PR builder?
I'm seeing consistent failures in the PR builder when touching SQL code:
https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4960/testReport/
org.apache.spark.sql.hive.thriftserver.SparkMetadataOperationSuite.Spark's
own GetSchemasOperation(SparkGetSchemasOperation)14 ms2
org.apache.spark.sql.hive.thriftserver.ThriftServerWithSparkContextSuite.(It
is not a test it is a sbt.testing.SuiteSelector)
Looks like this has failed about 6 builds in the past few days. Has anyone
seen this / has a clue what's causing it? errors are like ...
java.sql.SQLException: No suitable driver found for
jdbc:hive2://localhost:13694/?a=avalue;b=bvalue#c=cvalue;d=dvalue
Caused by: sbt.ForkMain$ForkError: java.lang.RuntimeException: class
org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl not
org.apache.hadoop.hive.metastore.MetaStoreFilterHook
Re: SQL test failures in PR builder?
Posted by Shane Knapp <sk...@berkeley.edu>.
yeah, totally weird.
i'm actually going to take this moment and clean up the build scripts
for both of these jobs. there's a lot of years-old cruft that i'll
delete and make things more readable.
On Sun, Dec 8, 2019 at 7:50 PM Sean Owen <sr...@gmail.com> wrote:
>
> Hm, so they look pretty similar except for minor differences in the
> actual script run. Is there any reason this should be different? Would
> it be reasonable to try making the 'new' one work like the 'old' one
> if the former isn't working?
>
> But I still can't figure out why it causes the same odd error every
> time on this one PR, which is a minor change to tooltips in the UI. I
> haven't seen other manually-triggered PR builds fail this way. Really
> mysterious so far!
>
> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4964/testReport/
>
>
> Old:
>
> #!/bin/bash
>
> set -e # fail on any non-zero exit code
> set -x
>
> export AMPLAB_JENKINS=1
> export PATH="$PATH:/home/anaconda/envs/py3k/bin"
>
> # Prepend JAVA_HOME/bin to fix issue where Zinc's embedded SBT
> incremental compiler seems to
> # ignore our JAVA_HOME and use the system javac instead.
> export PATH="$JAVA_HOME/bin:$PATH"
>
> # Add a pre-downloaded version of Maven to the path so that we avoid
> the flaky download step.
> export PATH="/home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.3.9/bin/:$PATH"
>
> echo "fixing target dir permissions"
> chmod -R +w target/* || true # stupid hack by sknapp to ensure that
> the chmod always exits w/0 and doesn't bork the script
>
> echo "running git clean -fdx"
> git clean -fdx
>
> # Configure per-build-executor Ivy caches to avoid SBT Ivy lock contention
> export HOME="/home/sparkivy/per-executor-caches/$EXECUTOR_NUMBER"
> mkdir -p "$HOME"
> export SBT_OPTS="-Duser.home=$HOME -Dsbt.ivy.home=$HOME/.ivy2"
> export SPARK_VERSIONS_SUITE_IVY_PATH="$HOME/.ivy2"
>
>
> ./dev/run-tests-jenkins
>
>
> # Hack to ensure that at least one JVM suite always runs in order to
> prevent spurious errors from the
> # Jenkins JUnit test reporter plugin
> ./build/sbt unsafe/test > /dev/null 2>&1
>
>
>
> New:
>
> #!/bin/bash
>
> set -e
> export AMPLAB_JENKINS=1
> export PATH="$PATH:/home/anaconda/envs/py3k/bin"
> git clean -fdx
>
> # Prepend JAVA_HOME/bin to fix issue where Zinc's embedded SBT
> incremental compiler seems to
> # ignore our JAVA_HOME and use the system javac instead.
> export PATH="$JAVA_HOME/bin:$PATH"
>
> # Add a pre-downloaded version of Maven to the path so that we avoid
> the flaky download step.
> export PATH="/home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.3.9/bin/:$PATH"
>
> # Configure per-build-executor Ivy caches to avoid SBT Ivy lock contention
> export HOME="/home/sparkivy/per-executor-caches/$EXECUTOR_NUMBER"
> mkdir -p "$HOME"
> export SBT_OPTS="-Duser.home=$HOME -Dsbt.ivy.home=$HOME/.ivy2"
> export SPARK_VERSIONS_SUITE_IVY_PATH="$HOME/.ivy2"
>
> # This is required for tests of backport patches.
> # We need to download the run-tests-codes.sh file because it's
> imported by run-tests-jenkins.
> # When running tests on branch-1.0 (and earlier), the older version of
> run-tests won't set CURRENT_BLOCK, so
> # the Jenkins scripts will report all failures as "some tests failed"
> rather than a more specific
> # error message.
> if [ ! -f "dev/run-tests-jenkins" ]; then
> wget https://raw.githubusercontent.com/apache/spark/master/dev/run-tests-jenkins
> wget https://raw.githubusercontent.com/apache/spark/master/dev/run-tests-codes.sh
> mv run-tests-jenkins dev/
> mv run-tests-codes.sh dev/
> chmod 755 dev/run-tests-jenkins
> chmod 755 dev/run-tests-codes.sh
> fi
>
> ./dev/run-tests-jenkins
>
>
> On Wed, Dec 4, 2019 at 5:53 PM Shane Knapp <sk...@berkeley.edu> wrote:
> >
> > ++yin huai for more insight in to the NewSparkPullRequestBuilder job...
> >
> > tbh, i never (or still) really understand the exact use for that job,
> > except that it's triggered by https://spark-prs.appspot.com/
> >
> > shane
> >
> >
> > On Wed, Dec 4, 2019 at 3:34 PM Sean Owen <sr...@gmail.com> wrote:
> > >
> > > BTW does anyone know why there are two PR builder jobs? I'm confused
> > > about why different ones would execute.
> > >
> > > Yes I see NewSparkPullRequestBuilder failing on a variety of PRs.
> > > I don't think it has anything to do with Hive; these PRs touch
> > > different parts of code but all not related to this failure.
> > >
> > > On Wed, Dec 4, 2019 at 12:40 PM Dongjoon Hyun <do...@gmail.com> wrote:
> > > >
> > > > Hi, Sean.
> > > >
> > > > It seems that there is no failure on your other SQL PR.
> > > >
> > > > https://github.com/apache/spark/pull/26748
> > > >
> > > > Does the sequential failure happen only at `NewSparkPullRequestBuilder`?
> > > > Since `NewSparkPullRequestBuilder` is not the same with `SparkPullRequestBuilder`,
> > > > there might be a root cause inside it if it happens only at `NewSparkPullRequestBuilder`.
> > > >
> > > > For `org.apache.hive.service.ServiceException: Failed to Start HiveServer2`,
> > > > I've observed them before, but the root cause might be different from this one.
> > > >
> > > > BTW, to reduce the scope of investigation, could you try with `[hive-1.2]` tag in your PR?
> > > >
> > > > Bests,
> > > > Dongjoon.
> > > >
> > > >
> > > > On Wed, Dec 4, 2019 at 6:29 AM Sean Owen <sr...@gmail.com> wrote:
> > > >>
> > > >> I'm seeing consistent failures in the PR builder when touching SQL code:
> > > >>
> > > >> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4960/testReport/
> > > >>
> > > >> org.apache.spark.sql.hive.thriftserver.SparkMetadataOperationSuite.Spark's own GetSchemasOperation(SparkGetSchemasOperation)14 ms2
> > > >> org.apache.spark.sql.hive.thriftserver.ThriftServerWithSparkContextSuite.(It is not a test it is a sbt.testing.SuiteSelector)
> > > >>
> > > >> Looks like this has failed about 6 builds in the past few days. Has anyone seen this / has a clue what's causing it? errors are like ...
> > > >>
> > > >> java.sql.SQLException: No suitable driver found for jdbc:hive2://localhost:13694/?a=avalue;b=bvalue#c=cvalue;d=dvalue
> > > >>
> > > >>
> > > >> Caused by: sbt.ForkMain$ForkError: java.lang.RuntimeException: class org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl not org.apache.hadoop.hive.metastore.MetaStoreFilterHook
> >
> >
> >
> > --
> > Shane Knapp
> > UC Berkeley EECS Research / RISELab Staff Technical Lead
> > https://rise.cs.berkeley.edu
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: SQL test failures in PR builder?
Posted by Sean Owen <sr...@gmail.com>.
Hm, so they look pretty similar except for minor differences in the
actual script run. Is there any reason this should be different? Would
it be reasonable to try making the 'new' one work like the 'old' one
if the former isn't working?
But I still can't figure out why it causes the same odd error every
time on this one PR, which is a minor change to tooltips in the UI. I
haven't seen other manually-triggered PR builds fail this way. Really
mysterious so far!
https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4964/testReport/
Old:
#!/bin/bash
set -e # fail on any non-zero exit code
set -x
export AMPLAB_JENKINS=1
export PATH="$PATH:/home/anaconda/envs/py3k/bin"
# Prepend JAVA_HOME/bin to fix issue where Zinc's embedded SBT
incremental compiler seems to
# ignore our JAVA_HOME and use the system javac instead.
export PATH="$JAVA_HOME/bin:$PATH"
# Add a pre-downloaded version of Maven to the path so that we avoid
the flaky download step.
export PATH="/home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.3.9/bin/:$PATH"
echo "fixing target dir permissions"
chmod -R +w target/* || true # stupid hack by sknapp to ensure that
the chmod always exits w/0 and doesn't bork the script
echo "running git clean -fdx"
git clean -fdx
# Configure per-build-executor Ivy caches to avoid SBT Ivy lock contention
export HOME="/home/sparkivy/per-executor-caches/$EXECUTOR_NUMBER"
mkdir -p "$HOME"
export SBT_OPTS="-Duser.home=$HOME -Dsbt.ivy.home=$HOME/.ivy2"
export SPARK_VERSIONS_SUITE_IVY_PATH="$HOME/.ivy2"
./dev/run-tests-jenkins
# Hack to ensure that at least one JVM suite always runs in order to
prevent spurious errors from the
# Jenkins JUnit test reporter plugin
./build/sbt unsafe/test > /dev/null 2>&1
New:
#!/bin/bash
set -e
export AMPLAB_JENKINS=1
export PATH="$PATH:/home/anaconda/envs/py3k/bin"
git clean -fdx
# Prepend JAVA_HOME/bin to fix issue where Zinc's embedded SBT
incremental compiler seems to
# ignore our JAVA_HOME and use the system javac instead.
export PATH="$JAVA_HOME/bin:$PATH"
# Add a pre-downloaded version of Maven to the path so that we avoid
the flaky download step.
export PATH="/home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.3.9/bin/:$PATH"
# Configure per-build-executor Ivy caches to avoid SBT Ivy lock contention
export HOME="/home/sparkivy/per-executor-caches/$EXECUTOR_NUMBER"
mkdir -p "$HOME"
export SBT_OPTS="-Duser.home=$HOME -Dsbt.ivy.home=$HOME/.ivy2"
export SPARK_VERSIONS_SUITE_IVY_PATH="$HOME/.ivy2"
# This is required for tests of backport patches.
# We need to download the run-tests-codes.sh file because it's
imported by run-tests-jenkins.
# When running tests on branch-1.0 (and earlier), the older version of
run-tests won't set CURRENT_BLOCK, so
# the Jenkins scripts will report all failures as "some tests failed"
rather than a more specific
# error message.
if [ ! -f "dev/run-tests-jenkins" ]; then
wget https://raw.githubusercontent.com/apache/spark/master/dev/run-tests-jenkins
wget https://raw.githubusercontent.com/apache/spark/master/dev/run-tests-codes.sh
mv run-tests-jenkins dev/
mv run-tests-codes.sh dev/
chmod 755 dev/run-tests-jenkins
chmod 755 dev/run-tests-codes.sh
fi
./dev/run-tests-jenkins
On Wed, Dec 4, 2019 at 5:53 PM Shane Knapp <sk...@berkeley.edu> wrote:
>
> ++yin huai for more insight in to the NewSparkPullRequestBuilder job...
>
> tbh, i never (or still) really understand the exact use for that job,
> except that it's triggered by https://spark-prs.appspot.com/
>
> shane
>
>
> On Wed, Dec 4, 2019 at 3:34 PM Sean Owen <sr...@gmail.com> wrote:
> >
> > BTW does anyone know why there are two PR builder jobs? I'm confused
> > about why different ones would execute.
> >
> > Yes I see NewSparkPullRequestBuilder failing on a variety of PRs.
> > I don't think it has anything to do with Hive; these PRs touch
> > different parts of code but all not related to this failure.
> >
> > On Wed, Dec 4, 2019 at 12:40 PM Dongjoon Hyun <do...@gmail.com> wrote:
> > >
> > > Hi, Sean.
> > >
> > > It seems that there is no failure on your other SQL PR.
> > >
> > > https://github.com/apache/spark/pull/26748
> > >
> > > Does the sequential failure happen only at `NewSparkPullRequestBuilder`?
> > > Since `NewSparkPullRequestBuilder` is not the same with `SparkPullRequestBuilder`,
> > > there might be a root cause inside it if it happens only at `NewSparkPullRequestBuilder`.
> > >
> > > For `org.apache.hive.service.ServiceException: Failed to Start HiveServer2`,
> > > I've observed them before, but the root cause might be different from this one.
> > >
> > > BTW, to reduce the scope of investigation, could you try with `[hive-1.2]` tag in your PR?
> > >
> > > Bests,
> > > Dongjoon.
> > >
> > >
> > > On Wed, Dec 4, 2019 at 6:29 AM Sean Owen <sr...@gmail.com> wrote:
> > >>
> > >> I'm seeing consistent failures in the PR builder when touching SQL code:
> > >>
> > >> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4960/testReport/
> > >>
> > >> org.apache.spark.sql.hive.thriftserver.SparkMetadataOperationSuite.Spark's own GetSchemasOperation(SparkGetSchemasOperation)14 ms2
> > >> org.apache.spark.sql.hive.thriftserver.ThriftServerWithSparkContextSuite.(It is not a test it is a sbt.testing.SuiteSelector)
> > >>
> > >> Looks like this has failed about 6 builds in the past few days. Has anyone seen this / has a clue what's causing it? errors are like ...
> > >>
> > >> java.sql.SQLException: No suitable driver found for jdbc:hive2://localhost:13694/?a=avalue;b=bvalue#c=cvalue;d=dvalue
> > >>
> > >>
> > >> Caused by: sbt.ForkMain$ForkError: java.lang.RuntimeException: class org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl not org.apache.hadoop.hive.metastore.MetaStoreFilterHook
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: SQL test failures in PR builder?
Posted by Shane Knapp <sk...@berkeley.edu>.
++yin huai for more insight in to the NewSparkPullRequestBuilder job...
tbh, i never (or still) really understand the exact use for that job,
except that it's triggered by https://spark-prs.appspot.com/
shane
On Wed, Dec 4, 2019 at 3:34 PM Sean Owen <sr...@gmail.com> wrote:
>
> BTW does anyone know why there are two PR builder jobs? I'm confused
> about why different ones would execute.
>
> Yes I see NewSparkPullRequestBuilder failing on a variety of PRs.
> I don't think it has anything to do with Hive; these PRs touch
> different parts of code but all not related to this failure.
>
> On Wed, Dec 4, 2019 at 12:40 PM Dongjoon Hyun <do...@gmail.com> wrote:
> >
> > Hi, Sean.
> >
> > It seems that there is no failure on your other SQL PR.
> >
> > https://github.com/apache/spark/pull/26748
> >
> > Does the sequential failure happen only at `NewSparkPullRequestBuilder`?
> > Since `NewSparkPullRequestBuilder` is not the same with `SparkPullRequestBuilder`,
> > there might be a root cause inside it if it happens only at `NewSparkPullRequestBuilder`.
> >
> > For `org.apache.hive.service.ServiceException: Failed to Start HiveServer2`,
> > I've observed them before, but the root cause might be different from this one.
> >
> > BTW, to reduce the scope of investigation, could you try with `[hive-1.2]` tag in your PR?
> >
> > Bests,
> > Dongjoon.
> >
> >
> > On Wed, Dec 4, 2019 at 6:29 AM Sean Owen <sr...@gmail.com> wrote:
> >>
> >> I'm seeing consistent failures in the PR builder when touching SQL code:
> >>
> >> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4960/testReport/
> >>
> >> org.apache.spark.sql.hive.thriftserver.SparkMetadataOperationSuite.Spark's own GetSchemasOperation(SparkGetSchemasOperation)14 ms2
> >> org.apache.spark.sql.hive.thriftserver.ThriftServerWithSparkContextSuite.(It is not a test it is a sbt.testing.SuiteSelector)
> >>
> >> Looks like this has failed about 6 builds in the past few days. Has anyone seen this / has a clue what's causing it? errors are like ...
> >>
> >> java.sql.SQLException: No suitable driver found for jdbc:hive2://localhost:13694/?a=avalue;b=bvalue#c=cvalue;d=dvalue
> >>
> >>
> >> Caused by: sbt.ForkMain$ForkError: java.lang.RuntimeException: class org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl not org.apache.hadoop.hive.metastore.MetaStoreFilterHook
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: SQL test failures in PR builder?
Posted by Sean Owen <sr...@gmail.com>.
BTW does anyone know why there are two PR builder jobs? I'm confused
about why different ones would execute.
Yes I see NewSparkPullRequestBuilder failing on a variety of PRs.
I don't think it has anything to do with Hive; these PRs touch
different parts of code but all not related to this failure.
On Wed, Dec 4, 2019 at 12:40 PM Dongjoon Hyun <do...@gmail.com> wrote:
>
> Hi, Sean.
>
> It seems that there is no failure on your other SQL PR.
>
> https://github.com/apache/spark/pull/26748
>
> Does the sequential failure happen only at `NewSparkPullRequestBuilder`?
> Since `NewSparkPullRequestBuilder` is not the same with `SparkPullRequestBuilder`,
> there might be a root cause inside it if it happens only at `NewSparkPullRequestBuilder`.
>
> For `org.apache.hive.service.ServiceException: Failed to Start HiveServer2`,
> I've observed them before, but the root cause might be different from this one.
>
> BTW, to reduce the scope of investigation, could you try with `[hive-1.2]` tag in your PR?
>
> Bests,
> Dongjoon.
>
>
> On Wed, Dec 4, 2019 at 6:29 AM Sean Owen <sr...@gmail.com> wrote:
>>
>> I'm seeing consistent failures in the PR builder when touching SQL code:
>>
>> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4960/testReport/
>>
>> org.apache.spark.sql.hive.thriftserver.SparkMetadataOperationSuite.Spark's own GetSchemasOperation(SparkGetSchemasOperation)14 ms2
>> org.apache.spark.sql.hive.thriftserver.ThriftServerWithSparkContextSuite.(It is not a test it is a sbt.testing.SuiteSelector)
>>
>> Looks like this has failed about 6 builds in the past few days. Has anyone seen this / has a clue what's causing it? errors are like ...
>>
>> java.sql.SQLException: No suitable driver found for jdbc:hive2://localhost:13694/?a=avalue;b=bvalue#c=cvalue;d=dvalue
>>
>>
>> Caused by: sbt.ForkMain$ForkError: java.lang.RuntimeException: class org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl not org.apache.hadoop.hive.metastore.MetaStoreFilterHook
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: SQL test failures in PR builder?
Posted by Dongjoon Hyun <do...@gmail.com>.
Hi, Sean.
It seems that there is no failure on your other SQL PR.
https://github.com/apache/spark/pull/26748
Does the sequential failure happen only at `NewSparkPullRequestBuilder`?
Since `NewSparkPullRequestBuilder` is not the same with
`SparkPullRequestBuilder`,
there might be a root cause inside it if it happens only at
`NewSparkPullRequestBuilder`.
For `org.apache.hive.service.ServiceException: Failed to Start HiveServer2`,
I've observed them before, but the root cause might be different from this
one.
BTW, to reduce the scope of investigation, could you try with `[hive-1.2]`
tag in your PR?
Bests,
Dongjoon.
On Wed, Dec 4, 2019 at 6:29 AM Sean Owen <sr...@gmail.com> wrote:
> I'm seeing consistent failures in the PR builder when touching SQL code:
>
>
> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4960/testReport/
>
> org.apache.spark.sql.hive.thriftserver.SparkMetadataOperationSuite.Spark's
> own GetSchemasOperation(SparkGetSchemasOperation)14 ms2
> org.apache.spark.sql.hive.thriftserver.ThriftServerWithSparkContextSuite.(It
> is not a test it is a sbt.testing.SuiteSelector)
>
> Looks like this has failed about 6 builds in the past few days. Has anyone
> seen this / has a clue what's causing it? errors are like ...
>
> java.sql.SQLException: No suitable driver found for jdbc:hive2://localhost:13694/?a=avalue;b=bvalue#c=cvalue;d=dvalue
>
>
> Caused by: sbt.ForkMain$ForkError: java.lang.RuntimeException: class org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl not org.apache.hadoop.hive.metastore.MetaStoreFilterHook
>
>