You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by brennonyork <gi...@git.apache.org> on 2015/04/10 00:40:34 UTC
[GitHub] spark pull request: [WIP][HOTFIX][SPARK-4123]: Fix bug in dependen...
GitHub user brennonyork opened a pull request:
https://github.com/apache/spark/pull/5443
[WIP][HOTFIX][SPARK-4123]: Fix bug in dependency removal issue
We're seeing a bug sporadically in the new PR dependency comparison test whereby it notes that *all* dependencies are removed. This happens when the current PR is built, but the final, sorted, dependency file is left blank. I believe this is an error either in the way the `git checkout` calls have been or an error within the `mvn` build for that PR (again, likely related to the `git checkout`). As such I've set the checkouts to now force (with `-f` flag) which is more in line with what Jenkins currently does on the initial checkout.
Setting this as a WIP for now to trigger the build process myriad times to see if the issue still arises.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/brennonyork/spark HOTFIX2-SPARK-4123
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5443.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5443
----
commit 710c8d18b5631757efbb45f0aeb32417ebc27ac0
Author: Brennon York <br...@capitalone.com>
Date: 2015-04-08T20:00:25Z
added 30 minutes to the test benchmark
commit e3f63c7df7f821bf19b0975b03ad4ea035b2a4a8
Author: Brennon York <br...@capitalone.com>
Date: 2015-04-09T22:31:14Z
added '-f' to the checkout flags for git
commit 07765a6e3209b0add601900ddf77dbac3bef59ae
Author: Brennon York <br...@capitalone.com>
Date: 2015-04-09T22:34:55Z
updated the diff logic to reference the filenames rather than hardlink
commit 3f073d671c148d91b386d6df7bc9d5c048d9abc0
Author: Brennon York <br...@capitalone.com>
Date: 2015-04-09T22:37:21Z
removed the git checkouts piping to dev null
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][HOTFIX][SPARK-4123]: Fix bug in PR depen...
Posted by brennonyork <gi...@git.apache.org>.
Github user brennonyork commented on the pull request:
https://github.com/apache/spark/pull/5443#issuecomment-91373793
jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][HOTFIX][SPARK-4123]: Fix bug in PR depen...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5443#issuecomment-91386466
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29978/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][HOTFIX][SPARK-4123]: Fix bug in PR depen...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/5443#issuecomment-91629017
@brennonyork let me know when this is not WIP and I'll take a look. Seems like you are still testing a bit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][HOTFIX][SPARK-4123]: Fix bug in PR depen...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/5443#issuecomment-92607590
I think you may need to look at the source code to figure out what is going on:
https://github.com/jenkinsci/ghprb-plugin/blob/master/src/main/java/org/jenkinsci/plugins/ghprb/GhprbTrigger.java#L156
The current set of changes seem pretty straightforward, so maybe I can merge this now to see if the "-f" helps.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][HOTFIX][SPARK-4123]: Fix bug in PR depen...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5443#issuecomment-91374215
[Test build #29978 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29978/consoleFull) for PR 5443 at commit [`3f073d6`](https://github.com/apache/spark/commit/3f073d671c148d91b386d6df7bc9d5c048d9abc0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][HOTFIX][SPARK-4123]: Fix bug in PR depen...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5443#issuecomment-91646169
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30040/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][HOTFIX][SPARK-4123]: Fix bug in PR depen...
Posted by brennonyork <gi...@git.apache.org>.
Github user brennonyork commented on the pull request:
https://github.com/apache/spark/pull/5443#issuecomment-92518053
@shaneknapp can you help me understand how Jenkins is doing the checkouts? I'm seeing the PR builder outputting:
```
Building remotely on amp-jenkins-worker-06 (centos) in workspace /home/jenkins/workspace/SparkPullRequestBuilder
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://github.com/apache/spark.git # timeout=10
Fetching upstream changes from https://github.com/apache/spark.git
> git --version # timeout=10
> git fetch --tags --progress https://github.com/apache/spark.git +refs/pull/5443/*:refs/remotes/origin/pr/5443/* # timeout=15
> git rev-parse origin/pr/5443/merge^{commit} # timeout=10
> git branch -a --contains c5916336e6aff94dd3abfc9a0a41a2528c765fce # timeout=10
> git rev-parse remotes/origin/pr/5443/merge^{commit} # timeout=10
Checking out Revision c5916336e6aff94dd3abfc9a0a41a2528c765fce (origin/pr/5443/merge)
> git config core.sparsecheckout # timeout=10
> git checkout -f c5916336e6aff94dd3abfc9a0a41a2528c765fce
```
although I'm a bit confused what checkout I should switch between if, say, I want to, from a PR, checkout the `master` branch, then switch back to the given PR branch, then possibly back to `master`, and finally back to the PR again.
I'm currently doing what I believe is correct [here](https://github.com/apache/spark/blob/master/dev/tests/pr_new_dependencies.sh#L42) although there are times when the checkout from `master` back to the current PR fails, producing odd dependency reports. I've noticed that Jenkins is using the `-f` flag which I've added, but wanted to see if you had any thoughts into the matter.
Further, I've added `echo` statements to dump the `ghprbActualCommit`, the `sha1`, and the output of `git rev-parse HEAD`. Each are different commit hashes which makes me further think this is the cause for all the errors. Again, any advice?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][HOTFIX][SPARK-4123]: Fix bug in PR depen...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/5443
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][HOTFIX][SPARK-4123]: Fix bug in PR depen...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5443#issuecomment-91386460
[Test build #29978 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29978/consoleFull) for PR 5443 at commit [`3f073d6`](https://github.com/apache/spark/commit/3f073d671c148d91b386d6df7bc9d5c048d9abc0).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][HOTFIX][SPARK-4123]: Fix bug in PR depen...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5443#issuecomment-91646146
[Test build #30040 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30040/consoleFull) for PR 5443 at commit [`f2186be`](https://github.com/apache/spark/commit/f2186be090c9c73847e97cc9abf37fc62bbf7368).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][HOTFIX][SPARK-4123]: Fix bug in PR depen...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5443#issuecomment-91622470
[Test build #30040 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30040/consoleFull) for PR 5443 at commit [`f2186be`](https://github.com/apache/spark/commit/f2186be090c9c73847e97cc9abf37fc62bbf7368).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][HOTFIX][SPARK-4123]: Fix bug in PR depen...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5443#issuecomment-91373621
[Test build #29977 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29977/consoleFull) for PR 5443 at commit [`3f073d6`](https://github.com/apache/spark/commit/3f073d671c148d91b386d6df7bc9d5c048d9abc0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][HOTFIX][SPARK-4123]: Fix bug in PR depen...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5443#issuecomment-91384500
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29977/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][HOTFIX][SPARK-4123]: Fix bug in PR depen...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5443#issuecomment-91384493
[Test build #29977 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29977/consoleFull) for PR 5443 at commit [`3f073d6`](https://github.com/apache/spark/commit/3f073d671c148d91b386d6df7bc9d5c048d9abc0).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][HOTFIX][SPARK-4123]: Fix bug in dependen...
Posted by brennonyork <gi...@git.apache.org>.
Github user brennonyork commented on the pull request:
https://github.com/apache/spark/pull/5443#issuecomment-91373411
/cc @pwendell @nchammas @shaneknapp
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org