You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by JoshRosen <gi...@git.apache.org> on 2016/04/05 08:58:10 UTC
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/12171
[SPARK-14399] Remove unnecessary excludes from POMs and simplify Hive POM
This patch aims to simplify our build by removing a number of unnecessary excludes from the build. The individual commit messages describe the changes here in more detail, but there are a few key themes:
- **Remove many excludes in the root POM:** According to the [Maven documentation](ttps://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Management), "dependency management takes precedence over dependency mediation". In many cases, we were both excluding transitive dependencies while also using those dependencies in `spark-core` and pinning those dependencies' versions in the `dependencyManagment` section of our POM.
As a rough guideline, I think like we should only exclude dependencies under the following scenarios:
- We don't want the dependency at all, in which case we should define a Maven Enforcer rule to ban it.
- We want to include _some_ version of the dependency but don't want to explicitly specify a fixed version. For instance, imagine that libraries A and B depend on some dependency C. We might exclude C from library B so that library A's version of C becomes the final effective version.
- We want to include the dependency only when certain profiles are enabled, so we ban it from all of our transitive dependencies and explicitly re-add it in a profile.
- **Remove explicitly-promoted transitive dependencies from Hive POM:** in an early revision of #7191, it looks like we tried to use the `core`-classified version of `hive-exec`, which is published as a thin-JAR which does not bundle third-party dependencies' classes. However, both the regular and classified versions of the dependency share the same effective POM, which, in this case, happens to be the dependency-reduced-POM produced by the Maven shade plugin. As a result, users of the `core`-classified JAR must add direct dependencies on what are logically transitive dependencies of `hive-exec`, so #7191 added several such dependencies in `hive/pom.xml`.
However, by the end of #7191 we abandoned the idea of using an existing Hive JAR and chose to republish our own JAR, so this promotion of transitive dependencies is no longer necessary. As a result, this patch was able to significantly cut down the size of the `hive` POM by removing those direct dependencies. This had the side-effect of exposing a problem where one of our Janino JARs was being pulled to a lower version as a side-effect of this unnecessary dependency promotion.
- **Use Maven's new wildcard exclude support:** Maven now supports wildcard exclusion patterns, so I used that to simplify the exclusion of `org.ow2.asm` artifacts.
- **Improve `dev/test-dependencies`:** a bad regex caused the `org.spark-project:hive` JARs to not be listed under `dev/deps`; this is now fixed.
- **Remove `joda-time` dependency:** Spark does not directly depend on `joda-time`, so I don't think that it makes sense to keep it in our POM.
- **Remove explicit `io.netty` dependency**: we don't seem to need this anymore. I did leave the `dependencyManagement` entry for now in order to prevent an unexpected downgrade.
/cc @srowen @steveloughran @pwendell @rxin for review.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark build-cleanup
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12171.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12171
----
commit ca2aef540954d2a2b29c00fd792e1f73c8600e6d
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-04T23:42:25Z
Update test-dependeices script to not accidentally filter out Spark hive deps.
commit 87dd382710af1b2d9e89025590f66d9192154a1e
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T00:21:34Z
Remove explicit jodd dependency
commit ed40fa3d11a896ef08dcfb484488b1ad3628117a
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T00:25:22Z
Remove explicit Datanucleus dependency.
commit 38ed734c787692c59ff67e363e0d57e9b7348972
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T00:46:29Z
Add Maven Enforcer rule to document org.hsqldb:hsqldb exclusion
commit aec2f209abc6aa358564bf6669b6c86a2c16b3dc
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T01:07:02Z
Remove seemingly-unnecessary Calcite exclude.
commit 02dec1be34e7bfbe7079d6bb35ea55ea6d9a5467
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T01:18:13Z
Remove explicit Calcite dependency.
It seems that the explicit dep. was pulling our version of Janino's commons-compiler
a bit lower than it should have been.
commit 1cf73d4894be4ab6c7f75e417761307b1693c46c
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T01:25:27Z
Remove unused Joda dependency.
commit e8409d02b67523806b81b5298cb74f53f982c4a1
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T01:34:50Z
Remove unnecessary Jackson excludes:
See https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Management
> "dependency management takes precedence over dependency mediation"
commit bcd486792e33c6a2703de5d94d52db94f0484c4d
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T01:40:47Z
Remove unnecessary Jackson excludes.
commit 44ae3e07420a5164676d893ec96f083e01b9ae7a
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T01:42:55Z
Remove unnecessary jackson-mapper-asl direct dependency in Hive
commit e5c221e6e1626c15ef53b9a3918569796adac465
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T01:49:20Z
Remove unnecessary httpclient exclusions.
commit 3deb2c2af9cc6530d1c970480e725d710c02f782
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T02:15:06Z
Remove unnecessary jsr305 excludes.
commit 7a24ae86d08f420b5a17a3ab52e2a2e6ae37008f
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T02:19:08Z
Clean up commons-codec excludes.
commit 55e7cc8e89b1ce3a8faa3e41f4b0affae5e8c8af
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T02:20:00Z
Remove commented-out portions of the Hive build.
commit 482840a31d1675bd9caf9a0c7aa14404db91fdbc
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T02:23:39Z
Remove unnecessary Janino exclude.
commit 9eb49f2afbb5460b2ddc0d71a23a425f1aa9f03c
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T02:28:18Z
Remove unnecessary libfb303 direct dependency.
commit 35d539d6c58292739c256fba8aa089a1097a7b2e
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T02:35:00Z
Remove a bunch of unnecessary logging-related excludes.
commit 805f55be217f78df4288f4c91568487e4758c516
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T02:35:37Z
Remove a stray httpclient exclude.
commit 277e56a5e7a43dd73e49df7eb2e3efbd291d8257
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T03:06:57Z
Document org.ow2.asm exclusions.
commit 4f7c0c55e074d51822690bfdeafef1a9a37c8282
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T03:21:07Z
Clarify comments surrounding Hive's Avro dep. classifier.
commit 1abd1f0b6e118eeabcc6e20ef0d423fe06d4beda
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T05:01:31Z
Remove unnecessary io.netty exclusions; add clarifying comment.
commit 8eb56cd322b929c6cde271b6516879b7096bafc0
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T05:06:44Z
Add enforcer rule for excluding javax.servlet:servlet-api
commit 6b64539c164e3e3f9cf03e8ac05c205e7bd44ecd
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T05:09:42Z
Add enforcer rule for mockito-all exclusion.
commit b1809b7e17e8d02a42be4866b9351ad3ac1a22b8
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T06:11:39Z
Remove another stray slf4j exclude.
commit e5ca86e93508187b3c93d681ba907e43703535d1
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T06:21:55Z
More unnecessary exclude removal; reudce scope of Kryo exclude.
commit 393696cf4c9580782cb9753cebf8b74faeaabf30
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T06:30:04Z
Remove unnecessary Zookeeper excludes
commit 17afbe23b92520228e5d004648bd2d9f0a56e12c
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T06:36:28Z
Remove unnecessary curator excludes
commit d736e7e26863af4cc382fe644f9e60c6193f7687
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T06:39:31Z
Remove inherited exclusion.
commit 51853c7d18c6a85a669a2d62639dfba30ed29dcd
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-05T06:53:27Z
Fix typo.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-207551094
Now that we no longer publish assembly JARs for Spark, I don't think that shading is going to require exclusions because the only reason to do that would be if we wanted to completely prevent Jetty and Guava from appearing in our transitive dependency classpath, a task which currently isn't possible unless we republish all of our deps. which depend on Guava and Jetty.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen closed the pull request at:
https://github.com/apache/spark/pull/12171
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205999807
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55026/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-206005481
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205922367
**[Test build #55002 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55002/consoleFull)** for PR 12171 at commit [`0bb11be`](https://github.com/apache/spark/commit/0bb11be97113cecb30c642b6d4797f9e131ba095).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205780064
**[Test build #54972 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54972/consoleFull)** for PR 12171 at commit [`36430de`](https://github.com/apache/spark/commit/36430de7b4ca793b43577ac0c73f506712413a31).
* This patch **fails from timeout after a configured wait of \`250m\`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205714343
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54971/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205714338
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205719312
**[Test build #54967 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54967/consoleFull)** for PR 12171 at commit [`51853c7`](https://github.com/apache/spark/commit/51853c7d18c6a85a669a2d62639dfba30ed29dcd).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205712227
In fact, being able to use the nice wildcard syntax may be a way's off because the official sbt-pom-reader doesn't support Maven 3.3.x due to API incompatibilities: https://github.com/sbt/sbt-pom-reader/blob/master/project/dependencies.scala#L5
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205686151
**[Test build #54963 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54963/consoleFull)** for PR 12171 at commit [`51853c7`](https://github.com/apache/spark/commit/51853c7d18c6a85a669a2d62639dfba30ed29dcd).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205685535
A few quick questions:
- [ ] Can we remove all of the Guava excludes now? They're all over the place but it's not clear whether it's necessary anymore. If we don't want Guava to appear at all, then I think we should use Enforcer to ban it. Otherwise, I think we should remove the excludes and let `dependencyManagement` take care of fixing the version.
- [ ] Why do we exclude `org.mortbay.jetty` all over the place? I tried adding an enforcer rule but this proved to be a bit tricky because of how a lot of Hadoop components used in tests seem to depend on it in one way or another.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205719362
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54967/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399][test-maven] Remove unnecessary e...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205699286
**[Test build #54967 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54967/consoleFull)** for PR 12171 at commit [`51853c7`](https://github.com/apache/spark/commit/51853c7d18c6a85a669a2d62639dfba30ed29dcd).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205695469
```
[warn] module not found: org.pentaho#pentaho-aggdesigner-algorithm;5.1.5-jhyde
[warn] ==== public: tried
[warn] https://repo1.maven.org/maven2/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom
[warn] ==== Maven2 Local: tried
[warn] file:/home/sparkivy/per-executor-caches/6/.m2/repository/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom
[warn] ==== local: tried
[warn] /home/sparkivy/per-executor-caches/6/.ivy2/local/org.pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/ivys/ivy.xml
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.pentaho#pentaho-aggdesigner-algorithm;5.1.5-jhyde: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Unresolved dependencies path:
[warn] org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde
[warn] +- org.apache.calcite:calcite-core:1.2.0-incubating
[warn] +- org.spark-project.hive:hive-exec:1.2.1.spark ((com.typesafe.sbt.pom.MavenHelper) MavenHelper.scala#L76)
[warn] +- org.apache.spark:spark-yarn_2.11:2.0.0-SNAPSHOT
```
It looks like compilation failed because the `pentaho-aggdesigner-algorithm` transitive dependency isn't present in Maven Central; see also https://mail-archives.apache.org/mod_mbox/hive-dev/201412.mbox/%3C571034233.2432062.1419887855285.JavaMail.yahoo@jws10685.mail.bf1.yahoo.com%3E
This is a great illustration of why Spark's build bans the use of dependencies which aren't present in Maven Central: non-central repos can go offline and break a previously-good build.
I _did_ keep the exclusion which is supposed to be preventing this dep. from being pulled in, so I wonder whether this is a problem specific to the SBT build. Let me try re-running with Maven to see if our POM reader plugin could be to blame.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205688006
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54963/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-207240207
@JoshRosen Guava and Jetty excludes might have been done when they were shaded:
https://issues.apache.org/jira/browse/SPARK-2848 - shade Guava
https://issues.apache.org/jira/browse/SPARK-3996 - shade Jetty
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205687993
**[Test build #54963 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54963/consoleFull)** for PR 12171 at commit [`51853c7`](https://github.com/apache/spark/commit/51853c7d18c6a85a669a2d62639dfba30ed29dcd).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205858120
It looks like Hive might actually need Joda time:
```
- analyze MetastoreRelations *** FAILED ***
org.apache.spark.sql.execution.QueryExecutionException: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org/joda/time/ReadWritableInstant
at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:455)
at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:440)
at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:226)
at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:173)
at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:172)
at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:215)
at org.apache.spark.sql.hive.client.HiveClientImpl.runHive(HiveClientImpl.scala:440)
at org.apache.spark.sql.hive.client.HiveClientImpl.runSqlHive(HiveClientImpl.scala:430)
at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:351)
at org.apache.spark.sql.hive.test.TestHiveContext.runSqlHive(TestHive.scala:183)
...
01:36:40.566 ERROR hive.ql.exec.DDLTask: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyPrimitiveObjectInspectorFactory
```
https://stackoverflow.com/questions/26259717/facing-java-lang-noclassdeffounderror-org-joda-time-readableinstant-error-even
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205719339
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205714331
**[Test build #54971 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54971/consoleFull)** for PR 12171 at commit [`fc938a0`](https://github.com/apache/spark/commit/fc938a099bbf3704bf9f55fc0b4f6a6c0d3db92b).
* This patch **fails build dependency tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-206005390
**[Test build #55002 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55002/consoleFull)** for PR 12171 at commit [`0bb11be`](https://github.com/apache/spark/commit/0bb11be97113cecb30c642b6d4797f9e131ba095).
* This patch **fails from timeout after a configured wait of \`250m\`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205717007
**[Test build #54972 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54972/consoleFull)** for PR 12171 at commit [`36430de`](https://github.com/apache/spark/commit/36430de7b4ca793b43577ac0c73f506712413a31).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12171#discussion_r58601918
--- Diff: pom.xml ---
@@ -840,14 +827,13 @@
<groupId>org.apache.avro</groupId>
<artifactId>avro-mapred</artifactId>
<version>${avro.version}</version>
+ <!-- use the build matching the hadoop api of avro-mapred (i.e. no classifier for hadoop 1
--- End diff --
Do we even need this anymore? We don't support hadoop1 in Spark 2.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-206099245
Hmm, the FlumePollingStreamSuite test looks broken because it has hung twice now. I'll try to reproduce offline.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/12171#discussion_r58610193
--- Diff: pom.xml ---
@@ -840,14 +827,13 @@
<groupId>org.apache.avro</groupId>
<artifactId>avro-mapred</artifactId>
<version>${avro.version}</version>
+ <!-- use the build matching the hadoop api of avro-mapred (i.e. no classifier for hadoop 1
--- End diff --
Good point; we can just hardcode `hadoop2` here for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205962888
LGTM if sbt and maven tests pass.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205687999
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205780099
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205921075
I re-added joda-time; let's see if that fixes Hive.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205712989
_Actually_, we might be able to use `sbt-pom-reader` 2.10-RC2, since we only require Maven 3.2.1+ and that artifact uses 3.2.2: https://github.com/sbt/sbt-pom-reader/issues/20#issuecomment-183949295
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-206005487
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55002/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399][test-maven] Remove unnecessary e...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205707080
From the Jenkins Maven output:
```
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ spark-hive_2.11 ---
Downloading: https://repo1.maven.org/maven2/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom
Downloading: http://www.datanucleus.org/downloads/maven2/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom
Downloading: http://conjars.org/repo/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom
2/2 KB
Downloaded: http://conjars.org/repo/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom (2 KB at 4.0 KB/sec)
Downloading: https://repo1.maven.org/maven2/org/pentaho/pentaho-aggdesigner/5.1.5-jhyde/pentaho-aggdesigner-5.1.5-jhyde.pom
Downloading: http://www.datanucleus.org/downloads/maven2/org/pentaho/pentaho-aggdesigner/5.1.5-jhyde/pentaho-aggdesigner-5.1.5-jhyde.pom
Downloading: http://conjars.org/repo/org/pentaho/pentaho-aggdesigner/5.1.5-jhyde/pentaho-aggdesigner-5.1.5-jhyde.pom
4/9 KB
8/9 KB
9/9 KB
Downloaded: http://conjars.org/repo/org/pentaho/pentaho-aggdesigner/5.1.5-jhyde/pentaho-aggdesigner-5.1.5-jhyde.pom (9 KB at 26.3 KB/sec)
Downloading: https://repo1.maven.org/maven2/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
Downloading: http://www.datanucleus.org/downloads/maven2/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
Downloading: http://conjars.org/repo/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
4/48 KB
7/48 KB
11/48 KB
14/48 KB
17/48 KB
21/48 KB
25/48 KB
29/48 KB
33/48 KB
34/48 KB
38/48 KB
42/48 KB
46/48 KB
48/48 KB
Downloaded: http://conjars.org/repo/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar (48 KB at 99.6 KB/sec)
```
It looks like our Maven build isn't as strict in its enforcement of only using Maven Central.
I guess maybe Maven will still try to resolve excluded transitive dependencies unless you add an explicit direct dependency? The extra download time by itself isn't a huge deal / that's not necessary what we want to optimize for (otherwise we'd just ban all transitive dependencies and explicitly promote everything to top-level). However, I don't want to rely on clojars so I'll look into restoring that direct dep. on calcite.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205712554
**[Test build #54971 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54971/consoleFull)** for PR 12171 at commit [`fc938a0`](https://github.com/apache/spark/commit/fc938a099bbf3704bf9f55fc0b4f6a6c0d3db92b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205780102
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54972/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205744467
without looking at the details, I appreciated the ambition —and like that wildcard artifact exclusion.
what is the dependency graph that SBT generates from this?
Joda time is an interesting point ... we've seen some problems related to spark + aws having joda time mismatches. the amazon aws/s3/kinesis libs do require joda time, and there was some conflict creeping in there between the hadoop installation and kinesis imports in the spark job.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12171#discussion_r58602465
--- Diff: tools/pom.xml ---
@@ -71,6 +71,16 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
</plugin>
+ <!-- This project is only used to hold the MiMa exclude generator, so we don't care
+ whether normally-banned dependencies are used here. Therefore, skpi enforcer: -->
--- End diff --
nit: skip
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399][test-maven] Remove unnecessary e...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205704283
Great cleanup!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399][test-maven] Remove unnecessary e...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/12171#discussion_r58498334
--- Diff: tools/pom.xml ---
@@ -71,6 +71,16 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
</plugin>
+ <!-- This project is only used to hold the MiMa exclude generator, so we don't care
+ whether normally-banned dependencies are used here. Therefore, skpi enforcer: -->
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-enforcer-plugin</artifactId>
+ <version>1.4.1</version>
--- End diff --
Nit: you don't need to repeat the version here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205684795
Oh, /cc @vanzin as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399][test-maven] Remove unnecessary e...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205695666
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14399] Remove unnecessary excludes from...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12171#issuecomment-205709605
```
org.apache.maven.model.building.ModelBuildingException: 10 problems were encountered while building the effective model for org.apache.spark:spark-hive_2.11:2.0.0-SNAPSHOT
[WARNING] 'dependencies.dependency.exclusions.exclusion.artifactId' for org.spark-project.hive:hive-exec:jar with value '*' does not match a valid id pattern. @ org.apache.spark:spark-parent_2.11:2.0.0-SNAPSHOT, /Users/joshrosen/Documents/spark2/pom.xml, line 1184, column 25
```
It looks like our custom `sbt-pom-reader` fork is using an ancient version of Maven which doesn't support this lovely wildcard exclusion syntax. Bummer! Yet another motivation to move back to a stock version; see https://issues.apache.org/jira/browse/SPARK-14401
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org