You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by srowen <gi...@git.apache.org> on 2017/10/02 18:39:40 UTC
[GitHub] spark pull request #19412: [SPARK-22142][BUILD][STREAMING] Move Flume suppor...
GitHub user srowen opened a pull request:
https://github.com/apache/spark/pull/19412
[SPARK-22142][BUILD][STREAMING] Move Flume support behind a profile, take 2
## What changes were proposed in this pull request?
Move flume behind a profile, take 2. See https://github.com/apache/spark/pull/19365 for most of the back-story.
This change should fix the problem by removing the examples module dependency and moving Flume examples to the module itself. It also adds deprecation messages, per a discussion on dev@ about deprecating for 2.3.0.
## How was this patch tested?
Existing tests, which still enable flume integration.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/srowen/spark SPARK-22142.2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19412.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19412
----
commit 4b45f9c30a6e06cd4a3509f93c6370374e8f4c05
Author: Sean Owen <so...@cloudera.com>
Date: 2017-09-30T16:06:34Z
Move flume behind a profile, take 2. This time also remove Flume examples and example dependency on flume.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19412: [SPARK-22142][BUILD][STREAMING] Move Flume support behin...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19412
I'm not really convinced that adding this profile is really achieving much. After all, the flume connector doesn't end up in the Spark packaging, only in maven central, and the new profile seems to be enabled in all the scripts anyway. So nothing seems to be changing much aside from deprecating the connector.
That being said, I don't really care one way or another as long as the code is there. Might want to test with maven too.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19412: [SPARK-22142][BUILD][STREAMING] Move Flume support behin...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/19412
I think the argument for it if anything is that it's a) deprecated, so should kinda be optional to build, and b) this would simply be consistent with how other external/* modules are handled. For Spark 2.x yes there isn't and shouldn't be an actual change to the outputs.
There's a legitimate separate question here about whether it should be deprecated? my sense is yes, to leave the option to remove it in Spark 3.0, which would probably follow 2.3. I recall something about flume-ng uses an old version of Netty and it's the thing blocking updating it for all of Spark, but I may be misremembering the detail there.
Yeah this passed a Maven test build as well as `dev/run-tests` now.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19412: [SPARK-22142][BUILD][STREAMING] Move Flume support behin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19412
**[Test build #82392 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82392/testReport)** for PR 19412 at commit [`4b45f9c`](https://github.com/apache/spark/commit/4b45f9c30a6e06cd4a3509f93c6370374e8f4c05).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19412: [SPARK-22142][BUILD][STREAMING] Move Flume support behin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19412
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82392/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19412: [SPARK-22142][BUILD][STREAMING] Move Flume support behin...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/19412
Merged to master
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19412: [SPARK-22142][BUILD][STREAMING] Move Flume support behin...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/19412
@vanzin might want a look on this second try, or @squito
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19412: [SPARK-22142][BUILD][STREAMING] Move Flume suppor...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19412
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19412: [SPARK-22142][BUILD][STREAMING] Move Flume support behin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19412
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19412: [SPARK-22142][BUILD][STREAMING] Move Flume support behin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19412
**[Test build #82392 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82392/testReport)** for PR 19412 at commit [`4b45f9c`](https://github.com/apache/spark/commit/4b45f9c30a6e06cd4a3509f93c6370374e8f4c05).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org