You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by holdenk <gi...@git.apache.org> on 2016/02/21 07:53:25 UTC
[GitHub] spark pull request: [SPARK-13399][STREAMING] Fix checkpointsuite t...
GitHub user holdenk opened a pull request:
https://github.com/apache/spark/pull/11286
[SPARK-13399][STREAMING] Fix checkpointsuite type erasure warnings
## What changes were proposed in this pull request?
Change the checkpointsuite getting the outputstreams to explicitly be unchecked on the generic type so as to avoid the warnings. This only impacts test code.
Alternatively we could encode the type tag in the TestOutputStreamWithPartitions and filter the type tag as well - but this is unnecessary since multiple testoutputstreams are not registered and the previous code was not actually checking this type.
## How was the this patch tested?
unit tests (streaming/testOnly org.apache.spark.streaming.CheckpointSuite)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/holdenk/spark SPARK-13399-checkpointsuite-type-erasure
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11286.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11286
----
commit dd70caddde0100a96a5581725d36c77d31b33c99
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-02-21T04:46:41Z
Fix the type erasure warning in the checkpoint suite, add the class tag informtion to the TestOutputStreamWithPartitions and use that in the suite to verify the correct type
commit 8e18eaf71a4073168c6fb490a879243ff62d3109
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-02-21T06:35:56Z
Wait we were not actually depending on checking the type of the elements in the stream before, just mark as unchecked and go for it.
commit 0dd337e5a0e4ade7b09c4cca0cb2a947006fb539
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-02-21T06:36:21Z
Clarify comment
commit 3f2980a5402277c56835e60c9bb399a35889c1b3
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-02-21T06:51:50Z
Move to base trait rather than obj
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13399][STREAMING] Fix checkpointsuite t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11286#issuecomment-186909589
**[Test build #51643 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51643/consoleFull)** for PR 11286 at commit [`41c0b90`](https://github.com/apache/spark/commit/41c0b9087eafedc595045ed166a5af0d3ffe277c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13399][STREAMING] Fix checkpointsuite t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11286#issuecomment-186775751
**[Test build #51613 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51613/consoleFull)** for PR 11286 at commit [`3f2980a`](https://github.com/apache/spark/commit/3f2980a5402277c56835e60c9bb399a35889c1b3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13399][STREAMING] Fix checkpointsuite t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11286#discussion_r53569281
--- Diff: streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -133,6 +133,17 @@ trait DStreamCheckpointTester { self: SparkFunSuite =>
new StreamingContext(SparkContext.getOrCreate(conf), batchDuration)
}
+ /**
+ * Get the first TestOutputStreamWithPartitions, does not check the provided generic type.
+ */
+ protected def getTestOutputStream[V: ClassTag](streams: Array[DStream[_]]):
+ TestOutputStreamWithPartitions[V] = {
+ streams.collect{
--- End diff --
Seems OK. I was going to ask if you can use `find()` here but I don't think it quite applies in the same way. Let's see if Jenkins makes you put a space before the brace
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13399][STREAMING] Fix checkpointsuite t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/11286#issuecomment-187099507
Merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13399][STREAMING] Fix checkpointsuite t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11286#issuecomment-186782985
**[Test build #51613 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51613/consoleFull)** for PR 11286 at commit [`3f2980a`](https://github.com/apache/spark/commit/3f2980a5402277c56835e60c9bb399a35889c1b3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13399][STREAMING] Fix checkpointsuite t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11286#issuecomment-186939366
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51643/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13399][STREAMING] Fix checkpointsuite t...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/11286
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13399][STREAMING] Fix checkpointsuite t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11286#issuecomment-186939363
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13399][STREAMING] Fix checkpointsuite t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11286#issuecomment-186783014
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13399][STREAMING] Fix checkpointsuite t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11286#issuecomment-186783016
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51613/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13399][STREAMING] Fix checkpointsuite t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11286#issuecomment-186939000
**[Test build #51643 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51643/consoleFull)** for PR 11286 at commit [`41c0b90`](https://github.com/apache/spark/commit/41c0b9087eafedc595045ed166a5af0d3ffe277c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org