You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by holdenk <gi...@git.apache.org> on 2016/01/20 00:33:47 UTC
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
GitHub user holdenk opened a pull request:
https://github.com/apache/spark/pull/10841
[SPARK-12469][CORE][RFC/WIP] Add Consistent Accumulators for Spark
This is an initial PR illustrating one of the possible approaches for providing consistent accumulators in Spark.
Tasks executed on Spark workers are unable to modify values from the driver, and accumulators are the one exception for this. Accumulators in Spark are implemented in such a way that when a stage is recomputed (say for cache eviction) the accumulator will be updated a second time. This makes accumulators inside of transformations more difficult to use for things like counting invalid records (one of the primary potential use cases of collecting side information during a transformation). However in some cases this counting during re-evaluation is exactly the behaviour we want (say in tracking total execution time for a particular function). Spark would benefit from a version of accumulators which did not double count even if stages were re-executed.
Motivating example:
```
val parseTime = sc.accumulator(0L)
val parseFailures = sc.accumulator(0L)
val parsedData = sc.textFile(...).flatMap { line =>
val start = System.currentTimeMillis()
val parsed = Try(parse(line))
if (parsed.isFailure) parseFailures += 1
parseTime += System.currentTimeMillis() - start
parsed.toOption
}
parsedData.cache()
val resultA = parsedData.map(...).filter(...).count()
// some intervening code. Almost anything could happen here -- some of parsedData may
// get kicked out of the cache, or an executor where data was cached might get lost
val resultB = parsedData.filter(...).map(...).flatMap(...).count()
// now we look at the accumulators
```
Here we would want parseFailures to only have been added to once for every line which failed to parse. Unfortunately, the current Spark accumulator API doesn’t support the current parseFailures use case since if some data had been evicted its possible that it will be double counted.
See the full design document at https://docs.google.com/document/d/1lR_l1g3zMVctZXrcVjFusq2iQVpr4XvRK_UUDsDr6nk/edit?usp=sharing
cc @squito
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/holdenk/spark SPARK-12469-consistent-accumulators-for-spark
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10841.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10841
----
commit 7c9f874d94085f60089cda8ab3b6c451bf793a1a
Author: Holden Karau <ho...@us.ibm.com>
Date: 2015-12-28T19:59:30Z
Start adding the proposed external API for consistent accumulators
commit fce92981371ddd7af8ad28f30537fb473bbee378
Author: Holden Karau <ho...@us.ibm.com>
Date: 2015-12-28T19:59:47Z
Start keeping track of the required information in the taskcontext
commit c0e2bfd9d3d5f22e9ea217a500b67e84679f519d
Author: Holden Karau <ho...@us.ibm.com>
Date: 2015-12-30T00:29:53Z
A bit more progress towards consistent accumulators, add an add function which checks the bitset, throw an exception on merges (perhaps this isn't reasonable and then we need to keep all of the previous values - but a quick skim looks like its intended to be a user accesiable function so just don't support).
commit e9c287f682cbefc531a8fc84b00a472f860c9d06
Author: Holden Karau <ho...@us.ibm.com>
Date: 2015-12-30T01:51:53Z
Waaait that was not thinking clearly. As we go with a consistent accumulator adding values add them to a hashmap of pending values to be merged based on the rdd id & partition id - then driver side when merging the accumulators check each rdd & partition id individually against the rdd & bitset of processed partitions for that rdd adding to a single accumulator. (This way we don't need to keep all of the values around forever but if we say have first() and then foreach() we can add the value for partition one then when the second task happens and we can skip just the adding partion one. Up next: start adding some simple tests for this to see if it works
commit cf32dc5d543d4c0bb883e00692d21f22a7470737
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-01-06T07:45:18Z
Merge branch 'master' into SPARK-12469-consistent-accumulators-for-spark
commit d9354b12977bdaa2bcd82730fb34bdae490ab02d
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-01-06T20:26:13Z
add consistentValue (temporary) to get the consistent value and fix the accumulator to work even when no merge happens
commit 28a5754b654441ad8904f6a8f6e4a06db3908541
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-01-06T20:58:07Z
Introduce a GenericAccumulable to allow for Accumulables which don't return the same value as the value accumulate (e.g. consistent accumulators accumulate a lot of book keeping information which the user shouldn't know about so hide that). Accumulable now extends GenericAccumulable to keep the API the same
commit 21387e1c5ebf10d6a2b26996ba0a54640925a903
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-01-07T07:37:31Z
Temporary debugging
commit 11ac8364c4077ee14c2284f6d3a17341efb79b61
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-01-07T07:38:36Z
Add a some tests for consistent accumulators
commit b76218d1645b58be587e9db1337b2867b20dcaf1
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-01-07T09:11:38Z
Switch to doing explicit passin in the API for now (can figure out the magic later if we want magic). Next up: handle partially consumed partitions.
commit 7fb981c9f1503747b687f31d37a11b22c87fcf66
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-01-07T19:22:39Z
Get rid of some old junk added when trying the old approach
commit c75b037cdffd05c415e8e3dbe247f921009e31d6
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-01-07T21:39:01Z
Have tasks report if they've processed the entire partition and skip incrementing the consistent accumulator if this is not the case
commit df89265241891380810860ad0f26e653efed298d
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-01-07T22:54:56Z
Regardless of how much of the iterator we read in the task, if we are using persistance than the entire partition gets computed
commit 33b45411c4f7ac45c439e020d82db95f8333e0bd
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-01-07T23:20:34Z
Some improvements
commit b2d0926eac596b0f075db7809df1130bd7025d00
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-01-11T23:07:13Z
Merge in master
commit 4fd97a0a85f4e2c3a0e9b7edf2deb0a957443c4d
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-01-12T18:53:01Z
Style fixes - todo check some indentation and such
commit f12608776d5359636065b2e9dd7c3c3d1eed930a
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-01-12T19:30:59Z
Fix some indentation
commit 11da3d30d148e474595e95465c1dd2d81ca1d43f
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-01-14T00:07:56Z
Fix whitespace, add an explicit test for a single partiton, remove setting the when adding an accumulator value since we always have a merge with the zero val
commit b7637c94f0fd694c61ccba2e8876352e472c3795
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-01-14T00:11:14Z
Remove unecessary printlns
commit 82e8cb33ca1aef621d11083a30c16baf881ec90e
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-01-17T20:27:55Z
Merge branch 'master' into SPARK-12469-consistent-accumulators-for-spark
commit 6304270f42bc9ad5e1aea6021b122d4d85d47c38
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-01-17T20:38:03Z
Fix indentation
commit 6557e5db63a59af527d0f90447d3a40309ef9d87
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-01-17T20:38:40Z
Fix documentation for consistent accumulators example, and be consistent about using splitID
commit 922d12c17cb213db596b60736f975f3a52d71196
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-01-17T20:39:07Z
Clarify when includeConsistent should be true/false in the javadoc of TaskContext
commit 2c7cba021cec3ea0f1f708e4bb0f4291648d12ac
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-01-17T20:45:55Z
Remove some unused imports
commit a842ed23b8994aa1a109f32a742e5f98c0661599
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-01-17T20:46:29Z
rename consistentaccumulators.scala to consistentaccumulatorssuite.scala
commit 1a7cce9d339bb0debf456e012f07f8b6a2dc5b69
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-01-17T20:52:41Z
Fix indentation on multiline params
commit aaa9b89fabed40fec33c33243166eeb8a4b3feed
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-01-19T23:09:37Z
Merge in master - first pass
commit 211b3e7a7c9d1e08df919c12fb8e7e6af76eb640
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-01-19T23:27:18Z
Fix merge and add flatMapWithAccumulator
commit 3f96429b5c6a6c5596d70a9d15de74ae6203c970
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-01-19T23:33:36Z
Remove extra line
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173026556
**[Test build #49723 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49723/consoleFull)** for PR 10841 at commit [`3f96429`](https://github.com/apache/spark/commit/3f96429b5c6a6c5596d70a9d15de74ae6203c970).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-178921591
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50619/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-178881831
**[Test build #50607 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50607/consoleFull)** for PR 10841 at commit [`a507f74`](https://github.com/apache/spark/commit/a507f7454c4f5ea872f0f417d9135fee38b4e191).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173525219
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49870/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-178928161
**[Test build #50622 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50622/consoleFull)** for PR 10841 at commit [`69c5c94`](https://github.com/apache/spark/commit/69c5c9487a3461b08f6eaf3870200b58c9338e32).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-174754737
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-176470848
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-176429734
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-179144260
**[Test build #50654 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50654/consoleFull)** for PR 10841 at commit [`69c5c94`](https://github.com/apache/spark/commit/69c5c9487a3461b08f6eaf3870200b58c9338e32).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-176324493
**[Test build #50284 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50284/consoleFull)** for PR 10841 at commit [`26db2c6`](https://github.com/apache/spark/commit/26db2c6e9dfd21a48803d3820c8d381156b5f01a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-176300039
Build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173506101
**[Test build #49873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49873/consoleFull)** for PR 10841 at commit [`f4100bc`](https://github.com/apache/spark/commit/f4100bc6dd165d025c92fc2853e6b3b075991791).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-180644758
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-179084732
looks like an internal jenkins issue, jenkins retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-180264031
**[Test build #50811 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50811/consoleFull)** for PR 10841 at commit [`133a358`](https://github.com/apache/spark/commit/133a358dfe4cb03dc9a4a8f3783dcf32b5508cfc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-178879266
**[Test build #50607 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50607/consoleFull)** for PR 10841 at commit [`a507f74`](https://github.com/apache/spark/commit/a507f7454c4f5ea872f0f417d9135fee38b4e191).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-178923873
looks like an internal jenkins issue, jenkins retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173537104
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49873/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-176378614
**[Test build #50291 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50291/consoleFull)** for PR 10841 at commit [`5524bc7`](https://github.com/apache/spark/commit/5524bc74422259b8df1e5e1bda27606764ea34b7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-174711333
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173472900
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49849/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-174711334
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50010/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173536939
**[Test build #49873 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49873/consoleFull)** for PR 10841 at commit [`f4100bc`](https://github.com/apache/spark/commit/f4100bc6dd165d025c92fc2853e6b3b075991791).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-179977274
jenkins retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-174665371
**[Test build #50010 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50010/consoleFull)** for PR 10841 at commit [`495c8b6`](https://github.com/apache/spark/commit/495c8b677cef7a38939c09cb7a796c4b1034aea3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-180554724
**[Test build #50842 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50842/consoleFull)** for PR 10841 at commit [`91c4f0e`](https://github.com/apache/spark/commit/91c4f0ec1e0956041a8cfee9e9bae3180e5c0dac).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-176429379
**[Test build #50291 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50291/consoleFull)** for PR 10841 at commit [`5524bc7`](https://github.com/apache/spark/commit/5524bc74422259b8df1e5e1bda27606764ea34b7).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173500763
**[Test build #49870 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49870/consoleFull)** for PR 10841 at commit [`cdfd0be`](https://github.com/apache/spark/commit/cdfd0be8ef6d4ee3b4a6656910e1a3cb049e1320).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-176334317
**[Test build #50285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50285/consoleFull)** for PR 10841 at commit [`3e014c7`](https://github.com/apache/spark/commit/3e014c7a07107dd864239b21d67d8fc6511c8529).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-179986860
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50756/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-174730261
seems unrelated, jenkins retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173029460
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49723/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173425544
That's a good point @codingcat , although the old behavior for regular accumulators is the same I'll add a note about the new API option.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-180733844
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50868/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173396095
**[Test build #49821 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49821/consoleFull)** for PR 10841 at commit [`ce60621`](https://github.com/apache/spark/commit/ce60621dd5048114a1793b5adfad3d72cc423fed).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-176506795
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50309/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-179144751
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-174754740
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50035/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173345083
**[Test build #49802 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49802/consoleFull)** for PR 10841 at commit [`0915634`](https://github.com/apache/spark/commit/0915634730e7984aa0e086eb10c37e9f68dc0f48).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-178962135
**[Test build #50622 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50622/consoleFull)** for PR 10841 at commit [`69c5c94`](https://github.com/apache/spark/commit/69c5c9487a3461b08f6eaf3870200b58c9338e32).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-180540823
@andrewor14 I changed it around a bit so we don't need to have the user explicitly pass the rdd/partition information to the consistent accumulator. It's still got the separate class since we need to keep track of some extra information for consistent accumulators. Think this is closer to the correct direction?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-174754603
**[Test build #50035 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50035/consoleFull)** for PR 10841 at commit [`495c8b6`](https://github.com/apache/spark/commit/495c8b677cef7a38939c09cb7a796c4b1034aea3).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-177045002
@holdenk Thanks for spending the time to put forth the detailed design. I agree 100% with the problem and have thought a great deal about how to fix it myself. However, I personally found the proposed API kind of clunky. It's not super clear what "consistent" means, especially to someone new to Spark. I think it would be cleaner to just implement this in the existing classes themselves and maybe expose only a flag.
It seems that on a high level we want to associate an accumulated value with a particular computation of an RDD. Can we store something like a map from RDD ID to a list of accumulated values in each accumulator? Then if the user calls `acc.value` we just sum all of them up. If the user wants an accumulated value associated with a particular RDD or even a particular computation then they can also do that. E.g.
```
class AccumulableInfo[R, T](...) {
private val valuesByRDD = mutable.HashMap[Int, ArrayBuffer[R]]
def value: R = valuesByRDD.values.reduce(mergeThemAll)
def valueByRDD(rddId: Int): R = valuesByRDD(rddId).reduce(mergeThemAll)
}
```
Even this may solve only part of the problem, where the same accumulator used across different RDDs, but maybe it's a better first step. I may be missing something but I don't see the need to introduce all these new classes; the existing set is confusing enough and in the past we've made the mistake of exposing all of them to the user.
Also, I believe @rxin has some thoughts on this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-177047887
Thanks for taking a look @andrewor14 :) This PR (and the JIRA & design doc) are all very much focused on enabling the _data property_ use case for accumulators.
I introduced the GenericAccumulable as a parent to Accumulable to try and maintain source compatibility between versions and also limit any cost to users of "regular" accumulators - and since we are coming up on Spark 2.0 that might not be worth the overhead so I'm more than happy to simplify that part away. Or if we change the current accumulator base class in such a way it meets the needs while keeping source compatibility all the better.
For the naming of the Consistent Accumulators (or naming of the flag) totally open to ideas - I just didn't have anything else come to mind.
So I'm assuming the clunky part of the API your referring to is the "withAccumulator" part of the transformations - which I do feel is pretty clunky myself. My initial draft attempted to avoid using this (I first tried Implementation Option 1 from the design doc ( https://docs.google.com/document/d/1lR_l1g3zMVctZXrcVjFusq2iQVpr4XvRK_UUDsDr6nk/edit?usp=sharing ) but when multiple transformations are chained together the TaskContext ends up having the RDD id of the inner most RDD. I'd really love to avoid explicitly having the user explicitly pass this information in and if there is a permanent way to do that I'm happy to change this around to give that a shot.
I like the idea of the map of RDD id to accumulated value, since an entire RDD might not be computed in the same task though I think we will need to keep track of the value by RDD ID & Partition ID (or at least keep track of value and bitmask of partitions accumulated per RDD ID).
cc @squito who did the initial accumulator work and we've gone over the design doc some together.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173423177
Build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-180644692
**[Test build #50842 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50842/consoleFull)** for PR 10841 at commit [`91c4f0e`](https://github.com/apache/spark/commit/91c4f0ec1e0956041a8cfee9e9bae3180e5c0dac).
* This patch **fails from timeout after a configured wait of \`250m\`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173356239
**[Test build #49807 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49807/consoleFull)** for PR 10841 at commit [`99acdff`](https://github.com/apache/spark/commit/99acdff674f880e15ecb4e1dbcae935b96bfaeaa).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-180264238
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50811/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-176403954
Build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-180264237
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173423180
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49821/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-176403742
**[Test build #50285 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50285/consoleFull)** for PR 10841 at commit [`3e014c7`](https://github.com/apache/spark/commit/3e014c7a07107dd864239b21d67d8fc6511c8529).
* This patch **fails Spark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-178881868
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173345095
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49802/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173029456
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173424483
In docs, you need to remove the following sentence
> For accumulator updates performed inside actions only, Spark guarantees that each task’s update to the accumulator will only be applied once, i.e. restarted tasks will not update the value. In transformations, users should be aware of that each task’s update may be applied more than once if tasks or job stages are re-executed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-176474992
**[Test build #50309 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50309/consoleFull)** for PR 10841 at commit [`59c211a`](https://github.com/apache/spark/commit/59c211ab899482946f302aad079bccd5f553baa6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-180733770
**[Test build #50868 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50868/consoleFull)** for PR 10841 at commit [`5a7bd6f`](https://github.com/apache/spark/commit/5a7bd6fad9cb2bfb054d1377841d641a6c5adb27).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173472830
**[Test build #49849 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49849/consoleFull)** for PR 10841 at commit [`5ff6b6a`](https://github.com/apache/spark/commit/5ff6b6a6fce974735fd0ad4450fdc382ce8590f9).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-176470853
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50295/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173356286
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173537102
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173450266
**[Test build #49849 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49849/consoleFull)** for PR 10841 at commit [`5ff6b6a`](https://github.com/apache/spark/commit/5ff6b6a6fce974735fd0ad4450fdc382ce8590f9).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173472899
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173525109
**[Test build #49870 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49870/consoleFull)** for PR 10841 at commit [`cdfd0be`](https://github.com/apache/spark/commit/cdfd0be8ef6d4ee3b4a6656910e1a3cb049e1320).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-176300048
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50278/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173345089
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-178921580
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173352831
**[Test build #49807 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49807/consoleFull)** for PR 10841 at commit [`99acdff`](https://github.com/apache/spark/commit/99acdff674f880e15ecb4e1dbcae935b96bfaeaa).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-174734436
**[Test build #50035 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50035/consoleFull)** for PR 10841 at commit [`495c8b6`](https://github.com/apache/spark/commit/495c8b677cef7a38939c09cb7a796c4b1034aea3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-180721409
**[Test build #50868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50868/consoleFull)** for PR 10841 at commit [`5a7bd6f`](https://github.com/apache/spark/commit/5a7bd6fad9cb2bfb054d1377841d641a6c5adb27).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173423035
**[Test build #49821 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49821/consoleFull)** for PR 10841 at commit [`ce60621`](https://github.com/apache/spark/commit/ce60621dd5048114a1793b5adfad3d72cc423fed).
* This patch **fails Spark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-180644759
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50842/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-178881873
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50607/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-178962414
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-178962416
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50622/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-180232682
**[Test build #50811 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50811/consoleFull)** for PR 10841 at commit [`133a358`](https://github.com/apache/spark/commit/133a358dfe4cb03dc9a4a8f3783dcf32b5508cfc).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-176325014
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50284/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-176325012
Build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-179144758
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50654/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173029415
**[Test build #49723 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49723/consoleFull)** for PR 10841 at commit [`3f96429`](https://github.com/apache/spark/commit/3f96429b5c6a6c5596d70a9d15de74ae6203c970).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-179986856
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-180733841
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-176299436
**[Test build #50278 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50278/consoleFull)** for PR 10841 at commit [`523ba43`](https://github.com/apache/spark/commit/523ba4365a947ccb862bfae158dbcb301ec5163d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-176469947
**[Test build #50295 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50295/consoleFull)** for PR 10841 at commit [`dabe8cf`](https://github.com/apache/spark/commit/dabe8cff5536dadf46481924ee88154e95a9609e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-176325002
**[Test build #50284 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50284/consoleFull)** for PR 10841 at commit [`26db2c6`](https://github.com/apache/spark/commit/26db2c6e9dfd21a48803d3820c8d381156b5f01a).
* This patch **fails Scala style tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-176429739
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50291/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk closed the pull request at:
https://github.com/apache/spark/pull/10841
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173525217
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-177045628
By the way, there's a section on the design doc attached to https://issues.apache.org/jira/browse/SPARK-10620 that describes the same problem by classifying accumulators into different use cases. I think it's simpler to think about this problem in terms of these use cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-176300027
**[Test build #50278 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50278/consoleFull)** for PR 10841 at commit [`523ba43`](https://github.com/apache/spark/commit/523ba4365a947ccb862bfae158dbcb301ec5163d).
* This patch **fails Scala style tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-179092907
**[Test build #50654 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50654/consoleFull)** for PR 10841 at commit [`69c5c94`](https://github.com/apache/spark/commit/69c5c9487a3461b08f6eaf3870200b58c9338e32).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-174432893
ping @andrewor14 if you have some time to look at this :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-180844069
I'm going to close this and re-open one with the "full auto" type approach suggested.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173356288
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49807/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-174711126
**[Test build #50010 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50010/consoleFull)** for PR 10841 at commit [`495c8b6`](https://github.com/apache/spark/commit/495c8b677cef7a38939c09cb7a796c4b1034aea3).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-176415571
**[Test build #50295 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50295/consoleFull)** for PR 10841 at commit [`dabe8cf`](https://github.com/apache/spark/commit/dabe8cff5536dadf46481924ee88154e95a9609e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-176506787
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-173343150
**[Test build #49802 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49802/consoleFull)** for PR 10841 at commit [`0915634`](https://github.com/apache/spark/commit/0915634730e7984aa0e086eb10c37e9f68dc0f48).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-176403957
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50285/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-176473361
I've updated this against master with @andrewor14's Accumulator changes. cc @squito
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-176506066
**[Test build #50309 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50309/consoleFull)** for PR 10841 at commit [`59c211a`](https://github.com/apache/spark/commit/59c211ab899482946f302aad079bccd5f553baa6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org