You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by ScrapCodes <gi...@git.apache.org> on 2016/02/08 14:33:46 UTC
[GitHub] spark pull request: [SPARK-13231] Make count failed values a user ...
GitHub user ScrapCodes opened a pull request:
https://github.com/apache/spark/pull/11115
[SPARK-13231] Make count failed values a user facing API.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ScrapCodes/spark 13231/accumulator
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11115.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11115
----
commit f84fc4dd25904f4a95f79cce789d1aba2f5f1a50
Author: Prashant Sharma <pr...@in.ibm.com>
Date: 2016-02-08T12:22:12Z
[SPARK-13231] Make count failed values a user facing API.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13231] Make count failed values a user ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11115#issuecomment-181374811
**[Test build #50920 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50920/consoleFull)** for PR 11115 at commit [`f84fc4d`](https://github.com/apache/spark/commit/f84fc4dd25904f4a95f79cce789d1aba2f5f1a50).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13231] Make count failed values a user ...
Posted by ScrapCodes <gi...@git.apache.org>.
Github user ScrapCodes commented on the pull request:
https://github.com/apache/spark/pull/11115#issuecomment-210324204
@andrewor14 As far as I knew, it worked fine. There is a check in Task.collectAccumulatorUpdates, you mean that check is not sufficient ? Ahh looking at the jira, I understood there are a lot more ways than just task failure that a stage is recomputed e.g. cache eviction. Correct me if I am wrong.
Alright closing this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13231] Make count failed values a user ...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/11115#issuecomment-202639815
I just talked with @rxin and we decided that this flag isn't quite ready to be exposed yet. The problem is the default value (false) has weird semantics on stage retries; if you're counting the number of rows in your data and your stage was run twice, then you might get 2X the actual number of rows. This is what #11105 is looking at, and we might proceed from there.
Either way I think we should try to fix the default behavior before exposing a flag that we won't be able to change in the future. Let's keep the issue open but close this PR for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13231] Make count failed values a user ...
Posted by ScrapCodes <gi...@git.apache.org>.
Github user ScrapCodes commented on the pull request:
https://github.com/apache/spark/pull/11115#issuecomment-181716006
@JoshRosen Thanks for taking a look. I will update it, did you also consider renaming - or may be just updating the doc suffices ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13231] Make count failed values a user ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11115#issuecomment-181475086
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50923/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13231] Make count failed values a user ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11115#issuecomment-181375401
**[Test build #50920 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50920/consoleFull)** for PR 11115 at commit [`f84fc4d`](https://github.com/apache/spark/commit/f84fc4dd25904f4a95f79cce789d1aba2f5f1a50).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13231] Make count failed values a user ...
Posted by ScrapCodes <gi...@git.apache.org>.
Github user ScrapCodes commented on the pull request:
https://github.com/apache/spark/pull/11115#issuecomment-181369924
I have left-out the renaming for now, not sure about it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13231] Make count failed values a user ...
Posted by ScrapCodes <gi...@git.apache.org>.
GitHub user ScrapCodes reopened a pull request:
https://github.com/apache/spark/pull/11115
[SPARK-13231] Make count failed values a user facing API.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ScrapCodes/spark 13231/accumulator
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11115.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11115
----
commit f84fc4dd25904f4a95f79cce789d1aba2f5f1a50
Author: Prashant Sharma <pr...@in.ibm.com>
Date: 2016-02-08T12:22:12Z
[SPARK-13231] Make count failed values a user facing API.
commit 5e5c58f6c88315cf320ecf83bdca85e9a089859a
Author: Prashant Sharma <pr...@in.ibm.com>
Date: 2016-02-08T14:52:02Z
fixed-style-issues
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13231] Make count failed values a user ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11115#issuecomment-181375409
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13231] Make count failed values a user ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11115#issuecomment-181375416
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50920/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13231] Make count failed values a user ...
Posted by ScrapCodes <gi...@git.apache.org>.
Github user ScrapCodes commented on the pull request:
https://github.com/apache/spark/pull/11115#issuecomment-193112524
@andrewor14 Can you take a look ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13231] Make count failed values a user ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11115#issuecomment-181473927
**[Test build #50923 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50923/consoleFull)** for PR 11115 at commit [`5e5c58f`](https://github.com/apache/spark/commit/5e5c58f6c88315cf320ecf83bdca85e9a089859a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13231] Make count failed values a user ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11115#issuecomment-181475078
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13231] Make count failed values a user ...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/11115#issuecomment-181514492
/cc @andrewor14, we should look at this PR since I know that we discussed making the metric-style accumulators a public API while working on that patch.
@ScrapCodes, if we do move forward with this I think we'll want to update the programming guide in order to help users discover this feature.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13231] Make count failed values a user ...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/11115#issuecomment-195133179
I'm hesitant about this because we would need to maintain backward compatibility if we make it fully public. All the exposed constructors will have to be there forever. If we do decide to do this we need to make sure we like the name because that can't change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13231] Make count failed values a user ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11115#issuecomment-181417813
**[Test build #50923 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50923/consoleFull)** for PR 11115 at commit [`5e5c58f`](https://github.com/apache/spark/commit/5e5c58f6c88315cf320ecf83bdca85e9a089859a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13231] Make count failed values a user ...
Posted by ScrapCodes <gi...@git.apache.org>.
Github user ScrapCodes closed the pull request at:
https://github.com/apache/spark/pull/11115
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13231] Make count failed values a user ...
Posted by ScrapCodes <gi...@git.apache.org>.
Github user ScrapCodes closed the pull request at:
https://github.com/apache/spark/pull/11115
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org