You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by sarutak <gi...@git.apache.org> on 2017/03/11 00:03:53 UTC
[GitHub] spark pull request #17252: [SPARK-19913][SS] Log warning rather than throw A...
GitHub user sarutak opened a pull request:
https://github.com/apache/spark/pull/17252
[SPARK-19913][SS] Log warning rather than throw AnalysisException when output is partitioned although format is memory, console or foreach
## What changes were proposed in this pull request?
When batches are executed with memory, console or foreach format, `assertNotPartitioned` will check whether output is not partitioned and throw AnalysisException in case it is.
But I wonder it's better to log warning rather than throw the exception because partitioning does not affect output for those formats but also does not bring any negative impacts.
Also, this assertion is not applied when the format is `console`. I think in this case too, we should assert that .
By fixing them, we can easily switch the format to memory or console for debug purposes.
## How was this patch tested?
I tested manually for memory, foreach and console formats and confirm that warning is displayed.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sarutak/spark SPARK-19913
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17252.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17252
----
commit 12b251c8d32d79f74f74b65eb62d08994f83da5a
Author: Kousuke Saruta <sa...@oss.nttdata.co.jp>
Date: 2017-03-10T23:35:41Z
Log warning rather than throw exception when MemorySink or ForeachSink is used with partitioning
commit 7372aa509453d94db5cf267cc8006c78c1be54ef
Author: Kousuke Saruta <sa...@oss.nttdata.co.jp>
Date: 2017-03-10T23:36:01Z
Log warning when output is partitioned though the format is console
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17252: [SPARK-19913][SS] Log warning rather than throw Analysis...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17252
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74357/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17252: [SPARK-19913][SS] Log warning rather than throw Analysis...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17252
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17252: [SPARK-19913][SS] Log warning rather than throw Analysis...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17252
**[Test build #74554 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74554/testReport)** for PR 17252 at commit [`01ad04a`](https://github.com/apache/spark/commit/01ad04a84655d779940bdc0487a58028d1ef3771).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17252: [SPARK-19913][SS] Log warning rather than throw A...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak closed the pull request at:
https://github.com/apache/spark/pull/17252
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17252: [SPARK-19913][SS] Log warning rather than throw Analysis...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17252
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17252: [SPARK-19913][SS] Log warning rather than throw Analysis...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17252
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74554/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17252: [SPARK-19913][SS] Log warning rather than throw Analysis...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17252
**[Test build #74357 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74357/testReport)** for PR 17252 at commit [`7372aa5`](https://github.com/apache/spark/commit/7372aa509453d94db5cf267cc8006c78c1be54ef).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17252: [SPARK-19913][SS] Log warning rather than throw Analysis...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/17252
Thanks for working on this, but I think this is inconsistent with other APIs in Spark. Also for things like the foreach sink, you might actually be expecting the option to affect the partitioning for some correctness reason. As such I think we should close this issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17252: [SPARK-19913][SS] Log warning rather than throw Analysis...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17252
**[Test build #74357 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74357/testReport)** for PR 17252 at commit [`7372aa5`](https://github.com/apache/spark/commit/7372aa509453d94db5cf267cc8006c78c1be54ef).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17252: [SPARK-19913][SS] Log warning rather than throw Analysis...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on the issue:
https://github.com/apache/spark/pull/17252
Thanks for the comment. I understand the concern relevant to the consistency.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17252: [SPARK-19913][SS] Log warning rather than throw Analysis...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17252
**[Test build #74554 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74554/testReport)** for PR 17252 at commit [`01ad04a`](https://github.com/apache/spark/commit/01ad04a84655d779940bdc0487a58028d1ef3771).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org