You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jerryshao <gi...@git.apache.org> on 2015/01/23 03:45:33 UTC

[GitHub] spark pull request: [SPARK-5372][Streaming] Change the default sto...

GitHub user jerryshao opened a pull request:

    https://github.com/apache/spark/pull/4167

    [SPARK-5372][Streaming] Change the default storage level in window operators

    The default storage level in window operators is MEMORY_ONLY_SER, which will lead to unexpected behavior when memory is not enough. Besides for the reason of aligning this to the storage level of input DStream, here I'd change the storage level to MEMORY_AND_DISK_SER.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/jerryshao/apache-spark SPARK-5372

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/4167.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #4167
    
----
commit 927a7d768c895b2f7f315c7d6cd095f1a559ec35
Author: jerryshao <sa...@intel.com>
Date:   2015-01-23T02:26:04Z

    Change the default storage level in window operators

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5372][Streaming] Change the default sto...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4167#issuecomment-71139617
  
      [Test build #25992 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25992/consoleFull) for   PR 4167 at commit [`927a7d7`](https://github.com/apache/spark/commit/927a7d768c895b2f7f315c7d6cd095f1a559ec35).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5372][Streaming] Change the default sto...

Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the pull request:

    https://github.com/apache/spark/pull/4167#issuecomment-72165580
  
    OK, got it, I will close this PR, thanks @tdas for your review :).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5372][Streaming] Change the default sto...

Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:

    https://github.com/apache/spark/pull/4167#issuecomment-72144338
  
    Well, it generally considered outside the scope of Spark Streaming to windows that large! If you have to process data that is a day old, then you probably need a dedicated storage system. Spark Streaming is not a storage system, so using it for long term data storage is using it outside its design space. Breaking default-case performance for those out-of-the-design-space scenarios is not the right solution. Those should be handled by changing the storage level directly. And users who need that sort of performance across such large windows, obviously need to learn a bit more about Spark Streaming. We can probably help them learn. maybe add some stuff in the programming guide?
    
    An alternate, more sophisticated solution is to detect when such spillover is continuously happening and print suggestion (log4j warnings) saying "not enough memory to store the whole window, consider using memory_and_disk for the windowed stream". This is definitely trickier to do but a safer solution that does not involve regression. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5372][Streaming] Change the default sto...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4167#issuecomment-71144194
  
      [Test build #25992 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25992/consoleFull) for   PR 4167 at commit [`927a7d7`](https://github.com/apache/spark/commit/927a7d768c895b2f7f315c7d6cd095f1a559ec35).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5372][Streaming] Change the default sto...

Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the pull request:

    https://github.com/apache/spark/pull/4167#issuecomment-72139922
  
    And normally we met some customers who set the window size very large, like several hours, or even one day, so in these situations, normally memory might not be enough to hold all the data in the window. So how to handle this situation, to set the storage level explicitly? But sometimes user even do not know window operators will cache the data, so what is your suggestion?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5372][Streaming] Change the default sto...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/4167#issuecomment-71144198
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25992/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5372][Streaming] Change the default sto...

Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:

    https://github.com/apache/spark/pull/4167#issuecomment-72137779
  
    This is a tricky change. I had kept it in MEMORY_ONLY_SER to make the default usecase fast (when stuff fits in memory) fast, and the corner case (stuff not in memory) will be slow (recomputed from raw data or intermediate shuffle files). However writing to MEMORY_AND_DISK_SER makes even the default usecase slow, and corner case equally fast as the default (so effectively, no corner case). This is, for the majority of the users, a signiicant performance regression, especially if they have not even configured their streaming application to use fast and/or multiple disks (which is in most cases not necessary).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5372][Streaming] Change the default sto...

Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao closed the pull request at:

    https://github.com/apache/spark/pull/4167


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5372][Streaming] Change the default sto...

Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the pull request:

    https://github.com/apache/spark/pull/4167#issuecomment-72139397
  
    Hi TD, AFAIK  StorageLevel `MEMORY_AND_DISK_SER` has no difference to `MEMORY_ONLY_SER` if memory is enough, BM will only put data into memory, and only spill to disk when memory is not enough.
    
    Besides all the receiver's default storage level is `MEMORY_AND_DISK_SER_2`, if performance is the first consideration, why not `MEMORY_ONLY_SER_2`?
    
    Actually the behavior of `DISK` will only be triggered when memory is not enough, so I think for normal case, the performance should be same to `MEMORY_ONLY_SER`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org