You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/07 05:45:17 UTC
[GitHub] [spark] cloud-fan commented on a change in pull request #29242: [SPARK-31448] [PYTHON] Fix storage level used in cache() in dataframe.py
cloud-fan commented on a change in pull request #29242:
URL: https://github.com/apache/spark/pull/29242#discussion_r484193839
##########
File path: python/pyspark/sql/dataframe.py
##########
@@ -678,13 +678,14 @@ def cache(self):
return self
@since(1.3)
- def persist(self, storageLevel=StorageLevel.MEMORY_AND_DISK):
+ def persist(self, storageLevel=StorageLevel.MEMORY_AND_DISK_DESER):
Review comment:
Now I see the confusion. In Scala, `MEMORY_AND_DISK` means `deserialized=true`, while in Python, `MEMORY_AND_DISK` means `deserialized=false`.
##########
File path: python/pyspark/sql/dataframe.py
##########
@@ -678,13 +678,14 @@ def cache(self):
return self
@since(1.3)
- def persist(self, storageLevel=StorageLevel.MEMORY_AND_DISK):
+ def persist(self, storageLevel=StorageLevel.MEMORY_AND_DISK_DESER):
"""Sets the storage level to persist the contents of the :class:`DataFrame` across
operations after the first time it is computed. This can only be used to assign
a new storage level if the :class:`DataFrame` does not have a storage level set yet.
- If no storage level is specified defaults to (`MEMORY_AND_DISK`).
+ If no storage level is specified defaults to (`MEMORY_AND_DISK_DESER`)
- .. note:: The default storage level has changed to `MEMORY_AND_DISK` to match Scala in 2.0.
+ .. note:: The default storage level has changed to `MEMORY_AND_DISK_DESER` to match Scala
+ in 2.0.
Review comment:
`in 3.0`?
##########
File path: python/pyspark/sql/dataframe.py
##########
@@ -678,13 +678,14 @@ def cache(self):
return self
@since(1.3)
- def persist(self, storageLevel=StorageLevel.MEMORY_AND_DISK):
+ def persist(self, storageLevel=StorageLevel.MEMORY_AND_DISK_DESER):
"""Sets the storage level to persist the contents of the :class:`DataFrame` across
operations after the first time it is computed. This can only be used to assign
a new storage level if the :class:`DataFrame` does not have a storage level set yet.
- If no storage level is specified defaults to (`MEMORY_AND_DISK`).
+ If no storage level is specified defaults to (`MEMORY_AND_DISK_DESER`)
- .. note:: The default storage level has changed to `MEMORY_AND_DISK` to match Scala in 2.0.
+ .. note:: The default storage level has changed to `MEMORY_AND_DISK_DESER` to match Scala
+ in 2.0.
Review comment:
`in 3.0`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org