You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Luke Miner (JIRA)" <ji...@apache.org> on 2016/04/04 20:54:25 UTC
[jira] [Comment Edited] (SPARK-14141) Let user specify datatypes of
pandas dataframe in toPandas()
[ https://issues.apache.org/jira/browse/SPARK-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15218930#comment-15218930 ]
Luke Miner edited comment on SPARK-14141 at 4/4/16 6:54 PM:
------------------------------------------------------------
Anecdotally, at least, it seems like a pretty common workflow for data scientists is to use Spark to preprocess data down to a size where it can be sent through scikit-learn/theano/nltk/tensorflow/etc. So I'd imagine that there'd be a fair amount of uptake on a feature that would make it painless to get large datasets from Spark into pandas on a single machine. I don't know if this means that it belongs in Spark, but I'd find it very useful even if it is a little slow!
was (Author: lminer):
Anecdotally, at least, it seems like a pretty common workflow for data scientists is to use Spark to preprocess data down to a size where it can be sent through scikit-learn/theano/nltk/tensorflow/etc. So I'd imagine that there'd be a fair amount of uptake on a feature that would make it painless to get large datasets from Spark into pandas on a single machine. I don't know if this means that it belongs in Spark, but I'd find it very useful even if it is a little slow!
Incidentally, I added an issue to Pandas to allow dtypes to be specified in the `from_records()` constructor. I figured this would be useful regardless, if only to make it easier preserve some more type information during the conversion (e.g. datetime columns): https://github.com/pydata/pandas/issues/12751
> Let user specify datatypes of pandas dataframe in toPandas()
> ------------------------------------------------------------
>
> Key: SPARK-14141
> URL: https://issues.apache.org/jira/browse/SPARK-14141
> Project: Spark
> Issue Type: New Feature
> Components: Input/Output, PySpark, SQL
> Reporter: Luke Miner
> Priority: Minor
>
> Would be nice to specify the dtypes of the pandas dataframe during the toPandas() call. Something like:
> bq. pdf = df.toPandas(dtypes={'a': 'float64', 'b': 'datetime64', 'c': 'bool', 'd': 'category'})
> Since dtypes like `category` are more memory efficient, you could potentially load many more rows into a pandas dataframe with this option without running out of memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org