You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "antonkulaga (Jira)" <ji...@apache.org> on 2019/10/10 19:23:00 UTC
[jira] [Commented] (SPARK-28547) Make it work for wide (> 10K
columns data)
[ https://issues.apache.org/jira/browse/SPARK-28547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948897#comment-16948897 ]
antonkulaga commented on SPARK-28547:
-------------------------------------
[~hyukjin.kwon] what is not clear for you? I think it is really clear that Spark performs miserably (freezing or taking many hours) whenever the data frame has 10-20K and more columns and I gave GTEX dataset as an example (however any gene or transcript expression dataset will be ok to demonstrate it). In many fields (like big part of bioinformatics) wide data frames are common, right now Spark is totally useless there.
> Make it work for wide (> 10K columns data)
> ------------------------------------------
>
> Key: SPARK-28547
> URL: https://issues.apache.org/jira/browse/SPARK-28547
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.0.0
> Environment: Ubuntu server, Spark 2.4.3 Scala with >64GB RAM per node, 32 cores (tried different configurations of executors)
> Reporter: antonkulaga
> Priority: Critical
>
> Spark is super-slow for all wide data (when there are >15kb columns and >15kb rows). Most of the genomics/transcriptomic data is wide because number of genes is usually >20kb and number of samples ass well. Very popular GTEX dataset is a good example ( see for instance RNA-Seq data at https://storage.googleapis.com/gtex_analysis_v7/rna_seq_data where gct is just a .tsv file with two comments in the beginning). Everything done in wide tables (even simple "describe" functions applied to all the genes-columns) either takes hours or gets frozen (because of lost executors) irrespective of memory and numbers of cores. While the same operations work fast (minutes) and well with pure pandas (without any spark involved).
> f
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org