You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Andrew Davidson <ae...@ucsc.edu.INVALID> on 2022/01/20 22:20:32 UTC

java.lang.StackOverflow Error How to sum across rows in a data frame with a large number of columns

Hi

I have a dataframe of integers. It has 10409 columns. How can I sum across each row?

I get a very long stack trace

rowSums BEGIN
2022-01-20 22:11:24 ERROR __main__:? - An error occurred while calling o93935.withColumn.
: java.lang.StackOverflowError
                        at scala.collection.immutable.Set$SetBuilderImpl.$plus$eq(Set.scala:349)
                        at scala.collection.immutable.Set$SetBuilderImpl.$plus$eq(Set.scala:329)


###############################################################################

def rowSums( self, countsSparkDF, columnNames ):

    self.logger.warn( "rowSums BEGIN" )



    # https://stackoverflow.com/a/54283997/4586180

    retDF = countsSparkDF.na.fill( 0 ).withColumn( "rowSum" , reduce( add, [col( x ) for x in columnNames] ) )



    self.logger.warn( "rowSums retDF numRows:{} numCols:{}"\

                         .format( retDF.count(), len( retDF.columns ) ) )



    self.logger.warn( "rowSums END\n" )

    return retDF