You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sa...@wellsfargo.com on 2015/10/01 20:44:50 UTC
Accumulator of rows?
Hi all,
I need to repeat a couple rows from a dataframe by n times each. To do so, I plan to create a new Data Frame, but I am being unable to find a way to accumulate "Rows" somewhere, as this might get huge, I can't accumulate into a mutable Array, I think?.
Thanks,
Saif
RE: Accumulator of rows?
Posted by Sa...@wellsfargo.com.
Thank you, exactly what I was looking for. I have read of it before but never associated.
Saif
From: Adrian Tanase [mailto:atanase@adobe.com]
Sent: Friday, October 02, 2015 8:24 AM
To: Ellafi, Saif A.; user@spark.apache.org
Subject: Re: Accumulator of rows?
Have you seen window functions?
https://databricks.com/blog/2015/07/15/introducing-window-functions-in-spark-sql.html
From: "Saif.A.Ellafi@wellsfargo.com<ma...@wellsfargo.com>"
Date: Thursday, October 1, 2015 at 9:44 PM
To: "user@spark.apache.org<ma...@spark.apache.org>"
Subject: Accumulator of rows?
Hi all,
I need to repeat a couple rows from a dataframe by n times each. To do so, I plan to create a new Data Frame, but I am being unable to find a way to accumulate “Rows” somewhere, as this might get huge, I can’t accumulate into a mutable Array, I think?.
Thanks,
Saif
Re: Accumulator of rows?
Posted by Adrian Tanase <at...@adobe.com>.
Have you seen window functions?
https://databricks.com/blog/2015/07/15/introducing-window-functions-in-spark-sql.html
From: "Saif.A.Ellafi@wellsfargo.com<ma...@wellsfargo.com>"
Date: Thursday, October 1, 2015 at 9:44 PM
To: "user@spark.apache.org<ma...@spark.apache.org>"
Subject: Accumulator of rows?
Hi all,
I need to repeat a couple rows from a dataframe by n times each. To do so, I plan to create a new Data Frame, but I am being unable to find a way to accumulate “Rows” somewhere, as this might get huge, I can’t accumulate into a mutable Array, I think?.
Thanks,
Saif