How to retrieve multiple columns values (in one row) to variables in Spark Scala method

[Mich Talebzadeh <mi...@gmail.com>]

Hi,

Pretty basic question.

I use Spark on Hbase to retrieve the last  14 prices average and standard
deviation  for a security (ticker) from an Hbase table.

However, the call is expensive in Spark streaming where these values are
used to indicate buy and sell and subsequently the high value prices are
stored to MongoDB in the same spark streaming program.

I have defined these two methods to get the average and STDDEV of prices.

This method gets the average of prices

  def tickerAvg(dfHbase: org.apache.spark.sql.DataFrame, ticker: String):
Double = {
    var priceTicker = dfHbase.filter(col("ticker") ===
ticker).sort(col("timeissued").desc).
                                   limit(14).
                                   select(round(avg(col("price")),3)).
                                   collect.apply(0).getDouble(0)
    return priceTicker
  }
And this one STDDEV

  def tickerStddev(dfHbase: org.apache.spark.sql.DataFrame, ticker:
String): Double = {
    var priceTicker = dfHbase.filter(col("ticker") ===
ticker).sort(col("timeissued").desc).
                                   limit(14).
                                   select(round(stddev(col("price")),3)).
                                   collect.apply(0).getDouble(0)
    return priceTicker
  }

Note that each returns one value

However as these calls are expensive,  I want to return both AVG and STDDEV
in one call and from one method as follows


  def tickerGeneric(dfHbase: org.apache.spark.sql.DataFrame, ticker:
String): Double = {
    var priceTicker = dfHbase.filter(col("ticker") ===
ticker).sort(col("timeissued").desc).
                                   limit(14).

select(round(avg(col("price")),3),round(stddev(col("price")),3)).
                                   collect.?

   return ?
}

How this can be achieved to return both AVG and STDDEV values with one call
to method?

thanks

Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.

Re: How to retrieve multiple columns values (in one row) to variables in Spark Scala method

[Mich Talebzadeh <mi...@gmail.com>]

Thanks Ayan!

Much appreciated your help.

Regards,


Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Fri, 5 Apr 2019 at 23:56, ayan guha <gu...@gmail.com> wrote:

> Try like this:
>
> val primitiveDS = spark.sql("select 1.2 avg ,2.3
> stddev").collect().apply(0)
> val arr = Array(primitiveDS.getDecimal(0), primitiveDS.getDecimal(1))
>
> primitiveDS: org.apache.spark.sql.Row = [1.2,2.3] arr:
> Array[java.math.BigDecimal] = Array(1.2, 2.3)
>

Re: How to retrieve multiple columns values (in one row) to variables in Spark Scala method

[ayan guha <gu...@gmail.com>]

Try like this:

val primitiveDS = spark.sql("select 1.2 avg ,2.3 stddev").collect().apply(0)
val arr = Array(primitiveDS.getDecimal(0), primitiveDS.getDecimal(1))

primitiveDS: org.apache.spark.sql.Row = [1.2,2.3] arr:
Array[java.math.BigDecimal] = Array(1.2, 2.3)