You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by umargeek <um...@gmail.com> on 2018/05/23 04:59:26 UTC

How to validate orc vectorization is working within spark application?

Hi Folks,

I have enabled below listed configurations within my spark streaming
application but I did not gain performance benefit even after setting these
parameters ,can you please help me is there a way to validate whether
vectorization is working as expeced/enabled correctly !

Note: I am using Spark 2.3 and converted all the data within my application
in orc format.

    sparkSqlCtx.setConf("spark.sql.orc.filterPushdown", "true")
    sparkSqlCtx.setConf("spark.sql.orc.enabled", "true")
    sparkSqlCtx.setConf("spark.sql.hive.convertMetastoreOrc", "true")
    sparkSqlCtx.setConf("spark.sql.orc.char.enabled", "true")
    sparkSqlCtx.setConf("spark.sql.orc.impl","native")
    sparkSqlCtx.setConf("spark.sql.orc.enableVectorizedReader","true")

Thanks,
Umar



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: How to validate orc vectorization is working within spark application?

Posted by umargeek <um...@gmail.com>.
Hello Jorn,

I am unable to post the entire code due to some data sharing related issues.

Use Case: I am performing aggregations after reading data from HDFS file
every min would like to understand how to perform using vectorisation 
enabled and what are pre requisite to successfully to enable the same.

Thanks,
Umar



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: How to validate orc vectorization is working within spark application?

Posted by Jörn Franke <jo...@gmail.com>.
Full code? What is expected performance and actual ?
What is the use case?

> On 20. Jun 2018, at 05:33, umargeek <um...@gmail.com> wrote:
> 
> Hi Folks,
> 
> I would just require few pointers on the above query w.r.t vectorization
> looking forward for support from the community.
> 
> Thanks,
> Umar
> 
> 
> 
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
> 
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> 

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: How to validate orc vectorization is working within spark application?

Posted by umargeek <um...@gmail.com>.
Hi Folks,

I would just require few pointers on the above query w.r.t vectorization
looking forward for support from the community.

Thanks,
Umar



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org