You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Sudhir Babu Pothineni <sb...@gmail.com> on 2016/09/27 19:43:48 UTC

ORC file stripe statistics in Spark

I am trying to get number of rows each stripe of ORC file?

hivecontext.orcFile doesn't exist anymore? I am using Spark 1.6.0

scala> val hiveSqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
hiveSqlContext: org.apache.spark.sql.hive.HiveContext =
org.apache.spark.sql.hive.HiveContext@83223af

scala> hiveSqlContext.
analyze                   applySchema
asInstanceOf              baseRelationToDataFrame   cacheTable
clearCache                createDataFrame
createDataset             createExternalTable       dropTempTable
emptyDataFrame            experimental
getAllConfs               getConf                   implicits
isCached                  isInstanceOf
isRootContext             jdbc                      jsonFile
jsonRDD                   listenerManager
load                      newSession                parquetFile
range                     read
refreshTable              setConf                   sparkContext
sql                       table
tableNames                tables                    toString
udf                       uncacheTable


Thanks
Sudhir