HiveContext.getOrCreate not accessible

[Praseetha <pr...@gmail.com>]

Hi All,


I have a streaming app and when i try invoking the HiveContext.getOrCreate,
it errors out with the following stmt. 'object HiveContext in package hive
cannot be accessed in package org.apache.spark.sql.hive'

I would require HiveContext instead of SQLContext for my application and
creating new HiveContext everytime would not be a feasible solution.

Here is my code snippet:
object sampleStreamingApp  {

  def createStreamingContext(checkpointDirectory: String): StreamingContext
= {
    val conf = new SparkConf().setAppName("sampleStreaming")
    val sc = new SparkContext(conf)
    val ssc = new StreamingContext(sc, Milliseconds(5000))
    ssc.checkpoint(checkpointDirectory)
val smDStream = ssc.textFileStream("/user/hdpuser/data")
val smSplitted = smDStream.map( x => x.split(";") ).map( x => Row.fromSeq(
x ) )
    smSplitted.foreachRDD { rdd =>
                 val sqlContext = HiveContext.getOrCreate(rdd.sparkContext)
                import sqlContext.implicits._
                <other code logic goes here>
    }
    }
    ssc
  }

  def main(args: Array[String]) {
  val checkpointDirectory =
"hdfs://localhost:8020/user/dfml/checkpointing/AAA"
  val ssc = StreamingContext.getActiveOrCreate(checkpointDirectory, () =>
createStreamingContext(checkpointDirectory))

  ssc.start()
  ssc.awaitTermination()
  }
}

Any help would be appreciated.

Regds,
--Praseetha

Re: HiveContext.getOrCreate not accessible

["Shixiong(Ryan) Zhu" <sh...@databricks.com>]

`SQLContext.getOrCreate` will return the HiveContext you created.

On Mon, Nov 14, 2016 at 11:17 PM, Praseetha <pr...@gmail.com> wrote:

>
> Hi All,
>
>
> I have a streaming app and when i try invoking the
> HiveContext.getOrCreate, it errors out with the following stmt. 'object
> HiveContext in package hive cannot be accessed in package
> org.apache.spark.sql.hive'
>
> I would require HiveContext instead of SQLContext for my application and
> creating new HiveContext everytime would not be a feasible solution.
>
> Here is my code snippet:
> object sampleStreamingApp  {
>
>   def createStreamingContext(checkpointDirectory: String):
> StreamingContext = {
>     val conf = new SparkConf().setAppName("sampleStreaming")
>     val sc = new SparkContext(conf)
>     val ssc = new StreamingContext(sc, Milliseconds(5000))
>     ssc.checkpoint(checkpointDirectory)
> val smDStream = ssc.textFileStream("/user/hdpuser/data")
> val smSplitted = smDStream.map( x => x.split(";") ).map( x => Row.fromSeq(
> x ) )
>     smSplitted.foreachRDD { rdd =>
>                  val sqlContext = HiveContext.getOrCreate(rdd.
> sparkContext)
>                 import sqlContext.implicits._
>                 <other code logic goes here>
>     }
>     }
>     ssc
>   }
>
>   def main(args: Array[String]) {
>   val checkpointDirectory = "hdfs://localhost:8020/user/
> dfml/checkpointing/AAA"
>   val ssc = StreamingContext.getActiveOrCreate(checkpointDirectory, () =>
> createStreamingContext(checkpointDirectory))
>
>   ssc.start()
>   ssc.awaitTermination()
>   }
> }
>
> Any help would be appreciated.
>
> Regds,
> --Praseetha
>