Spark SQL query

[AJT <at...@currenex.com>]

From what I have read on Spark SQL - you need to already have a dataframe
which you can then query on - e.g. select * from myDataframe where
<conditions>
Where the dataframe is either a Hive table or Avro file etc.

What if you want to create a dataframe from your underlying data on the fly
with input parameters passed into your job. 
i.e. 
1. Read my data files (e.g. avro) into a dataframe dependent on what
arguments are passed (e.g. date range)
2. perform map / mapPartitions / filter / GroupBy functions on the dataframe
to create a new dataframe
3. output this dataframe

I can see how to do this in a standard spark application (e.g. run via
spark-submit) but what if I want to use one of the myriad of tools
(Tableau/Qlik etc) that are SparkSQL compliant and run my job from there? Is
there a way I can do:

select * from
functions_on_dataframe_which_output_dataframe(dataframe_built_from_input_arguments)

Appreciate any help



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-query-tp27850.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org