Spark on Mesos fine-grained - has one core less per executor

[hbogert <ha...@gmail.com>]

I'm doing a performance analysis for Spark on Mesos and I can see that the
Coarse-grained backend simply launches tasks in wave size of the amount of
cores available. But it seems Fine-grained mode the Mesos executor takes 1
core for itself (so -1 core per mesos slave). Shouldn't fine- and
coarse-grained mode have the same behaviour?




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-Mesos-fine-grained-has-one-core-less-per-executor-tp23273.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: spark stream and spark sql with data warehouse

[Akhil Das <ak...@sigmoidanalytics.com>]

This is a good start, if you haven't read it already
http://spark.apache.org/docs/latest/streaming-programming-guide.html#dataframe-and-sql-operations

Thanks
Best Regards

On Thu, Jun 11, 2015 at 8:17 PM, 唐思成 <ja...@qq.com> wrote:

> Hi all:
>        We are trying to using spark to do some real time data processing.
> I need do some sql-like query and analytical tasks with the real time data
> against historical normalized data stored in data bases. Is there anyone
> has done this kind of work or design? Any suggestion or material would be
> truly welcomed.
>
>
>

Re: spark stream and spark sql with data warehouse

[Yi Tian <ti...@gmail.com>]

Here is an example:

|val sc = new SparkContext(new SparkConf)

// access hive tables
val hqlc = new HiveContext(sc)
import hqlc.implicits._

// access files on hdfs
val sqlc = new SQLContext(sc)
import sqlc.implicits._
sqlc.jsonFile("xxx").registerTempTable("xxx")

// access other DB
sqlc.jdbc("url", "tablename").registerTempTable("xxx")

// create streams
val ssc = new StreamingContext(sc, Seconds(30))
val stream = ssc.textFileStream("xxx")

// you could use foreachRDD or transform
// foreachRDD return Unit
// transform return DStream
stream.foreachRDD( streamRDD => {
   streamRDD.map(aaa(_)).toDF.registerTempTable("xxx")
   sqlc.sql("xxx").saveAsParquetFile("xxx")
})
ssc.start()
ssc.awaitTermination()
|


On Thu, Jun 11, 2015 at 10:47 PM, 唐思成 jadetangcn@qq.com 
<ht...@qq.com> wrote:
Hi all:
We are trying to using spark to do some real time data processing. I 
need do some sql-like query and analytical tasks with the real time data 
against historical normalized data stored in data bases. Is there anyone 
has done this kind of work or design? Any suggestion or material would 
be truly welcomed.

On 6/11/15 22:47, 唐思成 wrote:

> Hi all:
>        We are trying to using spark to do some real time data 
> processing. I need do some sql-like query and analytical tasks with 
> the real time data against historical normalized data stored in data 
> bases. Is there anyone has done this kind of work or design? Any 
> suggestion or material would be truly welcomed.
>
​

spark stream and spark sql with data warehouse

[唐思成 <ja...@qq.com>]

Hi all:
       We are trying to using spark to do some real time data processing. I need do some sql-like query and analytical tasks with the real time data against historical normalized data stored in data bases. Is there anyone has done this kind of work or design? Any suggestion or material would be truly welcomed.