You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by James King <ja...@gmail.com> on 2015/03/19 15:50:26 UTC
Writing Spark Streaming Programs
Hello All,
I'm using Spark for streaming but I'm unclear one which implementation
language to use Java, Scala or Python.
I don't know anything about Python, familiar with Scala and have been doing
Java for a long time.
I think the above shouldn't influence my decision on which language to use
because I believe the tool should, fit the problem.
In terms of performance Java and Scala are comparable. However Java is OO
and Scala is FP, no idea what Python is.
If using Scala and not applying a consistent style of programming Scala
code can become unreadable, but I do like the fact it seems to be possible
to do so much work with so much less code, that's a strong selling point
for me. Also it could be that the type of programming done in Spark is best
implemented in Scala as FP language, not sure though.
The question I would like your good help with is are there any other
considerations I need to think about when deciding this? are there any
recommendations you can make in regards to this?
Regards
jk
Re: Writing Spark Streaming Programs
Posted by Charles Feduke <ch...@gmail.com>.
Scala is the language used to write Spark so there's never a situation in
which features introduced in a newer version of Spark cannot be taken
advantage of if you write your code in Scala. (This is mostly true of Java,
but it may be a little more legwork if a Java-friendly adapter isn't
available alongside new features.)
Scala is also OO; its a functional hybrid OO language.
Although much of my organization's codebase is written in Java and we've
recently transitioned to Java 8 I still write all of my Spark code using
Scala. (I also squeeze in Scala where I can in other parts of the
organization.) Additionally I use both Python and R for local data
analysis, though I haven't used Python with Spark in production.
On Thu, Mar 19, 2015 at 10:51 AM James King <ja...@gmail.com> wrote:
> Hello All,
>
> I'm using Spark for streaming but I'm unclear one which implementation
> language to use Java, Scala or Python.
>
> I don't know anything about Python, familiar with Scala and have been
> doing Java for a long time.
>
> I think the above shouldn't influence my decision on which language to use
> because I believe the tool should, fit the problem.
>
> In terms of performance Java and Scala are comparable. However Java is OO
> and Scala is FP, no idea what Python is.
>
> If using Scala and not applying a consistent style of programming Scala
> code can become unreadable, but I do like the fact it seems to be possible
> to do so much work with so much less code, that's a strong selling point
> for me. Also it could be that the type of programming done in Spark is best
> implemented in Scala as FP language, not sure though.
>
> The question I would like your good help with is are there any other
> considerations I need to think about when deciding this? are there any
> recommendations you can make in regards to this?
>
> Regards
> jk
>
>
>
>
>
>
>
Re: Writing Spark Streaming Programs
Posted by Jeffrey Jedele <je...@gmail.com>.
I second what has been said already.
We just built a streaming app in Java and I would definitely choose Scala
this time.
Regards,
Jeff
2015-03-19 16:34 GMT+01:00 Emre Sevinc <em...@gmail.com>:
> Hello James,
>
> I've been working with Spark Streaming for the last 6 months, and I'm
> coding in Java 7. Even though I haven't encountered any blocking issues
> with that combination, I'd definitely pick Scala if the decision was up to
> me.
>
> I agree with Gerard and Charles on this one. If you can, go with Scala for
> Spark Streaming applications.
>
> Cheers,
>
> Emre Sevinç
> http://www.bigindustries.be/
>
>
>
> On Thu, Mar 19, 2015 at 4:09 PM, James King <ja...@gmail.com> wrote:
>
>> Many thanks Gerard, this is very helpful. Cheers!
>>
>> On Thu, Mar 19, 2015 at 4:02 PM, Gerard Maas <ge...@gmail.com>
>> wrote:
>>
>>> Try writing this Spark Streaming idiom in Java and you'll choose Scala
>>> soon enough:
>>>
>>> dstream.foreachRDD{rdd =>
>>> rdd.foreachPartition( partition => ....)
>>> }
>>>
>>> When deciding between Java and Scala for Spark, IMHO Scala has the
>>> upperhand. If you're concerned with readability, have a look at the Scala
>>> coding style recently open sourced by DataBricks:
>>> https://github.com/databricks/scala-style-guide (btw, I don't agree a
>>> good part of it, but recognize that it can keep the most complex Scala
>>> constructions out of your code)
>>>
>>>
>>>
>>> On Thu, Mar 19, 2015 at 3:50 PM, James King <ja...@gmail.com>
>>> wrote:
>>>
>>>> Hello All,
>>>>
>>>> I'm using Spark for streaming but I'm unclear one which implementation
>>>> language to use Java, Scala or Python.
>>>>
>>>> I don't know anything about Python, familiar with Scala and have been
>>>> doing Java for a long time.
>>>>
>>>> I think the above shouldn't influence my decision on which language to
>>>> use because I believe the tool should, fit the problem.
>>>>
>>>> In terms of performance Java and Scala are comparable. However Java is
>>>> OO and Scala is FP, no idea what Python is.
>>>>
>>>> If using Scala and not applying a consistent style of programming Scala
>>>> code can become unreadable, but I do like the fact it seems to be possible
>>>> to do so much work with so much less code, that's a strong selling point
>>>> for me. Also it could be that the type of programming done in Spark is best
>>>> implemented in Scala as FP language, not sure though.
>>>>
>>>> The question I would like your good help with is are there any other
>>>> considerations I need to think about when deciding this? are there any
>>>> recommendations you can make in regards to this?
>>>>
>>>> Regards
>>>> jk
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
>
> --
> Emre Sevinc
>
Re: Writing Spark Streaming Programs
Posted by Emre Sevinc <em...@gmail.com>.
Hello James,
I've been working with Spark Streaming for the last 6 months, and I'm
coding in Java 7. Even though I haven't encountered any blocking issues
with that combination, I'd definitely pick Scala if the decision was up to
me.
I agree with Gerard and Charles on this one. If you can, go with Scala for
Spark Streaming applications.
Cheers,
Emre Sevinç
http://www.bigindustries.be/
On Thu, Mar 19, 2015 at 4:09 PM, James King <ja...@gmail.com> wrote:
> Many thanks Gerard, this is very helpful. Cheers!
>
> On Thu, Mar 19, 2015 at 4:02 PM, Gerard Maas <ge...@gmail.com>
> wrote:
>
>> Try writing this Spark Streaming idiom in Java and you'll choose Scala
>> soon enough:
>>
>> dstream.foreachRDD{rdd =>
>> rdd.foreachPartition( partition => ....)
>> }
>>
>> When deciding between Java and Scala for Spark, IMHO Scala has the
>> upperhand. If you're concerned with readability, have a look at the Scala
>> coding style recently open sourced by DataBricks:
>> https://github.com/databricks/scala-style-guide (btw, I don't agree a
>> good part of it, but recognize that it can keep the most complex Scala
>> constructions out of your code)
>>
>>
>>
>> On Thu, Mar 19, 2015 at 3:50 PM, James King <ja...@gmail.com>
>> wrote:
>>
>>> Hello All,
>>>
>>> I'm using Spark for streaming but I'm unclear one which implementation
>>> language to use Java, Scala or Python.
>>>
>>> I don't know anything about Python, familiar with Scala and have been
>>> doing Java for a long time.
>>>
>>> I think the above shouldn't influence my decision on which language to
>>> use because I believe the tool should, fit the problem.
>>>
>>> In terms of performance Java and Scala are comparable. However Java is
>>> OO and Scala is FP, no idea what Python is.
>>>
>>> If using Scala and not applying a consistent style of programming Scala
>>> code can become unreadable, but I do like the fact it seems to be possible
>>> to do so much work with so much less code, that's a strong selling point
>>> for me. Also it could be that the type of programming done in Spark is best
>>> implemented in Scala as FP language, not sure though.
>>>
>>> The question I would like your good help with is are there any other
>>> considerations I need to think about when deciding this? are there any
>>> recommendations you can make in regards to this?
>>>
>>> Regards
>>> jk
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
--
Emre Sevinc
Re: Writing Spark Streaming Programs
Posted by James King <ja...@gmail.com>.
Many thanks Gerard, this is very helpful. Cheers!
On Thu, Mar 19, 2015 at 4:02 PM, Gerard Maas <ge...@gmail.com> wrote:
> Try writing this Spark Streaming idiom in Java and you'll choose Scala
> soon enough:
>
> dstream.foreachRDD{rdd =>
> rdd.foreachPartition( partition => ....)
> }
>
> When deciding between Java and Scala for Spark, IMHO Scala has the
> upperhand. If you're concerned with readability, have a look at the Scala
> coding style recently open sourced by DataBricks:
> https://github.com/databricks/scala-style-guide (btw, I don't agree a
> good part of it, but recognize that it can keep the most complex Scala
> constructions out of your code)
>
>
>
> On Thu, Mar 19, 2015 at 3:50 PM, James King <ja...@gmail.com> wrote:
>
>> Hello All,
>>
>> I'm using Spark for streaming but I'm unclear one which implementation
>> language to use Java, Scala or Python.
>>
>> I don't know anything about Python, familiar with Scala and have been
>> doing Java for a long time.
>>
>> I think the above shouldn't influence my decision on which language to
>> use because I believe the tool should, fit the problem.
>>
>> In terms of performance Java and Scala are comparable. However Java is OO
>> and Scala is FP, no idea what Python is.
>>
>> If using Scala and not applying a consistent style of programming Scala
>> code can become unreadable, but I do like the fact it seems to be possible
>> to do so much work with so much less code, that's a strong selling point
>> for me. Also it could be that the type of programming done in Spark is best
>> implemented in Scala as FP language, not sure though.
>>
>> The question I would like your good help with is are there any other
>> considerations I need to think about when deciding this? are there any
>> recommendations you can make in regards to this?
>>
>> Regards
>> jk
>>
>>
>>
>>
>>
>>
>>
>
Re: Writing Spark Streaming Programs
Posted by Gerard Maas <ge...@gmail.com>.
Try writing this Spark Streaming idiom in Java and you'll choose Scala soon
enough:
dstream.foreachRDD{rdd =>
rdd.foreachPartition( partition => ....)
}
When deciding between Java and Scala for Spark, IMHO Scala has the
upperhand. If you're concerned with readability, have a look at the Scala
coding style recently open sourced by DataBricks:
https://github.com/databricks/scala-style-guide (btw, I don't agree a good
part of it, but recognize that it can keep the most complex Scala
constructions out of your code)
On Thu, Mar 19, 2015 at 3:50 PM, James King <ja...@gmail.com> wrote:
> Hello All,
>
> I'm using Spark for streaming but I'm unclear one which implementation
> language to use Java, Scala or Python.
>
> I don't know anything about Python, familiar with Scala and have been
> doing Java for a long time.
>
> I think the above shouldn't influence my decision on which language to use
> because I believe the tool should, fit the problem.
>
> In terms of performance Java and Scala are comparable. However Java is OO
> and Scala is FP, no idea what Python is.
>
> If using Scala and not applying a consistent style of programming Scala
> code can become unreadable, but I do like the fact it seems to be possible
> to do so much work with so much less code, that's a strong selling point
> for me. Also it could be that the type of programming done in Spark is best
> implemented in Scala as FP language, not sure though.
>
> The question I would like your good help with is are there any other
> considerations I need to think about when deciding this? are there any
> recommendations you can make in regards to this?
>
> Regards
> jk
>
>
>
>
>
>
>