You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Maximilian Alber <al...@gmail.com> on 2015/06/29 12:19:27 UTC

Python vs Scala - Performance

Hi Flinksters,

we had recently a discussion in our working group which Language we should
use with Flink. To bring it to the point: most people would like to use
Python because the are familiar with it and there is a nice scientific
stack to f.e. print and analyse the results. But our concern is that Python
is far less efficient than Scala.

Is that true? If yes, there is an estimate on the penalty?
For a better understanding it would be nice if someone could describe how
Python fits into the Java/Scala environment of Flink!

Thank you!
Cheers,
Max

Re: Python vs Scala - Performance

Posted by Maximilian Alber <al...@gmail.com>.
Ok, thanks. That I wanted to (not) hear :-)

Cheers,
Max

On Tue, Jun 30, 2015 at 10:17 AM, Fabian Hueske <fh...@gmail.com> wrote:

> Hi Max,
>
> the Python API is still in an early beta state and builds on the Java API
> as said before. All framework processing (sorting, joining, data shipping,
> etc.) is done in Java. Whenever, a Python user function needs to be called,
> the data is given to an external Python process and later received back.
> The Python API tries to reduce the number of switches between Java and
> Python, but the cost is quite high.
>
> You can expect a slowdown of at least 2x, probably more. That depends of
> course on your job.
>
> Cheers, Fabian
>
> 2015-06-30 10:05 GMT+02:00 Maximilian Alber <al...@gmail.com>:
>
>> Thank you very much! Good to know that it is build on the Java API.
>>
>> I would be still interested to know, it it is a "serious" impact to use
>> Python instead of Java or if on the long run the runtime is amortized the
>> same?
>> That there is no definite answer I'm aware of, but still I would
>> appreciate your opinions!
>>
>> Cheers,
>> Max
>>
>> On Tue, Jun 30, 2015 at 3:12 AM, Hawin Jiang <ha...@gmail.com>
>> wrote:
>>
>>> Hi  Max
>>>
>>> I think you have to learn Java or Scala if you want to use Flink in your
>>> project.
>>>
>>> The below picture from Marton's slides.  Hopefully, that is a good
>>> reference for you.
>>>
>>>
>>> [image: Inline image 1]
>>>
>>>
>>> Best regards
>>> Hawin
>>>
>>> On Mon, Jun 29, 2015 at 3:19 AM, Maximilian Alber <
>>> alber.maximilian@gmail.com> wrote:
>>>
>>>> Hi Flinksters,
>>>>
>>>> we had recently a discussion in our working group which Language we
>>>> should use with Flink. To bring it to the point: most people would like to
>>>> use Python because the are familiar with it and there is a nice scientific
>>>> stack to f.e. print and analyse the results. But our concern is that Python
>>>> is far less efficient than Scala.
>>>>
>>>> Is that true? If yes, there is an estimate on the penalty?
>>>> For a better understanding it would be nice if someone could describe
>>>> how Python fits into the Java/Scala environment of Flink!
>>>>
>>>> Thank you!
>>>> Cheers,
>>>> Max
>>>>
>>>
>>>
>>
>

Re: Python vs Scala - Performance

Posted by Fabian Hueske <fh...@gmail.com>.
Hi Max,

the Python API is still in an early beta state and builds on the Java API
as said before. All framework processing (sorting, joining, data shipping,
etc.) is done in Java. Whenever, a Python user function needs to be called,
the data is given to an external Python process and later received back.
The Python API tries to reduce the number of switches between Java and
Python, but the cost is quite high.

You can expect a slowdown of at least 2x, probably more. That depends of
course on your job.

Cheers, Fabian

2015-06-30 10:05 GMT+02:00 Maximilian Alber <al...@gmail.com>:

> Thank you very much! Good to know that it is build on the Java API.
>
> I would be still interested to know, it it is a "serious" impact to use
> Python instead of Java or if on the long run the runtime is amortized the
> same?
> That there is no definite answer I'm aware of, but still I would
> appreciate your opinions!
>
> Cheers,
> Max
>
> On Tue, Jun 30, 2015 at 3:12 AM, Hawin Jiang <ha...@gmail.com>
> wrote:
>
>> Hi  Max
>>
>> I think you have to learn Java or Scala if you want to use Flink in your
>> project.
>>
>> The below picture from Marton's slides.  Hopefully, that is a good
>> reference for you.
>>
>>
>> [image: Inline image 1]
>>
>>
>> Best regards
>> Hawin
>>
>> On Mon, Jun 29, 2015 at 3:19 AM, Maximilian Alber <
>> alber.maximilian@gmail.com> wrote:
>>
>>> Hi Flinksters,
>>>
>>> we had recently a discussion in our working group which Language we
>>> should use with Flink. To bring it to the point: most people would like to
>>> use Python because the are familiar with it and there is a nice scientific
>>> stack to f.e. print and analyse the results. But our concern is that Python
>>> is far less efficient than Scala.
>>>
>>> Is that true? If yes, there is an estimate on the penalty?
>>> For a better understanding it would be nice if someone could describe
>>> how Python fits into the Java/Scala environment of Flink!
>>>
>>> Thank you!
>>> Cheers,
>>> Max
>>>
>>
>>
>

Re: Python vs Scala - Performance

Posted by Maximilian Alber <al...@gmail.com>.
Thank you very much! Good to know that it is build on the Java API.

I would be still interested to know, it it is a "serious" impact to use
Python instead of Java or if on the long run the runtime is amortized the
same?
That there is no definite answer I'm aware of, but still I would appreciate
your opinions!

Cheers,
Max

On Tue, Jun 30, 2015 at 3:12 AM, Hawin Jiang <ha...@gmail.com> wrote:

> Hi  Max
>
> I think you have to learn Java or Scala if you want to use Flink in your
> project.
>
> The below picture from Marton's slides.  Hopefully, that is a good
> reference for you.
>
>
> [image: Inline image 1]
>
>
> Best regards
> Hawin
>
> On Mon, Jun 29, 2015 at 3:19 AM, Maximilian Alber <
> alber.maximilian@gmail.com> wrote:
>
>> Hi Flinksters,
>>
>> we had recently a discussion in our working group which Language we
>> should use with Flink. To bring it to the point: most people would like to
>> use Python because the are familiar with it and there is a nice scientific
>> stack to f.e. print and analyse the results. But our concern is that Python
>> is far less efficient than Scala.
>>
>> Is that true? If yes, there is an estimate on the penalty?
>> For a better understanding it would be nice if someone could describe how
>> Python fits into the Java/Scala environment of Flink!
>>
>> Thank you!
>> Cheers,
>> Max
>>
>
>

Re: Python vs Scala - Performance

Posted by Hawin Jiang <ha...@gmail.com>.
Hi  Max

I think you have to learn Java or Scala if you want to use Flink in your
project.

The below picture from Marton's slides.  Hopefully, that is a good
reference for you.


[image: Inline image 1]


Best regards
Hawin

On Mon, Jun 29, 2015 at 3:19 AM, Maximilian Alber <
alber.maximilian@gmail.com> wrote:

> Hi Flinksters,
>
> we had recently a discussion in our working group which Language we should
> use with Flink. To bring it to the point: most people would like to use
> Python because the are familiar with it and there is a nice scientific
> stack to f.e. print and analyse the results. But our concern is that Python
> is far less efficient than Scala.
>
> Is that true? If yes, there is an estimate on the penalty?
> For a better understanding it would be nice if someone could describe how
> Python fits into the Java/Scala environment of Flink!
>
> Thank you!
> Cheers,
> Max
>