Running Apache Flink on the GraalVM as a Native Image

["ivo.knabe@t-online.de" <iv...@t-online.de>]

Whats up guys,
 
I'm trying to run an Apache Flink Application with the GraalVM Native Image 
but I get the following error: (check attached file)
 
I suppose this happens, because Flink uses a lot of low-level-code and is 
highly optimized.
 
When I googled the combination of GraalVM Native Image and Apache Flink I 
get no results.
 
Did anyone ever succeeded in making it work and how?
 
Best regards,
 
Ivo


Re: Running Apache Flink on the GraalVM as a Native Image

[Stephen Connolly <st...@gmail.com>]

On Sun 28 Jun 2020 at 01:34, Stephen Connolly <
stephen.alan.connolly@gmail.com> wrote:

>
>
> On Thu 25 Jun 2020 at 12:48, ivo.knabe@t-online.de <iv...@t-online.de>
> wrote:
>
>> Whats up guys,
>>
>>
>>
>> I'm trying to run an Apache Flink Application with the GraalVM Native
>> Image but I get the following error: (check attached file)
>>
>>
>>
>> I suppose this happens, because Flink uses a lot of low-level-code and is
>> highly optimized.
>>
>
> Actually I suspect the reason is that Flink uses dynamic classloading.
>
> GraalVM requires all the code available in order to produce a native image.
>
> You’d need to pre-bind the topology you want Flink to run into the native
> image.
>
> More fun, you’ll actually need two images, one for the job manager and one
> for the task manager.
>
> And you’ll need to convince GraalVM that the entry-point is your topology
> needs reflection support enabled... plus whatever other classes use
> reflection in Flink.
>
> Sounds rather complex to me. If native images are what is important to
> you, there seemed to be a strong contender in the Rust language community,
> didn’t provide as strong management as Flink, and you’d probably have more
> work managing things like checkpointing, but if native code is important
> that’s where I’d be looking. Sadly I cannot remember the name and my
> google-foo is weak tonight
>

I think it might have been
https://github.com/grippy/tempest but that looks less actively developed
than the one I thought I saw...

I’d also check out frameworks for Go even if I dislike Go... if you want
native code it’s either Rust or Go in my book


>
>>
>> When I googled the combination of GraalVM Native Image and Apache Flink I
>> get no results.
>>
>>
>>
>> Did anyone ever succeeded in making it work and how?
>>
>>
>>
>> Best regards,
>>
>>
>>
>> Ivo
>> 
>>
> --
> Sent from my phone
>
-- 
Sent from my phone

Re: Running Apache Flink on the GraalVM as a Native Image

[Stephen Connolly <st...@gmail.com>]

On Thu 25 Jun 2020 at 12:48, ivo.knabe@t-online.de <iv...@t-online.de>
wrote:

> Whats up guys,
>
>
>
> I'm trying to run an Apache Flink Application with the GraalVM Native
> Image but I get the following error: (check attached file)
>
>
>
> I suppose this happens, because Flink uses a lot of low-level-code and is
> highly optimized.
>

Actually I suspect the reason is that Flink uses dynamic classloading.

GraalVM requires all the code available in order to produce a native image.

You’d need to pre-bind the topology you want Flink to run into the native
image.

More fun, you’ll actually need two images, one for the job manager and one
for the task manager.

And you’ll need to convince GraalVM that the entry-point is your topology
needs reflection support enabled... plus whatever other classes use
reflection in Flink.

Sounds rather complex to me. If native images are what is important to you,
there seemed to be a strong contender in the Rust language community,
didn’t provide as strong management as Flink, and you’d probably have more
work managing things like checkpointing, but if native code is important
that’s where I’d be looking. Sadly I cannot remember the name and my
google-foo is weak tonight


>
> When I googled the combination of GraalVM Native Image and Apache Flink I
> get no results.
>
>
>
> Did anyone ever succeeded in making it work and how?
>
>
>
> Best regards,
>
>
>
> Ivo
> 
>
-- 
Sent from my phone

Re: Running Apache Flink on the GraalVM as a Native Image

[Arvid Heise <ar...@ververica.com>]

Using Flink in a NI does not make much sense. However, it's definitely
worthwhile to enable Flink applications to use Flink as a library and
bundle that in a NI.

Btw reflection is not the root problem. You can use reflection just fine in
a NI. The issue with NI is (next to Unsafe) that NI has a close-world
assumption. That means we would only run Flink without any UDFs (in the
sense of real user code, not UDFs provided by Flink). However, in theory,
it should be possible to compile Flink against a specific application and
build a NI when everything that is needed for a particular Flink
application is known to the Graal NI compiler. In practise, Flink would
need that compiler to figure out, which classes/methods are actually needed
and which not, since some Flink magic would break the close world
assumption.
Finally, Table API with code generation will probably not work with Graal
for the time being at all. Not sure if there is any effort to support that
as well on Graal side.

On Tue, Jun 30, 2020 at 9:51 AM Roman Grebennikov <gr...@dfdx.me> wrote:

> Hi,
>
> the error it the original message means that inside a Kryo serializer,
> there is something that uses sun.misc.Unsafe to compute field offset in raw
> memory to access class fields. The problem is that memory layout in JVM and
> in NI is different and most probably will result in a segfault if you leave
> it as is.
>
> If you really want to go further, there is a @RecomputeFieldValue(kind =
> Kind.FieldOffset) annotation in the graalvm to update the offset for NI.
>
> But I guess the overall perspective of running Flink as a NI is not that
> good for the following reasons:
> * GraalVM NI requires that the whole application is completely static and
> there is no dynamic classloading happening in runtime. NI still can handle
> some sort of dynamic classloading, but only if you define all possible
> combinations of loaded classes on compile time. Flink itself uses dynamic
> classloading when you submit a job there as a fat jar file. So you will
> need to statically compile a bundle of your app fat jar AND taskmanager
> code at once into a single binary.
> * Flink also relies quite a lot on dynamic reflection in the serialization
> code, so you also have to build static reflection configuration at
> compile-time (by probably running your job+taskmanager with tracing agent
> on JVM)
> * As GraalVM NI has no way of doing JIT compilation (as everything must be
> compiled statically), you should expect a lower overall job performance.
>
> GraalVM and Flink can be quite a nice code gymnastics exercise, but what
> is your final business goal?
>
> Roman Grebennikov | grv@dfdx.me
>
>
> On Thu, Jun 25, 2020, at 11:48, ivo.knabe@t-online.de wrote:
>
> Whats up guys,
>
>
>
> I'm trying to run an Apache Flink Application with the GraalVM Native
> Image but I get the following error: (check attached file)
>
>
>
> I suppose this happens, because Flink uses a lot of low-level-code and is
> highly optimized.
>
>
>
> When I googled the combination of GraalVM Native Image and Apache Flink I
> get no results.
>
>
>
> Did anyone ever succeeded in making it work and how?
>
>
>
> Best regards,
>
>
>
> Ivo
> 
>
> *Attachments:*
>
>    - Flink GraalVM Native Image Error
>
>
>

-- 

Arvid Heise | Senior Java Developer

<https://www.ververica.com/>

Follow us @VervericaData

--

Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Toni) Cheng

Re: Running Apache Flink on the GraalVM as a Native Image

[Roman Grebennikov <gr...@dfdx.me>]

Hi,

the error it the original message means that inside a Kryo serializer, there is something that uses sun.misc.Unsafe to compute field offset in raw memory to access class fields. The problem is that memory layout in JVM and in NI is different and most probably will result in a segfault if you leave it as is.

If you really want to go further, there is a @RecomputeFieldValue(kind = Kind.FieldOffset) annotation in the graalvm to update the offset for NI.

But I guess the overall perspective of running Flink as a NI is not that good for the following reasons:
* GraalVM NI requires that the whole application is completely static and there is no dynamic classloading happening in runtime. NI still can handle some sort of dynamic classloading, but only if you define all possible combinations of loaded classes on compile time. Flink itself uses dynamic classloading when you submit a job there as a fat jar file. So you will need to statically compile a bundle of your app fat jar AND taskmanager code at once into a single binary.
* Flink also relies quite a lot on dynamic reflection in the serialization code, so you also have to build static reflection configuration at compile-time (by probably running your job+taskmanager with tracing agent on JVM)
* As GraalVM NI has no way of doing JIT compilation (as everything must be compiled statically), you should expect a lower overall job performance.

GraalVM and Flink can be quite a nice code gymnastics exercise, but what is your final business goal?

Roman Grebennikov | grv@dfdx.me


On Thu, Jun 25, 2020, at 11:48, ivo.knabe@t-online.de wrote:
> Whats up guys,

> 

> I'm trying to run an Apache Flink Application with the GraalVM Native Image but I get the following error: (check attached file)

> 

> I suppose this happens, because Flink uses a lot of low-level-code and is highly optimized.

> 

> When I googled the combination of GraalVM Native Image and Apache Flink I get no results.

> 

> Did anyone ever succeeded in making it work and how?

> 

> Best regards,

> 

> Ivo

> 
> 
> *Attachments:*
>  * Flink GraalVM Native Image Error