Ship compiled code with broadcast stream ?

[Maxim Parkachov <la...@gmail.com>]

Hi everyone,

I have a job with event stream and control stream delivering rules for
event transformation. Rules are broadcasted and used in flatMat-like
coProcessFunction. Rules are defined in custom JSON format. Amount of rules
and complexity rises significantly with every new feature.

What I would like is to ship compiled (serialized ?) code instead of JSON
rules to control stream and use this compiled classes directly without
additional transformation. This will allow to get more robust testing and
will allow to implement much more complex rules. But I'm struggling to
understand how to achieve this.

Did someone implement system like this ? Is this possible at all ?

Any help is greatly appreciated,
Maxim.

Re: Ship compiled code with broadcast stream ?

[Gyula Fóra <gy...@gmail.com>]

You should not try sending the compiled code anywhere but you can use it
from within the processor.

You can do the same thing with the jar, you compile your jar, store it on
HDFS. Send the jar path to the processor which can download the jar and
instantiate the rule. No need to resubmit the job.

Gyula

Maxim Parkachov <la...@gmail.com> ezt írta (időpont: 2018. okt. 9.,
K, 15:08):

> Hi,
>
> This is certainly possible. What you can do is use a
>> BroadcastProcessFunction where you receive the rule code on the broadcast
>> side.
>>
>
> Yes, this part works, no problem.
>
>
>> You probably cannot send newly compiled objects this way but what you can
>> do is either send a reference to some compiled jars and load them with the
>> URLClassloader or send the actual String code and invoke the java compiler
>> from your function.
>>
>
> Compiling from source code ... actually, I can implement rules in nashorn
> JavaScript, but I'm pretty sure I'll get not serializable exception, but
> I'll check this.
>
> Referencing existing jar will not solve problem as I would need to
> re-submit job, which I want to avoid in the first place. I actually wanted
> exactly first scenario, send newly compiled objects.
>
> Regards,
> Maxim.
>
>>

Re: Ship compiled code with broadcast stream ?

[Maxim Parkachov <la...@gmail.com>]

Hi,

This is certainly possible. What you can do is use a
> BroadcastProcessFunction where you receive the rule code on the broadcast
> side.
>

Yes, this part works, no problem.


> You probably cannot send newly compiled objects this way but what you can
> do is either send a reference to some compiled jars and load them with the
> URLClassloader or send the actual String code and invoke the java compiler
> from your function.
>

Compiling from source code ... actually, I can implement rules in nashorn
JavaScript, but I'm pretty sure I'll get not serializable exception, but
I'll check this.

Referencing existing jar will not solve problem as I would need to
re-submit job, which I want to avoid in the first place. I actually wanted
exactly first scenario, send newly compiled objects.

Regards,
Maxim.

>

Re: Ship compiled code with broadcast stream ?

[Gyula Fóra <gy...@gmail.com>]

Hi,

This is certainly possible. What you can do is use a
BroadcastProcessFunction where you receive the rule code on the broadcast
side.

You probably cannot send newly compiled objects this way but what you can
do is either send a reference to some compiled jars and load them with the
URLClassloader or send the actual String code and invoke the java compiler
from your function.

Hope that helps :)

Gyula

Maxim Parkachov <la...@gmail.com> ezt írta (időpont: 2018. okt. 9.,
K, 9:38):

> Hi everyone,
>
> I have a job with event stream and control stream delivering rules for
> event transformation. Rules are broadcasted and used in flatMat-like
> coProcessFunction. Rules are defined in custom JSON format. Amount of rules
> and complexity rises significantly with every new feature.
>
> What I would like is to ship compiled (serialized ?) code instead of JSON
> rules to control stream and use this compiled classes directly without
> additional transformation. This will allow to get more robust testing and
> will allow to implement much more complex rules. But I'm struggling to
> understand how to achieve this.
>
> Did someone implement system like this ? Is this possible at all ?
>
> Any help is greatly appreciated,
> Maxim.
>