You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Giacomo Licari <gi...@gmail.com> on 2015/09/21 18:41:06 UTC
Compile flink file with external jar
Hi guys,
I'm developing a dataflow client whose backend, exported as jar, allows
users to convert dataflows to a Flink file .java with some code inside. The
generated file naturally calls some classes I have in jar file, like
MapFunction, DataSet.
My question is:
How to compile the generated file and pack it into a jar file, so I can
execute it with Flink?
I have tried several methods like:
- javac -cp backend.jar:. GeneratedClass.java
- javac -cp ".:backend.jar" GeneratedClass.java
- javac -cp ".:./backend.jar" GeneratedClass.java
with no success.
I receive errors like:
GeneratedClass.java:2: error: package org.apache.flink.api.common.functions
does not exist GeneratedClass.java:3: error: package
org.apache.flink.api.java does not exist
Thanks a lot,
Giacomo
Re: Compile flink file with external jar
Posted by Giacomo Licari <gi...@gmail.com>.
Hi guys,
Janino works perfectly!
It allows to create entire classes on-the-fly and to use them.
Thank you a lot.
Cheers,
Giacomo
On Mon, Sep 21, 2015 at 7:49 PM, Giacomo Licari <gi...@gmail.com>
wrote:
> Thanks a lot Fabian,
> I will try it.
>
> Cheers,
> Giacomo
>
> On Mon, Sep 21, 2015 at 7:35 PM, Fabian Hueske <fh...@gmail.com> wrote:
>
>> Hi Giacomo,
>>
>> you could use Janino [1] to directly compile the code string into a class
>> and execute it. The program does not need to be shipped to the cluster if
>> all user functions are contained in the jar.
>>
>> Cheers, Fabian
>>
>> [1] http://unkrig.de/w/Janino
>> On Sep 21, 2015 7:08 PM, "Giacomo Licari" <gi...@gmail.com>
>> wrote:
>>
>>> Hi Robert,
>>> thanks for the reply.
>>> I receive a JSON from my client interface, which contains the dataflow
>>> description.
>>> Then I parse that JSON and the parser creates a string which contains
>>> the Flink code, as the user can modify the dataflow, the description can
>>> change every time it calls "Execute Dataflow".
>>>
>>> Now I should be able to execute the generated String code, so I was
>>> first thinking about to write it to a .java file.
>>> What could you suggest to me? Is it possible to execute the string code
>>> directly from the backend.jar I have?
>>>
>>>
>>> On Mon, Sep 21, 2015 at 6:54 PM, Robert Metzger <rm...@apache.org>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> you have to make sure that the Flink classes are contained in your
>>>> class path.
>>>> Either add the flink-dist jar from the binary distribution to your
>>>> class path, or use maven to build the backend.jar as a fat jar.
>>>>
>>>> Why are you generating a java class from your dataflows?
>>>> Isn't it easier to just call the Flink APIs when parsing your flow
>>>> description language?
>>>>
>>>> Robert
>>>>
>>>>
>>>> On Mon, Sep 21, 2015 at 6:41 PM, Giacomo Licari <
>>>> giacomo.licari@gmail.com> wrote:
>>>>
>>>>> Hi guys,
>>>>>
>>>>> I'm developing a dataflow client whose backend, exported as jar,
>>>>> allows users to convert dataflows to a Flink file .java with some code
>>>>> inside. The generated file naturally calls some classes I have in jar file,
>>>>> like MapFunction, DataSet.
>>>>>
>>>>> My question is:
>>>>> How to compile the generated file and pack it into a jar file, so I
>>>>> can execute it with Flink?
>>>>>
>>>>> I have tried several methods like:
>>>>>
>>>>> - javac -cp backend.jar:. GeneratedClass.java
>>>>> - javac -cp ".:backend.jar" GeneratedClass.java
>>>>> - javac -cp ".:./backend.jar" GeneratedClass.java
>>>>>
>>>>> with no success.
>>>>>
>>>>> I receive errors like:
>>>>> GeneratedClass.java:2: error: package
>>>>> org.apache.flink.api.common.functions does not exist GeneratedClass.java:3:
>>>>> error: package org.apache.flink.api.java does not exist
>>>>>
>>>>>
>>>>> Thanks a lot,
>>>>>
>>>>> Giacomo
>>>>>
>>>>
>>>>
>>>
>
Re: Compile flink file with external jar
Posted by Giacomo Licari <gi...@gmail.com>.
Thanks a lot Fabian,
I will try it.
Cheers,
Giacomo
On Mon, Sep 21, 2015 at 7:35 PM, Fabian Hueske <fh...@gmail.com> wrote:
> Hi Giacomo,
>
> you could use Janino [1] to directly compile the code string into a class
> and execute it. The program does not need to be shipped to the cluster if
> all user functions are contained in the jar.
>
> Cheers, Fabian
>
> [1] http://unkrig.de/w/Janino
> On Sep 21, 2015 7:08 PM, "Giacomo Licari" <gi...@gmail.com>
> wrote:
>
>> Hi Robert,
>> thanks for the reply.
>> I receive a JSON from my client interface, which contains the dataflow
>> description.
>> Then I parse that JSON and the parser creates a string which contains the
>> Flink code, as the user can modify the dataflow, the description can change
>> every time it calls "Execute Dataflow".
>>
>> Now I should be able to execute the generated String code, so I was first
>> thinking about to write it to a .java file.
>> What could you suggest to me? Is it possible to execute the string code
>> directly from the backend.jar I have?
>>
>>
>> On Mon, Sep 21, 2015 at 6:54 PM, Robert Metzger <rm...@apache.org>
>> wrote:
>>
>>> Hi,
>>>
>>> you have to make sure that the Flink classes are contained in your class
>>> path.
>>> Either add the flink-dist jar from the binary distribution to your class
>>> path, or use maven to build the backend.jar as a fat jar.
>>>
>>> Why are you generating a java class from your dataflows?
>>> Isn't it easier to just call the Flink APIs when parsing your flow
>>> description language?
>>>
>>> Robert
>>>
>>>
>>> On Mon, Sep 21, 2015 at 6:41 PM, Giacomo Licari <
>>> giacomo.licari@gmail.com> wrote:
>>>
>>>> Hi guys,
>>>>
>>>> I'm developing a dataflow client whose backend, exported as jar, allows
>>>> users to convert dataflows to a Flink file .java with some code inside. The
>>>> generated file naturally calls some classes I have in jar file, like
>>>> MapFunction, DataSet.
>>>>
>>>> My question is:
>>>> How to compile the generated file and pack it into a jar file, so I can
>>>> execute it with Flink?
>>>>
>>>> I have tried several methods like:
>>>>
>>>> - javac -cp backend.jar:. GeneratedClass.java
>>>> - javac -cp ".:backend.jar" GeneratedClass.java
>>>> - javac -cp ".:./backend.jar" GeneratedClass.java
>>>>
>>>> with no success.
>>>>
>>>> I receive errors like:
>>>> GeneratedClass.java:2: error: package
>>>> org.apache.flink.api.common.functions does not exist GeneratedClass.java:3:
>>>> error: package org.apache.flink.api.java does not exist
>>>>
>>>>
>>>> Thanks a lot,
>>>>
>>>> Giacomo
>>>>
>>>
>>>
>>
Re: Compile flink file with external jar
Posted by Fabian Hueske <fh...@gmail.com>.
Hi Giacomo,
you could use Janino [1] to directly compile the code string into a class
and execute it. The program does not need to be shipped to the cluster if
all user functions are contained in the jar.
Cheers, Fabian
[1] http://unkrig.de/w/Janino
On Sep 21, 2015 7:08 PM, "Giacomo Licari" <gi...@gmail.com> wrote:
> Hi Robert,
> thanks for the reply.
> I receive a JSON from my client interface, which contains the dataflow
> description.
> Then I parse that JSON and the parser creates a string which contains the
> Flink code, as the user can modify the dataflow, the description can change
> every time it calls "Execute Dataflow".
>
> Now I should be able to execute the generated String code, so I was first
> thinking about to write it to a .java file.
> What could you suggest to me? Is it possible to execute the string code
> directly from the backend.jar I have?
>
>
> On Mon, Sep 21, 2015 at 6:54 PM, Robert Metzger <rm...@apache.org>
> wrote:
>
>> Hi,
>>
>> you have to make sure that the Flink classes are contained in your class
>> path.
>> Either add the flink-dist jar from the binary distribution to your class
>> path, or use maven to build the backend.jar as a fat jar.
>>
>> Why are you generating a java class from your dataflows?
>> Isn't it easier to just call the Flink APIs when parsing your flow
>> description language?
>>
>> Robert
>>
>>
>> On Mon, Sep 21, 2015 at 6:41 PM, Giacomo Licari <giacomo.licari@gmail.com
>> > wrote:
>>
>>> Hi guys,
>>>
>>> I'm developing a dataflow client whose backend, exported as jar, allows
>>> users to convert dataflows to a Flink file .java with some code inside. The
>>> generated file naturally calls some classes I have in jar file, like
>>> MapFunction, DataSet.
>>>
>>> My question is:
>>> How to compile the generated file and pack it into a jar file, so I can
>>> execute it with Flink?
>>>
>>> I have tried several methods like:
>>>
>>> - javac -cp backend.jar:. GeneratedClass.java
>>> - javac -cp ".:backend.jar" GeneratedClass.java
>>> - javac -cp ".:./backend.jar" GeneratedClass.java
>>>
>>> with no success.
>>>
>>> I receive errors like:
>>> GeneratedClass.java:2: error: package
>>> org.apache.flink.api.common.functions does not exist GeneratedClass.java:3:
>>> error: package org.apache.flink.api.java does not exist
>>>
>>>
>>> Thanks a lot,
>>>
>>> Giacomo
>>>
>>
>>
>
Re: Compile flink file with external jar
Posted by Giacomo Licari <gi...@gmail.com>.
Hi Robert,
thanks for the reply.
I receive a JSON from my client interface, which contains the dataflow
description.
Then I parse that JSON and the parser creates a string which contains the
Flink code, as the user can modify the dataflow, the description can change
every time it calls "Execute Dataflow".
Now I should be able to execute the generated String code, so I was first
thinking about to write it to a .java file.
What could you suggest to me? Is it possible to execute the string code
directly from the backend.jar I have?
On Mon, Sep 21, 2015 at 6:54 PM, Robert Metzger <rm...@apache.org> wrote:
> Hi,
>
> you have to make sure that the Flink classes are contained in your class
> path.
> Either add the flink-dist jar from the binary distribution to your class
> path, or use maven to build the backend.jar as a fat jar.
>
> Why are you generating a java class from your dataflows?
> Isn't it easier to just call the Flink APIs when parsing your flow
> description language?
>
> Robert
>
>
> On Mon, Sep 21, 2015 at 6:41 PM, Giacomo Licari <gi...@gmail.com>
> wrote:
>
>> Hi guys,
>>
>> I'm developing a dataflow client whose backend, exported as jar, allows
>> users to convert dataflows to a Flink file .java with some code inside. The
>> generated file naturally calls some classes I have in jar file, like
>> MapFunction, DataSet.
>>
>> My question is:
>> How to compile the generated file and pack it into a jar file, so I can
>> execute it with Flink?
>>
>> I have tried several methods like:
>>
>> - javac -cp backend.jar:. GeneratedClass.java
>> - javac -cp ".:backend.jar" GeneratedClass.java
>> - javac -cp ".:./backend.jar" GeneratedClass.java
>>
>> with no success.
>>
>> I receive errors like:
>> GeneratedClass.java:2: error: package
>> org.apache.flink.api.common.functions does not exist GeneratedClass.java:3:
>> error: package org.apache.flink.api.java does not exist
>>
>>
>> Thanks a lot,
>>
>> Giacomo
>>
>
>
Re: Compile flink file with external jar
Posted by Robert Metzger <rm...@apache.org>.
Hi,
you have to make sure that the Flink classes are contained in your class
path.
Either add the flink-dist jar from the binary distribution to your class
path, or use maven to build the backend.jar as a fat jar.
Why are you generating a java class from your dataflows?
Isn't it easier to just call the Flink APIs when parsing your flow
description language?
Robert
On Mon, Sep 21, 2015 at 6:41 PM, Giacomo Licari <gi...@gmail.com>
wrote:
> Hi guys,
>
> I'm developing a dataflow client whose backend, exported as jar, allows
> users to convert dataflows to a Flink file .java with some code inside. The
> generated file naturally calls some classes I have in jar file, like
> MapFunction, DataSet.
>
> My question is:
> How to compile the generated file and pack it into a jar file, so I can
> execute it with Flink?
>
> I have tried several methods like:
>
> - javac -cp backend.jar:. GeneratedClass.java
> - javac -cp ".:backend.jar" GeneratedClass.java
> - javac -cp ".:./backend.jar" GeneratedClass.java
>
> with no success.
>
> I receive errors like:
> GeneratedClass.java:2: error: package
> org.apache.flink.api.common.functions does not exist GeneratedClass.java:3:
> error: package org.apache.flink.api.java does not exist
>
>
> Thanks a lot,
>
> Giacomo
>