You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Niklas Semmler <ns...@inet.tu-berlin.de> on 2015/09/04 11:46:17 UTC
Updating the flink JAR with a single module
Hi there,
I have a question on the setup of the development environment for flink.
Is it possible to update the big JAR file
(flink-dist/target/.../lib/flink-dist-0.9-SNAPSHOT.jar), which flink
apparently uses for deployment, with a single module (flink-runtime)?
Recompiling all of the modules takes several minutes on my laptop, while
compiling only the module is a matter of seconds.
My knowledge of maven is pretty limited, so I compile flink with
> mvn package -Dmaven.test.skip=true
This process is really slowing down my development cycle. I would
appreciate your help :)
Best regards,
Niklas
--
PhD Student / Research Assistant
INET, TU Berlin
Room 4.029
Marchstr 23
10587 Berlin
Tel: +49 30 314 78752
Re: Updating the flink JAR with a single module
Posted by Niklas Semmler <ns...@inet.tu-berlin.de>.
Hello Fabian, Hello Matthias,
Thank you very much! I am not making API changes at the moment, so I
should be fine with this approach.
Unfortunately the command "mvn -DskipTests clean install" in the main
folder didn't work for me (it hanged in the middle of the operation), so
I am using a more brute force method. I compile the flink-runtime jar
file just as Fabian described and then use it to overwrite the content
in the "lib/flink-dist-0.9-SNAPSHOT.jar".
It is not really a clean path, but this method has reduced my compile
time down to 1 minute, which is a great improvement :)
Best,
Niklas
On 04.09.2015 12:59, Matthias J. Sax wrote:
> One more remark on this:
>
> You need to be a little bit more careful. This approach works only, if
> the initially changed module has no other dependent modules.
>
> For example, flink-clients also depends on flink-runtime and thus, you
> *might* need to rebuild flink-clients, too, before building flink-dist.
> However, only if you did API changes in flink-runtime.
>
> The basic message is, you need to understand the dependency tree. If you
> did API changes, you need to rebuild all modules that depend on the
> changed module.
>
>
> -Matthias
>
> On 09/04/2015 12:24 PM, Fabian Hueske wrote:
>> Hi Niklas,
>>
>> if you only want to update individual submodules you can do:
>>
>> cd flink-runtime // build this submodule
>> mvn -DskipTests clean install // will put the new flink-runtime jar into
>> the local Maven repository
>> cd ../flink-dist
>> mvn -DskipTests clean install // will fetch all required dependencies from
>> the local Maven repository without rebuilding them.
>>
>> You need to have run mvn -DskipTests clean install before to have all
>> required dependencies in the local repository.
>>
>> Cheers, Fabian
>>
>> 2015-09-04 11:46 GMT+02:00 Niklas Semmler <ns...@inet.tu-berlin.de>:
>>
>>> Hi there,
>>>
>>> I have a question on the setup of the development environment for flink.
>>>
>>> Is it possible to update the big JAR file
>>> (flink-dist/target/.../lib/flink-dist-0.9-SNAPSHOT.jar), which flink
>>> apparently uses for deployment, with a single module (flink-runtime)?
>>>
>>> Recompiling all of the modules takes several minutes on my laptop, while
>>> compiling only the module is a matter of seconds.
>>>
>>> My knowledge of maven is pretty limited, so I compile flink with
>>>
>>>> mvn package -Dmaven.test.skip=true
>>>
>>> This process is really slowing down my development cycle. I would
>>> appreciate your help :)
>>>
>>> Best regards,
>>> Niklas
>>>
>>> --
>>> PhD Student / Research Assistant
>>> INET, TU Berlin
>>> Room 4.029
>>> Marchstr 23
>>> 10587 Berlin
>>> Tel: +49 30 314 78752
>>>
>>
>
--
PhD Student / Research Assistant
INET, TU Berlin
Room 4.029
Marchstr 23
10587 Berlin
Tel: +49 30 314 78752
Re: Updating the flink JAR with a single module
Posted by "Matthias J. Sax" <mj...@apache.org>.
One more remark on this:
You need to be a little bit more careful. This approach works only, if
the initially changed module has no other dependent modules.
For example, flink-clients also depends on flink-runtime and thus, you
*might* need to rebuild flink-clients, too, before building flink-dist.
However, only if you did API changes in flink-runtime.
The basic message is, you need to understand the dependency tree. If you
did API changes, you need to rebuild all modules that depend on the
changed module.
-Matthias
On 09/04/2015 12:24 PM, Fabian Hueske wrote:
> Hi Niklas,
>
> if you only want to update individual submodules you can do:
>
> cd flink-runtime // build this submodule
> mvn -DskipTests clean install // will put the new flink-runtime jar into
> the local Maven repository
> cd ../flink-dist
> mvn -DskipTests clean install // will fetch all required dependencies from
> the local Maven repository without rebuilding them.
>
> You need to have run mvn -DskipTests clean install before to have all
> required dependencies in the local repository.
>
> Cheers, Fabian
>
> 2015-09-04 11:46 GMT+02:00 Niklas Semmler <ns...@inet.tu-berlin.de>:
>
>> Hi there,
>>
>> I have a question on the setup of the development environment for flink.
>>
>> Is it possible to update the big JAR file
>> (flink-dist/target/.../lib/flink-dist-0.9-SNAPSHOT.jar), which flink
>> apparently uses for deployment, with a single module (flink-runtime)?
>>
>> Recompiling all of the modules takes several minutes on my laptop, while
>> compiling only the module is a matter of seconds.
>>
>> My knowledge of maven is pretty limited, so I compile flink with
>>
>>> mvn package -Dmaven.test.skip=true
>>
>> This process is really slowing down my development cycle. I would
>> appreciate your help :)
>>
>> Best regards,
>> Niklas
>>
>> --
>> PhD Student / Research Assistant
>> INET, TU Berlin
>> Room 4.029
>> Marchstr 23
>> 10587 Berlin
>> Tel: +49 30 314 78752
>>
>
Re: Updating the flink JAR with a single module
Posted by Fabian Hueske <fh...@gmail.com>.
Hi Niklas,
if you only want to update individual submodules you can do:
cd flink-runtime // build this submodule
mvn -DskipTests clean install // will put the new flink-runtime jar into
the local Maven repository
cd ../flink-dist
mvn -DskipTests clean install // will fetch all required dependencies from
the local Maven repository without rebuilding them.
You need to have run mvn -DskipTests clean install before to have all
required dependencies in the local repository.
Cheers, Fabian
2015-09-04 11:46 GMT+02:00 Niklas Semmler <ns...@inet.tu-berlin.de>:
> Hi there,
>
> I have a question on the setup of the development environment for flink.
>
> Is it possible to update the big JAR file
> (flink-dist/target/.../lib/flink-dist-0.9-SNAPSHOT.jar), which flink
> apparently uses for deployment, with a single module (flink-runtime)?
>
> Recompiling all of the modules takes several minutes on my laptop, while
> compiling only the module is a matter of seconds.
>
> My knowledge of maven is pretty limited, so I compile flink with
>
> > mvn package -Dmaven.test.skip=true
>
> This process is really slowing down my development cycle. I would
> appreciate your help :)
>
> Best regards,
> Niklas
>
> --
> PhD Student / Research Assistant
> INET, TU Berlin
> Room 4.029
> Marchstr 23
> 10587 Berlin
> Tel: +49 30 314 78752
>