You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Henry Saputra <he...@gmail.com> on 2014/07/21 09:18:55 UTC
Guide/ tips to do preferred development for Flink?
HI Guys,
What are your recommended setup to run and develop Flink locally?
I have been doing local Flink by going to flink-dist/target/ and run
Flink from there.
I was wondering if you guys have preferred way to do Flink development?
Maybe we could add section about "Developing Flink" in the website?
- Henry
Re: Guide/ tips to do preferred development for Flink?
Posted by Henry Saputra <he...@gmail.com>.
Thanks Ufuk, yeah originally I wanted to start local standalone dev
env without actually going to target directory to start the server.
But I will use local environment for dev for now to get more familiar
about the flow.
Thanks guys!
- Henry
On Fri, Aug 8, 2014 at 12:36 AM, Ufuk Celebi <u....@fu-berlin.de> wrote:
> Hey Henry,
>
> sorry for not getting back to you earlier.
>
> The difference is that the network stack does NOT start up and all data transfers are made locally via memory copies (no TCP channels). The default parallelism is set depending on the number of cores, so stuff will still run in parallel.
>
> If you have any ideas for improvement, feel free to post them. :-)
>
> Best,
>
> Ufuk
>
>
> On 08 Aug 2014, at 07:26, Henry Saputra <he...@gmail.com> wrote:
>
>> Thanks Robert, it seems like it does work.
>>
>> So if it is running via local environment context it will just execute
>> the local executor?
>> How much is the difference does the flow in local vs cluster?
>>
>> - Henry
>>
>>
>> On Wed, Aug 6, 2014 at 6:41 AM, Robert Metzger <rm...@apache.org> wrote:
>>> The dev setup for developing Flink with IntelliJ is quite simple: Just
>>> checkout the source code and import it as a Maven project into IntelliJ.
>>> To verify the correctness of your setup, try running the main() method of
>>> the WordCount example. This should always work without troubles.
>>> You can use the WordCount as an example for running Flink programs out of
>>> your IDE.
>>>
>>>
>>> The only time you really have to use the build in "flink-dist/target" is
>>> for manual testing of a version (for example the web interfaces) or
>>> deploying binary archives.
>>>
>>>
>>> On Thu, Jul 31, 2014 at 9:09 AM, Henry Saputra <he...@gmail.com>
>>> wrote:
>>>
>>>> Hi Ufuk,
>>>>
>>>> Sorry I missed this email reply from you.
>>>>
>>>> Yeah, I also used IntelliJ for development and trying to do remote
>>>> debugging instead.
>>>>
>>>> Could you share your dev setup to run Flink using IntelliJ instead?
>>>>
>>>> - Henry
>>>>
>>>> On Mon, Jul 21, 2014 at 12:29 AM, Ufuk Celebi <u....@fu-berlin.de>
>>>> wrote:
>>>>> Hey Henry,
>>>>>
>>>>> On 21 Jul 2014, at 09:18, Henry Saputra <he...@gmail.com> wrote:
>>>>>> What are your recommended setup to run and develop Flink locally?
>>>>>
>>>>> I use IntelliJ for day-to-day dev and testing and run everything inside
>>>> the IDE.
>>>>>
>>>>> If you want to use real programs to test your code you can use
>>>> JavaProgramTestBase for local startup of the system.
>>>> o.a.flink.test.exampleJavaPrograms.WordCountITCase is a good example to see
>>>> how it works.
>>>>>
>>>>> You can also write tests, which use LocalEnvironment. If you want to use
>>>> an existing Flink installation, e.g. in flink-dist/target or remote, you
>>>> can use the RemoteEnvironment to submit a JAR.
>>>>>
>>>>> Does any of this help?
>>>>>
>>>>>> I have been doing local Flink by going to flink-dist/target/ and run
>>>>>> Flink from there.
>>>>>> I was wondering if you guys have preferred way to do Flink development?
>>>>>> Maybe we could add section about "Developing Flink" in the website?
>>>>>
>>>>> +1
>>>>>
>>>>> We had something along those lines a while back for Eclipse and I
>>>> remember that it was helpful when I started with the project. Thanks for
>>>> pointing this out. :-)
>>>>
>
Re: Guide/ tips to do preferred development for Flink?
Posted by Ufuk Celebi <u....@fu-berlin.de>.
Hey Henry,
sorry for not getting back to you earlier.
The difference is that the network stack does NOT start up and all data transfers are made locally via memory copies (no TCP channels). The default parallelism is set depending on the number of cores, so stuff will still run in parallel.
If you have any ideas for improvement, feel free to post them. :-)
Best,
Ufuk
On 08 Aug 2014, at 07:26, Henry Saputra <he...@gmail.com> wrote:
> Thanks Robert, it seems like it does work.
>
> So if it is running via local environment context it will just execute
> the local executor?
> How much is the difference does the flow in local vs cluster?
>
> - Henry
>
>
> On Wed, Aug 6, 2014 at 6:41 AM, Robert Metzger <rm...@apache.org> wrote:
>> The dev setup for developing Flink with IntelliJ is quite simple: Just
>> checkout the source code and import it as a Maven project into IntelliJ.
>> To verify the correctness of your setup, try running the main() method of
>> the WordCount example. This should always work without troubles.
>> You can use the WordCount as an example for running Flink programs out of
>> your IDE.
>>
>>
>> The only time you really have to use the build in "flink-dist/target" is
>> for manual testing of a version (for example the web interfaces) or
>> deploying binary archives.
>>
>>
>> On Thu, Jul 31, 2014 at 9:09 AM, Henry Saputra <he...@gmail.com>
>> wrote:
>>
>>> Hi Ufuk,
>>>
>>> Sorry I missed this email reply from you.
>>>
>>> Yeah, I also used IntelliJ for development and trying to do remote
>>> debugging instead.
>>>
>>> Could you share your dev setup to run Flink using IntelliJ instead?
>>>
>>> - Henry
>>>
>>> On Mon, Jul 21, 2014 at 12:29 AM, Ufuk Celebi <u....@fu-berlin.de>
>>> wrote:
>>>> Hey Henry,
>>>>
>>>> On 21 Jul 2014, at 09:18, Henry Saputra <he...@gmail.com> wrote:
>>>>> What are your recommended setup to run and develop Flink locally?
>>>>
>>>> I use IntelliJ for day-to-day dev and testing and run everything inside
>>> the IDE.
>>>>
>>>> If you want to use real programs to test your code you can use
>>> JavaProgramTestBase for local startup of the system.
>>> o.a.flink.test.exampleJavaPrograms.WordCountITCase is a good example to see
>>> how it works.
>>>>
>>>> You can also write tests, which use LocalEnvironment. If you want to use
>>> an existing Flink installation, e.g. in flink-dist/target or remote, you
>>> can use the RemoteEnvironment to submit a JAR.
>>>>
>>>> Does any of this help?
>>>>
>>>>> I have been doing local Flink by going to flink-dist/target/ and run
>>>>> Flink from there.
>>>>> I was wondering if you guys have preferred way to do Flink development?
>>>>> Maybe we could add section about "Developing Flink" in the website?
>>>>
>>>> +1
>>>>
>>>> We had something along those lines a while back for Eclipse and I
>>> remember that it was helpful when I started with the project. Thanks for
>>> pointing this out. :-)
>>>
Re: Guide/ tips to do preferred development for Flink?
Posted by Henry Saputra <he...@gmail.com>.
Thanks Robert, it seems like it does work.
So if it is running via local environment context it will just execute
the local executor?
How much is the difference does the flow in local vs cluster?
- Henry
On Wed, Aug 6, 2014 at 6:41 AM, Robert Metzger <rm...@apache.org> wrote:
> The dev setup for developing Flink with IntelliJ is quite simple: Just
> checkout the source code and import it as a Maven project into IntelliJ.
> To verify the correctness of your setup, try running the main() method of
> the WordCount example. This should always work without troubles.
> You can use the WordCount as an example for running Flink programs out of
> your IDE.
>
>
> The only time you really have to use the build in "flink-dist/target" is
> for manual testing of a version (for example the web interfaces) or
> deploying binary archives.
>
>
> On Thu, Jul 31, 2014 at 9:09 AM, Henry Saputra <he...@gmail.com>
> wrote:
>
>> Hi Ufuk,
>>
>> Sorry I missed this email reply from you.
>>
>> Yeah, I also used IntelliJ for development and trying to do remote
>> debugging instead.
>>
>> Could you share your dev setup to run Flink using IntelliJ instead?
>>
>> - Henry
>>
>> On Mon, Jul 21, 2014 at 12:29 AM, Ufuk Celebi <u....@fu-berlin.de>
>> wrote:
>> > Hey Henry,
>> >
>> > On 21 Jul 2014, at 09:18, Henry Saputra <he...@gmail.com> wrote:
>> >> What are your recommended setup to run and develop Flink locally?
>> >
>> > I use IntelliJ for day-to-day dev and testing and run everything inside
>> the IDE.
>> >
>> > If you want to use real programs to test your code you can use
>> JavaProgramTestBase for local startup of the system.
>> o.a.flink.test.exampleJavaPrograms.WordCountITCase is a good example to see
>> how it works.
>> >
>> > You can also write tests, which use LocalEnvironment. If you want to use
>> an existing Flink installation, e.g. in flink-dist/target or remote, you
>> can use the RemoteEnvironment to submit a JAR.
>> >
>> > Does any of this help?
>> >
>> >> I have been doing local Flink by going to flink-dist/target/ and run
>> >> Flink from there.
>> >> I was wondering if you guys have preferred way to do Flink development?
>> >> Maybe we could add section about "Developing Flink" in the website?
>> >
>> > +1
>> >
>> > We had something along those lines a while back for Eclipse and I
>> remember that it was helpful when I started with the project. Thanks for
>> pointing this out. :-)
>>
Re: Guide/ tips to do preferred development for Flink?
Posted by Robert Metzger <rm...@apache.org>.
The dev setup for developing Flink with IntelliJ is quite simple: Just
checkout the source code and import it as a Maven project into IntelliJ.
To verify the correctness of your setup, try running the main() method of
the WordCount example. This should always work without troubles.
You can use the WordCount as an example for running Flink programs out of
your IDE.
The only time you really have to use the build in "flink-dist/target" is
for manual testing of a version (for example the web interfaces) or
deploying binary archives.
On Thu, Jul 31, 2014 at 9:09 AM, Henry Saputra <he...@gmail.com>
wrote:
> Hi Ufuk,
>
> Sorry I missed this email reply from you.
>
> Yeah, I also used IntelliJ for development and trying to do remote
> debugging instead.
>
> Could you share your dev setup to run Flink using IntelliJ instead?
>
> - Henry
>
> On Mon, Jul 21, 2014 at 12:29 AM, Ufuk Celebi <u....@fu-berlin.de>
> wrote:
> > Hey Henry,
> >
> > On 21 Jul 2014, at 09:18, Henry Saputra <he...@gmail.com> wrote:
> >> What are your recommended setup to run and develop Flink locally?
> >
> > I use IntelliJ for day-to-day dev and testing and run everything inside
> the IDE.
> >
> > If you want to use real programs to test your code you can use
> JavaProgramTestBase for local startup of the system.
> o.a.flink.test.exampleJavaPrograms.WordCountITCase is a good example to see
> how it works.
> >
> > You can also write tests, which use LocalEnvironment. If you want to use
> an existing Flink installation, e.g. in flink-dist/target or remote, you
> can use the RemoteEnvironment to submit a JAR.
> >
> > Does any of this help?
> >
> >> I have been doing local Flink by going to flink-dist/target/ and run
> >> Flink from there.
> >> I was wondering if you guys have preferred way to do Flink development?
> >> Maybe we could add section about "Developing Flink" in the website?
> >
> > +1
> >
> > We had something along those lines a while back for Eclipse and I
> remember that it was helpful when I started with the project. Thanks for
> pointing this out. :-)
>
Re: Guide/ tips to do preferred development for Flink?
Posted by Henry Saputra <he...@gmail.com>.
Hi Ufuk,
Sorry I missed this email reply from you.
Yeah, I also used IntelliJ for development and trying to do remote
debugging instead.
Could you share your dev setup to run Flink using IntelliJ instead?
- Henry
On Mon, Jul 21, 2014 at 12:29 AM, Ufuk Celebi <u....@fu-berlin.de> wrote:
> Hey Henry,
>
> On 21 Jul 2014, at 09:18, Henry Saputra <he...@gmail.com> wrote:
>> What are your recommended setup to run and develop Flink locally?
>
> I use IntelliJ for day-to-day dev and testing and run everything inside the IDE.
>
> If you want to use real programs to test your code you can use JavaProgramTestBase for local startup of the system. o.a.flink.test.exampleJavaPrograms.WordCountITCase is a good example to see how it works.
>
> You can also write tests, which use LocalEnvironment. If you want to use an existing Flink installation, e.g. in flink-dist/target or remote, you can use the RemoteEnvironment to submit a JAR.
>
> Does any of this help?
>
>> I have been doing local Flink by going to flink-dist/target/ and run
>> Flink from there.
>> I was wondering if you guys have preferred way to do Flink development?
>> Maybe we could add section about "Developing Flink" in the website?
>
> +1
>
> We had something along those lines a while back for Eclipse and I remember that it was helpful when I started with the project. Thanks for pointing this out. :-)
Re: Guide/ tips to do preferred development for Flink?
Posted by Ufuk Celebi <u....@fu-berlin.de>.
Hey Henry,
On 21 Jul 2014, at 09:18, Henry Saputra <he...@gmail.com> wrote:
> What are your recommended setup to run and develop Flink locally?
I use IntelliJ for day-to-day dev and testing and run everything inside the IDE.
If you want to use real programs to test your code you can use JavaProgramTestBase for local startup of the system. o.a.flink.test.exampleJavaPrograms.WordCountITCase is a good example to see how it works.
You can also write tests, which use LocalEnvironment. If you want to use an existing Flink installation, e.g. in flink-dist/target or remote, you can use the RemoteEnvironment to submit a JAR.
Does any of this help?
> I have been doing local Flink by going to flink-dist/target/ and run
> Flink from there.
> I was wondering if you guys have preferred way to do Flink development?
> Maybe we could add section about "Developing Flink" in the website?
+1
We had something along those lines a while back for Eclipse and I remember that it was helpful when I started with the project. Thanks for pointing this out. :-)