You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Ingo Bürk <in...@ververica.com> on 2021/08/18 09:03:08 UTC
Configuring flink-python in IntelliJ
Hello @dev,
like probably most of you, I am using IntelliJ to work on Flink. Lately I
needed to also work with flink-python, and thus was wondering about how to
properly set it up to work in IntelliJ. So far, I have done the following:
1. Install the Python plugin
2. Set up a custom Python SDK using a virtualenv
3. Configure the flink-python module to use this Python SDK (rather than a
Java SDK)
4. Install as many of the Python dependencies as possible when IntelliJ
prompted me to do so
This got me into a mostly working state, for example I can run tests from
the IDE, at least the ones I tried so far. However, there are two concerns:
a) Step (3) will be undone on every Maven import since Maven sets the SDK
for the module
b) Step (4) installed most, but a couple of Python dependencies could not
be installed, though so far that didn't cause noticeable problems
I'm wondering if there are additional / different steps to do, specifically
for (a), and maybe how the PyFlink developers are configuring this in their
IDE. The IDE setup guide didn't seem to contain information about that
(only about separately setting up this module in PyCharm). Ideally, we
could even update the IDE setup guide.
Best
Ingo
Re: Configuring flink-python in IntelliJ
Posted by Ingo Bürk <in...@ververica.com>.
Hi Dian,
thank you for the explanation. I guess this is, at least as far as we know,
as good as it gets for an IntelliJ setup for all of Flink. I'll put up a
pull request to update the IDE setup guide, but also point out that for
specific work on PyFlink, a separate setup with PyCharm is recommended.
Best
Ingo
On Wed, Aug 18, 2021 at 4:20 PM Dian Fu <di...@gmail.com> wrote:
> Hi Ingo,
>
> There are both Java/Python source codes in PyFlink and I use two IDEs at
> the same time: IntelliJ IDEA & PyCharm.
>
> Regarding ONLY using IntelliJ IDEA for both Python & Java development, I
> have tried it and it works just as you said. I have done the following:
> - Install Python Plugin
> - Mark the module flink-python as a Python module: right click
> "flink-python" -> "Open Module Settings", change the Module SDK to Python
> - Click file setup.py under flink-python, then it will promote to install a
> few Python libraries, just install them
> - Run the Python tests under sub-directories of flink-python/pyflink
>
> Regarding “a) Step (3) will be undone on every Maven import since Maven
> sets the SDK for the module”, this has not occurred in my environment for
> now. Are there any differences for you regarding the above steps?
>
> PS: For me, I will still use both IDEs for PyFlink development as it
> contains both Java & Python source code under flink-python directory and it
> doesn't support configuring both Python & Java SDK for it. However, if
> one works
> with flink-python occasionally, using only IntelliJ IDEA is a good approach
> and should be enough.
>
> Regards,
> Dian
>
> On Wed, Aug 18, 2021 at 7:33 PM Ingo Bürk <in...@ververica.com> wrote:
>
> > Hi Dian,
> >
> > thanks for responding! No, I haven't tried, but I'm aware of it. I don't
> > work exclusively on PyFlink, but rather just occasionally. So instead of
> > having a whole separate IDE and project for one module I'm trying to get
> > the whole Flink project to work as one in IntelliJ. Do PyFlink developers
> > generally work solely on PyFlink and nothing else / do you switch IDEs
> for
> > everything?
> >
> >
> > Best
> > Ingo
> >
> > On Wed, Aug 18, 2021 at 1:20 PM Dian Fu <di...@apache.org> wrote:
> >
> > > Hi Ingo,
> > >
> > > Thanks a lot for starting up this discussion. I have not tried IntelliJ
> > and
> > > I’m using PyCharm for PyFlink development. Have you tried this
> guideline?
> > > [1]
> > > It’s written in 1.9 and I think it should still work.
> > >
> > > Regards,
> > > Dian
> > >
> > > [1]
> > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/flinkdev/ide_setup/#pycharm
> > >
> > > On Wed, Aug 18, 2021 at 5:38 PM Till Rohrmann <tr...@apache.org>
> > > wrote:
> > >
> > > > Thanks for starting this discussion Ingo. I guess that Dian can
> > probably
> > > > answer this question. I am big +1 for updating our documentation for
> > how
> > > to
> > > > set up the IDE since this problem will probably be encountered a
> couple
> > > of
> > > > times.
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Wed, Aug 18, 2021 at 11:03 AM Ingo Bürk <in...@ververica.com>
> wrote:
> > > >
> > > > > Hello @dev,
> > > > >
> > > > > like probably most of you, I am using IntelliJ to work on Flink.
> > > Lately I
> > > > > needed to also work with flink-python, and thus was wondering about
> > how
> > > > to
> > > > > properly set it up to work in IntelliJ. So far, I have done the
> > > > following:
> > > > >
> > > > > 1. Install the Python plugin
> > > > > 2. Set up a custom Python SDK using a virtualenv
> > > > > 3. Configure the flink-python module to use this Python SDK (rather
> > > than
> > > > a
> > > > > Java SDK)
> > > > > 4. Install as many of the Python dependencies as possible when
> > IntelliJ
> > > > > prompted me to do so
> > > > >
> > > > > This got me into a mostly working state, for example I can run
> tests
> > > from
> > > > > the IDE, at least the ones I tried so far. However, there are two
> > > > concerns:
> > > > >
> > > > > a) Step (3) will be undone on every Maven import since Maven sets
> the
> > > SDK
> > > > > for the module
> > > > > b) Step (4) installed most, but a couple of Python dependencies
> could
> > > not
> > > > > be installed, though so far that didn't cause noticeable problems
> > > > >
> > > > > I'm wondering if there are additional / different steps to do,
> > > > specifically
> > > > > for (a), and maybe how the PyFlink developers are configuring this
> in
> > > > their
> > > > > IDE. The IDE setup guide didn't seem to contain information about
> > that
> > > > > (only about separately setting up this module in PyCharm). Ideally,
> > we
> > > > > could even update the IDE setup guide.
> > > > >
> > > > >
> > > > > Best
> > > > > Ingo
> > > > >
> > > >
> > >
> >
>
Re: Configuring flink-python in IntelliJ
Posted by Dian Fu <di...@gmail.com>.
Hi Ingo,
There are both Java/Python source codes in PyFlink and I use two IDEs at
the same time: IntelliJ IDEA & PyCharm.
Regarding ONLY using IntelliJ IDEA for both Python & Java development, I
have tried it and it works just as you said. I have done the following:
- Install Python Plugin
- Mark the module flink-python as a Python module: right click
"flink-python" -> "Open Module Settings", change the Module SDK to Python
- Click file setup.py under flink-python, then it will promote to install a
few Python libraries, just install them
- Run the Python tests under sub-directories of flink-python/pyflink
Regarding “a) Step (3) will be undone on every Maven import since Maven
sets the SDK for the module”, this has not occurred in my environment for
now. Are there any differences for you regarding the above steps?
PS: For me, I will still use both IDEs for PyFlink development as it
contains both Java & Python source code under flink-python directory and it
doesn't support configuring both Python & Java SDK for it. However, if
one works
with flink-python occasionally, using only IntelliJ IDEA is a good approach
and should be enough.
Regards,
Dian
On Wed, Aug 18, 2021 at 7:33 PM Ingo Bürk <in...@ververica.com> wrote:
> Hi Dian,
>
> thanks for responding! No, I haven't tried, but I'm aware of it. I don't
> work exclusively on PyFlink, but rather just occasionally. So instead of
> having a whole separate IDE and project for one module I'm trying to get
> the whole Flink project to work as one in IntelliJ. Do PyFlink developers
> generally work solely on PyFlink and nothing else / do you switch IDEs for
> everything?
>
>
> Best
> Ingo
>
> On Wed, Aug 18, 2021 at 1:20 PM Dian Fu <di...@apache.org> wrote:
>
> > Hi Ingo,
> >
> > Thanks a lot for starting up this discussion. I have not tried IntelliJ
> and
> > I’m using PyCharm for PyFlink development. Have you tried this guideline?
> > [1]
> > It’s written in 1.9 and I think it should still work.
> >
> > Regards,
> > Dian
> >
> > [1]
> >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/flinkdev/ide_setup/#pycharm
> >
> > On Wed, Aug 18, 2021 at 5:38 PM Till Rohrmann <tr...@apache.org>
> > wrote:
> >
> > > Thanks for starting this discussion Ingo. I guess that Dian can
> probably
> > > answer this question. I am big +1 for updating our documentation for
> how
> > to
> > > set up the IDE since this problem will probably be encountered a couple
> > of
> > > times.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Wed, Aug 18, 2021 at 11:03 AM Ingo Bürk <in...@ververica.com> wrote:
> > >
> > > > Hello @dev,
> > > >
> > > > like probably most of you, I am using IntelliJ to work on Flink.
> > Lately I
> > > > needed to also work with flink-python, and thus was wondering about
> how
> > > to
> > > > properly set it up to work in IntelliJ. So far, I have done the
> > > following:
> > > >
> > > > 1. Install the Python plugin
> > > > 2. Set up a custom Python SDK using a virtualenv
> > > > 3. Configure the flink-python module to use this Python SDK (rather
> > than
> > > a
> > > > Java SDK)
> > > > 4. Install as many of the Python dependencies as possible when
> IntelliJ
> > > > prompted me to do so
> > > >
> > > > This got me into a mostly working state, for example I can run tests
> > from
> > > > the IDE, at least the ones I tried so far. However, there are two
> > > concerns:
> > > >
> > > > a) Step (3) will be undone on every Maven import since Maven sets the
> > SDK
> > > > for the module
> > > > b) Step (4) installed most, but a couple of Python dependencies could
> > not
> > > > be installed, though so far that didn't cause noticeable problems
> > > >
> > > > I'm wondering if there are additional / different steps to do,
> > > specifically
> > > > for (a), and maybe how the PyFlink developers are configuring this in
> > > their
> > > > IDE. The IDE setup guide didn't seem to contain information about
> that
> > > > (only about separately setting up this module in PyCharm). Ideally,
> we
> > > > could even update the IDE setup guide.
> > > >
> > > >
> > > > Best
> > > > Ingo
> > > >
> > >
> >
>
Re: Configuring flink-python in IntelliJ
Posted by Ingo Bürk <in...@ververica.com>.
Hi Dian,
thanks for responding! No, I haven't tried, but I'm aware of it. I don't
work exclusively on PyFlink, but rather just occasionally. So instead of
having a whole separate IDE and project for one module I'm trying to get
the whole Flink project to work as one in IntelliJ. Do PyFlink developers
generally work solely on PyFlink and nothing else / do you switch IDEs for
everything?
Best
Ingo
On Wed, Aug 18, 2021 at 1:20 PM Dian Fu <di...@apache.org> wrote:
> Hi Ingo,
>
> Thanks a lot for starting up this discussion. I have not tried IntelliJ and
> I’m using PyCharm for PyFlink development. Have you tried this guideline?
> [1]
> It’s written in 1.9 and I think it should still work.
>
> Regards,
> Dian
>
> [1]
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/flinkdev/ide_setup/#pycharm
>
> On Wed, Aug 18, 2021 at 5:38 PM Till Rohrmann <tr...@apache.org>
> wrote:
>
> > Thanks for starting this discussion Ingo. I guess that Dian can probably
> > answer this question. I am big +1 for updating our documentation for how
> to
> > set up the IDE since this problem will probably be encountered a couple
> of
> > times.
> >
> > Cheers,
> > Till
> >
> > On Wed, Aug 18, 2021 at 11:03 AM Ingo Bürk <in...@ververica.com> wrote:
> >
> > > Hello @dev,
> > >
> > > like probably most of you, I am using IntelliJ to work on Flink.
> Lately I
> > > needed to also work with flink-python, and thus was wondering about how
> > to
> > > properly set it up to work in IntelliJ. So far, I have done the
> > following:
> > >
> > > 1. Install the Python plugin
> > > 2. Set up a custom Python SDK using a virtualenv
> > > 3. Configure the flink-python module to use this Python SDK (rather
> than
> > a
> > > Java SDK)
> > > 4. Install as many of the Python dependencies as possible when IntelliJ
> > > prompted me to do so
> > >
> > > This got me into a mostly working state, for example I can run tests
> from
> > > the IDE, at least the ones I tried so far. However, there are two
> > concerns:
> > >
> > > a) Step (3) will be undone on every Maven import since Maven sets the
> SDK
> > > for the module
> > > b) Step (4) installed most, but a couple of Python dependencies could
> not
> > > be installed, though so far that didn't cause noticeable problems
> > >
> > > I'm wondering if there are additional / different steps to do,
> > specifically
> > > for (a), and maybe how the PyFlink developers are configuring this in
> > their
> > > IDE. The IDE setup guide didn't seem to contain information about that
> > > (only about separately setting up this module in PyCharm). Ideally, we
> > > could even update the IDE setup guide.
> > >
> > >
> > > Best
> > > Ingo
> > >
> >
>
Re: Configuring flink-python in IntelliJ
Posted by Dian Fu <di...@apache.org>.
Hi Ingo,
Thanks a lot for starting up this discussion. I have not tried IntelliJ and
I’m using PyCharm for PyFlink development. Have you tried this guideline?
[1]
It’s written in 1.9 and I think it should still work.
Regards,
Dian
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/flinkdev/ide_setup/#pycharm
On Wed, Aug 18, 2021 at 5:38 PM Till Rohrmann <tr...@apache.org> wrote:
> Thanks for starting this discussion Ingo. I guess that Dian can probably
> answer this question. I am big +1 for updating our documentation for how to
> set up the IDE since this problem will probably be encountered a couple of
> times.
>
> Cheers,
> Till
>
> On Wed, Aug 18, 2021 at 11:03 AM Ingo Bürk <in...@ververica.com> wrote:
>
> > Hello @dev,
> >
> > like probably most of you, I am using IntelliJ to work on Flink. Lately I
> > needed to also work with flink-python, and thus was wondering about how
> to
> > properly set it up to work in IntelliJ. So far, I have done the
> following:
> >
> > 1. Install the Python plugin
> > 2. Set up a custom Python SDK using a virtualenv
> > 3. Configure the flink-python module to use this Python SDK (rather than
> a
> > Java SDK)
> > 4. Install as many of the Python dependencies as possible when IntelliJ
> > prompted me to do so
> >
> > This got me into a mostly working state, for example I can run tests from
> > the IDE, at least the ones I tried so far. However, there are two
> concerns:
> >
> > a) Step (3) will be undone on every Maven import since Maven sets the SDK
> > for the module
> > b) Step (4) installed most, but a couple of Python dependencies could not
> > be installed, though so far that didn't cause noticeable problems
> >
> > I'm wondering if there are additional / different steps to do,
> specifically
> > for (a), and maybe how the PyFlink developers are configuring this in
> their
> > IDE. The IDE setup guide didn't seem to contain information about that
> > (only about separately setting up this module in PyCharm). Ideally, we
> > could even update the IDE setup guide.
> >
> >
> > Best
> > Ingo
> >
>
Re: Configuring flink-python in IntelliJ
Posted by Till Rohrmann <tr...@apache.org>.
Thanks for starting this discussion Ingo. I guess that Dian can probably
answer this question. I am big +1 for updating our documentation for how to
set up the IDE since this problem will probably be encountered a couple of
times.
Cheers,
Till
On Wed, Aug 18, 2021 at 11:03 AM Ingo Bürk <in...@ververica.com> wrote:
> Hello @dev,
>
> like probably most of you, I am using IntelliJ to work on Flink. Lately I
> needed to also work with flink-python, and thus was wondering about how to
> properly set it up to work in IntelliJ. So far, I have done the following:
>
> 1. Install the Python plugin
> 2. Set up a custom Python SDK using a virtualenv
> 3. Configure the flink-python module to use this Python SDK (rather than a
> Java SDK)
> 4. Install as many of the Python dependencies as possible when IntelliJ
> prompted me to do so
>
> This got me into a mostly working state, for example I can run tests from
> the IDE, at least the ones I tried so far. However, there are two concerns:
>
> a) Step (3) will be undone on every Maven import since Maven sets the SDK
> for the module
> b) Step (4) installed most, but a couple of Python dependencies could not
> be installed, though so far that didn't cause noticeable problems
>
> I'm wondering if there are additional / different steps to do, specifically
> for (a), and maybe how the PyFlink developers are configuring this in their
> IDE. The IDE setup guide didn't seem to contain information about that
> (only about separately setting up this module in PyCharm). Ideally, we
> could even update the IDE setup guide.
>
>
> Best
> Ingo
>