You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Benjamin Burkhardt <pr...@benjaminburkhardt.de> on 2019/02/13 11:15:13 UTC

Using RemoteExecutionEnvironment from IDE

Hi all,
I’m currently debugging and enjoy it executing my task directly from the IDE.
The IDE starts a LocalExecutionEnvironment and immediately runs the task.
Now my errors seem to only appear when executing in the RemoteExecutionEnvironment. (it’s around the RemoteInputChannels et. al.)
To run it and test my modifications I always need to recompile it with maven which takes about 2.5 minutes.
mvn install -Dcheckstyle.skip -Drat.skip=true -DskipTests -T 1
I know you can send the task to a running env like this:
final ExecutionEnvironment env = ExecutionEnvironment.createRemoteEnvironment("localhost", 8081, „target/task.jar“);
But thats not what I’m looking for….
Is there a way to start a RemoteExecutionEnvironment from the IDE from scratch?
Thanks.

Benjamin

Re: Using RemoteExecutionEnvironment from IDE

Posted by Benjamin Burkhardt <pr...@benjaminburkhardt.de>.

Hi Till,

thanks for the great explanation! This was exactly what I was looking for.

Have a great day!

Regards,

Benjamin
Am 15. Feb. 2019, 10:55 +0100 schrieb Till Rohrmann <tr...@apache.org>:
> Hi Benjamin,
>
> what do you mean with "start a RemoteExecutionEnvironment from the IDE from scratch"? Are you asking whether you can restart the cluster against which the RemoteExecutionEnvironment submits the job? This is not possible.
>
> If you want to debug `RemoteInputChannels` locally, you could use the MiniClusterWithClientResource to start a MiniCluster with more than 1 TaskExecutor. That way it would instantiate the RemoteInputChannel in order to communicate between the different TaskExecutors. The resource will automatically set a special TestStreamEnvironment such that jobs will be submitted against the MiniCluster.
>
> public class FoobarITCase {
> @ClassRule
> public static final MiniClusterWithClientResource MINI_CLUSTER = new MiniClusterWithClientResource(
> new MiniClusterResourceConfiguration.Builder()
> .setNumberTaskManagers(2)
> .setNumberSlotsPerTaskManager(2)
> .build());
>
> @Test
> public void testFoobar() {
> MyApplication.main(new String[0]);
> }
> }
>
> The class MiniClusterWithClientResource lives in the flink-test-utils package.
>
> Cheers,
> Till
>
> > On Fri, Feb 15, 2019 at 10:29 AM Benjamin Burkhardt <pr...@benjaminburkhardt.de> wrote:
> > > Hi Andrey,
> > >
> > > I made some bigger changes in the logic of the RemoteInputChannels, means that this class should be recompiled for every run.
> > >
> > > Attaching to the IDE is pretty helpful, I already tried that before.
> > > But I think that there is just no solution for this and I have rerun the environment manually.
> > >
> > > Thanks.
> > >
> > > Benjamin
> > > Am 13. Feb. 2019, 17:57 +0100 schrieb Andrey Zagrebin <an...@ververica.com>:
> > > > Hi Benjamin,
> > > >
> > > > what kind of problem with RemoteInputChannels are you debugging?
> > > >
> > > > If the problem is in a Task Manager, you can try enabling IDE debug in flink-conf.yaml [1] before starting the task manager and then attach IDE to the JVM process of the Task Manager.
> > > >
> > > > Best,
> > > > Andrey
> > > >
> > > > [1] https://cwiki.apache.org/confluence/display/FLINK/Remote+Debugging+of+Flink+Clusters
> > > >
> > > > > On Wed, Feb 13, 2019 at 12:15 PM Benjamin Burkhardt <pr...@benjaminburkhardt.de> wrote:
> > > > > > Hi all,
> > > > > > I’m currently debugging and enjoy it executing my task directly from the IDE.
> > > > > > The IDE starts a LocalExecutionEnvironment and immediately runs the task.
> > > > > > Now my errors seem to only appear when executing in the RemoteExecutionEnvironment. (it’s around the RemoteInputChannels et. al.)
> > > > > > To run it and test my modifications I always need to recompile it with maven which takes about 2.5 minutes.
> > > > > > mvn install -Dcheckstyle.skip -Drat.skip=true -DskipTests -T 1
> > > > > > I know you can send the task to a running env like this:
> > > > > > final ExecutionEnvironment env = ExecutionEnvironment.createRemoteEnvironment("localhost", 8081, „target/task.jar“);
> > > > > > But thats not what I’m looking for….
> > > > > > Is there a way to start a RemoteExecutionEnvironment from the IDE from scratch?
> > > > > > Thanks.
> > > > > >
> > > > > > Benjamin

Re: Using RemoteExecutionEnvironment from IDE

Posted by Till Rohrmann <tr...@apache.org>.

Hi Benjamin,

what do you mean with "start a RemoteExecutionEnvironment from the IDE from
scratch"? Are you asking whether you can restart the cluster against which
the RemoteExecutionEnvironment submits the job? This is not possible.

If you want to debug `RemoteInputChannels` locally, you could use the
MiniClusterWithClientResource to start a MiniCluster with more than 1
TaskExecutor. That way it would instantiate the RemoteInputChannel in order
to communicate between the different TaskExecutors. The resource will
automatically set a special TestStreamEnvironment such that jobs will be
submitted against the MiniCluster.

public class FoobarITCase {
@ClassRule
public static final MiniClusterWithClientResource MINI_CLUSTER = new
MiniClusterWithClientResource(
new MiniClusterResourceConfiguration.Builder()
.setNumberTaskManagers(2)
.setNumberSlotsPerTaskManager(2)
.build());

@Test
public void testFoobar() {
MyApplication.main(new String[0]);
}
}

The class MiniClusterWithClientResource lives in the flink-test-utils
package.

Cheers,
Till

On Fri, Feb 15, 2019 at 10:29 AM Benjamin Burkhardt <
privat@benjaminburkhardt.de> wrote:

> Hi Andrey,
>
> I made some bigger changes in the logic of the RemoteInputChannels, means
> that this class should be recompiled for every run.
>
> Attaching to the IDE is pretty helpful, I already tried that before.
> But I think that there is just no solution for this and I have rerun the
> environment manually.
>
> Thanks.
>
> Benjamin
> Am 13. Feb. 2019, 17:57 +0100 schrieb Andrey Zagrebin <
> andrey@ververica.com>:
>
> Hi Benjamin,
>
> what kind of problem with RemoteInputChannels are you debugging?
>
> If the problem is in a Task Manager, you can try enabling IDE debug
> in flink-conf.yaml [1] before starting the task manager and then attach IDE
> to the JVM process of the Task Manager.
>
> Best,
> Andrey
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/Remote+Debugging+of+Flink+Clusters
>
> On Wed, Feb 13, 2019 at 12:15 PM Benjamin Burkhardt <
> privat@benjaminburkhardt.de> wrote:
>
>> Hi all,
>>
>> I’m currently debugging and enjoy it executing my task directly from the
>> IDE.
>>
>> The IDE starts a LocalExecutionEnvironment and immediately runs the task.
>>
>> Now my errors seem to only appear when executing in the
>> RemoteExecutionEnvironment. (it’s around the RemoteInputChannels et. al.)
>>
>> To run it and test my modifications I always need to recompile it with
>> maven which takes about 2.5 minutes.
>>
>> *mvn install -Dcheckstyle.skip -Drat.skip=true -DskipTests -T 1*
>>
>> I know you can send the task to a running env like this:
>>
>> *final ExecutionEnvironment env =
>> ExecutionEnvironment.createRemoteEnvironment("localhost", 8081,
>> „target/task.jar“);*
>>
>> But thats not what I’m looking for….
>>
>> Is there a way to start a RemoteExecutionEnvironment *from the IDE from
>> scratch*?
>>
>> Thanks.
>>
>>
>> Benjamin
>>
>

Re: Using RemoteExecutionEnvironment from IDE

Posted by Benjamin Burkhardt <pr...@benjaminburkhardt.de>.

Hi Andrey,

I made some bigger changes in the logic of the RemoteInputChannels, means that this class should be recompiled for every run.

Attaching to the IDE is pretty helpful, I already tried that before.
But I think that there is just no solution for this and I have rerun the environment manually.

Thanks.

Benjamin
Am 13. Feb. 2019, 17:57 +0100 schrieb Andrey Zagrebin <an...@ververica.com>:
> Hi Benjamin,
>
> what kind of problem with RemoteInputChannels are you debugging?
>
> If the problem is in a Task Manager, you can try enabling IDE debug in flink-conf.yaml [1] before starting the task manager and then attach IDE to the JVM process of the Task Manager.
>
> Best,
> Andrey
>
> [1] https://cwiki.apache.org/confluence/display/FLINK/Remote+Debugging+of+Flink+Clusters
>
> > On Wed, Feb 13, 2019 at 12:15 PM Benjamin Burkhardt <pr...@benjaminburkhardt.de> wrote:
> > > Hi all,
> > > I’m currently debugging and enjoy it executing my task directly from the IDE.
> > > The IDE starts a LocalExecutionEnvironment and immediately runs the task.
> > > Now my errors seem to only appear when executing in the RemoteExecutionEnvironment. (it’s around the RemoteInputChannels et. al.)
> > > To run it and test my modifications I always need to recompile it with maven which takes about 2.5 minutes.
> > > mvn install -Dcheckstyle.skip -Drat.skip=true -DskipTests -T 1
> > > I know you can send the task to a running env like this:
> > > final ExecutionEnvironment env = ExecutionEnvironment.createRemoteEnvironment("localhost", 8081, „target/task.jar“);
> > > But thats not what I’m looking for….
> > > Is there a way to start a RemoteExecutionEnvironment from the IDE from scratch?
> > > Thanks.
> > >
> > > Benjamin

Re: Using RemoteExecutionEnvironment from IDE

Posted by Andrey Zagrebin <an...@ververica.com>.

Hi Benjamin,

what kind of problem with RemoteInputChannels are you debugging?

If the problem is in a Task Manager, you can try enabling IDE debug
in flink-conf.yaml [1] before starting the task manager and then attach IDE
to the JVM process of the Task Manager.

Best,
Andrey

[1]
https://cwiki.apache.org/confluence/display/FLINK/Remote+Debugging+of+Flink+Clusters

On Wed, Feb 13, 2019 at 12:15 PM Benjamin Burkhardt <
privat@benjaminburkhardt.de> wrote:

> Hi all,
>
> I’m currently debugging and enjoy it executing my task directly from the
> IDE.
>
> The IDE starts a LocalExecutionEnvironment and immediately runs the task.
>
> Now my errors seem to only appear when executing in the
> RemoteExecutionEnvironment. (it’s around the RemoteInputChannels et. al.)
>
> To run it and test my modifications I always need to recompile it with
> maven which takes about 2.5 minutes.
>
> *mvn install -Dcheckstyle.skip -Drat.skip=true -DskipTests -T 1*
>
> I know you can send the task to a running env like this:
>
> *final ExecutionEnvironment env =
> ExecutionEnvironment.createRemoteEnvironment("localhost", 8081,
> „target/task.jar“);*
>
> But thats not what I’m looking for….
>
> Is there a way to start a RemoteExecutionEnvironment *from the IDE from
> scratch*?
>
> Thanks.
>
>
> Benjamin
>