Help with .getExecutionEnvironment() method

[Gyula Fóra <gy...@apache.org>]

Hey Guys,

The .getExecutionEnvironment() method for the StreamingexecutionEnvironment
does not work properly because we always return LocalEnvironment for
running on the minicluster.

I was trying to figure out how to fix this but I got lost in the code
trying to find how you determine whether the program was executed
standalone or the command line client.

Could you help me out with some pointers here?

Regards,
Gyula

Re: Help with .getExecutionEnvironment() method

[Stephan Ewen <se...@apache.org>]

Nice!

On Fri, Nov 7, 2014 at 11:09 PM, Gyula Fóra <gy...@gmail.com> wrote:

> Okay, I got it working :)
> Turns out I only needed the client from the contextenvironment to get it
> working.
>
> Thanks for the help :)
>
> On Fri, Nov 7, 2014 at 4:26 PM, Aljoscha Krettek <al...@apache.org>
> wrote:
>
> > Ah ok, maybe you can expose methods for direct execution in the
> > regular ExecutionEnvironment and then use them from you wrapper.
> >
> > On Fri, Nov 7, 2014 at 4:18 PM, Gyula Fóra <gy...@gmail.com> wrote:
> > > Hey,
> > >
> > > Okay, I think this is something that I might be able to use :)
> > >
> > > The problem is that the StreamingExecutionEnvironment has nothing to do
> > > with the ExecutionEnvironment (only the name is similar) because we are
> > > completely skipping the optimizer stuff at the execution. And it seemed
> > > that the context is figured out somewhere around there.
> > >
> > > So we cannot do too much with calling the method from the
> > > ExecutionEnvironment :S But I can try figuring out something with that.
> > >
> > > Cheers,
> > > Gyula
> > >
> > > On Fri, Nov 7, 2014 at 4:00 PM, Aljoscha Krettek <al...@apache.org>
> > > wrote:
> > >
> > >> Hi,
> > >> ExecutionEnvironment has a static field contextEnvironment. When you
> > >> call getExecutionEnvironment it checks whether this is null. If it is
> > >> null it will return a LocalEnvironment, otherwise it will return the
> > >> context environment.
> > >>
> > >> In Client.run() the Client creates a ContextEnvironment and calls
> > >> setAsContext() on it. This will put the env into the
> > >> contextEnvironment field of ExecutionEnvironment.
> > >>
> > >> I think you don't have to deal with all this though. Can you just wrap
> > >> a normal ExecutionEnvironment inside your streaming-specific
> > >> environment? In that case, you could just use
> > >> ExecutionEnvironemnt.getExecutionEnvironment and wrap the result of
> > >> that. This is how it works in the Scala API. Have a look at
> > >> ExecutionEnvironment.scala, it's really just a wrapper.
> > >>
> > >> Hope that helps. :D
> > >>
> > >> Cheers,
> > >> Aljoscha
> > >>
> > >> On Fri, Nov 7, 2014 at 3:24 PM, Gyula Fóra <gy...@apache.org> wrote:
> > >> > Hey Guys,
> > >> >
> > >> > The .getExecutionEnvironment() method for the
> > >> StreamingexecutionEnvironment
> > >> > does not work properly because we always return LocalEnvironment for
> > >> > running on the minicluster.
> > >> >
> > >> > I was trying to figure out how to fix this but I got lost in the
> code
> > >> > trying to find how you determine whether the program was executed
> > >> > standalone or the command line client.
> > >> >
> > >> > Could you help me out with some pointers here?
> > >> >
> > >> > Regards,
> > >> > Gyula
> > >>
> >
>

Re: Help with .getExecutionEnvironment() method

[Gyula Fóra <gy...@gmail.com>]

Okay, I got it working :)
Turns out I only needed the client from the contextenvironment to get it
working.

Thanks for the help :)

On Fri, Nov 7, 2014 at 4:26 PM, Aljoscha Krettek <al...@apache.org>
wrote:

> Ah ok, maybe you can expose methods for direct execution in the
> regular ExecutionEnvironment and then use them from you wrapper.
>
> On Fri, Nov 7, 2014 at 4:18 PM, Gyula Fóra <gy...@gmail.com> wrote:
> > Hey,
> >
> > Okay, I think this is something that I might be able to use :)
> >
> > The problem is that the StreamingExecutionEnvironment has nothing to do
> > with the ExecutionEnvironment (only the name is similar) because we are
> > completely skipping the optimizer stuff at the execution. And it seemed
> > that the context is figured out somewhere around there.
> >
> > So we cannot do too much with calling the method from the
> > ExecutionEnvironment :S But I can try figuring out something with that.
> >
> > Cheers,
> > Gyula
> >
> > On Fri, Nov 7, 2014 at 4:00 PM, Aljoscha Krettek <al...@apache.org>
> > wrote:
> >
> >> Hi,
> >> ExecutionEnvironment has a static field contextEnvironment. When you
> >> call getExecutionEnvironment it checks whether this is null. If it is
> >> null it will return a LocalEnvironment, otherwise it will return the
> >> context environment.
> >>
> >> In Client.run() the Client creates a ContextEnvironment and calls
> >> setAsContext() on it. This will put the env into the
> >> contextEnvironment field of ExecutionEnvironment.
> >>
> >> I think you don't have to deal with all this though. Can you just wrap
> >> a normal ExecutionEnvironment inside your streaming-specific
> >> environment? In that case, you could just use
> >> ExecutionEnvironemnt.getExecutionEnvironment and wrap the result of
> >> that. This is how it works in the Scala API. Have a look at
> >> ExecutionEnvironment.scala, it's really just a wrapper.
> >>
> >> Hope that helps. :D
> >>
> >> Cheers,
> >> Aljoscha
> >>
> >> On Fri, Nov 7, 2014 at 3:24 PM, Gyula Fóra <gy...@apache.org> wrote:
> >> > Hey Guys,
> >> >
> >> > The .getExecutionEnvironment() method for the
> >> StreamingexecutionEnvironment
> >> > does not work properly because we always return LocalEnvironment for
> >> > running on the minicluster.
> >> >
> >> > I was trying to figure out how to fix this but I got lost in the code
> >> > trying to find how you determine whether the program was executed
> >> > standalone or the command line client.
> >> >
> >> > Could you help me out with some pointers here?
> >> >
> >> > Regards,
> >> > Gyula
> >>
>

Re: Help with .getExecutionEnvironment() method

[Aljoscha Krettek <al...@apache.org>]

Ah ok, maybe you can expose methods for direct execution in the
regular ExecutionEnvironment and then use them from you wrapper.

On Fri, Nov 7, 2014 at 4:18 PM, Gyula Fóra <gy...@gmail.com> wrote:
> Hey,
>
> Okay, I think this is something that I might be able to use :)
>
> The problem is that the StreamingExecutionEnvironment has nothing to do
> with the ExecutionEnvironment (only the name is similar) because we are
> completely skipping the optimizer stuff at the execution. And it seemed
> that the context is figured out somewhere around there.
>
> So we cannot do too much with calling the method from the
> ExecutionEnvironment :S But I can try figuring out something with that.
>
> Cheers,
> Gyula
>
> On Fri, Nov 7, 2014 at 4:00 PM, Aljoscha Krettek <al...@apache.org>
> wrote:
>
>> Hi,
>> ExecutionEnvironment has a static field contextEnvironment. When you
>> call getExecutionEnvironment it checks whether this is null. If it is
>> null it will return a LocalEnvironment, otherwise it will return the
>> context environment.
>>
>> In Client.run() the Client creates a ContextEnvironment and calls
>> setAsContext() on it. This will put the env into the
>> contextEnvironment field of ExecutionEnvironment.
>>
>> I think you don't have to deal with all this though. Can you just wrap
>> a normal ExecutionEnvironment inside your streaming-specific
>> environment? In that case, you could just use
>> ExecutionEnvironemnt.getExecutionEnvironment and wrap the result of
>> that. This is how it works in the Scala API. Have a look at
>> ExecutionEnvironment.scala, it's really just a wrapper.
>>
>> Hope that helps. :D
>>
>> Cheers,
>> Aljoscha
>>
>> On Fri, Nov 7, 2014 at 3:24 PM, Gyula Fóra <gy...@apache.org> wrote:
>> > Hey Guys,
>> >
>> > The .getExecutionEnvironment() method for the
>> StreamingexecutionEnvironment
>> > does not work properly because we always return LocalEnvironment for
>> > running on the minicluster.
>> >
>> > I was trying to figure out how to fix this but I got lost in the code
>> > trying to find how you determine whether the program was executed
>> > standalone or the command line client.
>> >
>> > Could you help me out with some pointers here?
>> >
>> > Regards,
>> > Gyula
>>

Re: Help with .getExecutionEnvironment() method

[Gyula Fóra <gy...@gmail.com>]

Hey,

Okay, I think this is something that I might be able to use :)

The problem is that the StreamingExecutionEnvironment has nothing to do
with the ExecutionEnvironment (only the name is similar) because we are
completely skipping the optimizer stuff at the execution. And it seemed
that the context is figured out somewhere around there.

So we cannot do too much with calling the method from the
ExecutionEnvironment :S But I can try figuring out something with that.

Cheers,
Gyula

On Fri, Nov 7, 2014 at 4:00 PM, Aljoscha Krettek <al...@apache.org>
wrote:

> Hi,
> ExecutionEnvironment has a static field contextEnvironment. When you
> call getExecutionEnvironment it checks whether this is null. If it is
> null it will return a LocalEnvironment, otherwise it will return the
> context environment.
>
> In Client.run() the Client creates a ContextEnvironment and calls
> setAsContext() on it. This will put the env into the
> contextEnvironment field of ExecutionEnvironment.
>
> I think you don't have to deal with all this though. Can you just wrap
> a normal ExecutionEnvironment inside your streaming-specific
> environment? In that case, you could just use
> ExecutionEnvironemnt.getExecutionEnvironment and wrap the result of
> that. This is how it works in the Scala API. Have a look at
> ExecutionEnvironment.scala, it's really just a wrapper.
>
> Hope that helps. :D
>
> Cheers,
> Aljoscha
>
> On Fri, Nov 7, 2014 at 3:24 PM, Gyula Fóra <gy...@apache.org> wrote:
> > Hey Guys,
> >
> > The .getExecutionEnvironment() method for the
> StreamingexecutionEnvironment
> > does not work properly because we always return LocalEnvironment for
> > running on the minicluster.
> >
> > I was trying to figure out how to fix this but I got lost in the code
> > trying to find how you determine whether the program was executed
> > standalone or the command line client.
> >
> > Could you help me out with some pointers here?
> >
> > Regards,
> > Gyula
>

Re: Help with .getExecutionEnvironment() method

[Aljoscha Krettek <al...@apache.org>]

Hi,
ExecutionEnvironment has a static field contextEnvironment. When you
call getExecutionEnvironment it checks whether this is null. If it is
null it will return a LocalEnvironment, otherwise it will return the
context environment.

In Client.run() the Client creates a ContextEnvironment and calls
setAsContext() on it. This will put the env into the
contextEnvironment field of ExecutionEnvironment.

I think you don't have to deal with all this though. Can you just wrap
a normal ExecutionEnvironment inside your streaming-specific
environment? In that case, you could just use
ExecutionEnvironemnt.getExecutionEnvironment and wrap the result of
that. This is how it works in the Scala API. Have a look at
ExecutionEnvironment.scala, it's really just a wrapper.

Hope that helps. :D

Cheers,
Aljoscha

On Fri, Nov 7, 2014 at 3:24 PM, Gyula Fóra <gy...@apache.org> wrote:
> Hey Guys,
>
> The .getExecutionEnvironment() method for the StreamingexecutionEnvironment
> does not work properly because we always return LocalEnvironment for
> running on the minicluster.
>
> I was trying to figure out how to fix this but I got lost in the code
> trying to find how you determine whether the program was executed
> standalone or the command line client.
>
> Could you help me out with some pointers here?
>
> Regards,
> Gyula