You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Kathleen Sharp <ka...@signavio.com> on 2017/02/03 11:20:53 UTC
Restoring from an external checkpoint / savepoint with a local stream environment
Hi,
is it possible to restore from an external checkpoint / savepoint
while using a local stream environment?
I ask because I want to play around with some concepts from within my
IDE, without building a jar and deploying my job.
Thanks,
Kat
--
_______________________________________________________
Besuchen Sie uns / Meet us:
OPEX-Woche DACH: January 17-19, Wiesbaden, Germany
OPEX Week World Summit: January 23-27, Orlando, USA
CIO EDGE Experience: Febuary 22-23, Melbourne, Australia
Business Transformation and Operational Excellence World Summit: March
21-24, Orlando, USA
Folgen Sie uns / Follow us:
Twitter | Facebook | LinkedIn | Xing | Youtube
________________________________________________________
HRB 121584 B Amtsgericht Charlottenburg, Ust-ID: DE265675123
Geschäftsführer: Dr. Gero Decker, Guido Sachs
Re: Restoring from an external checkpoint / savepoint with a local
stream environment
Posted by Kathleen Sharp <ka...@signavio.com>.
Thanks a lot for the detailed answer. From reading the code i
suspected something would be possible, but this would have taken me a
while to figure out.
On Fri, Feb 3, 2017 at 2:08 PM, Ufuk Celebi <uc...@apache.org> wrote:
> Hey Kathleen,
>
> this is not supported "natively" in the APIs but the following is
> possible to work around it:
>
> int jobManagerPort = 6123;
> int numTaskSlots = 8;
> String savepointPath = "/Users/uce/Desktop/savepoint-xAsd";
>
> Configuration config = new Configuration();
> config.setInteger(ConfigConstants.JOB_MANAGER_IPC_PORT_KEY, jobManagerPort);
> config.setInteger(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS, numTaskSlots);
>
> LocalFlinkMiniCluster cluster = new LocalFlinkMiniCluster(config, false);
> cluster.start(true);
>
> // The "remote environment" submits to the local cluster
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.createRemoteEnvironment("localhost",
> jobManagerPort);
>
> // Create your program as usual
> // ...
>
> // Instead of env.execute do the following:
>
> JobGraph jobGraph = env.getStreamGraph().getJobGraph();
> jobGraph.setSavepointRestoreSettings(SavepointRestoreSettings.forPath(savepointPath));
>
> boolean printUpdates = true;
> cluster.submitJobAndWait(jobGraph, printUpdates);
>
> ----
>
> We could think about exposing the SavepointSettings to the
> StreamGraph. Then you don't need all the boilerplate for this.
>
>
> On Fri, Feb 3, 2017 at 12:20 PM, Kathleen Sharp
> <ka...@signavio.com> wrote:
>> Hi,
>>
>> is it possible to restore from an external checkpoint / savepoint
>> while using a local stream environment?
>>
>> I ask because I want to play around with some concepts from within my
>> IDE, without building a jar and deploying my job.
>>
>> Thanks,
>> Kat
>>
>> --
>> _______________________________________________________
>>
>> Besuchen Sie uns / Meet us:
>> OPEX-Woche DACH: January 17-19, Wiesbaden, Germany
>> OPEX Week World Summit: January 23-27, Orlando, USA
>> CIO EDGE Experience: Febuary 22-23, Melbourne, Australia
>> Business Transformation and Operational Excellence World Summit: March
>> 21-24, Orlando, USA
>>
>> Folgen Sie uns / Follow us:
>> Twitter | Facebook | LinkedIn | Xing | Youtube
>> ________________________________________________________
>>
>>
>> HRB 121584 B Amtsgericht Charlottenburg, Ust-ID: DE265675123
>> Geschäftsführer: Dr. Gero Decker, Guido Sachs
--
_______________________________________________________
Besuchen Sie uns / Meet us:
OPEX-Woche DACH: January 17-19, Wiesbaden, Germany
OPEX Week World Summit: January 23-27, Orlando, USA
CIO EDGE Experience: Febuary 22-23, Melbourne, Australia
Business Transformation and Operational Excellence World Summit: March
21-24, Orlando, USA
Folgen Sie uns / Follow us:
Twitter | Facebook | LinkedIn | Xing | Youtube
________________________________________________________
HRB 121584 B Amtsgericht Charlottenburg, Ust-ID: DE265675123
Geschäftsführer: Dr. Gero Decker, Guido Sachs
Re: Restoring from an external checkpoint / savepoint with a local
stream environment
Posted by Ufuk Celebi <uc...@apache.org>.
Hey Kathleen,
this is not supported "natively" in the APIs but the following is
possible to work around it:
int jobManagerPort = 6123;
int numTaskSlots = 8;
String savepointPath = "/Users/uce/Desktop/savepoint-xAsd";
Configuration config = new Configuration();
config.setInteger(ConfigConstants.JOB_MANAGER_IPC_PORT_KEY, jobManagerPort);
config.setInteger(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS, numTaskSlots);
LocalFlinkMiniCluster cluster = new LocalFlinkMiniCluster(config, false);
cluster.start(true);
// The "remote environment" submits to the local cluster
StreamExecutionEnvironment env =
StreamExecutionEnvironment.createRemoteEnvironment("localhost",
jobManagerPort);
// Create your program as usual
// ...
// Instead of env.execute do the following:
JobGraph jobGraph = env.getStreamGraph().getJobGraph();
jobGraph.setSavepointRestoreSettings(SavepointRestoreSettings.forPath(savepointPath));
boolean printUpdates = true;
cluster.submitJobAndWait(jobGraph, printUpdates);
----
We could think about exposing the SavepointSettings to the
StreamGraph. Then you don't need all the boilerplate for this.
On Fri, Feb 3, 2017 at 12:20 PM, Kathleen Sharp
<ka...@signavio.com> wrote:
> Hi,
>
> is it possible to restore from an external checkpoint / savepoint
> while using a local stream environment?
>
> I ask because I want to play around with some concepts from within my
> IDE, without building a jar and deploying my job.
>
> Thanks,
> Kat
>
> --
> _______________________________________________________
>
> Besuchen Sie uns / Meet us:
> OPEX-Woche DACH: January 17-19, Wiesbaden, Germany
> OPEX Week World Summit: January 23-27, Orlando, USA
> CIO EDGE Experience: Febuary 22-23, Melbourne, Australia
> Business Transformation and Operational Excellence World Summit: March
> 21-24, Orlando, USA
>
> Folgen Sie uns / Follow us:
> Twitter | Facebook | LinkedIn | Xing | Youtube
> ________________________________________________________
>
>
> HRB 121584 B Amtsgericht Charlottenburg, Ust-ID: DE265675123
> Geschäftsführer: Dr. Gero Decker, Guido Sachs