You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Andrew Ge Wu <an...@eniro.com> on 2016/02/19 18:08:14 UTC
state.backend.fs.checkpointdir setting
Hi All,
I have been experiencing an error stopping my HA standalone setup.
The cluster startup just fine, but when i deploy an application to it, I got this exception:
java.lang.Exception: Call to registerInputOutput() of invokable failed
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:529)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flink.configuration.IllegalConfigurationException: Cannot create the file system state backend: The configuration does not specify the checkpoint directory 'state.backend.fs.checkpointdir'
at org.apache.flink.runtime.state.filesystem.FsStateBackendFactory.createFromConfig(FsStateBackendFactory.java:41)
at org.apache.flink.streaming.runtime.tasks.StreamTask.createStateBackend(StreamTask.java:517)
at org.apache.flink.streaming.runtime.tasks.StreamTask.registerInputOutput(StreamTask.java:171)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:526)
Here’s my configuration:
…………….
#
# Note: You need to set the state backend to 'filesystem' and the checkpoint
# directory (see above) before configuring the storageDir.
#
# recovery.zookeeper.storageDir: hdfs:///recovery
recovery.zookeeper.path.root: /flink
state.backend: filesystem
state.backend.fs.checkpointdir: file:///apps/flink/checkpoints/
recovery.zookeeper.storageDir: file:///apps/flink/recovery/
And here’s my flink folder.
drwxr-xr-x 11 {user} {group} 4.0K Feb 19 17:31 .
drwxrwxr-x 6 {user} {group} 4.0K Feb 19 11:25 ..
-rw-r--r-- 1 {user} {group} 17K Nov 22 13:52 LICENSE
-rw-r--r-- 1 {user} {group} 779 Nov 22 13:52 NOTICE
-rw-r--r-- 1 {user} {group} 1.3K Nov 22 13:52 README.txt
drwxr-xr-x 2 {user} {group} 4.0K Nov 22 13:52 bin
drwxr-xr-x 2 {user} {group} 4.0K Feb 19 17:31 checkpoints
drwxr-xr-x 2 {user} {group} 4.0K Feb 19 17:37 conf
drwxr-xr-x 2 {user} {group} 4.0K Nov 22 13:52 examples
drwxr-xr-x 2 {user} {group} 4.0K Nov 22 13:52 lib
drwxr-xr-x 2 {user} {group} 4.0K Feb 19 17:48 log
drwxr-xr-x 3 {user} {group} 4.0K Feb 19 17:48 recovery
drwxr-xr-x 3 {user} {group} 4.0K Nov 22 13:52 resources
drwxr-xr-x 5 {user} {group} 4.0K Nov 22 13:52 tools
The recovery folder works just fine, with blobs in it.
Thanks!
Andrew
--
Confidentiality Notice: This e-mail transmission may contain confidential
or legally privileged information that is intended only for the individual
or entity named in the e-mail address. If you are not the intended
recipient, you are hereby notified that any disclosure, copying,
distribution, or reliance upon the contents of this e-mail is strictly
prohibited and may be unlawful. If you have received this e-mail in error,
please notify the sender immediately by return e-mail and delete all copies
of this message.
Re: state.backend.fs.checkpointdir setting
Posted by Andrew Ge Wu <an...@eniro.com>.
Hi Robert
I just checked my settings in Task Managers (they were configured separately), they are misconfigured.
My job now runs correctly, after reconfigured them.
Thanks!
Andrew
> On 22 Feb 2016, at 09:41, Robert Metzger <rm...@apache.org> wrote:
>
> Hi,
>
> how is your cluster setup? Do you have multiple machines, or only one?
> Did you copy the configuration to all machines?
>
>
>
> On Fri, Feb 19, 2016 at 6:08 PM, Andrew Ge Wu <andrew.ge-wu@eniro.com <ma...@eniro.com>> wrote:
> Hi All,
>
> I have been experiencing an error stopping my HA standalone setup.
>
> The cluster startup just fine, but when i deploy an application to it, I got this exception:
>
>
> java.lang.Exception: Call to registerInputOutput() of invokable failed
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:529)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.flink.configuration.IllegalConfigurationException: Cannot create the file system state backend: The configuration does not specify the checkpoint directory 'state.backend.fs.checkpointdir'
> at org.apache.flink.runtime.state.filesystem.FsStateBackendFactory.createFromConfig(FsStateBackendFactory.java:41)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.createStateBackend(StreamTask.java:517)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.registerInputOutput(StreamTask.java:171)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:526)
>
> Here’s my configuration:
>
> …………….
> #
> # Note: You need to set the state backend to 'filesystem' and the checkpoint
> # directory (see above) before configuring the storageDir.
> #
> # recovery.zookeeper.storageDir: hdfs:///recovery
> recovery.zookeeper.path.root: /flink
> state.backend: filesystem
>
> state.backend.fs.checkpointdir: file:///apps/flink/checkpoints/
> recovery.zookeeper.storageDir: file:///apps/flink/recovery/
>
>
>
> And here’s my flink folder.
>
> drwxr-xr-x 11 {user} {group} 4.0K Feb 19 17:31 .
> drwxrwxr-x 6 {user} {group} 4.0K Feb 19 11:25 ..
> -rw-r--r-- 1 {user} {group} 17K Nov 22 13:52 LICENSE
> -rw-r--r-- 1 {user} {group} 779 Nov 22 13:52 NOTICE
> -rw-r--r-- 1 {user} {group} 1.3K Nov 22 13:52 README.txt
> drwxr-xr-x 2 {user} {group} 4.0K Nov 22 13:52 bin
> drwxr-xr-x 2 {user} {group} 4.0K Feb 19 17:31 checkpoints
> drwxr-xr-x 2 {user} {group} 4.0K Feb 19 17:37 conf
> drwxr-xr-x 2 {user} {group} 4.0K Nov 22 13:52 examples
> drwxr-xr-x 2 {user} {group} 4.0K Nov 22 13:52 lib
> drwxr-xr-x 2 {user} {group} 4.0K Feb 19 17:48 log
> drwxr-xr-x 3 {user} {group} 4.0K Feb 19 17:48 recovery
> drwxr-xr-x 3 {user} {group} 4.0K Nov 22 13:52 resources
> drwxr-xr-x 5 {user} {group} 4.0K Nov 22 13:52 tools
>
> The recovery folder works just fine, with blobs in it.
>
>
>
> Thanks!
>
>
>
> Andrew
> --
> Confidentiality Notice: This e-mail transmission may contain confidential
> or legally privileged information that is intended only for the individual
> or entity named in the e-mail address. If you are not the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution, or reliance upon the contents of this e-mail is strictly
> prohibited and may be unlawful. If you have received this e-mail in error,
> please notify the sender immediately by return e-mail and delete all copies
> of this message.
>
--
Confidentiality Notice: This e-mail transmission may contain confidential
or legally privileged information that is intended only for the individual
or entity named in the e-mail address. If you are not the intended
recipient, you are hereby notified that any disclosure, copying,
distribution, or reliance upon the contents of this e-mail is strictly
prohibited and may be unlawful. If you have received this e-mail in error,
please notify the sender immediately by return e-mail and delete all copies
of this message.
Re: state.backend.fs.checkpointdir setting
Posted by Robert Metzger <rm...@apache.org>.
Hi,
how is your cluster setup? Do you have multiple machines, or only one?
Did you copy the configuration to all machines?
On Fri, Feb 19, 2016 at 6:08 PM, Andrew Ge Wu <an...@eniro.com>
wrote:
> Hi All,
>
> I have been experiencing an error stopping my HA standalone setup.
>
> The cluster startup just fine, but when i deploy an application to it, I
> got this exception:
>
>
> java.lang.Exception: Call to registerInputOutput() of invokable failed
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:529)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.flink.configuration.IllegalConfigurationException:
> Cannot create the file system state backend: The configuration does not
> specify the checkpoint directory 'state.backend.fs.checkpointdir'
> at
> org.apache.flink.runtime.state.filesystem.FsStateBackendFactory.createFromConfig(FsStateBackendFactory.java:41)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.createStateBackend(StreamTask.java:517)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.registerInputOutput(StreamTask.java:171)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:526)
>
> Here’s my configuration:
>
> …………….
> #
> # Note: You need to set the state backend to 'filesystem' and the
> checkpoint
> # directory (see above) before configuring the storageDir.
> #
> # recovery.zookeeper.storageDir: hdfs:///recovery
> recovery.zookeeper.path.root: /flink
> state.backend: filesystem
>
> state.backend.fs.checkpointdir: file:///apps/flink/checkpoints/
> recovery.zookeeper.storageDir: file:///apps/flink/recovery/
>
>
>
> And here’s my flink folder.
>
> drwxr-xr-x 11 {user} {group} 4.0K Feb 19 17:31 .
> drwxrwxr-x 6 {user} {group} 4.0K Feb 19 11:25 ..
> -rw-r--r-- 1 {user} {group} 17K Nov 22 13:52 LICENSE
> -rw-r--r-- 1 {user} {group} 779 Nov 22 13:52 NOTICE
> -rw-r--r-- 1 {user} {group} 1.3K Nov 22 13:52 README.txt
> drwxr-xr-x 2 {user} {group} 4.0K Nov 22 13:52 bin
> drwxr-xr-x 2 {user} {group} 4.0K Feb 19 17:31 checkpoints
> drwxr-xr-x 2 {user} {group} 4.0K Feb 19 17:37 conf
> drwxr-xr-x 2 {user} {group} 4.0K Nov 22 13:52 examples
> drwxr-xr-x 2 {user} {group} 4.0K Nov 22 13:52 lib
> drwxr-xr-x 2 {user} {group} 4.0K Feb 19 17:48 log
> drwxr-xr-x 3 {user} {group} 4.0K Feb 19 17:48 recovery
> drwxr-xr-x 3 {user} {group} 4.0K Nov 22 13:52 resources
> drwxr-xr-x 5 {user} {group} 4.0K Nov 22 13:52 tools
>
> The recovery folder works just fine, with blobs in it.
>
>
>
> Thanks!
>
>
>
> Andrew
> --
> Confidentiality Notice: This e-mail transmission may contain confidential
> or legally privileged information that is intended only for the individual
> or entity named in the e-mail address. If you are not the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution, or reliance upon the contents of this e-mail is strictly
> prohibited and may be unlawful. If you have received this e-mail in error,
> please notify the sender immediately by return e-mail and delete all copies
> of this message.
>