You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Julio Biason <ju...@azion.com> on 2018/02/12 19:11:15 UTC
Ceph configuration for checkpoints?
Hello people,
I'm looking for the configuration options required to use Ceph as a backend
for checkpoints.
So far, I can see that `state.backend.fs.checkpointdir` and
`state.checkpoints.dir` should be something like
`ceph:///my-checkpoint-bucket`. But where do I define server and acess and
secret keys? I couldn't find anything remotely related to that in the
docs...
--
*Julio Biason*, Sofware Engineer
*AZION* | Deliver. Accelerate. Protect.
Office: +55 51 3083 8101 <callto:+555130838101> | Mobile: +55 51
<callto:+5551996209291>*99907 0554*
Re: Ceph configuration for checkpoints?
Posted by Martin Pompéry <m...@pompery.net>.
Hello Julio,
how we deploy Ceph:
1) make sure Ceph is mounted through all the nodes that make up your Flink
cluster; ensure path(s) are the same for each node
2) simply reference the Ceph mountpoint(s) via file://<your path>
Martin
On Tue, Feb 13, 2018 at 10:18 AM Piotr Nowojski <pi...@data-artisans.com>
wrote:
> Hi,
>
> Have you tried to refer to ceph documentation?
> http://docs.ceph.com/docs/jewel/cephfs/hadoop/ It claims to be:
>
> > a drop-in replacement for the Hadoop File System (HDFS)
>
> So I would first try to configure ceph according to their documentation
> and try to use Flink’s built in Hadoop connector.
>
> Piotrek
>
> On 12 Feb 2018, at 20:11, Julio Biason <ju...@azion.com> wrote:
>
> Hello people,
>
> I'm looking for the configuration options required to use Ceph as a
> backend for checkpoints.
>
> So far, I can see that `state.backend.fs.checkpointdir` and
> `state.checkpoints.dir` should be something like `
> ceph:///my-checkpoint-bucket`. But where do I define server and acess and
> secret keys? I couldn't find anything remotely related to that in the
> docs...
>
> --
> *Julio Biason*, Sofware Engineer
> *AZION* | Deliver. Accelerate. Protect.
> Office: +55 51 3083 8101 <callto:+555130838101> | Mobile: +55 51
> <callto:+5551996209291>*99907 0554*
>
>
>
Re: Ceph configuration for checkpoints?
Posted by Piotr Nowojski <pi...@data-artisans.com>.
Hi,
Have you tried to refer to ceph documentation? http://docs.ceph.com/docs/jewel/cephfs/hadoop/ <http://docs.ceph.com/docs/jewel/cephfs/hadoop/> It claims to be:
> a drop-in replacement for the Hadoop File System (HDFS)
So I would first try to configure ceph according to their documentation and try to use Flink’s built in Hadoop connector.
Piotrek
> On 12 Feb 2018, at 20:11, Julio Biason <ju...@azion.com> wrote:
>
> Hello people,
>
> I'm looking for the configuration options required to use Ceph as a backend for checkpoints.
>
> So far, I can see that `state.backend.fs.checkpointdir` and `state.checkpoints.dir` should be something like `ceph:///my-checkpoint-bucket`. But where do I define server and acess and secret keys? I couldn't find anything remotely related to that in the docs...
>
> --
> Julio Biason, Sofware Engineer
> AZION | Deliver. Accelerate. Protect.
> Office: +55 51 3083 8101 <callto:+555130838101> | Mobile: +55 51 <callto:+5551996209291>99907 0554