You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Jordan Kuan <jo...@gmail.com> on 2017/11/07 02:34:43 UTC
Re: MODERATE for dev@flink.apache.org
Dear Flink Dev Team
I have encountered a problem and can't find any solution in Google.
And I have created a thread in stackoverflow.com but no response.
https://stackoverflow.com/questions/47123371/flink-windows-ha
I would really appreciate it if you could give some suggestions to me.
Thanks,
Jordan
On Tue, Nov 7, 2017 at 1:53 AM, Robert Metzger <rm...@apache.org> wrote:
> Hi,
> I would suggest to send your question to the user@flink.apache.org list
> (make sure to subscribe first)
>
> ---------- Forwarded message ----------
> From: <de...@flink.apache.org>
> Date: Mon, Nov 6, 2017 at 5:23 PM
> Subject: MODERATE for dev@flink.apache.org
> To:
> Cc: dev-allow-tc.1509985381.dcccgaimcaiiefbkiapi-jordan.kuan=
> gmail.com@flink.apache.org
>
>
>
> To approve:
> dev-accept-1509985381.14933.cigohlkhhjaclimegmod@flink.apache.org
> To reject:
> dev-reject-1509985381.14933.cigohlkhhjaclimegmod@flink.apache.org
> To give a reason to reject:
> %%% Start comment
> %%% End comment
>
>
>
> ---------- Forwarded message ----------
> From: Jordan Kuan <jo...@gmail.com>
> To: dev@flink.apache.org
> Cc:
> Bcc:
> Date: Tue, 7 Nov 2017 00:22:53 +0800
> Subject: Flink Windows HA Issue.
> Dear Flink Dev Team
>
> I have encountered a problem and can't find any solution in Google.
> And I have created a thread in stackoverflow.com but no response.
> https://stackoverflow.com/questions/47123371/flink-windows-ha
>
> I would really appreciate it if you could give some suggestions to me.
>
> Thanks,
>
> Jordan
>
>
>
>
--
Best Regards,
Jordan Kuan
Re: MODERATE for dev@flink.apache.org
Posted by Chesnay Schepler <ch...@apache.org>.
Have you tried removing the "cygdrive" portion from the path?
Something along the lines of
“state.backend.fs.checkpointdir: file:///Y:/flink-checkpoint-dir
state.checkpoints.dir: Y:/flink-checkpoints
state.backend: filesystem
high-availability.storageDir: file:///Y:/flink-recovery
"
On 07.11.2017 12:36, Jordan Kuan wrote:
> Dear Ufuk,
>
> Sorry, I still can’t get that work, I have double checked all the
> nodes can access the mapped drive Y.
> Here is my state backend configuration, and it writes files to local C
> drive.
>
> “state.backend.fs.checkpointdir: file:///cygdrive/Y/flink-checkpoint-dir
> state.checkpoints.dir: /cygdrive/Y/flink-checkpoints
> state.backend: filesystem
> high-availability.storageDir: file:///cygdrive/Y/flink-recovery
> "
>
> I have attached the flink-conf.yaml and screenshots, Is here any wrong
> setting?
>
> Thanks,
> Jordan
>
>> On 7 Nov 2017, at 6:49 PM, Ufuk Celebi <uce@apache.org
>> <ma...@apache.org>> wrote:
>>
>> Hey Jordan,
>>
>> yeah, that should just work. Check out the state backend configuration
>> here:
>> https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/state/state_backends.html
>> <https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/state/state_backends.html>
>>
>> – Ufuk
>>
>> On Tue, Nov 7, 2017 at 11:44 AM, Jordan Kuan <jordan.kuan@gmail.com
>> <ma...@gmail.com>> wrote:
>>> Dear Ufuk,
>>>
>>> Thank you for your reply.
>>>
>>> All the cluster Flink servers are able to access network drive, and
>>> it mapped as drive Y in all nodes.
>>> Do I need to provide more information?
>>>
>>> Thanks,
>>> Jordan
>>>
>>>
>>>> On 7 Nov 2017, at 6:36 PM, Ufuk Celebi <uce@apache.org
>>>> <ma...@apache.org>> wrote:
>>>>
>>>> As answered by David on SO, the files need to be accessible by all
>>>> nodes. In your setup this seems not to be the case, therefore it won't
>>>> work. You need a distributed file system (e.g. NFS or HDFS) or object
>>>> store (e.g. S3) that is accessible from all nodes.
>>>>
>>>> – Ufuk
>>>>
>>>>
>>>> On Tue, Nov 7, 2017 at 3:34 AM, Jordan Kuan <jordan.kuan@gmail.com
>>>> <ma...@gmail.com>> wrote:
>>>>> Dear Flink Dev Team
>>>>>
>>>>> I have encountered a problem and can't find any solution in Google.
>>>>> And I have created a thread in stackoverflow.com
>>>>> <http://stackoverflow.com> but no response.
>>>>> https://stackoverflow.com/questions/47123371/flink-windows-ha
>>>>> <https://stackoverflow.com/questions/47123371/flink-windows-ha>
>>>>>
>>>>> I would really appreciate it if you could give some suggestions to me.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jordan
>>>>>
>>>>> On Tue, Nov 7, 2017 at 1:53 AM, Robert Metzger
>>>>> <rmetzger@apache.org <ma...@apache.org>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>> I would suggest to send your question to the
>>>>>> user@flink.apache.org <ma...@flink.apache.org> list
>>>>>> (make sure to subscribe first)
>>>>>>
>>>>>> ---------- Forwarded message ----------
>>>>>> From:
>>>>>> <dev-reject-1509985381.14933.cigohlkhhjaclimegmod@flink.apache.org
>>>>>> <ma...@flink.apache.org>>
>>>>>> Date: Mon, Nov 6, 2017 at 5:23 PM
>>>>>> Subject: MODERATE for dev@flink.apache.org
>>>>>> <ma...@flink.apache.org>
>>>>>> To:
>>>>>> Cc:
>>>>>> dev-allow-tc.1509985381.dcccgaimcaiiefbkiapi-jordan.kuan=gmail.com@flink.apache.org
>>>>>> <ma...@flink.apache.org>
>>>>>>
>>>>>>
>>>>>>
>>>>>> To approve:
>>>>>> dev-accept-1509985381.14933.cigohlkhhjaclimegmod@flink.apache.org
>>>>>> <ma...@flink.apache.org>
>>>>>> To reject:
>>>>>> dev-reject-1509985381.14933.cigohlkhhjaclimegmod@flink.apache.org
>>>>>> <ma...@flink.apache.org>
>>>>>> To give a reason to reject:
>>>>>> %%% Start comment
>>>>>> %%% End comment
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------- Forwarded message ----------
>>>>>> From: Jordan Kuan <jordan.kuan@gmail.com
>>>>>> <ma...@gmail.com>>
>>>>>> To: dev@flink.apache.org <ma...@flink.apache.org>
>>>>>> Cc:
>>>>>> Bcc:
>>>>>> Date: Tue, 7 Nov 2017 00:22:53 +0800
>>>>>> Subject: Flink Windows HA Issue.
>>>>>> Dear Flink Dev Team
>>>>>>
>>>>>> I have encountered a problem and can't find any solution in Google.
>>>>>> And I have created a thread in stackoverflow.com
>>>>>> <http://stackoverflow.com> but no response.
>>>>>> https://stackoverflow.com/questions/47123371/flink-windows-ha
>>>>>> <https://stackoverflow.com/questions/47123371/flink-windows-ha>
>>>>>>
>>>>>> I would really appreciate it if you could give some suggestions
>>>>>> to me.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Jordan
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Jordan Kuan
>>>
>
Re: MODERATE for dev@flink.apache.org
Posted by Jordan Kuan <jo...@gmail.com>.
Dear Ufuk,
Sorry, I still can’t get that work, I have double checked all the nodes can
access the mapped drive Y.
Here is my state backend configuration, and it writes files to local C
drive.
“state.backend.fs.checkpointdir: file:///cygdrive/Y/flink-checkpoint-dir
state.checkpoints.dir: /cygdrive/Y/flink-checkpoints
state.backend: filesystem
high-availability.storageDir: file:///cygdrive/Y/flink-recovery
"
I have attached the flink-conf.yaml and screenshots, Is here any wrong
setting?
Thanks,
Jordan
On 7 Nov 2017, at 6:49 PM, Ufuk Celebi <uc...@apache.org> wrote:
Hey Jordan,
yeah, that should just work. Check out the state backend configuration
here: https://ci.apache.org/projects/flink/flink-docs-release-1.4/
ops/state/state_backends.html
– Ufuk
On Tue, Nov 7, 2017 at 11:44 AM, Jordan Kuan <jo...@gmail.com> wrote:
Dear Ufuk,
Thank you for your reply.
All the cluster Flink servers are able to access network drive, and it
mapped as drive Y in all nodes.
Do I need to provide more information?
Thanks,
Jordan
On 7 Nov 2017, at 6:36 PM, Ufuk Celebi <uc...@apache.org> wrote:
As answered by David on SO, the files need to be accessible by all
nodes. In your setup this seems not to be the case, therefore it won't
work. You need a distributed file system (e.g. NFS or HDFS) or object
store (e.g. S3) that is accessible from all nodes.
– Ufuk
On Tue, Nov 7, 2017 at 3:34 AM, Jordan Kuan <jo...@gmail.com> wrote:
Dear Flink Dev Team
I have encountered a problem and can't find any solution in Google.
And I have created a thread in stackoverflow.com but no response.
https://stackoverflow.com/questions/47123371/flink-windows-ha
I would really appreciate it if you could give some suggestions to me.
Thanks,
Jordan
On Tue, Nov 7, 2017 at 1:53 AM, Robert Metzger <rm...@apache.org> wrote:
Hi,
I would suggest to send your question to the user@flink.apache.org list
(make sure to subscribe first)
---------- Forwarded message ----------
From: <de...@flink.apache.org>
Date: Mon, Nov 6, 2017 at 5:23 PM
Subject: MODERATE for dev@flink.apache.org
To:
Cc:
dev-allow-tc.1509985381.dcccgaimcaiiefbkiapi-jordan.kuan=gma
il.com@flink.apache.org
To approve:
dev-accept-1509985381.14933.cigohlkhhjaclimegmod@flink.apache.org
To reject:
dev-reject-1509985381.14933.cigohlkhhjaclimegmod@flink.apache.org
To give a reason to reject:
%%% Start comment
%%% End comment
---------- Forwarded message ----------
From: Jordan Kuan <jo...@gmail.com>
To: dev@flink.apache.org
Cc:
Bcc:
Date: Tue, 7 Nov 2017 00:22:53 +0800
Subject: Flink Windows HA Issue.
Dear Flink Dev Team
I have encountered a problem and can't find any solution in Google.
And I have created a thread in stackoverflow.com but no response.
https://stackoverflow.com/questions/47123371/flink-windows-ha
I would really appreciate it if you could give some suggestions to me.
Thanks,
Jordan
--
Best Regards,
Jordan Kuan
Re: MODERATE for dev@flink.apache.org
Posted by Ufuk Celebi <uc...@apache.org>.
Hey Jordan,
yeah, that should just work. Check out the state backend configuration
here: https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/state/state_backends.html
– Ufuk
On Tue, Nov 7, 2017 at 11:44 AM, Jordan Kuan <jo...@gmail.com> wrote:
> Dear Ufuk,
>
> Thank you for your reply.
>
> All the cluster Flink servers are able to access network drive, and it mapped as drive Y in all nodes.
> Do I need to provide more information?
>
> Thanks,
> Jordan
>
>
>> On 7 Nov 2017, at 6:36 PM, Ufuk Celebi <uc...@apache.org> wrote:
>>
>> As answered by David on SO, the files need to be accessible by all
>> nodes. In your setup this seems not to be the case, therefore it won't
>> work. You need a distributed file system (e.g. NFS or HDFS) or object
>> store (e.g. S3) that is accessible from all nodes.
>>
>> – Ufuk
>>
>>
>> On Tue, Nov 7, 2017 at 3:34 AM, Jordan Kuan <jo...@gmail.com> wrote:
>>> Dear Flink Dev Team
>>>
>>> I have encountered a problem and can't find any solution in Google.
>>> And I have created a thread in stackoverflow.com but no response.
>>> https://stackoverflow.com/questions/47123371/flink-windows-ha
>>>
>>> I would really appreciate it if you could give some suggestions to me.
>>>
>>> Thanks,
>>>
>>> Jordan
>>>
>>> On Tue, Nov 7, 2017 at 1:53 AM, Robert Metzger <rm...@apache.org> wrote:
>>>>
>>>> Hi,
>>>> I would suggest to send your question to the user@flink.apache.org list
>>>> (make sure to subscribe first)
>>>>
>>>> ---------- Forwarded message ----------
>>>> From: <de...@flink.apache.org>
>>>> Date: Mon, Nov 6, 2017 at 5:23 PM
>>>> Subject: MODERATE for dev@flink.apache.org
>>>> To:
>>>> Cc:
>>>> dev-allow-tc.1509985381.dcccgaimcaiiefbkiapi-jordan.kuan=gmail.com@flink.apache.org
>>>>
>>>>
>>>>
>>>> To approve:
>>>> dev-accept-1509985381.14933.cigohlkhhjaclimegmod@flink.apache.org
>>>> To reject:
>>>> dev-reject-1509985381.14933.cigohlkhhjaclimegmod@flink.apache.org
>>>> To give a reason to reject:
>>>> %%% Start comment
>>>> %%% End comment
>>>>
>>>>
>>>>
>>>> ---------- Forwarded message ----------
>>>> From: Jordan Kuan <jo...@gmail.com>
>>>> To: dev@flink.apache.org
>>>> Cc:
>>>> Bcc:
>>>> Date: Tue, 7 Nov 2017 00:22:53 +0800
>>>> Subject: Flink Windows HA Issue.
>>>> Dear Flink Dev Team
>>>>
>>>> I have encountered a problem and can't find any solution in Google.
>>>> And I have created a thread in stackoverflow.com but no response.
>>>> https://stackoverflow.com/questions/47123371/flink-windows-ha
>>>>
>>>> I would really appreciate it if you could give some suggestions to me.
>>>>
>>>> Thanks,
>>>>
>>>> Jordan
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Jordan Kuan
>
Re: MODERATE for dev@flink.apache.org
Posted by Jordan Kuan <jo...@gmail.com>.
Dear Ufuk,
Thank you for your reply.
All the cluster Flink servers are able to access network drive, and it mapped as drive Y in all nodes.
Do I need to provide more information?
Thanks,
Jordan
> On 7 Nov 2017, at 6:36 PM, Ufuk Celebi <uc...@apache.org> wrote:
>
> As answered by David on SO, the files need to be accessible by all
> nodes. In your setup this seems not to be the case, therefore it won't
> work. You need a distributed file system (e.g. NFS or HDFS) or object
> store (e.g. S3) that is accessible from all nodes.
>
> – Ufuk
>
>
> On Tue, Nov 7, 2017 at 3:34 AM, Jordan Kuan <jo...@gmail.com> wrote:
>> Dear Flink Dev Team
>>
>> I have encountered a problem and can't find any solution in Google.
>> And I have created a thread in stackoverflow.com but no response.
>> https://stackoverflow.com/questions/47123371/flink-windows-ha
>>
>> I would really appreciate it if you could give some suggestions to me.
>>
>> Thanks,
>>
>> Jordan
>>
>> On Tue, Nov 7, 2017 at 1:53 AM, Robert Metzger <rm...@apache.org> wrote:
>>>
>>> Hi,
>>> I would suggest to send your question to the user@flink.apache.org list
>>> (make sure to subscribe first)
>>>
>>> ---------- Forwarded message ----------
>>> From: <de...@flink.apache.org>
>>> Date: Mon, Nov 6, 2017 at 5:23 PM
>>> Subject: MODERATE for dev@flink.apache.org
>>> To:
>>> Cc:
>>> dev-allow-tc.1509985381.dcccgaimcaiiefbkiapi-jordan.kuan=gmail.com@flink.apache.org
>>>
>>>
>>>
>>> To approve:
>>> dev-accept-1509985381.14933.cigohlkhhjaclimegmod@flink.apache.org
>>> To reject:
>>> dev-reject-1509985381.14933.cigohlkhhjaclimegmod@flink.apache.org
>>> To give a reason to reject:
>>> %%% Start comment
>>> %%% End comment
>>>
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: Jordan Kuan <jo...@gmail.com>
>>> To: dev@flink.apache.org
>>> Cc:
>>> Bcc:
>>> Date: Tue, 7 Nov 2017 00:22:53 +0800
>>> Subject: Flink Windows HA Issue.
>>> Dear Flink Dev Team
>>>
>>> I have encountered a problem and can't find any solution in Google.
>>> And I have created a thread in stackoverflow.com but no response.
>>> https://stackoverflow.com/questions/47123371/flink-windows-ha
>>>
>>> I would really appreciate it if you could give some suggestions to me.
>>>
>>> Thanks,
>>>
>>> Jordan
>>>
>>>
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Jordan Kuan
Re: MODERATE for dev@flink.apache.org
Posted by Ufuk Celebi <uc...@apache.org>.
As answered by David on SO, the files need to be accessible by all
nodes. In your setup this seems not to be the case, therefore it won't
work. You need a distributed file system (e.g. NFS or HDFS) or object
store (e.g. S3) that is accessible from all nodes.
– Ufuk
On Tue, Nov 7, 2017 at 3:34 AM, Jordan Kuan <jo...@gmail.com> wrote:
> Dear Flink Dev Team
>
> I have encountered a problem and can't find any solution in Google.
> And I have created a thread in stackoverflow.com but no response.
> https://stackoverflow.com/questions/47123371/flink-windows-ha
>
> I would really appreciate it if you could give some suggestions to me.
>
> Thanks,
>
> Jordan
>
> On Tue, Nov 7, 2017 at 1:53 AM, Robert Metzger <rm...@apache.org> wrote:
>>
>> Hi,
>> I would suggest to send your question to the user@flink.apache.org list
>> (make sure to subscribe first)
>>
>> ---------- Forwarded message ----------
>> From: <de...@flink.apache.org>
>> Date: Mon, Nov 6, 2017 at 5:23 PM
>> Subject: MODERATE for dev@flink.apache.org
>> To:
>> Cc:
>> dev-allow-tc.1509985381.dcccgaimcaiiefbkiapi-jordan.kuan=gmail.com@flink.apache.org
>>
>>
>>
>> To approve:
>> dev-accept-1509985381.14933.cigohlkhhjaclimegmod@flink.apache.org
>> To reject:
>> dev-reject-1509985381.14933.cigohlkhhjaclimegmod@flink.apache.org
>> To give a reason to reject:
>> %%% Start comment
>> %%% End comment
>>
>>
>>
>> ---------- Forwarded message ----------
>> From: Jordan Kuan <jo...@gmail.com>
>> To: dev@flink.apache.org
>> Cc:
>> Bcc:
>> Date: Tue, 7 Nov 2017 00:22:53 +0800
>> Subject: Flink Windows HA Issue.
>> Dear Flink Dev Team
>>
>> I have encountered a problem and can't find any solution in Google.
>> And I have created a thread in stackoverflow.com but no response.
>> https://stackoverflow.com/questions/47123371/flink-windows-ha
>>
>> I would really appreciate it if you could give some suggestions to me.
>>
>> Thanks,
>>
>> Jordan
>>
>>
>>
>
>
>
> --
> Best Regards,
> Jordan Kuan