You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Tony Chen <to...@robinhood.com> on 2023/10/04 19:25:25 UTC
Rolling back a bad deployment of FlinkDeployment on kubernetes
Hi Flink Community,
I am currently running Apache flink-kubernetes-operator on our kubernetes
clusters, and I have Flink applications that are deployed using the
FlinkDeployment Custom Resources (CR). I am trying to automate the process
of rollbacks and I am running into some issues.
I was testing out a bad deployment where the jobmanager never becomes
healthy. I simulated this bad deployment by creating a Flink image with a
bug in it. I see in the operator logs that the jobmanager is unhealthy:
[m [33m2023-10-02 22:14:34,874 [m
[36mo.a.f.k.o.r.d.AbstractFlinkResourceReconciler [m [32m[INFO
][flink-testing-service/flink-testing-service] UPGRADE change(s) detected
(FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJo]
differs from
FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJob]),
starting reconciliation.
...
[m [33m2023-10-02 22:15:09,001 [m [36mo.a.f.k.o.l.AuditUtils [m [32m[INFO
][flink-testing-service/flink-testing-service] >>> Status | Info |
UPGRADING | The resource is being upgraded
...
[m [33m2023-10-02 22:17:23,911 [m [36mo.a.f.k.o.l.AuditUtils [m [32m[INFO
][flink-testing-service/flink-testing-service] >>> Status | Error |
DEPLOYED |
{"type":"org.apache.flink.kubernetes.operator.exception.DeploymentFailedException","message":"back-off
20s restarting failed container=flink-main-container
pod=flink-testing-service-749dd97c75-4w9ps_flink-testing-service(6db1adb3-4ca4-4924-a8c3-57a417818d85)","additionalMetadata":{"reason":"CrashLoopBackOff"},"throwableList":[]}
...
[m [33m2023-10-02 22:17:33,576 [m [36mo.a.f.k.o.o.d.ApplicationObserver [m
[32m[INFO ][flink-testing-service/flink-testing-service] Observing
JobManager deployment. Previous status: ERROR
What I do next is I change the spec of the FlinkDeployment so that it uses
a Flink image that is healthy. The operator shows that the spec has changed:
[m [33m2023-10-02 22:45:37,445 [m [36mo.a.f.k.o.l.AuditUtils [m [32m[INFO
][flink-testing-service/flink-testing-service] >>> Event | Info |
SPECCHANGED | UPGRADE change(s) detected
(FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJob,job.initialSavepointPath=s3a://robinhood-dev-core-flink-states/flink-testing-service/savepoints/savepoint-329a14-2f8264206b1d]
differs from
FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJo,job.initialSavepointPath=s3a://robinhood-dev-core-flink-states/flink-testing-service/savepoints/savepoint-dc1077-134923759e30]),
starting reconciliation.
However, the Flink operator cannot reconcile this spec change, and the
jobmanager is now permanently failing because it's still running the bad
Flink image:
[m [33m2023-10-02 22:45:37,461 [m [36mo.a.f.k.o.l.AuditUtils [m [32m[INFO
][flink-testing-service/flink-testing-service] >>> Event | Warning |
UPGRADEFAILED | JobManager deployment is missing and HA data is not
available to make stateful upgrades. It is possible that the job has
finished or terminally failed, or the configmaps have been deleted. Manual
restore required.
I can simply delete this FlinkDeployment and redeploy with the healthy
Flink image, but I would like to avoid manual restores if possible. Is it
possible to recover by just changing the FlinkDeployment spec?
Thanks,
Tony
--
<http://www.robinhood.com/>
Tony Chen
Software Engineer
Menlo Park, CA
Don't copy, share, or use this email without permission. If you received it
by accident, please let us know and then delete it right away.
Re: Rolling back a bad deployment of FlinkDeployment on kubernetes
Posted by Tony Chen <to...@robinhood.com>.
So, I was able to get the rollback to work after I changed my upgradeMode
to *last-state*. Previously, my upgradeMode was *savepoint*, and when I
deployed a bad commit, the jobmanager-leader configmap would get deleted.
Once I changed the upgradeMode to *last-state*, the configmap was retained
when I deployed a bad commit, and the operator was able to rollback the
spec after the timeout.
On Thu, Oct 5, 2023 at 3:45 PM Gyula Fóra <gy...@gmail.com> wrote:
> Hi Tony!
>
> There are still a few corner cases when the operator cannot upgrade /
> rollback deployments due to the loss of HA metadata (and with that
> checkpoint information).
>
> Most of these issues are not related to the operator logic directly but to
> how Flink handles certain failures and are related to:
>
> https://issues.apache.org/jira/browse/FLINK-30444 and
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-360%3A+Merging+the+ExecutionGraphInfoStore+and+the+JobResultStore+into+a+single+component+CompletedJobStore
>
> Rollbacks are designed to allow automatic fallback to the last stable
> spec, but the mechanism doesn't work in these corner cases (in the same way
> spec upgrades also dont)
>
> I hope this helps to understand the problem.
> The solution in these cases is to manually recover the job from the last
> checkpoint/savepoint.
>
> Cheers,
> Gyula
>
>
> On Thu, Oct 5, 2023 at 7:56 PM Tony Chen <to...@robinhood.com> wrote:
>
>> I tried this out with operator version 1.4 and it didn't work for me. I
>> noticed that when I was deploying a bad version, the Kubernetes HA metadata
>> and configmaps were deleted:
>>
>> [m [33m2023-10-05 14:52:17,493 [m [36mo.a.f.k.o.l.AuditUtils [m [32m[INFO
>> ][flink-testing-service/flink-testing-service] >>> Event | Info |
>> SPECCHANGED | UPGRADE change(s) detected
>> (FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJo,job.initialSavepointPath=s3a://robinhood-prod-flink/flink-testing-service/savepoints/savepoint-b832ef-05b185cb5800]
>> differs from
>> FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJob,job.initialSavepointPath=<null>]),
>> starting reconciliation.
>> ...
>> [m [33m2023-10-05 14:52:51,054 [m [36mo.a.f.k.o.s.AbstractFlinkService [m
>> [32m[INFO ][flink-testing-service/flink-testing-service] Cluster shutdown
>> completed.
>> [m [33m2023-10-05 14:52:51,054 [m [36mo.a.f.k.o.s.AbstractFlinkService [m
>> [32m[INFO ][flink-testing-service/flink-testing-service] Deleting
>> Kubernetes HA metadata
>> [m [33m2023-10-05 14:52:51,196 [m [36mo.a.f.k.o.l.AuditUtils [m [32m[INFO
>> ][flink-testing-service/flink-testing-service] >>> Status | Info |
>> UPGRADING | The resource is being upgraded
>>
>>
>>
>> Eventually, the rollbak fails because the HA metadata is missing:
>>
>> [m [33m2023-10-05 14:58:16,119 [m
>> [36mo.a.f.k.o.r.d.AbstractFlinkResourceReconciler [m [33m[WARN
>> ][flink-testing-service/flink-testing-service] Rollback is not possible due
>> to missing HA metadata
>>
>>
>>
>> Besides setting kubernetes.operator.deployment.rollback.enabled: true,
>> is there anything else that I need to configure?
>>
>> On Thu, Oct 5, 2023 at 10:35 AM Tony Chen <to...@robinhood.com>
>> wrote:
>>
>>> I just saw this experimental feature in the documentation:
>>> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/job-management/#application-upgrade-rollbacks-experimental
>>>
>>> I'm guessing this is the only way to automate rollbacks for now.
>>>
>>> On Wed, Oct 4, 2023 at 3:25 PM Tony Chen <to...@robinhood.com>
>>> wrote:
>>>
>>>> Hi Flink Community,
>>>>
>>>> I am currently running Apache flink-kubernetes-operator on our
>>>> kubernetes clusters, and I have Flink applications that are deployed using
>>>> the FlinkDeployment Custom Resources (CR). I am trying to automate the
>>>> process of rollbacks and I am running into some issues.
>>>>
>>>> I was testing out a bad deployment where the jobmanager never becomes
>>>> healthy. I simulated this bad deployment by creating a Flink image with a
>>>> bug in it. I see in the operator logs that the jobmanager is unhealthy:
>>>>
>>>> [m [33m2023-10-02 22:14:34,874 [m
>>>> [36mo.a.f.k.o.r.d.AbstractFlinkResourceReconciler [m [32m[INFO
>>>> ][flink-testing-service/flink-testing-service] UPGRADE change(s) detected
>>>> (FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJo]
>>>> differs from
>>>> FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJob]),
>>>> starting reconciliation.
>>>> ...
>>>> [m [33m2023-10-02 22:15:09,001 [m [36mo.a.f.k.o.l.AuditUtils [m
>>>> [32m[INFO ][flink-testing-service/flink-testing-service] >>> Status | Info
>>>> | UPGRADING | The resource is being upgraded
>>>> ...
>>>> [m [33m2023-10-02 22:17:23,911 [m [36mo.a.f.k.o.l.AuditUtils [m
>>>> [32m[INFO ][flink-testing-service/flink-testing-service] >>> Status | Error
>>>> | DEPLOYED |
>>>> {"type":"org.apache.flink.kubernetes.operator.exception.DeploymentFailedException","message":"back-off
>>>> 20s restarting failed container=flink-main-container
>>>> pod=flink-testing-service-749dd97c75-4w9ps_flink-testing-service(6db1adb3-4ca4-4924-a8c3-57a417818d85)","additionalMetadata":{"reason":"CrashLoopBackOff"},"throwableList":[]}
>>>>
>>>> ...
>>>> [m [33m2023-10-02 22:17:33,576 [m [36mo.a.f.k.o.o.d.ApplicationObserver
>>>> [m [32m[INFO ][flink-testing-service/flink-testing-service] Observing
>>>> JobManager deployment. Previous status: ERROR
>>>>
>>>>
>>>> What I do next is I change the spec of the FlinkDeployment so that it
>>>> uses a Flink image that is healthy. The operator shows that the spec has
>>>> changed:
>>>>
>>>> [m [33m2023-10-02 22:45:37,445 [m [36mo.a.f.k.o.l.AuditUtils [m
>>>> [32m[INFO ][flink-testing-service/flink-testing-service] >>> Event | Info |
>>>> SPECCHANGED | UPGRADE change(s) detected
>>>> (FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJob,job.initialSavepointPath=s3a://robinhood-dev-core-flink-states/flink-testing-service/savepoints/savepoint-329a14-2f8264206b1d]
>>>> differs from
>>>> FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJo,job.initialSavepointPath=s3a://robinhood-dev-core-flink-states/flink-testing-service/savepoints/savepoint-dc1077-134923759e30]),
>>>> starting reconciliation.
>>>>
>>>>
>>>> However, the Flink operator cannot reconcile this spec change, and the
>>>> jobmanager is now permanently failing because it's still running the bad
>>>> Flink image:
>>>>
>>>> [m [33m2023-10-02 22:45:37,461 [m [36mo.a.f.k.o.l.AuditUtils [m
>>>> [32m[INFO ][flink-testing-service/flink-testing-service] >>> Event |
>>>> Warning | UPGRADEFAILED | JobManager deployment is missing and HA data is
>>>> not available to make stateful upgrades. It is possible that the job has
>>>> finished or terminally failed, or the configmaps have been deleted. Manual
>>>> restore required.
>>>>
>>>> I can simply delete this FlinkDeployment and redeploy with the healthy
>>>> Flink image, but I would like to avoid manual restores if possible. Is it
>>>> possible to recover by just changing the FlinkDeployment spec?
>>>>
>>>> Thanks,
>>>> Tony
>>>>
>>>> --
>>>>
>>>> <http://www.robinhood.com/>
>>>>
>>>> Tony Chen
>>>>
>>>> Software Engineer
>>>>
>>>> Menlo Park, CA
>>>>
>>>> Don't copy, share, or use this email without permission. If you
>>>> received it by accident, please let us know and then delete it right away.
>>>>
>>>
>>>
>>> --
>>>
>>> <http://www.robinhood.com/>
>>>
>>> Tony Chen
>>>
>>> Software Engineer
>>>
>>> Menlo Park, CA
>>>
>>> Don't copy, share, or use this email without permission. If you received
>>> it by accident, please let us know and then delete it right away.
>>>
>>
>>
>> --
>>
>> <http://www.robinhood.com/>
>>
>> Tony Chen
>>
>> Software Engineer
>>
>> Menlo Park, CA
>>
>> Don't copy, share, or use this email without permission. If you received
>> it by accident, please let us know and then delete it right away.
>>
>
--
<http://www.robinhood.com/>
Tony Chen
Software Engineer
Menlo Park, CA
Don't copy, share, or use this email without permission. If you received it
by accident, please let us know and then delete it right away.
Re: Rolling back a bad deployment of FlinkDeployment on kubernetes
Posted by Gyula Fóra <gy...@gmail.com>.
Hi Tony!
There are still a few corner cases when the operator cannot upgrade /
rollback deployments due to the loss of HA metadata (and with that
checkpoint information).
Most of these issues are not related to the operator logic directly but to
how Flink handles certain failures and are related to:
https://issues.apache.org/jira/browse/FLINK-30444 and
https://cwiki.apache.org/confluence/display/FLINK/FLIP-360%3A+Merging+the+ExecutionGraphInfoStore+and+the+JobResultStore+into+a+single+component+CompletedJobStore
Rollbacks are designed to allow automatic fallback to the last stable spec,
but the mechanism doesn't work in these corner cases (in the same way spec
upgrades also dont)
I hope this helps to understand the problem.
The solution in these cases is to manually recover the job from the last
checkpoint/savepoint.
Cheers,
Gyula
On Thu, Oct 5, 2023 at 7:56 PM Tony Chen <to...@robinhood.com> wrote:
> I tried this out with operator version 1.4 and it didn't work for me. I
> noticed that when I was deploying a bad version, the Kubernetes HA metadata
> and configmaps were deleted:
>
> [m [33m2023-10-05 14:52:17,493 [m [36mo.a.f.k.o.l.AuditUtils [m [32m[INFO
> ][flink-testing-service/flink-testing-service] >>> Event | Info |
> SPECCHANGED | UPGRADE change(s) detected
> (FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJo,job.initialSavepointPath=s3a://robinhood-prod-flink/flink-testing-service/savepoints/savepoint-b832ef-05b185cb5800]
> differs from
> FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJob,job.initialSavepointPath=<null>]),
> starting reconciliation.
> ...
> [m [33m2023-10-05 14:52:51,054 [m [36mo.a.f.k.o.s.AbstractFlinkService [m
> [32m[INFO ][flink-testing-service/flink-testing-service] Cluster shutdown
> completed.
> [m [33m2023-10-05 14:52:51,054 [m [36mo.a.f.k.o.s.AbstractFlinkService [m
> [32m[INFO ][flink-testing-service/flink-testing-service] Deleting
> Kubernetes HA metadata
> [m [33m2023-10-05 14:52:51,196 [m [36mo.a.f.k.o.l.AuditUtils [m [32m[INFO
> ][flink-testing-service/flink-testing-service] >>> Status | Info |
> UPGRADING | The resource is being upgraded
>
>
>
> Eventually, the rollbak fails because the HA metadata is missing:
>
> [m [33m2023-10-05 14:58:16,119 [m
> [36mo.a.f.k.o.r.d.AbstractFlinkResourceReconciler [m [33m[WARN
> ][flink-testing-service/flink-testing-service] Rollback is not possible due
> to missing HA metadata
>
>
>
> Besides setting kubernetes.operator.deployment.rollback.enabled: true, is
> there anything else that I need to configure?
>
> On Thu, Oct 5, 2023 at 10:35 AM Tony Chen <to...@robinhood.com>
> wrote:
>
>> I just saw this experimental feature in the documentation:
>> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/job-management/#application-upgrade-rollbacks-experimental
>>
>> I'm guessing this is the only way to automate rollbacks for now.
>>
>> On Wed, Oct 4, 2023 at 3:25 PM Tony Chen <to...@robinhood.com>
>> wrote:
>>
>>> Hi Flink Community,
>>>
>>> I am currently running Apache flink-kubernetes-operator on our
>>> kubernetes clusters, and I have Flink applications that are deployed using
>>> the FlinkDeployment Custom Resources (CR). I am trying to automate the
>>> process of rollbacks and I am running into some issues.
>>>
>>> I was testing out a bad deployment where the jobmanager never becomes
>>> healthy. I simulated this bad deployment by creating a Flink image with a
>>> bug in it. I see in the operator logs that the jobmanager is unhealthy:
>>>
>>> [m [33m2023-10-02 22:14:34,874 [m
>>> [36mo.a.f.k.o.r.d.AbstractFlinkResourceReconciler [m [32m[INFO
>>> ][flink-testing-service/flink-testing-service] UPGRADE change(s) detected
>>> (FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJo]
>>> differs from
>>> FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJob]),
>>> starting reconciliation.
>>> ...
>>> [m [33m2023-10-02 22:15:09,001 [m [36mo.a.f.k.o.l.AuditUtils [m
>>> [32m[INFO ][flink-testing-service/flink-testing-service] >>> Status | Info
>>> | UPGRADING | The resource is being upgraded
>>> ...
>>> [m [33m2023-10-02 22:17:23,911 [m [36mo.a.f.k.o.l.AuditUtils [m
>>> [32m[INFO ][flink-testing-service/flink-testing-service] >>> Status | Error
>>> | DEPLOYED |
>>> {"type":"org.apache.flink.kubernetes.operator.exception.DeploymentFailedException","message":"back-off
>>> 20s restarting failed container=flink-main-container
>>> pod=flink-testing-service-749dd97c75-4w9ps_flink-testing-service(6db1adb3-4ca4-4924-a8c3-57a417818d85)","additionalMetadata":{"reason":"CrashLoopBackOff"},"throwableList":[]}
>>>
>>> ...
>>> [m [33m2023-10-02 22:17:33,576 [m [36mo.a.f.k.o.o.d.ApplicationObserver
>>> [m [32m[INFO ][flink-testing-service/flink-testing-service] Observing
>>> JobManager deployment. Previous status: ERROR
>>>
>>>
>>> What I do next is I change the spec of the FlinkDeployment so that it
>>> uses a Flink image that is healthy. The operator shows that the spec has
>>> changed:
>>>
>>> [m [33m2023-10-02 22:45:37,445 [m [36mo.a.f.k.o.l.AuditUtils [m
>>> [32m[INFO ][flink-testing-service/flink-testing-service] >>> Event | Info |
>>> SPECCHANGED | UPGRADE change(s) detected
>>> (FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJob,job.initialSavepointPath=s3a://robinhood-dev-core-flink-states/flink-testing-service/savepoints/savepoint-329a14-2f8264206b1d]
>>> differs from
>>> FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJo,job.initialSavepointPath=s3a://robinhood-dev-core-flink-states/flink-testing-service/savepoints/savepoint-dc1077-134923759e30]),
>>> starting reconciliation.
>>>
>>>
>>> However, the Flink operator cannot reconcile this spec change, and the
>>> jobmanager is now permanently failing because it's still running the bad
>>> Flink image:
>>>
>>> [m [33m2023-10-02 22:45:37,461 [m [36mo.a.f.k.o.l.AuditUtils [m
>>> [32m[INFO ][flink-testing-service/flink-testing-service] >>> Event |
>>> Warning | UPGRADEFAILED | JobManager deployment is missing and HA data is
>>> not available to make stateful upgrades. It is possible that the job has
>>> finished or terminally failed, or the configmaps have been deleted. Manual
>>> restore required.
>>>
>>> I can simply delete this FlinkDeployment and redeploy with the healthy
>>> Flink image, but I would like to avoid manual restores if possible. Is it
>>> possible to recover by just changing the FlinkDeployment spec?
>>>
>>> Thanks,
>>> Tony
>>>
>>> --
>>>
>>> <http://www.robinhood.com/>
>>>
>>> Tony Chen
>>>
>>> Software Engineer
>>>
>>> Menlo Park, CA
>>>
>>> Don't copy, share, or use this email without permission. If you received
>>> it by accident, please let us know and then delete it right away.
>>>
>>
>>
>> --
>>
>> <http://www.robinhood.com/>
>>
>> Tony Chen
>>
>> Software Engineer
>>
>> Menlo Park, CA
>>
>> Don't copy, share, or use this email without permission. If you received
>> it by accident, please let us know and then delete it right away.
>>
>
>
> --
>
> <http://www.robinhood.com/>
>
> Tony Chen
>
> Software Engineer
>
> Menlo Park, CA
>
> Don't copy, share, or use this email without permission. If you received
> it by accident, please let us know and then delete it right away.
>
Re: Rolling back a bad deployment of FlinkDeployment on kubernetes
Posted by Tony Chen <to...@robinhood.com>.
I tried this out with operator version 1.4 and it didn't work for me. I
noticed that when I was deploying a bad version, the Kubernetes HA metadata
and configmaps were deleted:
[m [33m2023-10-05 14:52:17,493 [m [36mo.a.f.k.o.l.AuditUtils [m [32m[INFO
][flink-testing-service/flink-testing-service] >>> Event | Info |
SPECCHANGED | UPGRADE change(s) detected
(FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJo,job.initialSavepointPath=s3a://robinhood-prod-flink/flink-testing-service/savepoints/savepoint-b832ef-05b185cb5800]
differs from
FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJob,job.initialSavepointPath=<null>]),
starting reconciliation.
...
[m [33m2023-10-05 14:52:51,054 [m [36mo.a.f.k.o.s.AbstractFlinkService [m
[32m[INFO ][flink-testing-service/flink-testing-service] Cluster shutdown
completed.
[m [33m2023-10-05 14:52:51,054 [m [36mo.a.f.k.o.s.AbstractFlinkService [m
[32m[INFO ][flink-testing-service/flink-testing-service] Deleting
Kubernetes HA metadata
[m [33m2023-10-05 14:52:51,196 [m [36mo.a.f.k.o.l.AuditUtils [m [32m[INFO
][flink-testing-service/flink-testing-service] >>> Status | Info |
UPGRADING | The resource is being upgraded
Eventually, the rollbak fails because the HA metadata is missing:
[m [33m2023-10-05 14:58:16,119 [m
[36mo.a.f.k.o.r.d.AbstractFlinkResourceReconciler [m [33m[WARN
][flink-testing-service/flink-testing-service] Rollback is not possible due
to missing HA metadata
Besides setting kubernetes.operator.deployment.rollback.enabled: true, is
there anything else that I need to configure?
On Thu, Oct 5, 2023 at 10:35 AM Tony Chen <to...@robinhood.com> wrote:
> I just saw this experimental feature in the documentation:
> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/job-management/#application-upgrade-rollbacks-experimental
>
> I'm guessing this is the only way to automate rollbacks for now.
>
> On Wed, Oct 4, 2023 at 3:25 PM Tony Chen <to...@robinhood.com> wrote:
>
>> Hi Flink Community,
>>
>> I am currently running Apache flink-kubernetes-operator on our kubernetes
>> clusters, and I have Flink applications that are deployed using the
>> FlinkDeployment Custom Resources (CR). I am trying to automate the process
>> of rollbacks and I am running into some issues.
>>
>> I was testing out a bad deployment where the jobmanager never becomes
>> healthy. I simulated this bad deployment by creating a Flink image with a
>> bug in it. I see in the operator logs that the jobmanager is unhealthy:
>>
>> [m [33m2023-10-02 22:14:34,874 [m
>> [36mo.a.f.k.o.r.d.AbstractFlinkResourceReconciler [m [32m[INFO
>> ][flink-testing-service/flink-testing-service] UPGRADE change(s) detected
>> (FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJo]
>> differs from
>> FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJob]),
>> starting reconciliation.
>> ...
>> [m [33m2023-10-02 22:15:09,001 [m [36mo.a.f.k.o.l.AuditUtils [m [32m[INFO
>> ][flink-testing-service/flink-testing-service] >>> Status | Info |
>> UPGRADING | The resource is being upgraded
>> ...
>> [m [33m2023-10-02 22:17:23,911 [m [36mo.a.f.k.o.l.AuditUtils [m [32m[INFO
>> ][flink-testing-service/flink-testing-service] >>> Status | Error |
>> DEPLOYED |
>> {"type":"org.apache.flink.kubernetes.operator.exception.DeploymentFailedException","message":"back-off
>> 20s restarting failed container=flink-main-container
>> pod=flink-testing-service-749dd97c75-4w9ps_flink-testing-service(6db1adb3-4ca4-4924-a8c3-57a417818d85)","additionalMetadata":{"reason":"CrashLoopBackOff"},"throwableList":[]}
>>
>> ...
>> [m [33m2023-10-02 22:17:33,576 [m [36mo.a.f.k.o.o.d.ApplicationObserver
>> [m [32m[INFO ][flink-testing-service/flink-testing-service] Observing
>> JobManager deployment. Previous status: ERROR
>>
>>
>> What I do next is I change the spec of the FlinkDeployment so that it
>> uses a Flink image that is healthy. The operator shows that the spec has
>> changed:
>>
>> [m [33m2023-10-02 22:45:37,445 [m [36mo.a.f.k.o.l.AuditUtils [m [32m[INFO
>> ][flink-testing-service/flink-testing-service] >>> Event | Info |
>> SPECCHANGED | UPGRADE change(s) detected
>> (FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJob,job.initialSavepointPath=s3a://robinhood-dev-core-flink-states/flink-testing-service/savepoints/savepoint-329a14-2f8264206b1d]
>> differs from
>> FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJo,job.initialSavepointPath=s3a://robinhood-dev-core-flink-states/flink-testing-service/savepoints/savepoint-dc1077-134923759e30]),
>> starting reconciliation.
>>
>>
>> However, the Flink operator cannot reconcile this spec change, and the
>> jobmanager is now permanently failing because it's still running the bad
>> Flink image:
>>
>> [m [33m2023-10-02 22:45:37,461 [m [36mo.a.f.k.o.l.AuditUtils [m [32m[INFO
>> ][flink-testing-service/flink-testing-service] >>> Event | Warning |
>> UPGRADEFAILED | JobManager deployment is missing and HA data is not
>> available to make stateful upgrades. It is possible that the job has
>> finished or terminally failed, or the configmaps have been deleted. Manual
>> restore required.
>>
>> I can simply delete this FlinkDeployment and redeploy with the healthy
>> Flink image, but I would like to avoid manual restores if possible. Is it
>> possible to recover by just changing the FlinkDeployment spec?
>>
>> Thanks,
>> Tony
>>
>> --
>>
>> <http://www.robinhood.com/>
>>
>> Tony Chen
>>
>> Software Engineer
>>
>> Menlo Park, CA
>>
>> Don't copy, share, or use this email without permission. If you received
>> it by accident, please let us know and then delete it right away.
>>
>
>
> --
>
> <http://www.robinhood.com/>
>
> Tony Chen
>
> Software Engineer
>
> Menlo Park, CA
>
> Don't copy, share, or use this email without permission. If you received
> it by accident, please let us know and then delete it right away.
>
--
<http://www.robinhood.com/>
Tony Chen
Software Engineer
Menlo Park, CA
Don't copy, share, or use this email without permission. If you received it
by accident, please let us know and then delete it right away.
Re: Rolling back a bad deployment of FlinkDeployment on kubernetes
Posted by Tony Chen <to...@robinhood.com>.
I just saw this experimental feature in the documentation:
https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/job-management/#application-upgrade-rollbacks-experimental
I'm guessing this is the only way to automate rollbacks for now.
On Wed, Oct 4, 2023 at 3:25 PM Tony Chen <to...@robinhood.com> wrote:
> Hi Flink Community,
>
> I am currently running Apache flink-kubernetes-operator on our kubernetes
> clusters, and I have Flink applications that are deployed using the
> FlinkDeployment Custom Resources (CR). I am trying to automate the process
> of rollbacks and I am running into some issues.
>
> I was testing out a bad deployment where the jobmanager never becomes
> healthy. I simulated this bad deployment by creating a Flink image with a
> bug in it. I see in the operator logs that the jobmanager is unhealthy:
>
> [m [33m2023-10-02 22:14:34,874 [m
> [36mo.a.f.k.o.r.d.AbstractFlinkResourceReconciler [m [32m[INFO
> ][flink-testing-service/flink-testing-service] UPGRADE change(s) detected
> (FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJo]
> differs from
> FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJob]),
> starting reconciliation.
> ...
> [m [33m2023-10-02 22:15:09,001 [m [36mo.a.f.k.o.l.AuditUtils [m [32m[INFO
> ][flink-testing-service/flink-testing-service] >>> Status | Info |
> UPGRADING | The resource is being upgraded
> ...
> [m [33m2023-10-02 22:17:23,911 [m [36mo.a.f.k.o.l.AuditUtils [m [32m[INFO
> ][flink-testing-service/flink-testing-service] >>> Status | Error |
> DEPLOYED |
> {"type":"org.apache.flink.kubernetes.operator.exception.DeploymentFailedException","message":"back-off
> 20s restarting failed container=flink-main-container
> pod=flink-testing-service-749dd97c75-4w9ps_flink-testing-service(6db1adb3-4ca4-4924-a8c3-57a417818d85)","additionalMetadata":{"reason":"CrashLoopBackOff"},"throwableList":[]}
>
> ...
> [m [33m2023-10-02 22:17:33,576 [m [36mo.a.f.k.o.o.d.ApplicationObserver [m
> [32m[INFO ][flink-testing-service/flink-testing-service] Observing
> JobManager deployment. Previous status: ERROR
>
>
> What I do next is I change the spec of the FlinkDeployment so that it uses
> a Flink image that is healthy. The operator shows that the spec has changed:
>
> [m [33m2023-10-02 22:45:37,445 [m [36mo.a.f.k.o.l.AuditUtils [m [32m[INFO
> ][flink-testing-service/flink-testing-service] >>> Event | Info |
> SPECCHANGED | UPGRADE change(s) detected
> (FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJob,job.initialSavepointPath=s3a://robinhood-dev-core-flink-states/flink-testing-service/savepoints/savepoint-329a14-2f8264206b1d]
> differs from
> FlinkDeploymentSpec[job.entryClass=com.robinhood.flink.chaos.StreamingSumByKeyJo,job.initialSavepointPath=s3a://robinhood-dev-core-flink-states/flink-testing-service/savepoints/savepoint-dc1077-134923759e30]),
> starting reconciliation.
>
>
> However, the Flink operator cannot reconcile this spec change, and the
> jobmanager is now permanently failing because it's still running the bad
> Flink image:
>
> [m [33m2023-10-02 22:45:37,461 [m [36mo.a.f.k.o.l.AuditUtils [m [32m[INFO
> ][flink-testing-service/flink-testing-service] >>> Event | Warning |
> UPGRADEFAILED | JobManager deployment is missing and HA data is not
> available to make stateful upgrades. It is possible that the job has
> finished or terminally failed, or the configmaps have been deleted. Manual
> restore required.
>
> I can simply delete this FlinkDeployment and redeploy with the healthy
> Flink image, but I would like to avoid manual restores if possible. Is it
> possible to recover by just changing the FlinkDeployment spec?
>
> Thanks,
> Tony
>
> --
>
> <http://www.robinhood.com/>
>
> Tony Chen
>
> Software Engineer
>
> Menlo Park, CA
>
> Don't copy, share, or use this email without permission. If you received
> it by accident, please let us know and then delete it right away.
>
--
<http://www.robinhood.com/>
Tony Chen
Software Engineer
Menlo Park, CA
Don't copy, share, or use this email without permission. If you received it
by accident, please let us know and then delete it right away.