You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@streampark.apache.org by "wangkeshuai1 (via GitHub)" <gi...@apache.org> on 2023/06/21 04:52:05 UTC
[GitHub] [incubator-streampark] wangkeshuai1 opened a new issue, #2810: [Bug] Starting a job in K8S application mode may sometimes result in loading continuously, but the backend pod and deploy are normal
wangkeshuai1 opened a new issue, #2810:
URL: https://github.com/apache/incubator-streampark/issues/2810
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-streampark/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### Java Version
1.8
### Scala Version
2.12.x
### StreamPark Version
2.1.0
### Flink Version
1.14.3
### deploy mode
None
### What happened
Deploying FlinkSQL jobs in K8S application mode may sometimes remain in the INITIALIZING state, but in reality, it is already in the RUNNING state on K8S. I checked the network, deployment, and pod, and they all look normal.
Later, I debugged the code and printed the logs in it. I found that the two listening methods, subscribeJobStatusChange and subscribeMetricsChange, in the FlinkK8sChangeEventListener. java class, were simultaneously applied to t_ Flink_ The app table has been modified.
The following are the logs added to the code during debugging and the logs printed in the environment:
`@Subscribe
public void subscribeJobStatusChange(FlinkJobStatusChangeEvent event) {
JobStatusCV jobStatus = event.jobStatus();
TrackId trackId = event.trackId();
// get pre application record
Application app = applicationService.getById(trackId.appId());
if (app == null) {
return;
}
// update application record
setByJobStatusCV(app, jobStatus);
log.warn("wks---------update Status : {}", app.getState());
applicationService.persistMetrics(app);
Application app1 = applicationService.getById(trackId.appId());
log.warn("wks---------update Status : {}", app1.getState());
log.warn("wks---------update ModifyTime : {}", app1.getModifyTime());
log.warn("wks---------update Status Done : {}", app.getState());
// email alerts when necessary
FlinkAppState state = FlinkAppState.of(app.getState());
if (FlinkAppState.FAILED.equals(state)
|| FlinkAppState.LOST.equals(state)
|| FlinkAppState.RESTARTING.equals(state)
|| FlinkAppState.FINISHED.equals(state)) {
IngressController.deleteIngress(app.getClusterId(), app.getK8sNamespace());
executor.execute(() -> alertService.alert(app, state));
}
}
/**
* Catch FlinkClusterMetricChangeEvent then storage it persistently to db. Actually update
* org.apache.streampark.console.core.entity.Application records.
*/
@SuppressWarnings("UnstableApiUsage")
@Subscribe
public void subscribeMetricsChange(FlinkClusterMetricChangeEvent event) {
TrackId trackId = event.trackId();
ExecutionMode mode = FlinkK8sExecuteMode.toExecutionMode(trackId.executeMode());
// discard session mode change
if (ExecutionMode.KUBERNETES_NATIVE_SESSION.equals(mode)) {
return;
}
Application app = applicationService.getById(trackId.appId());
if (app == null) {
return;
}
log.warn("wks---------subscribeMetricsChange get app {}", app.getState());
FlinkMetricCV metrics = event.metrics();
app.setJmMemory(metrics.totalJmMemory());
app.setTmMemory(metrics.totalTmMemory());
app.setTotalTM(metrics.totalTm());
app.setTotalSlot(metrics.totalSlot());
app.setAvailableSlot(metrics.availableSlot());
app.setState(null);
applicationService.persistMetrics(app);
log.warn("wks--------subscribeMetricsChange update app {}", System.currentTimeMillis());
}`
It is recommended that in the subscribeMetricsChange method, the state state should not be modified when modifying the database
### Error Exception
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!(您是否要贡献这个PR?)
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] wangkeshuai1 commented on issue #2810: [Bug] Starting a job in K8S application mode may sometimes result in loading continuously, but the backend pod and deploy are normal
Posted by "wangkeshuai1 (via GitHub)" <gi...@apache.org>.
wangkeshuai1 commented on issue #2810:
URL: https://github.com/apache/incubator-streampark/issues/2810#issuecomment-1600082797
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Starting a job in K8S application mode may sometimes result in loading continuously, but the backend pod and deploy are normal [streampark]
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on issue #2810:
URL: https://github.com/apache/streampark/issues/2810#issuecomment-5523177195
Thanks for your feedback. We are closing this issue because it is outdated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Starting a job in K8S application mode may sometimes result in loading continuously, but the backend pod and deploy are normal [streampark]
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys closed issue #2810: [Bug] Starting a job in K8S application mode may sometimes result in loading continuously, but the backend pod and deploy are normal
URL: https://github.com/apache/streampark/issues/2810
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org