You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@streampark.apache.org by "sq8852161 (via GitHub)" <gi...@apache.org> on 2023/03/06 09:41:24 UTC
[GitHub] [incubator-streampark] sq8852161 opened a new issue, #2404: [Bug] Bug titleThe running status of the flink job is stuck in STARTING, but in fact the k8s job is running normally and k8s-ingress has been generated.
sq8852161 opened a new issue, #2404:
URL: https://github.com/apache/incubator-streampark/issues/2404
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-streampark/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### Java Version
1.8
### Scala Version
2.12.x
### StreamPark Version
v2.0.0
### Flink Version
1.15.1
### deploy mode
kubernetes-application
### What happened
The running status of the flink job is stuck in STARTING, but in fact the k8s job is running normally and k8s-ingress has been generated.
Background report warn
**ForkJoinPool-1-worker-1** | **org.apache.streampark.flink.kubernetes.watcher.FlinkJobStatusWatcher**:228] Failed to visit remote flink jobs on kubernetes-native-mode cluster, and the retry access logic is performed.
My action:
1. Configure the ingress domain name
2. A flink job is created, select ClusterIP and run.
3. The running status of the flink job is stuck in STARTING, but in fact the k8s job is running normally and k8s-ingress has been generated.
4.Background report warn
**ForkJoinPool-1-worker-1** | **org.apache.streampark.flink.kubernetes.watcher.FlinkJobStatusWatcher**:228] Failed to visit remote flink jobs on kubernetes-native-mode cluster, and the retry access logic is performed.
5.But after removing the ingress domain configuration. Re-establish the job and use NodePort and the task will run normally
The background will also report warn:
The retry fetch failed, final status failed, errorStack=Connect to http://xx.xx.xx.xxx:32185 [/xx.xx.xx.xxx] failed: Connection refused (Connection refused).
But after three reports, it will work normally.
### Error Exception
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!(您是否要贡献这个PR?)
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] sq8852161 commented on issue #2404: [Bug] Bug titleThe running status of the flink job is stuck in STARTING, but in fact the k8s job is running normally and k8s-ingress has been generated.
Posted by "sq8852161 (via GitHub)" <gi...@apache.org>.
sq8852161 commented on issue #2404:
URL: https://github.com/apache/incubator-streampark/issues/2404#issuecomment-1562380023
这是来自QQ邮箱的自动回复邮件。已经收到您的邮件。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] Shmilyqjj commented on issue #2404: [Bug] Bug titleThe running status of the flink job is stuck in STARTING, but in fact the k8s job is running normally and k8s-ingress has been generated.
Posted by "Shmilyqjj (via GitHub)" <gi...@apache.org>.
Shmilyqjj commented on issue #2404:
URL: https://github.com/apache/incubator-streampark/issues/2404#issuecomment-1562885158
This error can have many different reasons that I encountered and resolved. It's possible that your situation is different from mine, but I still hope to help you.
My Env: (streampark2.1.0 + flink1.14.5 + k8s Major:"1", Minor:"22+")
1.Check the version of K8s to ensure it is 1.19 or above.
2.Verify if .kube/config has the appropriate permissions (I encountered this issue due to insufficient permissions for the service account).
3.Flink REST endpoint cannot be accessed or requested .I found that I could make a successful REST request using curl, but streampark couldn't retrieve the REST address. After investigation, I discovered an error in the method **org.apache.streampark.flink.kubernetes.ingress.IngressController#ingressUrlAddress**, where the parsing of the K8s version was incorrect. The code in org.apache.streampark.flink.kubernetes.ingress.IngressStrategy was modified as follows:
`val version = s"${versionInfo.getMajor}.${versionInfo.getMinor}".toDouble`
Changed to:
`val version = s"${versionInfo.getMajor}.${versionInfo.getMinor}".replace("+", "").toDouble`
This resolved the issue.
4. When submitting the **Flink on K8s** job, it is important to ensure that the **streampark.workspace.remote** parameter is configured correctly. You can specify an HDFS (or file://) address, and the Flink parameter jobmanager.archive.fs.dir should also point to this address (streampark automatically appends this parameter). (In my case, I mounted a shared directory PVC that was accessible in both the Flink pod and streampark pod,then i solved the issue.)
希望对你有帮助.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Bug titleThe running status of the flink job is stuck in STARTING, but in fact the k8s job is running normally and k8s-ingress has been generated. [streampark]
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys closed issue #2404: [Bug] Bug titleThe running status of the flink job is stuck in STARTING, but in fact the k8s job is running normally and k8s-ingress has been generated.
URL: https://github.com/apache/streampark/issues/2404
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Bug titleThe running status of the flink job is stuck in STARTING, but in fact the k8s job is running normally and k8s-ingress has been generated. [streampark]
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on issue #2404:
URL: https://github.com/apache/streampark/issues/2404#issuecomment-5523227736
Thanks for your feedback. We are closing this issue because it is outdated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] moranrr commented on issue #2404: [Bug] Bug titleThe running status of the flink job is stuck in STARTING, but in fact the k8s job is running normally and k8s-ingress has been generated.
Posted by "moranrr (via GitHub)" <gi...@apache.org>.
moranrr commented on issue #2404:
URL: https://github.com/apache/incubator-streampark/issues/2404#issuecomment-1562379665
streamPark:2.0.0
flink:1.15.2 on k8s
+1
Sometimes the Running status remains “initializing” even though the job is running normally. When the job finishes or is stopped, the status does not show “FINISHED” or “CANCELED”, but instead shows “FAILED”.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] wolfboys commented on issue #2404: [Bug] Bug titleThe running status of the flink job is stuck in STARTING, but in fact the k8s job is running normally and k8s-ingress has been generated.
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on issue #2404:
URL: https://github.com/apache/incubator-streampark/issues/2404#issuecomment-1562385128
hi:
The reason has already been given in the log. Please check if the network connection is ok.
`The retry fetch failed, final status failed, errorStack=Connect to http://xx.xx.xx.xxx:32185/ [/xx.xx.xx.xxx] failed: Connection refused (Connection refused).`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] phoeph commented on issue #2404: [Bug] Bug titleThe running status of the flink job is stuck in STARTING, but in fact the k8s job is running normally and k8s-ingress has been generated.
Posted by "phoeph (via GitHub)" <gi...@apache.org>.
phoeph commented on issue #2404:
URL: https://github.com/apache/incubator-streampark/issues/2404#issuecomment-1486474980
streamPark:2.0.0
flink:1.16.1 on k8s
+1
stuck in INITIALIZING and STARTING.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] wolfboys commented on issue #2404: [Bug] Bug titleThe running status of the flink job is stuck in STARTING, but in fact the k8s job is running normally and k8s-ingress has been generated.
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on issue #2404:
URL: https://github.com/apache/incubator-streampark/issues/2404#issuecomment-1563833945
> This error can have many different reasons that I encountered and resolved. It's possible that your situation is different from mine, but I still hope to help you.
>
> My Env: (streampark2.1.0 + flink1.14.5 + k8s Major:"1", Minor:"22+")
>
> 1.Check the version of K8s to ensure it is 1.19 or above.
>
> 2.Verify if .kube/config has the appropriate permissions (I encountered this issue due to insufficient permissions for the service account).
>
> 3.Flink REST endpoint cannot be accessed or requested .I found that I could make a successful REST request using curl, but streampark couldn't retrieve the REST address. After investigation, I discovered an error in the method **org.apache.streampark.flink.kubernetes.ingress.IngressController#ingressUrlAddress**, where the parsing of the K8s version was incorrect. The code in org.apache.streampark.flink.kubernetes.ingress.IngressStrategy was modified as follows: `val version = s"${versionInfo.getMajor}.${versionInfo.getMinor}".toDouble` Changed to: `val version = s"${versionInfo.getMajor}.${versionInfo.getMinor}".replace("+", "").toDouble` This resolved the issue.
>
> 4. When submitting the **Flink on K8s** job, it is important to ensure that the **streampark.workspace.remote** parameter is configured correctly. You can specify an HDFS (or file://) address, and the Flink parameter jobmanager.archive.fs.dir should also point to this address (streampark automatically appends this parameter). (In my case, I mounted a shared directory PVC that was accessible in both the Flink pod and streampark pod,then i solved the issue.)
>
> 希望对你有帮助.
Thanks for your feedback. There was indeed a bug in the version parse, We have already fixed it. please see [here](https://github.com/apache/incubator-streampark/pull/2741)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] sq8852161 commented on issue #2404: [Bug] Bug titleThe running status of the flink job is stuck in STARTING, but in fact the k8s job is running normally and k8s-ingress has been generated.
Posted by "sq8852161 (via GitHub)" <gi...@apache.org>.
sq8852161 commented on issue #2404:
URL: https://github.com/apache/incubator-streampark/issues/2404#issuecomment-1486475494
这是来自QQ邮箱的自动回复邮件。已经收到您的邮件。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org