You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by GitBox <gi...@apache.org> on 2019/12/23 01:49:19 UTC
[GitHub] [camel-k] kameshsampath opened a new issue #1172: Knative
integration takes too long to make the service ready
kameshsampath opened a new issue #1172: Knative integration takes too long to make the service ready
URL: https://github.com/apache/camel-k/issues/1172
When trying to create a Knative integration I observe that the Knative service is taking too long to get to ready state and sometimes its struck in `RevisionMissing` state. This happens in both `--dev` mode as well as in `production` mode. This is highly inconsistent as sometimes it refreshes the state of the service soon but after few runs its starts to get hung .
It is also noted it takes very long time to trigger the service deployment after we call `kamel run`, wondering something to do with Knative Serving reconcilation ? There should be a way to force the knative recocilation, so that new reivision is updated with Ready state as soon as possible.
Example integation:
```yaml
- from:
uri: "knative:endpoint/echoer"
steps:
- log:
message: "Got Message: ${body}"
- convert-body: "java.lang.String"
- choice:
when:
- simple: "${body} != null && ${body.length} > 0"
steps:
- set-body:
simple: "${body.toUpperCase()}"
- set-header:
name: ContentType
simple: text/plain
- log:
message: "${body}"
otherwise:
steps:
- set-body:
constant: "no body received"
- set-header:
name: ContentType
simple: text/plain
- log:
message: "Otherwise::${body}"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [camel-k] kameshsampath commented on issue #1172: Knative
integration takes too long to make the service ready
Posted by GitBox <gi...@apache.org>.
kameshsampath commented on issue #1172: Knative integration takes too long to make the service ready
URL: https://github.com/apache/camel-k/issues/1172#issuecomment-568327565
Sample output of `ksvc`
```
NAME URL LATESTCREATED LATESTREADY READY REASON
echoer http://echoer.knativetutorial.example.com echoer-kpcmz Unknown RevisionMissing
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [camel-k] nicolaferraro commented on issue #1172: Knative
integration takes too long to make the service ready
Posted by GitBox <gi...@apache.org>.
nicolaferraro commented on issue #1172: Knative integration takes too long to make the service ready
URL: https://github.com/apache/camel-k/issues/1172#issuecomment-586296568
Fixed by #1259 (which reference images directly via digest, so that Knative does not try to contact the registry to find it)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [camel-k] kameshsampath commented on issue #1172: Knative
integration takes too long to make the service ready
Posted by GitBox <gi...@apache.org>.
kameshsampath commented on issue #1172: Knative integration takes too long to make the service ready
URL: https://github.com/apache/camel-k/issues/1172#issuecomment-569380759
> using the suggested minikube/kube version
you mean as per Knative upstream docs ? I am doing exactly the same my start script for minikube is
```shell
#!/bin/bash
set -eu
PROFILE_NAME=${PROFILE_NAME:-knativetutorial}
VM_DRIVER=${VM_DRIVER:-virtualbox}
MEMORY=${MEMORY:-8192}
CPUS=${CPUS:-6}
EXTRA_CONFIG="apiserver.enable-admission-plugins=\
LimitRanger,\
NamespaceExists,\
NamespaceLifecycle,\
ResourceQuota,\
ServiceAccount,\
DefaultStorageClass,\
MutatingAdmissionWebhook"
minikube start --memory=$MEMORY --cpus=$CPUS \
--kubernetes-version=v1.14.0 \
--vm-driver="$VM_DRIVER" \
--disk-size=50g \
--extra-config="$EXTRA_CONFIG" \
--insecure-registry='10.0.0.0/24'
```
I usually the problem more often when you try updating the service with updates either via redeploy or in `--dev` mode
> using the exact istio version, with no injection
yes injection is diabled and I use the `istio-lean` install
> installing the full monitoring stack
Dont know why this is needed ? With 8GB RAM this will not make us to run any other applications :(
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [camel-k] lburgazzoli commented on issue #1172: Knative integration
takes too long to make the service ready
Posted by GitBox <gi...@apache.org>.
lburgazzoli commented on issue #1172: Knative integration takes too long to make the service ready
URL: https://github.com/apache/camel-k/issues/1172#issuecomment-584521454
releated to #1172
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [camel-k] lburgazzoli commented on issue #1172: Knative integration
takes too long to make the service ready
Posted by GitBox <gi...@apache.org>.
lburgazzoli commented on issue #1172: Knative integration takes too long to make the service ready
URL: https://github.com/apache/camel-k/issues/1172#issuecomment-584521888
@kameshsampath @nak3 I remember I had some issues in the past similar to this one and were related to the default requests that knative sets-up. Lowering them did fix the issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [camel-k] nicolaferraro commented on issue #1172: Knative
integration takes too long to make the service ready
Posted by GitBox <gi...@apache.org>.
nicolaferraro commented on issue #1172: Knative integration takes too long to make the service ready
URL: https://github.com/apache/camel-k/issues/1172#issuecomment-569226494
I've hit the issue of services taking a long time to deploy on minikube, I've opened this: https://github.com/knative/serving/issues/5802, but no luck.
I'm not experiencing the issue anymore after I started to follow the exact procedure when installing Knative, like:
- using the suggested minikube/kube version
- using the exact istio version, with no injection
- installing the full monitoring stack
I've also noticed that sometimes the service state is not aligned with the pod state but didn't investigate becaus the service was running in my case. Will try to figure out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [camel-k] kameshsampath edited a comment on issue #1172: Knative
integration takes too long to make the service ready
Posted by GitBox <gi...@apache.org>.
kameshsampath edited a comment on issue #1172: Knative integration takes too long to make the service ready
URL: https://github.com/apache/camel-k/issues/1172#issuecomment-568327565
Sample output of `ksvc`
```
NAME URL LATESTCREATED LATESTREADY READY REASON
echoer http://echoer.knativetutorial.example.com echoer-kpcmz Unknown RevisionMissing
```
But when you see the pods of the namespace you will notice the Knative service pod is up
```
▶ kgpo
NAME READY STATUS RESTARTS AGE
camel-k-cache 0/1 Completed 0 10h
camel-k-kit-bnvnkhccmvshuqdcf8ng 0/1 Completed 0 11h
camel-k-kit-bnvnqlccmvshuqdcf8o0 0/1 Completed 0 11h
camel-k-kit-bnvodbccmvshuqdcf8og 0/1 Completed 0 10h
camel-k-kit-bnvodbccmvshuqdcf8og-builder 0/1 Completed 0 10h
camel-k-kit-bo0102kcmvshuqdcf8p0 0/1 Completed 0 62m
camel-k-kit-bo0102kcmvshuqdcf8p0-builder 0/1 Completed 0 62m
camel-k-operator-84d7896b68-6gv52 1/1 Running 0 11h
echoer-kpcmz-deployment-b9f85668c-ddzs4 2/2 Running 0 63s
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [camel-k] nicolaferraro closed issue #1172: Knative integration
takes too long to make the service ready
Posted by GitBox <gi...@apache.org>.
nicolaferraro closed issue #1172: Knative integration takes too long to make the service ready
URL: https://github.com/apache/camel-k/issues/1172
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [camel-k] kameshsampath commented on issue #1172: Knative
integration takes too long to make the service ready
Posted by GitBox <gi...@apache.org>.
kameshsampath commented on issue #1172: Knative integration takes too long to make the service ready
URL: https://github.com/apache/camel-k/issues/1172#issuecomment-568837966
I also observe that in these cases the `containerConcurrency` of the ksvc and its related configuration is set to `0`. Wondering this is causing the revision not to become ready immediately ??
CC: @nicolaferraro
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [camel-k] kameshsampath edited a comment on issue #1172: Knative
integration takes too long to make the service ready
Posted by GitBox <gi...@apache.org>.
kameshsampath edited a comment on issue #1172: Knative integration takes too long to make the service ready
URL: https://github.com/apache/camel-k/issues/1172#issuecomment-568327565
Sample output of `ksvc`
```
NAME URL LATESTCREATED LATESTREADY READY REASON
echoer http://echoer.knativetutorial.example.com echoer-kpcmz Unknown RevisionMissing
```
But when you see the pods of the namespace you will notice the Knative service pod is up
```▶ kgpo
NAME READY STATUS RESTARTS AGE
camel-k-cache 0/1 Completed 0 10h
camel-k-kit-bnvnkhccmvshuqdcf8ng 0/1 Completed 0 11h
camel-k-kit-bnvnqlccmvshuqdcf8o0 0/1 Completed 0 11h
camel-k-kit-bnvodbccmvshuqdcf8og 0/1 Completed 0 10h
camel-k-kit-bnvodbccmvshuqdcf8og-builder 0/1 Completed 0 10h
camel-k-kit-bo0102kcmvshuqdcf8p0 0/1 Completed 0 62m
camel-k-kit-bo0102kcmvshuqdcf8p0-builder 0/1 Completed 0 62m
camel-k-operator-84d7896b68-6gv52 1/1 Running 0 11h
echoer-kpcmz-deployment-b9f85668c-ddzs4 2/2 Running 0 63s```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [camel-k] nak3 commented on issue #1172: Knative integration takes
too long to make the service ready
Posted by GitBox <gi...@apache.org>.
nak3 commented on issue #1172: Knative integration takes too long to make the service ready
URL: https://github.com/apache/camel-k/issues/1172#issuecomment-583941124
Can we more narrow down what component/event takes long time? For example:
* Examine event logs during the deploy - `kubect get event -w`.
* Investigate controller logs - `kubectl -n knative-serving logs controller-xxx -c controller`. (like https://github.com/knative/serving/issues/5802 did.)
Also, it would be worth to look into some suspicious stuff like:
* CPU/memory usage on the host to verify no problem - `$ top`.
* The status of sub resource in the cluster. - eg. the log says `RevisionMissing`, but `kubect get rev -o yaml` shows anything?)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services