You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2021/01/20 11:38:18 UTC

[GitHub] [apisix] sober-wang opened a new issue #3367: bug: When the apisix running in kubernetes ,If etcd is stop, the apisix status is still running.

sober-wang opened a new issue #3367:
URL: https://github.com/apache/apisix/issues/3367


   ### Issue description
   When the apisix running in kubernetes ,If etcd cluster is stop, the apisix status is still running. 
   I have a k8s cluster and my apisix on k8s ,
   the etcd is outof the k8s cluster
   
   ### Environment
   k8s version: 1.18.2
   apisix version: 2.1
   etcd version: v3.4.3-0 
   
   ### Minimal test code / Steps to reproduce the issue
   I have a k8s cluster and my apisix on k8s ,
   the etcd is outof the k8s cluster
   
   this is my apisix-create.yaml
   
   ```yaml
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: apisix-gw
     name: apisix-gw-deployment
     namespace: ws
   spec:
     progressDeadlineSeconds: 600
     replicas: 1
     revisionHistoryLimit: 10
     selector:
       matchLabels:
         app: apisix-gw
     strategy:
       rollingUpdate:
         maxSurge: 25%
         maxUnavailable: 25%
       type: RollingUpdate
     template:
       metadata:
         creationTimestamp: null
         labels:
           app: apisix-gw
       spec:
         affinity:
           podAntiAffinity:
             preferredDuringSchedulingIgnoredDuringExecution:
             - podAffinityTerm:
                 labelSelector:
                   matchExpressions:
                   - key: ingress
                     operator: In
                     values:
                     - controller
                 topologyKey: kubernetes.io/hostname
               weight: 100
         containers:
         - env:
           - name: TZ
             value: Asia/Shanghai
           - name: POD_NAME
             valueFrom:
               fieldRef:
                 apiVersion: v1
                 fieldPath: metadata.name
           - name: POD_NAMESPACE
             valueFrom:
               fieldRef:
                 apiVersion: v1
                 fieldPath: metadata.namespace
           image: myregistry.com/ops/apisix:v2
           imagePullPolicy: IfNotPresent
           name: apisix-gw-deployment
           ports:
           - containerPort: 9080
             hostPort: 9080
             name: http
             protocol: TCP
           - containerPort: 9443
             hostPort: 9443
             name: https
             protocol: TCP
           readinessProbe:
             failureThreshold: 6
             initialDelaySeconds: 10
             periodSeconds: 10
             successThreshold: 1
             tcpSocket:
               port: 9080
             timeoutSeconds: 1
           resources: {}
           terminationMessagePath: /dev/termination-log
           terminationMessagePolicy: File
           volumeMounts:
           - mountPath: /usr/local/apisix/conf/config.yaml
             name: apisix-config-yaml-configmap
             subPath: config.yaml
           - mountPath: /etc/localtime
             name: localtime
             readOnly: true
         dnsPolicy: ClusterFirst
         hostNetwork: true
         initContainers:
         - command:
           - /bin/sh
           - -c
           - |
             sysctl -w net.core.somaxconn=65535
             sysctl -w net.ipv4.ip_local_port_range="1024 65535"
             sysctl -w net.ipv4.tcp_max_syn_backlog=8192
             sysctl -w fs.file-max=1048576
             sysctl -w fs.inotify.max_user_instances=16384
             sysctl -w fs.inotify.max_user_watches=524288
             sysctl -w fs.inotify.max_queued_events=16384
           image: busybox:latest
           imagePullPolicy: Always
           name: init-sysctl
           resources: {}
           securityContext:
             privileged: true
             procMount: Default
           terminationMessagePath: /dev/termination-log
           terminationMessagePolicy: File
         restartPolicy: Always
         schedulerName: default-scheduler
         securityContext: {}
         terminationGracePeriodSeconds: 30
         volumes:
         - configMap:
             defaultMode: 420
             name: apisix-gw-config.yaml
           name: apisix-config-yaml-configmap
         - hostPath:
             path: /etc/localtime
             type: File
           name: localtime
   ---
   apiVersion: v1
   data:
     config.yaml: |
     ----
   # Licensed to the Apache Software Foundation (ASF) under one or more
   # contributor license agreements.  See the NOTICE file distributed with
   # this work for additional information regarding copyright ownership.
   # The ASF licenses this file to You under the Apache License, Version 2.0
   # (the "License"); you may not use this file except in compliance with
   # the License.  You may obtain a copy of the License at
   #
   #     http://www.apache.org/licenses/LICENSE-2.0
   #
   # Unless required by applicable law or agreed to in writing, software
   # distributed under the License is distributed on an "AS IS" BASIS,
   # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   # See the License for the specific language governing permissions and
   # limitations under the License.
   #
   # If you want to set the specified configuration value, you can set the new
   # in this file. For example if you want to specify the etcd address:
   #
   etcd:
     host:
       - "http://192.168.1.1:2379"
   #
   # To configure via environment variables, you can use `${{VAR}}` syntax. For instance:
   #
   # etcd:
   #     host:
   #       - "http://${{ETCD_HOST}}:2379"
   #
   # If the configured environment variable can't be found, an error will be thrown.
   apisix:
     admin_key:
       -
         name: "admin"
         key: edd1c9f034335f136f87ad84b625c8f1 
   # using fixed API token has security risk, please
   # update it when you deploy to production environment
         role: admin
   
   Events:  <none>
   
   kind: ConfigMap
   metadata:
     name: apisix-gw-config.yaml
     namespace: ws
   
   ```
   
   ### Logs 
   we only enter the apisix pod , `cd logs/` the error.log have more error .
   ![图片](https://user-images.githubusercontent.com/35804630/105169484-8d3f8f00-5b56-11eb-9d49-7a027575830b.png)
   
   ### Execpt 
   We should build a health chek. and use k8s http check and the error log can print the /dev/stderr.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [apisix] tokers commented on issue #3367: bug: When the apisix running in kubernetes ,If etcd is stop, the apisix status is still running.

Posted by GitBox <gi...@apache.org>.
tokers commented on issue #3367:
URL: https://github.com/apache/apisix/issues/3367#issuecomment-763683875


   If you want to collect the error log, try the error-logger plugin:).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [apisix] sober-wang closed issue #3367: bug: When the apisix running in kubernetes ,If etcd is stop, the apisix status is still running.

Posted by GitBox <gi...@apache.org>.
sober-wang closed issue #3367:
URL: https://github.com/apache/apisix/issues/3367


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [apisix] Firstsawyou commented on issue #3367: bug: When the apisix running in kubernetes ,If etcd is stop, the apisix status is still running.

Posted by GitBox <gi...@apache.org>.
Firstsawyou commented on issue #3367:
URL: https://github.com/apache/apisix/issues/3367#issuecomment-763633835


   @gxthrj @tokers  If you are free, please take a look at this issue, thank you.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [apisix] sober-wang commented on issue #3367: bug: When the apisix running in kubernetes ,If etcd is stop, the apisix status is still running.

Posted by GitBox <gi...@apache.org>.
sober-wang commented on issue #3367:
URL: https://github.com/apache/apisix/issues/3367#issuecomment-768814306


   But the controller platform is bad .can't use apisix admin api. 
   The administrator and our service is not know , next we should speed more time go to troubleshooting.
   So When it have this problem , the apisix should call k8s events api or stdout or stderr.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [apisix] sober-wang commented on issue #3367: bug: When the apisix running in kubernetes ,If etcd is stop, the apisix status is still running.

Posted by GitBox <gi...@apache.org>.
sober-wang commented on issue #3367:
URL: https://github.com/apache/apisix/issues/3367#issuecomment-767980544


   No, My means is when the etcd connect is failed ,the apisix pod should print error log in the stdout or stderr .
   apisix server can used error-logger plugin 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [apisix] tokers commented on issue #3367: bug: When the apisix running in kubernetes ,If etcd is stop, the apisix status is still running.

Posted by GitBox <gi...@apache.org>.
tokers commented on issue #3367:
URL: https://github.com/apache/apisix/issues/3367#issuecomment-768223888


   @sober-wang All Nginx workers are running without a tty, it cannot use stdout, stderr.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [apisix] spacewander commented on issue #3367: bug: When the apisix running in kubernetes ,If etcd is stop, the apisix status is still running.

Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #3367:
URL: https://github.com/apache/apisix/issues/3367#issuecomment-822301447


   > So When it have this problem , the apisix should call k8s events api or stdout or stderr.
   
   You can write a component to convert error message in the error log to k8s events API or somewhere else. The APISIX may run in many different ways and there is not always k8s or stdout available.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org