You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@devlake.apache.org by "DarkoDevops (via GitHub)" <gi...@apache.org> on 2023/05/15 12:12:22 UTC

[GitHub] [incubator-devlake-helm-chart] DarkoDevops opened a new issue, #131: Helm Chart installation on AWS EKS: Unable to attach or mount volumes

DarkoDevops opened a new issue, #131:
URL: https://github.com/apache/incubator-devlake-helm-chart/issues/131

   I installed DevLake using Helm chart on my AWS EKS cluster. Used previous instructions from ticket: 
   https://github.com/apache/incubator-devlake-helm-chart/issues/87 but, without success. Still not working. 
   
   Helm values.yaml looks like: 
   
   ```
   replicaCount: 1
   imageTag: v0.16.0-beta17
   
   mysql:
     useExternal: false
     externalServer: 127.0.0.1
     externalPort: 3306
     username: merico
     password: merico
     database: lake
     rootPassword: admin
     storage:
       class: "gp2"
       size: 50Gi
   
     image:
       repository: mysql
       tag: 8
       pullPolicy: IfNotPresent
   
     resources: {}
   
     nodeSelector: {}
     tolerations: []
   
     affinity: {}
   
   
   grafana:
     image:
       repository: apache/devlake-dashboard
       pullPolicy: Always
   
     useExternal: false
   
     externalUrl: ''
   
     resources: {}
   
     nodeSelector: {}
   
     tolerations: []
   
     affinity: {}
   
   
   lake:
     image:
       repository: apache/devlake
       pullPolicy: Always
     storage:
       class: "gp2"
       size: 100Mi
     dotenv:
       API_TIMEOUT: "120s"
       API_RETRY: "3"
       API_REQUESTS_PER_HOUR: "10000"
       PIPELINE_MAX_PARALLEL: "1"
       IN_SECURE_SKIP_VERIFY: "false"
   
     hostNetwork: false
   
     resources: {}
   
     nodeSelector: {}
   
     tolerations: []
   
     affinity: {}
   
     loggingDir: "/app/logs"
     loggingLevel: "info"
   
   ui:
     image:
       repository: apache/devlake-config-ui
       pullPolicy: Always
   
     resources: {}
   
     nodeSelector: {}
   
     tolerations: []
   
     affinity: {}
   
     basicAuth:
       enabled: false
       user: admin
       password: admin
       useSecret: false
       autoCreateSecret: true
       secretName: devlake-auth
   
   
   alpine:
     image:
       repository: alpine
       tag: 3.16
       pullPolicy: IfNotPresent
   
   service:
     type: NodePort
     uiPort: 32001
     grafanaPort : 32002
   
   ingress:
     enabled: true
     enableHttps: false
     className: "alb"
     prefix: /
     tlsSecretName: ""
     httpPort: 80
     httpsPort: 443
     useDefaultNginx: false
     annotations:
       alb.ingress.kubernetes.io/scheme: internet-facing
       alb.ingress.kubernetes.io/target-type: ip
   
   option:
     localtime: /etc/localtime
   
     database: mysql
   
     useConnectionDetailsSecret: false
   
     connectionSecretName: devlake-db-connection
   
     autoCreateSecret: true
   
   ```
   
   I get multiple errors like: 
   ```
   Unable to attach or mount volumes: unmounted volumes=[devlake-lake-config], unattached volumes=[devlake-lake-localtime kube-api-access-r7dgn devlake-lake-config]: timed out waiting for the condition
   
   Unable to attach or mount volumes: unmounted volumes=[devlake-lake-config], unattached volumes=[kube-api-access-r7dgn devlake-lake-config devlake-lake-localtime]: timed out waiting for the condition
   
   Unable to attach or mount volumes: unmounted volumes=[devlake-lake-config], unattached volumes=[devlake-lake-config devlake-lake-localtime kube-api-access-r7dgn]: timed out waiting for the condition
   
   AttachVolume.Attach failed for volume "pvc-09b75f09-f0e4-4700-b42c-e9f901a13bf2" : volume attachment is being deleted
   
   MountVolume.WaitForAttach failed for volume "pvc-09b75f09-f0e4-4700-b42c-e9f901a13bf2" : volume attachment is being deleted
   ```
   
   Also tried to set storage class to empty (default) and same behaviour. Any idea?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake-helm-chart] DarkoDevops commented on issue #131: Helm Chart installation on AWS EKS: Unable to attach or mount volumes

Posted by "DarkoDevops (via GitHub)" <gi...@apache.org>.
DarkoDevops commented on issue #131:
URL: https://github.com/apache/incubator-devlake-helm-chart/issues/131#issuecomment-1552906498

   This is what I have of the pods for ebs:
   
   ```
   ebs-csi-controller-78bdbf98ff-574px   5/5     Running   0            2d5h
   ebs-csi-controller-78bdbf98ff-hwktx   5/5     Running   2 (8d ago)   8d
   ebs-csi-node-46nh7                    3/3     Running   0            8d
   ebs-csi-node-dthdq                    3/3     Running   0            2d5h
   ebs-csi-node-fpz98                    3/3     Running   0            8d
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake-helm-chart] BrookeKatalon commented on issue #131: Helm Chart installation on AWS EKS: Unable to attach or mount volumes

Posted by "BrookeKatalon (via GitHub)" <gi...@apache.org>.
BrookeKatalon commented on issue #131:
URL: https://github.com/apache/incubator-devlake-helm-chart/issues/131#issuecomment-1557648510

   Seeing the same issue with imageTag: v0.17.0-beta5
   
   containers have errors like:
   `MountVolume.SetUp failed for volume "devlake-mysql-localtime" : hostPath type check failed: /etc/localtime is not a file`
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake-helm-chart] JorgeGar commented on issue #131: Helm Chart installation on AWS EKS: Unable to attach or mount volumes

Posted by "JorgeGar (via GitHub)" <gi...@apache.org>.
JorgeGar commented on issue #131:
URL: https://github.com/apache/incubator-devlake-helm-chart/issues/131#issuecomment-1552713249

   I'm running similar values without any issues. I have mysql in an external database, but that's not the one giving you issues, as per what I see in your comment @DarkoDevops. The key difference I guess is that I have aws ebs csi driver installed in the cluster, rather than it being an add-on. Could you try and get logs from it?
   Also, which EKS version are you using?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake-helm-chart] DarkoDevops commented on issue #131: Helm Chart installation on AWS EKS: Unable to attach or mount volumes

Posted by "DarkoDevops (via GitHub)" <gi...@apache.org>.
DarkoDevops commented on issue #131:
URL: https://github.com/apache/incubator-devlake-helm-chart/issues/131#issuecomment-1552581051

   @IronCore864 I will run it and post it. 
   
   Yes, I have aws-ebs-csi-driver addon. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake-helm-chart] JorgeGar commented on issue #131: Helm Chart installation on AWS EKS: Unable to attach or mount volumes

Posted by "JorgeGar (via GitHub)" <gi...@apache.org>.
JorgeGar commented on issue #131:
URL: https://github.com/apache/incubator-devlake-helm-chart/issues/131#issuecomment-1552901754

   > @JorgeGar It's 1.25
   > 
   > It's installed, in cluster, also, added policy, not the add-on.
   
   Have you checked the logs of the attacher?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake-helm-chart] DarkoDevops commented on issue #131: Helm Chart installation on AWS EKS: Unable to attach or mount volumes

Posted by "DarkoDevops (via GitHub)" <gi...@apache.org>.
DarkoDevops commented on issue #131:
URL: https://github.com/apache/incubator-devlake-helm-chart/issues/131#issuecomment-1552895816

   @JorgeGar 
   It's 1.25 
   
   It's installed, in cluster, also, added policy, not the add-on. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake-helm-chart] IronCore864 commented on issue #131: Helm Chart installation on AWS EKS: Unable to attach or mount volumes

Posted by "IronCore864 (via GitHub)" <gi...@apache.org>.
IronCore864 commented on issue #131:
URL: https://github.com/apache/incubator-devlake-helm-chart/issues/131#issuecomment-1552564847

   @DarkoDevops can you run kubectl get/describe pv/pvc and see what you can get from the output?
   
   A wild guess: do you have the aws-ebs-csi-driver addon enabled in your EKS?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake-helm-chart] DarkoDevops commented on issue #131: Helm Chart installation on AWS EKS: Unable to attach or mount volumes

Posted by "DarkoDevops (via GitHub)" <gi...@apache.org>.
DarkoDevops commented on issue #131:
URL: https://github.com/apache/incubator-devlake-helm-chart/issues/131#issuecomment-1552677445

   pv/pvc describe: 
   
   ```
   pvc-09b75f09-f0e4-4700-b42c-e9f901a13bf2   1Gi        RWO            Delete           Bound    default/devlake-lake-config-devlake-lake-0   gp2                     65d
   pvc-16438fe5-4163-441b-be46-2a9d8920b99f   50Gi       RWO            Delete           Bound    default/devlake-mysql-data-devlake-mysql-0   gp2                     65d
   ```
   
   ```
   Name:              pvc-09b75f09-f0e4-4700-b42c-e9f901a13bf2
   Labels:            topology.kubernetes.io/region=eu-central-1
                      topology.kubernetes.io/zone=eu-central-1a
   Annotations:       kubernetes.io/createdby: aws-ebs-dynamic-provisioner
                      pv.kubernetes.io/bound-by-controller: yes
                      pv.kubernetes.io/migrated-to: ebs.csi.aws.com
                      pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
   Finalizers:        [kubernetes.io/pv-protection external-attacher/ebs-csi-aws-com]
   StorageClass:      gp2
   Status:            Bound
   Claim:             default/devlake-lake-config-devlake-lake-0
   Reclaim Policy:    Delete
   Access Modes:      RWO
   VolumeMode:        Filesystem
   Capacity:          1Gi
   Node Affinity:     
     Required Terms:  
       Term 0:        topology.kubernetes.io/zone in [eu-central-1a]
                      topology.kubernetes.io/region in [eu-central-1]
   Message:           
   Source:
       Type:       AWSElasticBlockStore (a Persistent Disk resource in AWS)
       VolumeID:   aws://eu-central-1a/vol-0b203af215cc352d9
       FSType:     ext4
       Partition:  0
       ReadOnly:   false
   Events:         <none>
   ```
   
   ```
   Name:              pvc-16438fe5-4163-441b-be46-2a9d8920b99f
   Labels:            topology.kubernetes.io/region=eu-central-1
                      topology.kubernetes.io/zone=eu-central-1b
   Annotations:       kubernetes.io/createdby: aws-ebs-dynamic-provisioner
                      pv.kubernetes.io/bound-by-controller: yes
                      pv.kubernetes.io/migrated-to: ebs.csi.aws.com
                      pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
   Finalizers:        [kubernetes.io/pv-protection external-attacher/ebs-csi-aws-com]
   StorageClass:      gp2
   Status:            Bound
   Claim:             default/devlake-mysql-data-devlake-mysql-0
   Reclaim Policy:    Delete
   Access Modes:      RWO
   VolumeMode:        Filesystem
   Capacity:          50Gi
   Node Affinity:     
     Required Terms:  
       Term 0:        topology.kubernetes.io/zone in [eu-central-1b]
                      topology.kubernetes.io/region in [eu-central-1]
   Message:           
   Source:
       Type:       AWSElasticBlockStore (a Persistent Disk resource in AWS)
       VolumeID:   aws://eu-central-1b/vol-07a37b8cc48ea301f
       FSType:     ext4
       Partition:  0
       ReadOnly:   false
   Events:         <none>
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake-helm-chart] JorgeGar commented on issue #131: Helm Chart installation on AWS EKS: Unable to attach or mount volumes

Posted by "JorgeGar (via GitHub)" <gi...@apache.org>.
JorgeGar commented on issue #131:
URL: https://github.com/apache/incubator-devlake-helm-chart/issues/131#issuecomment-1552911328

   Honestly, at this point, I'm following the [troubleshooting page from AWS](https://repost.aws/knowledge-center/eks-troubleshoot-ebs-volume-mounts), but...
   On the ebs-csi-controller, you should see a container called "csi-attacher". Try getting logs from there.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-devlake-helm-chart] klesh commented on issue #131: Helm Chart installation on AWS EKS: Unable to attach or mount volumes

Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #131:
URL: https://github.com/apache/incubator-devlake-helm-chart/issues/131#issuecomment-1552531884

   @IronCore864 @JorgeGar would  you mind take a look at this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org