You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@devlake.apache.org by "ankgupta99 (via GitHub)" <gi...@apache.org> on 2023/02/28 09:48:27 UTC
[GitHub] [incubator-devlake] ankgupta99 opened a new issue, #4545: [Bug][Devlake] panic: invalid encKey
ankgupta99 opened a new issue, #4545:
URL: https://github.com/apache/incubator-devlake/issues/4545
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar issues.
### What happened
Devlake container exiting in AWS ECS with following error -
28/2/2023, 3:01:27 pm | panic: invalid encKey | devlake
-- | -- | --
28/2/2023, 3:01:27 pm | goroutine 1 [running]: | devlake
28/2/2023, 3:01:27 pm | time="2023-02-28 09:31:27" level=info msg="plugin loaded zentao" | devlake
28/2/2023, 3:01:27 pm | time="2023-02-28 09:31:27" level=info msg="plugin loaded webhook" | devlake
28/2/2023, 3:01:27 pm | time="2023-02-28 09:31:27" level=info msg="plugin loaded tapd" | devlake
28/2/2023, 3:01:27 pm | time="2023-02-28 09:31:27" level=info msg="plugin loaded starrocks"
### What do you expect to happen
Devlake container stopped in AWS ECS with mentioned error
### How to reproduce
use the ECS task definition to spin up the containers.
### Anything else
_No response_
### Version
main
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] uderik commented on issue #4545: [Bug][Devlake] panic: invalid encKey
Posted by "uderik (via GitHub)" <gi...@apache.org>.
uderik commented on issue #4545:
URL: https://github.com/apache/incubator-devlake/issues/4545#issuecomment-1460284565
Hi, I have the same problem with devlake installed in k8s from helm chart, env file is stored in PVC, and I saw it was updated some days ago; I suspect the key has also been updated.
How could this happen, and how can I restore the data now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh commented on issue #4545: [Bug][Devlake] panic: invalid encKey
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #4545:
URL: https://github.com/apache/incubator-devlake/issues/4545#issuecomment-1455430254
@Startrekzky @likyh We should add a doc to explain the upgrade process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] iholovin commented on issue #4545: [Bug][Devlake] panic: invalid encKey
Posted by "iholovin (via GitHub)" <gi...@apache.org>.
iholovin commented on issue #4545:
URL: https://github.com/apache/incubator-devlake/issues/4545#issuecomment-1468086828
@klesh yeah, that's what I did multiple times — got this error even with the fresh deployment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh commented on issue #4545: [Bug][Devlake] panic: invalid encKey
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #4545:
URL: https://github.com/apache/incubator-devlake/issues/4545#issuecomment-1447902762
I think we need to add a doc to explain how the `encKey` works.
@ankgupta99 I see you have the "Submit PR" checked, may I ask how will you fix the bug? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh commented on issue #4545: [Bug][Devlake] panic: invalid encKey
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #4545:
URL: https://github.com/apache/incubator-devlake/issues/4545#issuecomment-1459344530
@ankgupta99 Did you mount the `.env` file into the containers? the generated `encKey` should be stored in the file, mounting the same file into the new containers or copying the `encKey` should fix your problem as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh commented on issue #4545: [Bug][Devlake] panic: invalid encKey
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #4545:
URL: https://github.com/apache/incubator-devlake/issues/4545#issuecomment-1461421944
@uderik That is weird, `devlake` it self wouldn't regenerate a new key if it already existed.
Are you using the official `helm` repo? @warren830 we have to confirm if it works correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh commented on issue #4545: [Bug][Devlake] panic: invalid encKey
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #4545:
URL: https://github.com/apache/incubator-devlake/issues/4545#issuecomment-1455429343
@iholovin the `.env` must be copied from the existing container volume if you are going to use the existing database since `devlake` writes the auto-generated `encKey` to the file.
The error occurred due to `devlake` trying to decrypt the existing data with the wrong key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh commented on issue #4545: [Bug][Devlake] panic: invalid encKey
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #4545:
URL: https://github.com/apache/incubator-devlake/issues/4545#issuecomment-1455723954
@ankgupta99 @iholovin May I ask which approach did you use to deploy and upgrade? Was it `docker-compose` or `helm`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] uderik commented on issue #4545: [Bug][Devlake] panic: invalid encKey
Posted by "uderik (via GitHub)" <gi...@apache.org>.
uderik commented on issue #4545:
URL: https://github.com/apache/incubator-devlake/issues/4545#issuecomment-1461671711
I did not update anything. I just restarted all the pods.
This is not the first time I've had this situation; I'll look later at what happens there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh closed issue #4545: [Bug][Devlake] panic: invalid encKey
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh closed issue #4545: [Bug][Devlake] panic: invalid encKey
URL: https://github.com/apache/incubator-devlake/issues/4545
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] iholovin commented on issue #4545: [Bug][Devlake] panic: invalid encKey
Posted by "iholovin (via GitHub)" <gi...@apache.org>.
iholovin commented on issue #4545:
URL: https://github.com/apache/incubator-devlake/issues/4545#issuecomment-1453250407
Experiencing the same issue with newly created infrustructure (including the DB) with ECS Task for the backend container:
For this one I had empty `.env` file copied into container:
Interestingly enough: there was no such error for the same exact setup and Postgres Aurora cluster (now I'm using MySQL Aurora)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] iholovin commented on issue #4545: [Bug][Devlake] panic: invalid encKey
Posted by "iholovin (via GitHub)" <gi...@apache.org>.
iholovin commented on issue #4545:
URL: https://github.com/apache/incubator-devlake/issues/4545#issuecomment-1466182254
> @ankgupta99 @iholovin May I ask which approach did you use to deploy and upgrade? Was it docker-compose or helm?
Because of the DevOps policies at my company, I had to deploy containers with Terraform and ECS tasks. For the database, I used Aurora MySQL.
It was working before with Aurora Postgres though 🤔
I checked the MySQL logs and discovered the following error corresponding to backend's requests:
Aborted connection 87 to db: 'devlake' user: 'root' host: '10.204.0.214' (Got an error reading communication packets). (sql_connect.cc:845)
Here is what I used for `aws_rds_cluster`:
```
...
instance_class = "db.r6g.large"
engine = "aurora-mysql"
engine_version = "8.0.mysql_aurora.3.03.0"
```
And `aws_rds_cluster_parameter_group`:
```
...
family = "aurora-mysql8.0"
parameter {
name = "character_set_server"
value = "utf8mb4"
}
parameter {
name = "collation_server"
value = "utf8mb4_bin"
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] ankgupta99 commented on issue #4545: [Bug][Devlake] panic: invalid encKey
Posted by "ankgupta99 (via GitHub)" <gi...@apache.org>.
ankgupta99 commented on issue #4545:
URL: https://github.com/apache/incubator-devlake/issues/4545#issuecomment-1457566103
> @ankgupta99 @iholovin May I ask which approach did you use to deploy and upgrade? Was it `docker-compose` or `helm`?
I took the devlake and config-ui containers details from the docker compose file and then created those 2 containers. For Grafana & mysql the services from aws were used. I was able to get around the error by creating new database.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] warren830 commented on issue #4545: [Bug][Devlake] panic: invalid encKey
Posted by "warren830 (via GitHub)" <gi...@apache.org>.
warren830 commented on issue #4545:
URL: https://github.com/apache/incubator-devlake/issues/4545#issuecomment-1461652781
Hi guys, I just upgraded my devlake pods by helm in minikube node, I found that encKey didnt change. I followed this link to upgrade: https://github.com/apache/incubator-devlake-helm-chart#update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh commented on issue #4545: [Bug][Devlake] panic: invalid encKey
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #4545:
URL: https://github.com/apache/incubator-devlake/issues/4545#issuecomment-1462190269
> I did not update anything. I just restarted all the pods. This is not the first time I've had this situation; I'll look later at what happens there.
Thanks, very much appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh commented on issue #4545: [Bug][Devlake] panic: invalid encKey
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #4545:
URL: https://github.com/apache/incubator-devlake/issues/4545#issuecomment-1467226016
> > @ankgupta99 @iholovin May I ask which approach did you use to deploy and upgrade? Was it docker-compose or helm?
>
> Because of the DevOps policies at my company, I had to deploy containers with Terraform and ECS tasks. For the database, I used Aurora MySQL.
>
> It was working before with Aurora Postgres though 🤔
>
> I checked the MySQL logs and discovered the following error corresponding to backend's requests:
>
> ```
> Aborted connection 87 to db: 'devlake' user: 'root' host: '10.204.0.214' (Got an error reading communication packets). (sql_connect.cc:845)
> ```
>
> Here is what I used for `aws_rds_cluster`:
>
> ```
> ...
>
> instance_class = "db.r6g.large"
> engine = "aurora-mysql"
> engine_version = "8.0.mysql_aurora.3.03.0"
> ```
>
> And `aws_rds_cluster_parameter_group`:
>
> ```
> ...
>
> family = "aurora-mysql8.0"
>
> parameter {
> name = "character_set_server"
> value = "utf8mb4"
> }
>
> parameter {
> name = "collation_server"
> value = "utf8mb4_bin"
> }
> ```
Since you had already switch to `mysql` from `pg`, I suggest that you can wipe out the old deployment and redeploy a new one to see if the problem persists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh commented on issue #4545: [Bug][Devlake] panic: invalid encKey
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #4545:
URL: https://github.com/apache/incubator-devlake/issues/4545#issuecomment-1469505869
@iholovin the `devlake` uses a connection pool with 100 connections, seems like there are some failed connections.
I suspect it is related to the aurora-mysql settings and this article from google seems likely related:
https://dba.stackexchange.com/questions/19135/mysql-error-reading-communication-packets
tag @daniel-hutao , he is working on similar task at this point, please help to reproduce the problem, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org