You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "mistercrunch (via GitHub)" <gi...@apache.org> on 2024/07/29 22:21:01 UTC
[PR] chore(docker): move mysql os-level deps (GPL) to dev image only [superset]
mistercrunch opened a new pull request, #29771:
URL: https://github.com/apache/superset/pull/29771
Since mysql client libs are GPL-licensed, we don't want to package them in our default images. Prior to this PR, the `default-libmysqlclient-dev` apt-get package would get installed in the `lean` docker image, though the pypi `mysqlclient` package that depends on this os-level dep was only installed in the `dev` layer.
After this PR, all GPL/mysql related package won't be in `lean`, but still be in `dev` for convenience and CI script that test against mysql.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore(docker): move mysql os-level deps (GPL) to dev image only [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on PR #29771:
URL: https://github.com/apache/superset/pull/29771#issuecomment-2261357135
Right, not sure what's up with it, it looks like the job is getting killed, presumably because it's abusing memory usage (?) The confusing thing is we're actually installing less things in the step that fails.
I tried a few things already:
- setting up swap space, doesn't seem to help, result of following a GPT-generated idea, I'll probably remove that part of the PR
- was looking for ways to reduce the parallelism in the docker build, currently it's building the `superset-node` (super memory heavy) while it builds the lean layer (`pip install requirements/base.txt`), but didn't find a clear/easy way to do it. Was looking for either a docker CLI flag or some env var, but nothing seemed to work, GPT hallucinated a few solutions but that didn't work
I'm wondering what in this PR is pushing the memory usage (assuming that's why the job gets killed) over the edge, and it seems to indicate that the `master` is probably fragile and on the edge too, meaning that this CI step is likely to fail on master soon too next time we touch something related.
Next steps/ideas:
- maybe running a docker command to build the superset-node target first, and then push for the other command which would reuse the cached layers from the first command
- find some other way to limit parallelism
- find a way to get more memory in GHA?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore(docker): move mysql os-level deps (GPL) to dev image only [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #29771:
URL: https://github.com/apache/superset/pull/29771#issuecomment-2259273239
Curious to see if `--no-cache-dir` will affect our PyPI stats and smooth things out at all.
Also, do you have any sense of what's causing the memory increase to the point of needing swap space? Probably unrelated to this, but a little disconcerting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore(docker): move mysql os-level deps (GPL) to dev image only [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas merged PR #29771:
URL: https://github.com/apache/superset/pull/29771
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org