You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@devlake.apache.org by "samcrang (via GitHub)" <gi...@apache.org> on 2023/03/27 09:16:12 UTC
[GitHub] [incubator-devlake] samcrang opened a new issue, #4785: [Feature][Docker] Docker images built from `main`
samcrang opened a new issue, #4785:
URL: https://github.com/apache/incubator-devlake/issues/4785
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar feature requirement.
### Use case
As someone writing plugins against Devlake I've seen bug fixes and new features appear in `main`. These appear to be cherry-picked to release branches at fairly regular intervals but it seems like things in `main` can end up taking a long time to end up in a released artifact.
It would be very helpful to have `latest` Docker images built from `main`, I'd be willing to accept the fact it would not necessarily meet the usual release standards and might be broken/buggy/eat-all-our-data.
### Description
A Docker images, built from `main`, tagged with `latest`.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh commented on issue #4785: [Feature][Docker] Docker images built from `main`
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #4785:
URL: https://github.com/apache/incubator-devlake/issues/4785#issuecomment-1485022486
Hi, @samcrang Thanks for your suggestion.
Our `main` branch is actually the `development` branch.
Breaking changes or halfway PRs might be merged into `main` at any time, so it is not very appropriate to update the `latest` tag on Docker hub base on the `main` branch.
And yes, you are correct, the `latest` tag is reserved for the most recent release version for users.
As a developer, I don't really see the point of a `dev` docker image, we have to import at least the `core` package anyway, might as well clone the repo for the sake of debugging and code reading. By the way, you don't need to bring up the whole `devlake` server/container for development. Check the [github.go](https://github.com/apache/incubator-devlake/blob/main/backend/plugins/github/github.go) you can run your plugin in a standalone mode which could increase your development pace quite a bit.
```
go run ./path/to/your/plugin.go
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] samcrang commented on issue #4785: [Feature][Docker] Docker images built from `main`
Posted by "samcrang (via GitHub)" <gi...@apache.org>.
samcrang commented on issue #4785:
URL: https://github.com/apache/incubator-devlake/issues/4785#issuecomment-1491666672
Cool, makes sense.
I think what we're going to do (at least in the short term) is build our own image off `main` and deal with any breakages as they happen. I think we're happy with the lack of stability whilst we're needing the very new, unreleased features. Worst case scenario (for us, at least) is that we have to trash the DB and rescrape which isn't too much of a hassle.
Feel free to close this issue—I suspect the usefulness of this request is really limited to people who are using _very_ new features.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] hezyin commented on issue #4785: [Feature][Docker] Docker images built from `main`
Posted by "hezyin (via GitHub)" <gi...@apache.org>.
hezyin commented on issue #4785:
URL: https://github.com/apache/incubator-devlake/issues/4785#issuecomment-1492834019
Hi @samcrang, if you run into any issues/bugs on main, let us know and we'll try to unblock you asap.
Closing this issue for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] samcrang commented on issue #4785: [Feature][Docker] Docker images built from `main`
Posted by "samcrang (via GitHub)" <gi...@apache.org>.
samcrang commented on issue #4785:
URL: https://github.com/apache/incubator-devlake/issues/4785#issuecomment-1488252675
Yeah, I appreciate this is very much of a "nice to have" for me as well as being a very big impact for you with how you build things.
Sounds like @keon94 has some ideas in #4805 though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh commented on issue #4785: [Feature][Docker] Docker images built from `main`
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #4785:
URL: https://github.com/apache/incubator-devlake/issues/4785#issuecomment-1487879470
Yes, I see the pain, it sounds like a common problem all projects are facing.
We may have our `main` branch more stable, and release a `nightly` build for people like you who need new fixes/features faster.
However, doing so would require massive automation/investment and we don't have enough resources at this point. We will get there 😂 someday...
@hezyin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] hezyin closed issue #4785: [Feature][Docker] Docker images built from `main`
Posted by "hezyin (via GitHub)" <gi...@apache.org>.
hezyin closed issue #4785: [Feature][Docker] Docker images built from `main`
URL: https://github.com/apache/incubator-devlake/issues/4785
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh commented on issue #4785: [Feature][Docker] Docker images built from `main`
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #4785:
URL: https://github.com/apache/incubator-devlake/issues/4785#issuecomment-1491461892
> Yeah, I appreciate this is very much of a "nice to have" for me as well as being a very big impact for you with how you build things.
>
> Sounds like @keon94 has some ideas in #4805 though.
It doesn't seem to be related IMO.
1. We should be updating the `latest` tag on the docker hub to the latest `Prod` image or there is a bug. It is not related to a more stable `nightly`(`main`/`dev`) that I was talking about.
2. It is easy to set up an auto-build that releases a `nightly` docker image, but it is hard to guarantee the quality of the build.
One more thing to note: depending on how you configure the `PullPolicy`, using the same tag name (e.g. `nightly`) can be unreliable and dangerous, either not using the latest image or the auto-updated image breaking your local setup. I stop using the `latest` tag on my home server because sometimes I lost track of which version the container is using, not a good idea in the long run. 🥲
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] samcrang commented on issue #4785: [Feature][Docker] Docker images built from `main`
Posted by "samcrang (via GitHub)" <gi...@apache.org>.
samcrang commented on issue #4785:
URL: https://github.com/apache/incubator-devlake/issues/4785#issuecomment-1484826432
Thinking about this a little bit more maybe it's better that it's tagged with something else like `dev`/`development`/`main`/something else. I suspect `latest` will want to be reserved as an additional tag for the most recent released version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] samcrang commented on issue #4785: [Feature][Docker] Docker images built from `main`
Posted by "samcrang (via GitHub)" <gi...@apache.org>.
samcrang commented on issue #4785:
URL: https://github.com/apache/incubator-devlake/issues/4785#issuecomment-1486998007
Hi thanks for getting back to me @klesh,
It's possible I'm doing something unusual (as well as working on stuff that is very new)--let me know if I'm chatting nonsense.
At the moment I'm building a Docker image that copies in and installs the dependencies for our own Python plugins. So our `Dockerfile` ends up looking something like this:
```
FROM apache/devlake:v0.16.0-beta10
RUN apt-get update && \
apt-get update && \
apt-get install -y make build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm \
libncurses5-dev libncursesw5-dev xz-utils tk-dev
RUN cd /tmp && \
wget https://www.python.org/ftp/python/3.11.2/Python-3.11.2.tgz && \
tar -xzvf Python-3.11.2.tgz && \
cd Python-3.11.2/ && \
./configure --with-ensurepip=install && \
make -j4 altinstall
RUN update-alternatives --install /usr/bin/python3 python3 /usr/local/bin/python3.11 1
RUN apt-get install -y python3-venv git libmariadb-dev && \
pip install pipx && \
pipx ensurepath && \
pipx install git+https://github.com/python-poetry/poetry.git
COPY plugins /app/python/plugins
RUN cd /app/python/plugins/hopper && poetry install
...
```
Much of this is due to the fact that [recent versions of `main` contain additional things required for `pydevlake`](https://github.com/apache/incubator-devlake/blob/29f0c5f8baa2c0e8d2efbcf261aa36adb2b8c46a/backend/Dockerfile#L114) whereas [the released versions don't](https://github.com/apache/incubator-devlake/blob/v0.16.0-beta10/backend/Dockerfile#L114).
However, even after that there appears to be a bug when we trigger a run of a Blueprint:
```
devlake-devlake-1 | Wraps: (2) Error running task 1.
devlake-devlake-1 | | Wraps: (2) subtask collectHopperReleases ended unexpectedly
devlake-devlake-1 | | Wraps: (3) remote error response:
devlake-devlake-1 | | | Traceback (most recent call last):
devlake-devlake-1 | | | File "/app/python/plugins/hopper/hopper/main.py", line 50, in <module>
devlake-devlake-1 | | | Hopper.start()
devlake-devlake-1 | | | File "/root/.cache/pypoetry/virtualenvs/hopper-LibYn79P-py3.11/src/incubator-devlake/backend/python/pydevlake/pydevlake/plugin.py", line 255, in start
devlake-devlake-1 | | | fire.Fire(PluginCommands(plugin))
devlake-devlake-1 | | | File "/root/.cache/pypoetry/virtualenvs/hopper-LibYn79P-py3.11/lib/python3.11/site-packages/fire/core.py", line 141, in Fire
devlake-devlake-1 | | | component_trace = _Fire(component, args, parsed_flag_args, context, name)
devlake-devlake-1 | | | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
devlake-devlake-1 | | | File "/root/.cache/pypoetry/virtualenvs/hopper-LibYn79P-py3.11/lib/python3.11/site-packages/fire/core.py", line 466, in _Fire
devlake-devlake-1 | | | component, remaining_args = _CallAndUpdateTrace(
devlake-devlake-1 | | | ^^^^^^^^^^^^^^^^^^^^
devlake-devlake-1 | | | File "/root/.cache/pypoetry/virtualenvs/hopper-LibYn79P-py3.11/lib/python3.11/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
devlake-devlake-1 | | | component = fn(*varargs, **kwargs)
devlake-devlake-1 | | | ^^^^^^^^^^^^^^^^^^^^^^
devlake-devlake-1 | | | File "/root/.cache/pypoetry/virtualenvs/hopper-LibYn79P-py3.11/src/incubator-devlake/backend/python/pydevlake/pydevlake/ipc.py", line 48, in wrapper
devlake-devlake-1 | | | for each in ret:
devlake-devlake-1 | | | File "/root/.cache/pypoetry/virtualenvs/hopper-LibYn79P-py3.11/src/incubator-devlake/backend/python/pydevlake/pydevlake/ipc.py", line 63, in collect
devlake-devlake-1 | | | yield from self._plugin.collect(self._mk_context(ctx), stream)
devlake-devlake-1 | | | ^^^^^^^^^^^^^^^^^^^^^
devlake-devlake-1 | | | File "/root/.cache/pypoetry/virtualenvs/hopper-LibYn79P-py3.11/src/incubator-devlake/backend/python/pydevlake/pydevlake/ipc.py", line 114, in _mk_context
devlake-devlake-1 | | | scope_dict = self._parse(data[\'scope\'])
devlake-devlake-1 | | | ~~~~^^^^^^^^^
devlake-devlake-1 | | | KeyError: \'scope\'
devlake-devlake-1 | | Wraps: (4) exit status 1
devlake-devlake-1 | | Wraps: (5) exit status 1
devlake-devlake-1 | | Error types: (1) *hintdetail.withDetail (2) *hintdetail.withDetail (3) *hintdetail.withDetail (4) *hintdetail.withDetail (5) *exec.ExitError
devlake-devlake-1 | Error types: (1) *hintdetail.withDetail (2) *errors.errorString',`error_name`='Error running pipeline 1.
```
This bug doesn't appear in recent versions of `main` so it appears to have been fixed somewhere already and therefore I've not raised a bug report.
I think the real issue I'm having here isn't that there's no `latest` Docker image but rather that it seems like it can take a long time for things on `main` to end up in a released artifact. I'd love to be able to take advantage of these fixes/features that appear without having to wait until they get cherry-picked for release.
tl;dr: releases tend to lag behind what's on `main`: is there a better way to get these changes in production that don't involve building the Docker images ourselves? Is it even safe to do this? Should I be waiting until things are released officially?
Sam
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org