You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by GitBox <gi...@apache.org> on 2022/07/05 15:16:46 UTC
[GitHub] [incubator-kvrocks] tisonkun opened a new issue, #718: Build docker image with Travis CI
tisonkun opened a new issue, #718:
URL: https://github.com/apache/incubator-kvrocks/issues/718
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-kvrocks/issues) and found no similar issues.
### Motivation
#550 introduces a CI task to build docker image, especially including build on ARM64 which is simulated via QEMU. This task generally takes more than 2 hours to complete and we currently have to run it daily in a dedicate job.
GitHub Actions support only x86 arch at the moment, and among CI platforms ASF provides, Travis CI seems a good fit to run CI tasks on other arches including ARM64. [I ask INFRA how to configure Travis CI settings](https://issues.apache.org/jira/browse/INFRA-23444), and the answer is directly pushing a `.travis.yml` file to the root path.
IIRC @git-hulk @PragmaTwice show your interests in this topic before, you may take a look and give it a try. I may find sometime later this month to give it a try if nobody moves forward.
cc @foxdalas as the original author of #550.
### Solution
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] PragmaTwice commented on issue #718: Build docker image with Travis CI
Posted by GitBox <gi...@apache.org>.
PragmaTwice commented on issue #718:
URL: https://github.com/apache/incubator-kvrocks/issues/718#issuecomment-1175826711
Thanks. I think it maybe unnecessary to build docker images for every commit in a PR: we've fully tested them in other workflows. I think we can consider building docker images for each new commit in the unstable branch and release it as a nightly version (of course we can do some pruning when releasing to ensure that only one nightly image is released per day)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] tisonkun commented on issue #718: Build docker image with Travis CI
Posted by GitBox <gi...@apache.org>.
tisonkun commented on issue #718:
URL: https://github.com/apache/incubator-kvrocks/issues/718#issuecomment-1175641504
@PragmaTwice currently, we run docker build daily due to it's slow. If using travis we can finish the task within an hour, we can run the task per PR. For cron settings, you can comment at the JIRA ticket, asking the best practice from INFRA team.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] tisonkun commented on issue #718: Build docker image with Travis CI
Posted by GitBox <gi...@apache.org>.
tisonkun commented on issue #718:
URL: https://github.com/apache/incubator-kvrocks/issues/718#issuecomment-1175912648
@kezhenxu94 the main argument about using QEMU for kvrocks is that it takes over 2 hours to complete, while I can see that the SkyWalking job take ~20 mins to complete. I have no idea yet how can kvrocks build behaves so different. Perhaps jar files are platform-agnostic while we should build kvrocks from scratch on ARM64 platform.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] kezhenxu94 commented on issue #718: Build docker image with Travis CI
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on issue #718:
URL: https://github.com/apache/incubator-kvrocks/issues/718#issuecomment-1175905750
Personally I'd prefer GitHub Actions over Travis CI as GHA is integrated smoothly in terms of development experience, and you don't need to ask Apache infra team to provide token and configure them in Travis CI (this is usually forbidden by infra team for the sake of security).
> Thanks. I think it maybe unnecessary to build docker images for every commit in a PR: we've fully tested them in other workflows.
I agree we don't need to build Docker images in every commit **in a PR**, not to say pushing/publishing to GitHub Container Registry, ghcr,
> I think we can consider building docker images for each new commit in the unstable branch and release it as a nightly version (of course we can do some pruning when releasing to ensure that only one nightly image is released per day).
In SkyWalking, we build and push a Docker image for every commit in the `master` branch, tagged with the commit hash as well as the `latest` tag, so we'd recommend testers (not users) to use the `:latest` tag when they want to try something new, or help to test the latest codes.
Also, to build multi-arch Docker images, you actually don't need an ARM-arch machine, by leveraging qemu and Docker build kit you can cross build Docker images for all architectures https://github.com/apache/skywalking/blob/47cc2f7bb99fefda37a3adcc85206e1743d32757/.github/workflows/publish-docker.yaml#L55-L58
And to just build for every commit in `master` branch, just set the `on` condition to `branch.master` so the workflow won't be triggered in PR commits
https://github.com/apache/skywalking/blob/47cc2f7bb99fefda37a3adcc85206e1743d32757/.github/workflows/publish-docker.yaml#L19-L22
All the works 👆 can be done on your own, without asking apache infra team to configure tokens, which is impossible in Travis CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] kezhenxu94 commented on issue #718: Build docker image with Travis CI
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on issue #718:
URL: https://github.com/apache/incubator-kvrocks/issues/718#issuecomment-1175943538
From [the logs](https://github.com/apache/incubator-kvrocks/runs/7206849766?check_suite_focus=true#step:6:4651) I can see the time in building ARM Docker image dominates (~2 hours), and [the time in building AMD image is about 20 minutes](https://github.com/apache/incubator-kvrocks/runs/7206849766?check_suite_focus=true#step:6:4150). I remember it indeed takes longer time to build ARM compared to AMD when I tried to build ARM Docker images for apache/pulsar, which includes some C++ codes for their C++ client.
> Perhaps jar files are platform-agnostic so that you can simply copy the generated file into the image
I agree this might be the main reason
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] PragmaTwice commented on issue #718: Build docker image with Travis CI
Posted by GitBox <gi...@apache.org>.
PragmaTwice commented on issue #718:
URL: https://github.com/apache/incubator-kvrocks/issues/718#issuecomment-1175947032
In this workflow run https://github.com/apache/incubator-kvrocks/runs/7206849766?check_suite_focus=true, we can see
build kvrocks on native amd64: 1161.3s ~ 20 min
build kvrocks on QEMU arm64 : 7678.7s ~ 2h 8min
more than x6 times longer in QEMU simulation, it is frustrating 😢
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] kezhenxu94 commented on issue #718: Build docker image with Travis CI
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on issue #718:
URL: https://github.com/apache/incubator-kvrocks/issues/718#issuecomment-1175908395
I think you already have the workflow https://github.com/apache/incubator-kvrocks/blob/c094ea46ae500169dcc3e9b3b28ee6c47ea098bc/.github/workflows/release.yaml#L82-L94 so you only need to add a trigger condition when pushed to main branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] git-hulk commented on issue #718: Build docker image with Travis CI
Posted by GitBox <gi...@apache.org>.
git-hulk commented on issue #718:
URL: https://github.com/apache/incubator-kvrocks/issues/718#issuecomment-1176014979
I'm wondering whether build the arm64 only in QMEMU can reduce the build time or not, and
build the amd64 in previous way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] PragmaTwice commented on issue #718: Build docker image with Travis CI
Posted by GitBox <gi...@apache.org>.
PragmaTwice commented on issue #718:
URL: https://github.com/apache/incubator-kvrocks/issues/718#issuecomment-1175299296
https://docs.travis-ci.com/user/cron-jobs/
Seems cron jobs in Travis CI can only be set on web UI rather than `.travis.yml`, which maybe more hard since we have no permission to access Travis web UI of the Apache org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] tisonkun commented on issue #718: Build docker image with Travis CI
Posted by GitBox <gi...@apache.org>.
tisonkun commented on issue #718:
URL: https://github.com/apache/incubator-kvrocks/issues/718#issuecomment-1175893107
> building docker images for each new commit in the unstable branch and release it as a nightly version
That's possible. Actually SkyWalking provides such features along with GitHub Packages (https://github.com/orgs/apache/packages?repo_name=skywalking). Also they support build on ARM64 arch while doesn't involve Travis CI dependency. It should be worth to investigate.
cc @kezhenxu94 could you share some background how SkyWalking implements the workflow here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] PragmaTwice commented on issue #718: Build docker image with Travis CI
Posted by GitBox <gi...@apache.org>.
PragmaTwice commented on issue #718:
URL: https://github.com/apache/incubator-kvrocks/issues/718#issuecomment-1179694208
> You can use cross-compiling and just copy binary inside arm-based container. It will be faster
Thanks. I think it is a greate idea, but I do not know if it will be more complicated than the travis ci way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] foxdalas commented on issue #718: Build docker image with Travis CI
Posted by GitBox <gi...@apache.org>.
foxdalas commented on issue #718:
URL: https://github.com/apache/incubator-kvrocks/issues/718#issuecomment-1179693146
You can use cross-compiling and just copy binary inside arm-based container. It will be faster
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] tisonkun commented on issue #718: Build docker image with Travis CI
Posted by GitBox <gi...@apache.org>.
tisonkun commented on issue #718:
URL: https://github.com/apache/incubator-kvrocks/issues/718#issuecomment-1175200448
Perhaps useful references:
* https://docs.travis-ci.com/user/docker/
* https://docs.travis-ci.com/user/installing-dependencies/
* https://docs.travis-ci.com/user/languages/cpp/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] tisonkun commented on issue #718: Build docker image with Travis CI
Posted by GitBox <gi...@apache.org>.
tisonkun commented on issue #718:
URL: https://github.com/apache/incubator-kvrocks/issues/718#issuecomment-1179742444
You can review #724 for an attempt.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] tisonkun closed issue #718: Build docker image with Travis CI
Posted by GitBox <gi...@apache.org>.
tisonkun closed issue #718: Build docker image with Travis CI
URL: https://github.com/apache/incubator-kvrocks/issues/718
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org