You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/05/18 08:19:35 UTC
[GitHub] [flink-docker] wangxiyuan opened a new pull request #23: [FLINK-14241][test]Add arm64 support for docker e2e test
wangxiyuan opened a new pull request #23:
URL: https://github.com/apache/flink-docker/pull/23
The docker image `openjdk:8-jre` only works for amd64.
When running test on arm64, use `arm64v8/openjdk:8-jre` instead.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-docker] wangxiyuan closed pull request #23: [FLINK-14241][test]Add arm64 support for docker e2e test
Posted by GitBox <gi...@apache.org>.
wangxiyuan closed pull request #23:
URL: https://github.com/apache/flink-docker/pull/23
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-docker] rmetzger commented on pull request #23: [FLINK-14241][test]Add arm64 support for docker e2e test
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #23:
URL: https://github.com/apache/flink-docker/pull/23#issuecomment-688850845
I'm not sure if this PR works as-is. The updated generator doesn't seem to set the architecture flag correctly: https://github.com/apache/flink-docker/pull/23/files#diff-0c5f825c10334c2eff0a90687f513c69R93
also, the architecture seems to depend on the architecture of the machine executing the `generate.sh` script. But this script only generates the Dockerfile and metadata file, not the actual docker image.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-docker] wangxiyuan closed pull request #23: [FLINK-14241][test]Add arm64 support for docker e2e test
Posted by GitBox <gi...@apache.org>.
wangxiyuan closed pull request #23:
URL: https://github.com/apache/flink-docker/pull/23
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-docker] wangxiyuan edited a comment on pull request #23: [FLINK-14241][test]Add arm64 support for docker e2e test
Posted by GitBox <gi...@apache.org>.
wangxiyuan edited a comment on pull request #23:
URL: https://github.com/apache/flink-docker/pull/23#issuecomment-689237514
> I'm not sure if this PR works as-is. The updated generator doesn't seem to set the architecture flag correctly: https://github.com/apache/flink-docker/pull/23/files#diff-0c5f825c10334c2eff0a90687f513c69R93
>
> also, the architecture seems to depend on the architecture of the machine executing the `generate.sh` script. But this script only generates the Dockerfile and metadata file, not the actual docker image.
Yeah, many docker images support multi arch now, for example `openjdk:11-jre` [1]. For this kind of image, we don't need to change any code here of cause. Different machine use different image automatically.
But there are still some images don't support arm64, for example, the image `openjdk:8-jre`[2] which the e2e test uses. For this kind of image, we should use the image from another official repo `arm64v8`[3].
Now openlab runs flink e2e test on arm64 with my forked `flink-docker` repo.[4] And the test runs well[5]
[1]: https://hub.docker.com/layers/openjdk/library/openjdk/11-jre/images/sha256-9ea29bab2ef2eebaf0d9693cfab073fbdc560f2abe37be5885eede04a41188f8?context=explore
[2]: https://hub.docker.com/layers/openjdk/library/openjdk/8-jre/images/sha256-41aefd23ef94a79df24c0d534d80eefd3c1fbc4f3810f3c2211dffb2e2736fe1?context=explore
[3]: https://hub.docker.com/layers/arm64v8/openjdk/8-jre/images/sha256-b915aa8067ed0f7bdadb94701109b4408ee6d90642181c87ad37ff4036e0aed5?context=explore
[4]: https://github.com/wangxiyuan/flink-docker/blob/dev-master/generator.sh#L18-L22
[5]: http://status.openlabtesting.org/builds?job_name=flink-end-to-end-test-cron-hadoop313
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-docker] wangxiyuan commented on pull request #23: [FLINK-14241][test]Add arm64 support for docker e2e test
Posted by GitBox <gi...@apache.org>.
wangxiyuan commented on pull request #23:
URL: https://github.com/apache/flink-docker/pull/23#issuecomment-641770430
@zentol can you take a look? Thanks very much
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-docker] rmetzger commented on pull request #23: [FLINK-14241][test]Add arm64 support for docker e2e test -- work in progress
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #23:
URL: https://github.com/apache/flink-docker/pull/23#issuecomment-1014448068
The latest frocksdb version Flink is using (https://search.maven.org/artifact/com.ververica/frocksdbjni/6.20.3-ververica-1.0/jar) supports Linux aarch64, so you should be able to run Flink on Docker on an M1 mac book (because Docker for Mac is effectively running in a (arm) Linux VM)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-docker] rmetzger commented on pull request #23: [FLINK-14241][test]Add arm64 support for docker e2e test
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #23:
URL: https://github.com/apache/flink-docker/pull/23#issuecomment-689347085
I'm sorry that I have to say it again, but I don't think the current approach is correct.
The `generate.sh` script generates a Dockerfile and metadata file, then this file is used to create a stackbrew file, which is committed here: https://github.com/docker-library/official-images/blob/master/library/flink.
Then DockerHub does some magic to turn this into the official images.
If we want to add support for arm64 to the official Flink docker images, we probably need to generate an additional Dockerfile and metadata file with an updated FROM, and a correct `release.metadata` file.
I believe that your change works on ARM64 with the Flink e2e tests, but this repository is mostly used for building the official Flink docker images, and we need to make sure we are not breaking them with this change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-docker] rmetzger commented on pull request #23: [FLINK-14241][test]Add arm64 support for docker e2e test -- work in progress
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #23:
URL: https://github.com/apache/flink-docker/pull/23#issuecomment-689357796
Great! Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-docker] wangxiyuan commented on pull request #23: [FLINK-14241][test]Add arm64 support for docker e2e test
Posted by GitBox <gi...@apache.org>.
wangxiyuan commented on pull request #23:
URL: https://github.com/apache/flink-docker/pull/23#issuecomment-689356241
Oh, I see. I didn't realize that this is used for offcial image build as well.
Yeah, If so, this PR is not correct. I'll find some way to only apply for e2e test. Thanks!!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-docker] wangxiyuan commented on pull request #23: [FLINK-14241][test]Add arm64 support for docker e2e test
Posted by GitBox <gi...@apache.org>.
wangxiyuan commented on pull request #23:
URL: https://github.com/apache/flink-docker/pull/23#issuecomment-689237514
> I'm not sure if this PR works as-is. The updated generator doesn't seem to set the architecture flag correctly: https://github.com/apache/flink-docker/pull/23/files#diff-0c5f825c10334c2eff0a90687f513c69R93
>
> also, the architecture seems to depend on the architecture of the machine executing the `generate.sh` script. But this script only generates the Dockerfile and metadata file, not the actual docker image.
Acutally many docker images support multi arch currently. For example `openjdk:11-jre`
https://github.com/wangxiyuan/flink-docker/blob/dev-master/generator.sh#L18-L22
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-docker] nferrario commented on pull request #23: [FLINK-14241][test]Add arm64 support for docker e2e test -- work in progress
Posted by GitBox <gi...@apache.org>.
nferrario commented on pull request #23:
URL: https://github.com/apache/flink-docker/pull/23#issuecomment-851507351
RocksDB is not currently being built for ARM. We had to manually compile it and override it in the classpath. Kinda hacky but it's the only solution we have until Flink supports it officially.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-docker] wangxiyuan commented on pull request #23: [FLINK-14241][test]Add arm64 support for docker e2e test
Posted by GitBox <gi...@apache.org>.
wangxiyuan commented on pull request #23:
URL: https://github.com/apache/flink-docker/pull/23#issuecomment-643870330
@rmetzger could you please take a look? This block arm64 test for a while. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-docker] wangxiyuan edited a comment on pull request #23: [FLINK-14241][test]Add arm64 support for docker e2e test
Posted by GitBox <gi...@apache.org>.
wangxiyuan edited a comment on pull request #23:
URL: https://github.com/apache/flink-docker/pull/23#issuecomment-689237514
> I'm not sure if this PR works as-is. The updated generator doesn't seem to set the architecture flag correctly: https://github.com/apache/flink-docker/pull/23/files#diff-0c5f825c10334c2eff0a90687f513c69R93
>
> also, the architecture seems to depend on the architecture of the machine executing the `generate.sh` script. But this script only generates the Dockerfile and metadata file, not the actual docker image.
Yeah, many docker images support multi arch now, for example `openjdk:11-jre` [1]. For this kind of image, we don't need to change any code here of cause. Different machine use different image automatically.
But there are still some images don't support arm64, for example, the image `openjdk:8-jre`[2] which the e2e test uses. For this kind of image, we should use the image from another official repo `arm64v8`[3].
Now openlab runs flink e2e test on arm64 with my forked `flink-docker` repo.[4] And the test runs well[5]
P.S. In openlab test, the `swr.ap-southeast-3.myhuaweicloud.com/openlab/arm64v8` is just a mirror for `arm64v8` to speed up the test.
[1]: https://hub.docker.com/layers/openjdk/library/openjdk/11-jre/images/sha256-9ea29bab2ef2eebaf0d9693cfab073fbdc560f2abe37be5885eede04a41188f8?context=explore
[2]: https://hub.docker.com/layers/openjdk/library/openjdk/8-jre/images/sha256-41aefd23ef94a79df24c0d534d80eefd3c1fbc4f3810f3c2211dffb2e2736fe1?context=explore
[3]: https://hub.docker.com/layers/arm64v8/openjdk/8-jre/images/sha256-b915aa8067ed0f7bdadb94701109b4408ee6d90642181c87ad37ff4036e0aed5?context=explore
[4]: https://github.com/wangxiyuan/flink-docker/blob/dev-master/generator.sh#L18-L22
[5]: http://status.openlabtesting.org/builds?job_name=flink-end-to-end-test-cron-hadoop313
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org