You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Di Xu <sc...@gmail.com> on 2020/09/23 05:29:23 UTC
How to run k8s-related end-to-end tests locally?
Hi,
I tried to run single k8s e-2-e test locally on my macos laptop (a minikube
vm has been started manually), and here is the command I run from the
source root.
```
$ flink-end-to-end-tests/run-single-test.sh
flink-end-to-end-tests/test-scripts/test_kubernetes_application.sh
```
Then it failed at the image build stage while downloading the flink.tgz
file from a local fileserver.
```
Step 10/14 : RUN set -ex; wget -nv -O flink.tgz "$FLINK_TGZ_URL"; if
[ "$CHECK_GPG" = "true" ]; then wget -nv -O flink.tgz.asc
"$FLINK_ASC_URL"; export GNUPGHOME="$(mktemp -d)"; for server in
ha.pool.sks-keyservers.net $(shuf -e hkp://
p80.pool.sks-keyservers.net:80
keyserver.ubuntu.com hkp://
keyserver.ubuntu.com:80 pgp.mit.edu) ; do
gpg --batch --keyserver "$server" --recv-keys "$GPG_KEY" && break || : ;
done && gpg --batch --verify flink.tgz.asc flink.tgz; gpgconf
--kill all; rm -rf "$GNUPGHOME" flink.tgz.asc; fi; tar -xf
flink.tgz --strip-components=1; rm flink.tgz; chown -R flink:flink .;
---> Running in 2225f4ea3406
+ wget -nv -O flink.tgz localhost:9999/flink.tgz
failed: Connection refused.
The command '/bin/sh -c set -ex; wget -nv -O flink.tgz "$FLINK_TGZ_URL";
if [ "$CHECK_GPG" = "true" ]; then wget -nv -O flink.tgz.asc
"$FLINK_ASC_URL"; export GNUPGHOME="$(mktemp -d)"; for server in
ha.pool.sks-keyservers.net $(shuf -e hkp://
p80.pool.sks-keyservers.net:80
keyserver.ubuntu.com hkp://
keyserver.ubuntu.com:80 pgp.mit.edu) ; do
gpg --batch --keyserver "$server" --recv-keys "$GPG_KEY" && break || : ;
done && gpg --batch --verify flink.tgz.asc flink.tgz; gpgconf
--kill all; rm -rf "$GNUPGHOME" flink.tgz.asc; fi; tar -xf
flink.tgz --strip-components=1; rm flink.tgz; chown -R flink:flink
.;' returned a non-zero code: 4
```
I checked the source code and found the python fileserver is here:
https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_docker.sh#L44
Then it uses 'docker build --net==host' to build image:
https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_docker.sh#L53
I am confused here, --net=host only makes services at the minikube vm
available to the docker build process, however, the python fileserver is
running on my local env rather than the minikube vm, how the docker build
process access that fileserver?
Should I:
- run the test script inside the minikube vm (so that the fileserver will
run on it), or
- did I miss some configuration to my minikube vm?
Thanks in advance.
Re: How to run k8s-related end-to-end tests locally?
Posted by Di Xu <sc...@gmail.com>.
Hi Wang Yang，
Thanks for your quick reply. I followed the discussion in FLINK-17857 and
now understand the problem and solution.
I will fix it locally and please let me know if there is anything I can
help with this issue.
Yang Wang <da...@gmail.com>于2020年9月24日 周四上午12:12写道：
> Hi Xu Di,
>
> Sorry for the inconvenience. This is a known issue FLINK-17857[1]. After we
> migrate all the e2e tests
> from flink-container/docker to apache/flink-docker, the docker and k8s
> related e2e tests could not work.
> I will revive this ticket and fix it ASAP.
>
> [1]. https://issues.apache.org/jira/browse/FLINK-17857
>
> Best,
> Yang
>
>
> Di Xu <sc...@gmail.com> 于2020年9月23日周三 下午4:54写道：
>
> > Hi,
> >
> > I tried to run single k8s e-2-e test locally on my macos laptop (a
> minikube
> > vm has been started manually), and here is the command I run from the
> > source root.
> >
> > ```
> > $ flink-end-to-end-tests/run-single-test.sh
> > flink-end-to-end-tests/test-scripts/test_kubernetes_application.sh
> > ```
> > Then it failed at the image build stage while downloading the flink.tgz
> > file from a local fileserver.
> >
> > ```
> > Step 10/14 : RUN set -ex; wget -nv -O flink.tgz "$FLINK_TGZ_URL";
> if
> > [ "$CHECK_GPG" = "true" ]; then wget -nv -O flink.tgz.asc
> > "$FLINK_ASC_URL"; export GNUPGHOME="$(mktemp -d)"; for server in
> > ha.pool.sks-keyservers.net $(shuf -e hkp://
> > p80.pool.sks-keyservers.net:80
> > keyserver.ubuntu.com hkp://
> > keyserver.ubuntu.com:80 pgp.mit.edu) ; do
> > gpg --batch --keyserver "$server" --recv-keys "$GPG_KEY" && break || :
> ;
> > done && gpg --batch --verify flink.tgz.asc flink.tgz; gpgconf
> > --kill all; rm -rf "$GNUPGHOME" flink.tgz.asc; fi; tar -xf
> > flink.tgz --strip-components=1; rm flink.tgz; chown -R flink:flink
> .;
> > ---> Running in 2225f4ea3406
> > + wget -nv -O flink.tgz localhost:9999/flink.tgz
> > failed: Connection refused.
> > The command '/bin/sh -c set -ex; wget -nv -O flink.tgz
> "$FLINK_TGZ_URL";
> > if [ "$CHECK_GPG" = "true" ]; then wget -nv -O flink.tgz.asc
> > "$FLINK_ASC_URL"; export GNUPGHOME="$(mktemp -d)"; for server in
> > ha.pool.sks-keyservers.net $(shuf -e hkp://
> > p80.pool.sks-keyservers.net:80
> > keyserver.ubuntu.com hkp://
> > keyserver.ubuntu.com:80 pgp.mit.edu) ; do
> > gpg --batch --keyserver "$server" --recv-keys "$GPG_KEY" && break || :
> ;
> > done && gpg --batch --verify flink.tgz.asc flink.tgz; gpgconf
> > --kill all; rm -rf "$GNUPGHOME" flink.tgz.asc; fi; tar -xf
> > flink.tgz --strip-components=1; rm flink.tgz; chown -R flink:flink
> > .;' returned a non-zero code: 4
> > ```
> >
> > I checked the source code and found the python fileserver is here:
> >
> >
> >
> https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_docker.sh#L44
> >
> > Then it uses 'docker build --net==host' to build image:
> >
> >
> >
> https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_docker.sh#L53
> >
> > I am confused here, --net=host only makes services at the minikube vm
> > available to the docker build process, however, the python fileserver is
> > running on my local env rather than the minikube vm, how the docker build
> > process access that fileserver?
> >
> > Should I:
> > - run the test script inside the minikube vm (so that the fileserver will
> > run on it), or
> > - did I miss some configuration to my minikube vm?
> >
> > Thanks in advance.
> >
>
Re: How to run k8s-related end-to-end tests locally?
Posted by Yang Wang <da...@gmail.com>.
Hi Xu Di,
Sorry for the inconvenience. This is a known issue FLINK-17857[1]. After we
migrate all the e2e tests
from flink-container/docker to apache/flink-docker, the docker and k8s
related e2e tests could not work.
I will revive this ticket and fix it ASAP.
[1]. https://issues.apache.org/jira/browse/FLINK-17857
Best,
Yang
Di Xu <sc...@gmail.com> 于2020年9月23日周三 下午4:54写道：
> Hi,
>
> I tried to run single k8s e-2-e test locally on my macos laptop (a minikube
> vm has been started manually), and here is the command I run from the
> source root.
>
> ```
> $ flink-end-to-end-tests/run-single-test.sh
> flink-end-to-end-tests/test-scripts/test_kubernetes_application.sh
> ```
> Then it failed at the image build stage while downloading the flink.tgz
> file from a local fileserver.
>
> ```
> Step 10/14 : RUN set -ex; wget -nv -O flink.tgz "$FLINK_TGZ_URL"; if
> [ "$CHECK_GPG" = "true" ]; then wget -nv -O flink.tgz.asc
> "$FLINK_ASC_URL"; export GNUPGHOME="$(mktemp -d)"; for server in
> ha.pool.sks-keyservers.net $(shuf -e hkp://
> p80.pool.sks-keyservers.net:80
> keyserver.ubuntu.com hkp://
> keyserver.ubuntu.com:80 pgp.mit.edu) ; do
> gpg --batch --keyserver "$server" --recv-keys "$GPG_KEY" && break || : ;
> done && gpg --batch --verify flink.tgz.asc flink.tgz; gpgconf
> --kill all; rm -rf "$GNUPGHOME" flink.tgz.asc; fi; tar -xf
> flink.tgz --strip-components=1; rm flink.tgz; chown -R flink:flink .;
> ---> Running in 2225f4ea3406
> + wget -nv -O flink.tgz localhost:9999/flink.tgz
> failed: Connection refused.
> The command '/bin/sh -c set -ex; wget -nv -O flink.tgz "$FLINK_TGZ_URL";
> if [ "$CHECK_GPG" = "true" ]; then wget -nv -O flink.tgz.asc
> "$FLINK_ASC_URL"; export GNUPGHOME="$(mktemp -d)"; for server in
> ha.pool.sks-keyservers.net $(shuf -e hkp://
> p80.pool.sks-keyservers.net:80
> keyserver.ubuntu.com hkp://
> keyserver.ubuntu.com:80 pgp.mit.edu) ; do
> gpg --batch --keyserver "$server" --recv-keys "$GPG_KEY" && break || : ;
> done && gpg --batch --verify flink.tgz.asc flink.tgz; gpgconf
> --kill all; rm -rf "$GNUPGHOME" flink.tgz.asc; fi; tar -xf
> flink.tgz --strip-components=1; rm flink.tgz; chown -R flink:flink
> .;' returned a non-zero code: 4
> ```
>
> I checked the source code and found the python fileserver is here:
>
>
> https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_docker.sh#L44
>
> Then it uses 'docker build --net==host' to build image:
>
>
> https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_docker.sh#L53
>
> I am confused here, --net=host only makes services at the minikube vm
> available to the docker build process, however, the python fileserver is
> running on my local env rather than the minikube vm, how the docker build
> process access that fileserver?
>
> Should I:
> - run the test script inside the minikube vm (so that the fileserver will
> run on it), or
> - did I miss some configuration to my minikube vm?
>
> Thanks in advance.
>