You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/11/16 15:20:16 UTC
[GitHub] [flink] gaborgsomogyi opened a new pull request, #21339: [FLINK-30024][tests] Build local test KDC docker image
gaborgsomogyi opened a new pull request, #21339:
URL: https://github.com/apache/flink/pull/21339
## What is the purpose of the change
YARN e2e tests are using a roughly 7 years old KDC docker image which is hard to maintain. In this PR I've created a `Dockerfile` to build KDC on-the-fly.
## Brief change log
* Restructured the docker-compose part (now it's possible to use `docker-compose build`)
* Added `kdc/Dockerfile` to build it locally
* Moved the original docker files/configs into hadoop directory
* Made `hadoop/Dockerfile` hadoop version agnostic
* Modified `README.md` for the docker images to reflect actual stand
## Verifying this change
Existing YARN e2e tests.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper:no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on a diff in pull request #21339: [FLINK-30024][tests] Build local test KDC docker image
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on code in PR #21339:
URL: https://github.com/apache/flink/pull/21339#discussion_r1025316443
##########
flink-end-to-end-tests/test-scripts/common_yarn_docker.sh:
##########
@@ -100,13 +100,10 @@ function build_image() {
echo "Pre-downloading Hadoop tarball"
local cache_path
cache_path=$(get_artifact "http://archive.apache.org/dist/hadoop/common/hadoop-2.8.5/hadoop-2.8.5.tar.gz")
- ln "${cache_path}" "${END_TO_END_DIR}/test-scripts/docker-hadoop-secure-cluster/hadoop-2.8.5.tar.gz"
+ ln "${cache_path}" "${END_TO_END_DIR}/test-scripts/docker-hadoop-secure-cluster/hadoop/hadoop.tar.gz"
Review Comment:
I thought the cache is per run. OK, then putting back the version here too...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on pull request #21339: [FLINK-30024][tests] Build local test KDC docker image
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on PR #21339:
URL: https://github.com/apache/flink/pull/21339#issuecomment-1318249802
> Do you mean the Bash based E2E tests? Yes, I share that annoyance. There is https://issues.apache.org/jira/browse/FLINK-11463 but there's little interest in resolving this tech debt.
Good to see that it has been started. Sooner or later we must make it run locally otherwise the development in this area is super expensive.
> See https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43213&view=results for the latest failed run.
Thanks. I'm kicking this PR to green and going to have a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #21339: [FLINK-30024][tests] Build local test KDC docker image
Posted by GitBox <gi...@apache.org>.
flinkbot commented on PR #21339:
URL: https://github.com/apache/flink/pull/21339#issuecomment-1317201283
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "66489dd3b2a70839e469c94725a4409579581b1e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "66489dd3b2a70839e469c94725a4409579581b1e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 66489dd3b2a70839e469c94725a4409579581b1e UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a diff in pull request #21339: [FLINK-30024][tests] Build local test KDC docker image
Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #21339:
URL: https://github.com/apache/flink/pull/21339#discussion_r1026130555
##########
flink-end-to-end-tests/test-scripts/common_yarn_docker.sh:
##########
@@ -100,13 +100,10 @@ function build_image() {
echo "Pre-downloading Hadoop tarball"
local cache_path
cache_path=$(get_artifact "http://archive.apache.org/dist/hadoop/common/hadoop-2.8.5/hadoop-2.8.5.tar.gz")
- ln "${cache_path}" "${END_TO_END_DIR}/test-scripts/docker-hadoop-secure-cluster/hadoop-2.8.5.tar.gz"
+ ln "${cache_path}" "${END_TO_END_DIR}/test-scripts/docker-hadoop-secure-cluster/hadoop/hadoop.tar.gz"
Review Comment:
> I thought the cache is per run. OK, then putting back the version here too...
That would limit the caching benefits significantly because many things are only required once.
> Wait a minute, get_artifact downloads the jar if needed and we just make a link to the cache entry.
[Kafka](https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/kafka-common.sh#L43) works the same way where several version bump already happened.
The code as-is in the PR is fine. The important thing here is that the URL passed to `get_artifact` contains the version, since we only create a symbolic link to the path returned by `get_artifact` (which also contains the version, extracted from the URL).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a diff in pull request #21339: [FLINK-30024][tests] Build local test KDC docker image
Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #21339:
URL: https://github.com/apache/flink/pull/21339#discussion_r1026130555
##########
flink-end-to-end-tests/test-scripts/common_yarn_docker.sh:
##########
@@ -100,13 +100,10 @@ function build_image() {
echo "Pre-downloading Hadoop tarball"
local cache_path
cache_path=$(get_artifact "http://archive.apache.org/dist/hadoop/common/hadoop-2.8.5/hadoop-2.8.5.tar.gz")
- ln "${cache_path}" "${END_TO_END_DIR}/test-scripts/docker-hadoop-secure-cluster/hadoop-2.8.5.tar.gz"
+ ln "${cache_path}" "${END_TO_END_DIR}/test-scripts/docker-hadoop-secure-cluster/hadoop/hadoop.tar.gz"
Review Comment:
> I thought the cache is per run. OK, then putting back the version here too...
That would limit the caching benefits significantly because many things are only required once.
> Wait a minute, get_artifact downloads the jar if needed and we just make a link to the cache entry.
[Kafka](https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/kafka-common.sh#L43) works the same way where several version bump already happened.
The code as-is in the PR is fine. The important thing here is that the URL passed to `get_artifact` contains the version, since we only create a symbolic link to the path returned by `get_artifact` (which also contains the version, extracted from the URL).
If you were to _copy_ the file to `cache_path` then @MartijnVisser would be correct though (kinda; caching just wouldn't work lol).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #21339: [FLINK-30024][tests] Build local test KDC docker image
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #21339:
URL: https://github.com/apache/flink/pull/21339#issuecomment-1318239944
> As a general note I think it's super hard that we're not able to execute the e2e tests locally.
Do you mean the Bash based E2E tests? Yes, I share that annoyance. There is https://issues.apache.org/jira/browse/FLINK-11463 but there's little interest in resolving this tech debt.
> The other thing what I can suggest is to merge this when ready and we can have a look at top of this.
Sounds like a plan.
> If you could show me a bad run then maybe I can have a look and help some way.
See https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43213&view=results for the latest failed run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a diff in pull request #21339: [FLINK-30024][tests] Build local test KDC docker image
Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #21339:
URL: https://github.com/apache/flink/pull/21339#discussion_r1026130555
##########
flink-end-to-end-tests/test-scripts/common_yarn_docker.sh:
##########
@@ -100,13 +100,10 @@ function build_image() {
echo "Pre-downloading Hadoop tarball"
local cache_path
cache_path=$(get_artifact "http://archive.apache.org/dist/hadoop/common/hadoop-2.8.5/hadoop-2.8.5.tar.gz")
- ln "${cache_path}" "${END_TO_END_DIR}/test-scripts/docker-hadoop-secure-cluster/hadoop-2.8.5.tar.gz"
+ ln "${cache_path}" "${END_TO_END_DIR}/test-scripts/docker-hadoop-secure-cluster/hadoop/hadoop.tar.gz"
Review Comment:
> I thought the cache is per run. OK, then putting back the version here too...
That would limit the caching benefits significantly because many things are only required once.
> Wait a minute, get_artifact downloads the jar if needed and we just make a link to the cache entry.
[Kafka](https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/kafka-common.sh#L43) works the same way where several version bump already happened.
The code as-is in the PR is fine. The important thing here is the URL passed to `get_artifact` contains the version, since we only create a symbolic link to the path returned by `get_artifact` (which also contains the version, extracted from the URL).
If you were to _copy_ the file to `cache_path` then @MartijnVisser would be correct though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on pull request #21339: [FLINK-30024][tests] Build local test KDC docker image
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on PR #21339:
URL: https://github.com/apache/flink/pull/21339#issuecomment-1319806287
@MartijnVisser @zentol Thanks for taking care!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #21339: [FLINK-30024][tests] Build local test KDC docker image
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #21339:
URL: https://github.com/apache/flink/pull/21339#issuecomment-1317657486
@gaborgsomogyi Looking interesting, especially since I've been working on the upgrade to a later Hadoop version with
https://github.com/apache/flink/pull/21128 which runs into issues with the Yarn tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on a diff in pull request #21339: [FLINK-30024][tests] Build local test KDC docker image
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on code in PR #21339:
URL: https://github.com/apache/flink/pull/21339#discussion_r1025282921
##########
flink-end-to-end-tests/test-scripts/common_yarn_docker.sh:
##########
@@ -100,13 +100,10 @@ function build_image() {
echo "Pre-downloading Hadoop tarball"
local cache_path
cache_path=$(get_artifact "http://archive.apache.org/dist/hadoop/common/hadoop-2.8.5/hadoop-2.8.5.tar.gz")
- ln "${cache_path}" "${END_TO_END_DIR}/test-scripts/docker-hadoop-secure-cluster/hadoop-2.8.5.tar.gz"
+ ln "${cache_path}" "${END_TO_END_DIR}/test-scripts/docker-hadoop-secure-cluster/hadoop/hadoop.tar.gz"
Review Comment:
IIRC the danger of this is that now any Hadoop version will get cached, because there's no version number anymore. That means that if an upgrade is done from Hadoop 2.8.5 to 2.10.2, no new Hadoop TAR will be downloaded because it will be cached from a previous run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on pull request #21339: [FLINK-30024][tests] Build local test KDC docker image
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on PR #21339:
URL: https://github.com/apache/flink/pull/21339#issuecomment-1318652666
@MartijnVisser The tests passed. If you think the PR is good to go we can merge it and retry https://github.com/apache/flink/pull/21128 at top of this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on a diff in pull request #21339: [FLINK-30024][tests] Build local test KDC docker image
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on code in PR #21339:
URL: https://github.com/apache/flink/pull/21339#discussion_r1025324405
##########
flink-end-to-end-tests/test-scripts/common_yarn_docker.sh:
##########
@@ -100,13 +100,10 @@ function build_image() {
echo "Pre-downloading Hadoop tarball"
local cache_path
cache_path=$(get_artifact "http://archive.apache.org/dist/hadoop/common/hadoop-2.8.5/hadoop-2.8.5.tar.gz")
- ln "${cache_path}" "${END_TO_END_DIR}/test-scripts/docker-hadoop-secure-cluster/hadoop-2.8.5.tar.gz"
+ ln "${cache_path}" "${END_TO_END_DIR}/test-scripts/docker-hadoop-secure-cluster/hadoop/hadoop.tar.gz"
Review Comment:
Wait a minute, `get_artifact` downloads the jar if needed and we just make a link to the cache entry.
[Kafka](https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/kafka-common.sh#L43) works the same way where several version bump already happened.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a diff in pull request #21339: [FLINK-30024][tests] Build local test KDC docker image
Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #21339:
URL: https://github.com/apache/flink/pull/21339#discussion_r1026130555
##########
flink-end-to-end-tests/test-scripts/common_yarn_docker.sh:
##########
@@ -100,13 +100,10 @@ function build_image() {
echo "Pre-downloading Hadoop tarball"
local cache_path
cache_path=$(get_artifact "http://archive.apache.org/dist/hadoop/common/hadoop-2.8.5/hadoop-2.8.5.tar.gz")
- ln "${cache_path}" "${END_TO_END_DIR}/test-scripts/docker-hadoop-secure-cluster/hadoop-2.8.5.tar.gz"
+ ln "${cache_path}" "${END_TO_END_DIR}/test-scripts/docker-hadoop-secure-cluster/hadoop/hadoop.tar.gz"
Review Comment:
> I thought the cache is per run. OK, then putting back the version here too...
That would limit the caching benefits significantly because many things are only required once.
> Wait a minute, get_artifact downloads the jar if needed and we just make a link to the cache entry.
[Kafka](https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/kafka-common.sh#L43) works the same way where several version bump already happened.
The code as-is in the PR is fine. The important thing here is that the URL passed to `get_artifact` contains the version, since we only create a symbolic link to the path returned by `get_artifact` (which also contains the version, extracted from the URL).
If you were to _copy_ the file to `cache_path` then @MartijnVisser would be correct though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on pull request #21339: [FLINK-30024][tests] Build local test KDC docker image
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on PR #21339:
URL: https://github.com/apache/flink/pull/21339#issuecomment-1318206668
@MartijnVisser Yeah, this PR is intended to add more stability to the test (though I'm not saying it will solve the version bump issues). Additionally one needs to change the Hadoop version only in 1-2 places. I'm aware of the bump PR but I've not tracked the latest news. If you could show me a bad run then maybe I can have a look and help some way. At the same time I'm fixing the missing `hadoop.tar.gz` issue here. The other thing what I can suggest is to merge this when ready and we can have a look at top of this.
As a general note I think it's super hard that we're not able to execute the e2e tests locally. I'm thinking about to fix it somehow, but since I'm working on time pressured features this is best effort for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser merged pull request #21339: [FLINK-30024][tests] Build local test KDC docker image
Posted by GitBox <gi...@apache.org>.
MartijnVisser merged PR #21339:
URL: https://github.com/apache/flink/pull/21339
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org