You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "see-quick (via GitHub)" <gi...@apache.org> on 2025/09/17 12:42:25 UTC
[PR] MINOR: Add support for podman when running system tests [kafka]
see-quick opened a new pull request, #20546:
URL: https://github.com/apache/kafka/pull/20546
This PR adds support for podman within system tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add support for podman when running system tests [kafka]
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on code in PR #20546:
URL: https://github.com/apache/kafka/pull/20546#discussion_r2372326107
##########
tests/docker/run_tests.sh:
##########
@@ -20,6 +20,15 @@ KAFKA_NUM_CONTAINERS=${KAFKA_NUM_CONTAINERS:-14}
TC_PATHS=${TC_PATHS:-./kafkatest/}
REBUILD=${REBUILD:f}
+# Auto-detect container runtime if not set
+if [[ -z "${CONTAINER_RUNTIME}" ]]; then
Review Comment:
What about something like?
```
if [[ -z "${CONTAINER_RUNTIME}" ]]; then
export CONTAINER_RUNTIME="docker"
if command -v podman &> /dev/null; then
export CONTAINER_RUNTIME="podman"
fi
fi
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19745: Add support for podman when running system tests [kafka]
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on code in PR #20546:
URL: https://github.com/apache/kafka/pull/20546#discussion_r2398017127
##########
tests/docker/ducker-ak:
##########
@@ -142,6 +148,29 @@ require_commands() {
done
}
+# Detect and set the container runtime (docker or podman).
+# Sets the global variable container_runtime.
+# Can be overridden by setting CONTAINER_RUNTIME environment variable.
+detect_container_runtime() {
+ if [[ -n "${container_runtime}" ]]; then
+ return # Already set
+ fi
+
+ # Allow override via environment variable
+ if [[ -n "${CONTAINER_RUNTIME}" ]]; then
+ container_runtime="${CONTAINER_RUNTIME}"
+ which "${container_runtime}" &> /dev/null || die "Specified container runtime '${container_runtime}' not found."
+ return
+ fi
+
+ container_runtime="docker"
+ if command -v podman &> /dev/null; then
+ container_runtime="podman"
+ else
+ die "No supported container runtime found. Please install docker or podman, or set CONTAINER_RUNTIME environment variable."
Review Comment:
If this right? If `podman` is not available, aren't we going to enter this `else` block and always fail?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add support for podman when running system tests [kafka]
Posted by "see-quick (via GitHub)" <gi...@apache.org>.
see-quick commented on code in PR #20546:
URL: https://github.com/apache/kafka/pull/20546#discussion_r2371539061
##########
tests/docker/run_tests.sh:
##########
@@ -20,6 +20,15 @@ KAFKA_NUM_CONTAINERS=${KAFKA_NUM_CONTAINERS:-14}
TC_PATHS=${TC_PATHS:-./kafkatest/}
REBUILD=${REBUILD:f}
+# Auto-detect container runtime if not set
+if [[ -z "${CONTAINER_RUNTIME}" ]]; then
Review Comment:
I think the same applies to what I have described above? It would applies goes to `docker` if someone's enviroment is using `docker` (the one requirememt is that system has `which` utility).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add support for podman when running system tests [kafka]
Posted by "see-quick (via GitHub)" <gi...@apache.org>.
see-quick commented on code in PR #20546:
URL: https://github.com/apache/kafka/pull/20546#discussion_r2372470855
##########
tests/docker/run_tests.sh:
##########
@@ -20,6 +20,15 @@ KAFKA_NUM_CONTAINERS=${KAFKA_NUM_CONTAINERS:-14}
TC_PATHS=${TC_PATHS:-./kafkatest/}
REBUILD=${REBUILD:f}
+# Auto-detect container runtime if not set
+if [[ -z "${CONTAINER_RUNTIME}" ]]; then
Review Comment:
Yeah, also works for me. I will update it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add support for podman when running system tests [kafka]
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on code in PR #20546:
URL: https://github.com/apache/kafka/pull/20546#discussion_r2368792510
##########
tests/docker/run_tests.sh:
##########
@@ -20,6 +20,15 @@ KAFKA_NUM_CONTAINERS=${KAFKA_NUM_CONTAINERS:-14}
TC_PATHS=${TC_PATHS:-./kafkatest/}
REBUILD=${REBUILD:f}
+# Auto-detect container runtime if not set
+if [[ -z "${CONTAINER_RUNTIME}" ]]; then
Review Comment:
Same here, should it default to docker?
##########
tests/docker/ducker-ak:
##########
@@ -55,6 +55,9 @@ default_image_name="ducker-ak"
# The default kafka server mode.
default_kafka_mode="jvm"
+# Container runtime command (docker or podman)
+container_runtime=""
Review Comment:
Should it default to "docker" to ensure we don't break any environments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add support for podman when running system tests [kafka]
Posted by "see-quick (via GitHub)" <gi...@apache.org>.
see-quick commented on code in PR #20546:
URL: https://github.com/apache/kafka/pull/20546#discussion_r2371532843
##########
tests/docker/ducker-ak:
##########
@@ -55,6 +55,9 @@ default_image_name="ducker-ak"
# The default kafka server mode.
default_kafka_mode="jvm"
+# Container runtime command (docker or podman)
+container_runtime=""
Review Comment:
I mean, the function [1] defaults to Docker, since `detect_container_runtime` checks for it before Podman. It also respects the `CONTAINER_RUNTIME` environment variable if set, or a previously assigned value of `container_runtime` during the current shell execution. I think, this would not break any environments or do you think that is there some system, which does not have `which`?
[1] -
```bash
detect_container_runtime() {
if [[ -n "${container_runtime}" ]]; then
return # Already set
fi
# Allow override via environment variable
if [[ -n "${CONTAINER_RUNTIME}" ]]; then
container_runtime="${CONTAINER_RUNTIME}"
which "${container_runtime}" &> /dev/null || die "Specified container runtime '${container_runtime}' not found."
return
fi
if which docker &> /dev/null; then
container_runtime="docker"
elif which podman &> /dev/null; then
container_runtime="podman"
else
die "No supported container runtime found. Please install docker or podman, or set CONTAINER_RUNTIME environment variable."
fi
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add support for podman when running system tests [kafka]
Posted by "see-quick (via GitHub)" <gi...@apache.org>.
see-quick commented on PR #20546:
URL: https://github.com/apache/kafka/pull/20546#issuecomment-3350387148
> Can you open a Jira for this PR? I consider this worthy to appear in the release notes.
Sure, I have created [1].
[1] - https://issues.apache.org/jira/browse/KAFKA-19745
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19745: Add support for podman when running system tests [kafka]
Posted by "fvaleri (via GitHub)" <gi...@apache.org>.
fvaleri commented on PR #20546:
URL: https://github.com/apache/kafka/pull/20546#issuecomment-3414831414
Nice one. Do you also get this message when running a ST from scratch?
```
[1/2] STEP 1/12: FROM openjdk:17-bullseye AS build-native-image
? Please select an image:
▸ registry.fedoraproject.org/openjdk:17-bullseye
registry.access.redhat.com/openjdk:17-bullseye
docker.io/library/openjdk:17-bullseye
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19745: Add support for podman when running system tests [kafka]
Posted by "see-quick (via GitHub)" <gi...@apache.org>.
see-quick commented on code in PR #20546:
URL: https://github.com/apache/kafka/pull/20546#discussion_r2398049808
##########
tests/docker/ducker-ak:
##########
@@ -142,6 +148,29 @@ require_commands() {
done
}
+# Detect and set the container runtime (docker or podman).
+# Sets the global variable container_runtime.
+# Can be overridden by setting CONTAINER_RUNTIME environment variable.
+detect_container_runtime() {
+ if [[ -n "${container_runtime}" ]]; then
+ return # Already set
+ fi
+
+ # Allow override via environment variable
+ if [[ -n "${CONTAINER_RUNTIME}" ]]; then
+ container_runtime="${CONTAINER_RUNTIME}"
+ which "${container_runtime}" &> /dev/null || die "Specified container runtime '${container_runtime}' not found."
+ return
+ fi
+
+ container_runtime="docker"
+ if command -v podman &> /dev/null; then
+ container_runtime="podman"
+ else
+ die "No supported container runtime found. Please install docker or podman, or set CONTAINER_RUNTIME environment variable."
Review Comment:
Ah...true. I should also add an `elif` to check Docker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19745: Add support for podman when running system tests [kafka]
Posted by "see-quick (via GitHub)" <gi...@apache.org>.
see-quick commented on code in PR #20546:
URL: https://github.com/apache/kafka/pull/20546#discussion_r2398066450
##########
tests/docker/ducker-ak:
##########
@@ -142,6 +148,29 @@ require_commands() {
done
}
+# Detect and set the container runtime (docker or podman).
+# Sets the global variable container_runtime.
+# Can be overridden by setting CONTAINER_RUNTIME environment variable.
+detect_container_runtime() {
+ if [[ -n "${container_runtime}" ]]; then
+ return # Already set
+ fi
+
+ # Allow override via environment variable
+ if [[ -n "${CONTAINER_RUNTIME}" ]]; then
+ container_runtime="${CONTAINER_RUNTIME}"
+ which "${container_runtime}" &> /dev/null || die "Specified container runtime '${container_runtime}' not found."
+ return
+ fi
+
+ container_runtime="docker"
+ if command -v podman &> /dev/null; then
+ container_runtime="podman"
+ else
+ die "No supported container runtime found. Please install docker or podman, or set CONTAINER_RUNTIME environment variable."
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19745: Add support for podman when running system tests [kafka]
Posted by "see-quick (via GitHub)" <gi...@apache.org>.
see-quick commented on PR #20546:
URL: https://github.com/apache/kafka/pull/20546#issuecomment-3414504874
> Can you add a note in https://github.com/apache/kafka/blob/trunk/tests/README.md to mention users can set `CONTAINER_RUNTIME` to docker or podman?
I have added a mention there, good call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19745: Add support for podman when running system tests [kafka]
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison merged PR #20546:
URL: https://github.com/apache/kafka/pull/20546
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add support for podman when running system tests [kafka]
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on PR #20546:
URL: https://github.com/apache/kafka/pull/20546#issuecomment-3319487016
Nice one, it's a pain to remove `--memory-swappiness` each time I want to run tests locally!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add support for podman when running system tests [kafka]
Posted by "see-quick (via GitHub)" <gi...@apache.org>.
see-quick commented on code in PR #20546:
URL: https://github.com/apache/kafka/pull/20546#discussion_r2371671677
##########
tests/docker/run_tests.sh:
##########
@@ -20,6 +20,15 @@ KAFKA_NUM_CONTAINERS=${KAFKA_NUM_CONTAINERS:-14}
TC_PATHS=${TC_PATHS:-./kafkatest/}
REBUILD=${REBUILD:f}
+# Auto-detect container runtime if not set
+if [[ -z "${CONTAINER_RUNTIME}" ]]; then
Review Comment:
Hmm, so what if we add a fallback to `docker' in the else branch?
```
if [[ -z "${CONTAINER_RUNTIME}" ]]; then
if command -v docker &> /dev/null; then
export CONTAINER_RUNTIME="docker"
elif command -v podman &> /dev/null; then
export CONTAINER_RUNTIME="podman"
else
export CONTAINER_RUNTIME="docker"
fi
fi
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19745: Add support for podman when running system tests [kafka]
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on PR #20546:
URL: https://github.com/apache/kafka/pull/20546#issuecomment-3411272935
Can you add a note in https://github.com/apache/kafka/blob/trunk/tests/README.md to mention users can set `CONTAINER_RUNTIME` to docker or podman?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add support for podman when running system tests [kafka]
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on code in PR #20546:
URL: https://github.com/apache/kafka/pull/20546#discussion_r2371645208
##########
tests/docker/run_tests.sh:
##########
@@ -20,6 +20,15 @@ KAFKA_NUM_CONTAINERS=${KAFKA_NUM_CONTAINERS:-14}
TC_PATHS=${TC_PATHS:-./kafkatest/}
REBUILD=${REBUILD:f}
+# Auto-detect container runtime if not set
+if [[ -z "${CONTAINER_RUNTIME}" ]]; then
Review Comment:
My point is, is it possible that in some environments we might not enter the `if` nor the `elif` below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add support for podman when running system tests [kafka]
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on PR #20546:
URL: https://github.com/apache/kafka/pull/20546#issuecomment-3347181600
Can you open a Jira for this PR? I consider this worthy to appear in the release notes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19745: Add support for podman when running system tests [kafka]
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on code in PR #20546:
URL: https://github.com/apache/kafka/pull/20546#discussion_r2429814616
##########
tests/docker/run_tests.sh:
##########
@@ -20,6 +20,14 @@ KAFKA_NUM_CONTAINERS=${KAFKA_NUM_CONTAINERS:-14}
TC_PATHS=${TC_PATHS:-./kafkatest/}
REBUILD=${REBUILD:f}
+# Auto-detect container runtime if not set
+if [[ -z "${CONTAINER_RUNTIME}" ]]; then
+ export CONTAINER_RUNTIME="docker"
+ if command -v podman &> /dev/null; then
+ export CONTAINER_RUNTIME="podman"
+ fi
Review Comment:
If you have both installed, this will pick `podman`. I wonder if this should still default to `docker`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19745: Add support for podman when running system tests [kafka]
Posted by "see-quick (via GitHub)" <gi...@apache.org>.
see-quick commented on code in PR #20546:
URL: https://github.com/apache/kafka/pull/20546#discussion_r2434928146
##########
tests/docker/run_tests.sh:
##########
@@ -20,6 +20,14 @@ KAFKA_NUM_CONTAINERS=${KAFKA_NUM_CONTAINERS:-14}
TC_PATHS=${TC_PATHS:-./kafkatest/}
REBUILD=${REBUILD:f}
+# Auto-detect container runtime if not set
+if [[ -z "${CONTAINER_RUNTIME}" ]]; then
+ export CONTAINER_RUNTIME="docker"
+ if command -v podman &> /dev/null; then
+ export CONTAINER_RUNTIME="podman"
+ fi
Review Comment:
I have modified the logic to always prefer Docker over Podman. Let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19745: Add support for podman when running system tests [kafka]
Posted by "fvaleri (via GitHub)" <gi...@apache.org>.
fvaleri commented on PR #20546:
URL: https://github.com/apache/kafka/pull/20546#issuecomment-3415236807
It's happening on this branch.
The problem is that when using Podman, the permission issue with `.ducktape` directory creation occurs because of user namespace mapping. Podman by default uses user namespaces which can cause permission issues with mounted volumes.
If you enable me, I can push a patch for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19745: Add support for podman when running system tests [kafka]
Posted by "see-quick (via GitHub)" <gi...@apache.org>.
see-quick commented on PR #20546:
URL: https://github.com/apache/kafka/pull/20546#issuecomment-3414926095
> transactions_test
Hmm, did such an error occur on `trunk`? => That would also tell us if it's caused by this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org