[jira] [Commented] (KAFKA-12847) Dockerfile needed for kafka system tests needs changes

["Chia-Ping Tsai (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-12847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17354293#comment-17354293 ] 

Chia-Ping Tsai commented on KAFKA-12847:
----------------------------------------

Could you try to use non-root to run kafka tests? IIRC, Not only system tests but also UT/IT expect to be run by non-root. 

 

>  Also, I wonder whether or not upstream Dockerfile & System tests are part of CI/CD and get tested for every PR. If so, this issue should have been caught.

We don't run system tests for each PR since it is too expensive. Instead, we make sure all system tests pass before release.

> Dockerfile needed for kafka system tests needs changes
> ------------------------------------------------------
>
>                 Key: KAFKA-12847
>                 URL: https://issues.apache.org/jira/browse/KAFKA-12847
>             Project: Kafka
>          Issue Type: Bug
>          Components: system tests
>    Affects Versions: 2.8.0, 2.7.1
>         Environment: Issue tested in environments below but is independent of h/w arch. or Linux flavor: -
> 1.) RHEL-8.3 on x86_64 
> 2.) RHEL-8.3 on IBM Power (ppc64le)
> 3.) apache/kafka branch tested: trunk (master)
>            Reporter: Abhijit Mane
>            Assignee: Abhijit Mane
>            Priority: Major
>              Labels: easyfix
>         Attachments: Dockerfile.upstream
>
>
> Hello,
> I tried apache/kafka system tests as per documentation: -
> ([https://github.com/apache/kafka/tree/trunk/tests#readme|https://github.com/apache/kafka/tree/trunk/tests#readme_])
> =========================================================
>  PROBLEM
>  ~~~~~~
> 1.) As root user, clone kafka github repo and start "kafka system tests"
>  # git clone [https://github.com/apache/kafka.git]
>  # cd kafka
>  # ./gradlew clean systemTestLibs
>  # bash tests/docker/run_tests.sh
> 2.) Dockerfile issue - [https://github.com/apache/kafka/blob/trunk/tests/docker/Dockerfile]
> This file has an *UID* entry as shown below: -
>  -----------
>  ARG *UID*="1000"
>  RUN useradd -u $*UID* ducker
> // {color:#de350b}*Error during docker build*{color} => useradd: UID 0 is not unique, root user id is 0
>  -----------
>  I ran everything as root which means the built-in bash environment variable 'UID' always
> resolves to 0 and can't be changed. Hence, the docker build fails. The issue should be seen even if run as non-root.
> 3.) Next, as root, as per README, I ran: -
> server:/kafka> *bash tests/docker/run_tests.sh*
> The ducker tool builds the container images & switches to user '*ducker*' inside the container
> & maps kafka root dir ('kafka') from host to '/opt/kafka-dev' in the container.
> Ref: [https://github.com/apache/kafka/blob/trunk/tests/docker/ducker-ak|https://github.com/apache/kafka/blob/trunk/tests/docker/ducker-ak]
> Ex:  docker run -d *-v "${kafka_dir}:/opt/kafka-dev"* <img_name>
> This fails as the 'ducker' user has *no write permissions* to create files under 'kafka' root dir. Hence, it needs to be made writeable.
> // *chmod -R a+w kafka* 
>  – needed as container is run as 'ducker' and needs write access since kafka root volume from host is mapped to container as "/opt/kafka-dev" where the 'ducker' user writes logs
>  =========================================================
> =========================================================
>  *FIXES needed*
>  ~~~~~~~~~
>  1.) Dockerfile - [https://github.com/apache/kafka/blob/trunk/tests/docker/Dockerfile]
>  Change 'UID' to '*UID_DUCKER*'.
> This won't conflict with built in bash env. var UID and the docker image build should succeed.
>  -----------
>  ARG *UID_DUCKER*="1000"
>  RUN useradd -u $*UID_DUCKER* ducker
> // *{color:#57d9a3}No Error{color}* => No conflict with built-in UID
>  -----------
> 2.) README needs an update where we must ensure the kafka root dir from where the tests 
>  are launched is writeable to allow the 'ducker' user to create results/logs.
>  # chmod -R a+w kafka
> With this, I was able to get the docker images built and system tests started successfully.
>  =========================================================
> Also, I wonder whether or not upstream Dockerfile & System tests are part of CI/CD and get tested for every PR. If so, this issue should have been caught.
>  
> *Question to kafka SME*
>  -------------------------
>  Do you believe this is a valid problem with the Dockerfile and the fix is acceptable? 
>  Please let me know and I am happy to submit a PR with this fix.
> Thanks,
>  Abhijit



--
This message was sent by Atlassian Jira
(v8.3.4#803005)