You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Anto Aravinth <an...@gmail.com> on 2020/06/04 01:55:45 UTC
How to resolve Kafka within docker environment?
I have a spring application, which should connect to the kafka. This is
what my Docker file looks for spring application:
```
FROM maven:3-jdk-11 as builder
# create app folder for sources
RUN mkdir -p /build
WORKDIR /build
COPY pom.xml /build
#Download all required dependencies into one layer
RUN mvn -B dependency:resolve dependency:resolve-plugins
#Copy source code
COPY src /build/src
# Build application
RUN mvn package
FROM openjdk:8-jdk-alpine
RUN addgroup -S spring && adduser -S spring -G spring
USER spring:spring
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-jar","/app.jar"]
```
The kafka/Postgres/zookeeper everything other, comes from Docker images. So
thought will run the application in docker compose, so that it looks like
the following:
version: '3.1'
services:
postgres:
image: debezium/postgres
environment:
POSTGRES_PASSWORD: qwerty
POSTGRES_USER: appuser
volumes:
- ./postgres:/data/postgres
ports:
- 6532:6532
zookeeper:
image: confluentinc/cp-zookeeper
ports:
- "2181:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
kafka:
image: confluentinc/cp-kafka
depends_on:
- zookeeper
- postgres
ports:
- "9092:9092"
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_LOG_CLEANER_DELETE_RETENTION_MS: 5000
KAFKA_BROKER_ID: 1
KAFKA_MIN_INSYNC_REPLICAS: 1
KAFKA_ADVERTISED_HOST_NAME: kafka
connector:
image: debezium/connect:latest
ports:
- "8083:8083"
environment:
GROUP_ID: 1
CONFIG_STORAGE_TOPIC: my_connect_configs
OFFSET_STORAGE_TOPIC: my_connect_offsets
BOOTSTRAP_SERVERS: kafka:9092
depends_on:
- zookeeper
- postgres
- kafka
app-server:
# Configuration for building the docker image for the backend
service
build:
context: . # Use an image built from the specified dockerfile
in the `polling-app-server` directory.
dockerfile: Dockerfile
ports:
- "8080:8080" # Forward the exposed port 8080 on the
container to port 8080 on the host machine
restart: always
depends_on:
- kafka
- zookeeper
- postgres
- connector
environment:
BOOTSTRAP_SERVERS: kafka:9092
I pass an environment variable `BOOTSTRAP_SERVERS` with value as
`kafka:9092` (since `localhost:9092` doesn't work inside my docker
environment).
And in my spring code, I get the value like the following:
```
System.getenv("bootstrap.servers")
// or
System.getenv("BOOTSTRAP_SERVERS")
```
however, when I run `docker-compose up`, I get the value as `null` for the
above Java code. Not sure, what is the best way to get the docker resolved
network for my kafka.
So how to fix it so that java code picks up the Kafka broker inside the
docker environment?
Re: How to resolve Kafka within docker environment?
Posted by Anto Aravinth <an...@gmail.com>.
Hello Ricardo,
Thanks much for your help. In deed it have fixed the problem.
On Thu, Jun 4, 2020 at 8:31 PM Ricardo Ferreira <ri...@riferrei.com>
wrote:
> Anto,
>
> I am not 100% familiar with this image `confluentinc/cp-kafka` but there
> is a few things that you should try:
>
> 1) Make sure your `kafka` containers has this name set appropriately
>
> ```
>
> kafka:
> image: confluentinc/cp-enterprise-kafka:5.5.0
> container_name: *kafka*
> depends_on:
> - zookeeper
>
> ```
>
> 2) Set the list of protocols that Kafka will communicate externally
>
> ```
>
> KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:
> PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
>
> ```
>
> 3) In your app container; provide the bootstrap server value with double
> quotes
>
> ```
>
> depends_on:
> - kafka
> - zookeeper
> - postgres
> - connector
> environment:
> BOOTSTRAP_SERVERS: *"kafka:9092"*
>
> ```
>
> Thanks,
>
> -- riferrei
> On 6/3/20 9:55 PM, Anto Aravinth wrote:
>
> I have a spring application, which should connect to the kafka. This is
> what my Docker file looks for spring application:
>
> ```
> FROM maven:3-jdk-11 as builder
> # create app folder for sources
> RUN mkdir -p /build
> WORKDIR /build
> COPY pom.xml /build
> #Download all required dependencies into one layer
> RUN mvn -B dependency:resolve dependency:resolve-plugins
> #Copy source code
> COPY src /build/src
> # Build application
> RUN mvn package
>
> FROM openjdk:8-jdk-alpine
> RUN addgroup -S spring && adduser -S spring -G spring
> USER spring:spring
> ARG JAR_FILE=target/*.jar
> COPY ${JAR_FILE} app.jar
> ENTRYPOINT ["java","-jar","/app.jar"]
> ```
>
> The kafka/Postgres/zookeeper everything other, comes from Docker images. So
> thought will run the application in docker compose, so that it looks like
> the following:
>
> version: '3.1'
> services:
> postgres:
> image: debezium/postgres
> environment:
> POSTGRES_PASSWORD: qwerty
> POSTGRES_USER: appuser
> volumes:
> - ./postgres:/data/postgres
> ports:
> - 6532:6532
> zookeeper:
> image: confluentinc/cp-zookeeper
> ports:
> - "2181:2181"
> environment:
> ZOOKEEPER_CLIENT_PORT: 2181
> kafka:
> image: confluentinc/cp-kafka
> depends_on:
> - zookeeper
> - postgres
> ports:
> - "9092:9092"
> environment:
> KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
> KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
> KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
> KAFKA_LOG_CLEANER_DELETE_RETENTION_MS: 5000
> KAFKA_BROKER_ID: 1
> KAFKA_MIN_INSYNC_REPLICAS: 1
> KAFKA_ADVERTISED_HOST_NAME: kafka
> connector:
> image: debezium/connect:latest
> ports:
> - "8083:8083"
> environment:
> GROUP_ID: 1
> CONFIG_STORAGE_TOPIC: my_connect_configs
> OFFSET_STORAGE_TOPIC: my_connect_offsets
> BOOTSTRAP_SERVERS: kafka:9092
> depends_on:
> - zookeeper
> - postgres
> - kafka
> app-server:
> # Configuration for building the docker image for the backend
> service
> build:
> context: . # Use an image built from the specified dockerfile
> in the `polling-app-server` directory.
> dockerfile: Dockerfile
> ports:
> - "8080:8080" # Forward the exposed port 8080 on the
> container to port 8080 on the host machine
> restart: always
> depends_on:
> - kafka
> - zookeeper
> - postgres
> - connector
> environment:
> BOOTSTRAP_SERVERS: kafka:9092
>
> I pass an environment variable `BOOTSTRAP_SERVERS` with value as
> `kafka:9092` (since `localhost:9092` doesn't work inside my docker
> environment).
>
> And in my spring code, I get the value like the following:
>
> ```
> System.getenv("bootstrap.servers")
> // or
> System.getenv("BOOTSTRAP_SERVERS")
> ```
>
> however, when I run `docker-compose up`, I get the value as `null` for the
> above Java code. Not sure, what is the best way to get the docker resolved
> network for my kafka.
>
> So how to fix it so that java code picks up the Kafka broker inside the
> docker environment?
>
>
>
Re: How to resolve Kafka within docker environment?
Posted by Ricardo Ferreira <ri...@riferrei.com>.
Anto,
I am not 100% familiar with this image `confluentinc/cp-kafka` but there
is a few things that you should try:
1) Make sure your `kafka` containers has this name set appropriately
```
kafka:
image: confluentinc/cp-enterprise-kafka:5.5.0
container_name: *kafka*
depends_on:
- zookeeper
```
2) Set the list of protocols that Kafka will communicate externally
```
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:
PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
```
3) In your app container; provide the bootstrap server value with double
quotes
```
depends_on:
- kafka
- zookeeper
- postgres
- connector
environment:
BOOTSTRAP_SERVERS: *"kafka:9092"*
```
Thanks,
-- riferrei
On 6/3/20 9:55 PM, Anto Aravinth wrote:
> I have a spring application, which should connect to the kafka. This is
> what my Docker file looks for spring application:
>
> ```
> FROM maven:3-jdk-11 as builder
> # create app folder for sources
> RUN mkdir -p /build
> WORKDIR /build
> COPY pom.xml /build
> #Download all required dependencies into one layer
> RUN mvn -B dependency:resolve dependency:resolve-plugins
> #Copy source code
> COPY src /build/src
> # Build application
> RUN mvn package
>
> FROM openjdk:8-jdk-alpine
> RUN addgroup -S spring && adduser -S spring -G spring
> USER spring:spring
> ARG JAR_FILE=target/*.jar
> COPY ${JAR_FILE} app.jar
> ENTRYPOINT ["java","-jar","/app.jar"]
> ```
>
> The kafka/Postgres/zookeeper everything other, comes from Docker images. So
> thought will run the application in docker compose, so that it looks like
> the following:
>
> version: '3.1'
> services:
> postgres:
> image: debezium/postgres
> environment:
> POSTGRES_PASSWORD: qwerty
> POSTGRES_USER: appuser
> volumes:
> - ./postgres:/data/postgres
> ports:
> - 6532:6532
> zookeeper:
> image: confluentinc/cp-zookeeper
> ports:
> - "2181:2181"
> environment:
> ZOOKEEPER_CLIENT_PORT: 2181
> kafka:
> image: confluentinc/cp-kafka
> depends_on:
> - zookeeper
> - postgres
> ports:
> - "9092:9092"
> environment:
> KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
> KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
> KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
> KAFKA_LOG_CLEANER_DELETE_RETENTION_MS: 5000
> KAFKA_BROKER_ID: 1
> KAFKA_MIN_INSYNC_REPLICAS: 1
> KAFKA_ADVERTISED_HOST_NAME: kafka
> connector:
> image: debezium/connect:latest
> ports:
> - "8083:8083"
> environment:
> GROUP_ID: 1
> CONFIG_STORAGE_TOPIC: my_connect_configs
> OFFSET_STORAGE_TOPIC: my_connect_offsets
> BOOTSTRAP_SERVERS: kafka:9092
> depends_on:
> - zookeeper
> - postgres
> - kafka
> app-server:
> # Configuration for building the docker image for the backend
> service
> build:
> context: . # Use an image built from the specified dockerfile
> in the `polling-app-server` directory.
> dockerfile: Dockerfile
> ports:
> - "8080:8080" # Forward the exposed port 8080 on the
> container to port 8080 on the host machine
> restart: always
> depends_on:
> - kafka
> - zookeeper
> - postgres
> - connector
> environment:
> BOOTSTRAP_SERVERS: kafka:9092
>
> I pass an environment variable `BOOTSTRAP_SERVERS` with value as
> `kafka:9092` (since `localhost:9092` doesn't work inside my docker
> environment).
>
> And in my spring code, I get the value like the following:
>
> ```
> System.getenv("bootstrap.servers")
> // or
> System.getenv("BOOTSTRAP_SERVERS")
> ```
>
> however, when I run `docker-compose up`, I get the value as `null` for the
> above Java code. Not sure, what is the best way to get the docker resolved
> network for my kafka.
>
> So how to fix it so that java code picks up the Kafka broker inside the
> docker environment?
>