You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "MehulBatra (via GitHub)" <gi...@apache.org> on 2026/02/02 19:13:25 UTC
[PR] fix iceberg quickstart [fluss]
MehulBatra opened a new pull request, #2555:
URL: https://github.com/apache/fluss/pull/2555
<!--
*Thank you very much for contributing to Fluss - we are happy that you want to help us improve Fluss. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [GitHub issue](https://github.com/apache/fluss/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no issue.
- Name the pull request in the format "[component] Title of the pull request", where *[component]* should be replaced by the name of the component being changed. Typically, this corresponds to the component label assigned to the issue (e.g., [kv], [log], [client], [flink]). Skip *[component]* if you are unsure about which is the best component.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes.
- Each pull request should address only one issue, not mix up code from multiple issues.
**(The sections below can be removed for hotfixes or typos)**
-->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #2543
<!-- What is the purpose of the change -->
### Brief change log
Mkdirs failed error is coming while writing iceberg data. The core issue is that the standard Flink image drops privileges, and user: "flink:root" doesn't prevent that.
custom images with pre-modified entrypoints. For the
standard Flink image to work, we need to replicate that, so need to make a docker file to create an image.
<!-- Please describe the changes made in this pull request and explain how they address the issue -->
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [doc/build] Fix iceberg quickstart to build on flink standard image [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #2555:
URL: https://github.com/apache/fluss/pull/2555#discussion_r2760532678
##########
website/docs/quickstart/lakehouse.md:
##########
@@ -220,9 +242,11 @@ services:
datalake.iceberg.warehouse: /tmp/iceberg
volumes:
- shared-tmpfs:/tmp/iceberg
Review Comment:
why is that, can you please help me understand?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [doc/build] Fix iceberg quickstart to build on flink standard image [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on PR #2555:
URL: https://github.com/apache/fluss/pull/2555#issuecomment-3851452466
> @MehulBatra Thanks for update. LGTM!
Thank you @luoyuxia, for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [doc/build] Fix iceberg quickstart to build on flink standard image [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on PR #2555:
URL: https://github.com/apache/fluss/pull/2555#issuecomment-3843118136
Also, @luoyuxia, I was thinking of testing 1.10.1, and then we can close this one too #2415, can you help me with this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [doc/build] Fix iceberg quickstart to build on flink standard image [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #2555:
URL: https://github.com/apache/fluss/pull/2555#discussion_r2758804785
##########
website/docs/quickstart/lakehouse.md:
##########
Review Comment:
I'll also encounter parquet related conflict, after shade parquet, I can resolve this problem:
```
<relocation>
<pattern>org.apache.parquet</pattern> <shadedPattern>org.apache.iceberg.shaded.org.apache.parquet</shadedPattern>
</relocation>
```
##########
website/docs/quickstart/lakehouse.md:
##########
Review Comment:
we can remove this line since it's already in `streaming` mode
##########
website/docs/quickstart/lakehouse.md:
##########
Review Comment:
May in the next release, we'll consider not bundle iceberg-releated class in `fluss-lake-fluss` just like what we do for paimon in #2531
##########
website/docs/quickstart/lakehouse.md:
##########
@@ -155,37 +155,60 @@ mkdir fluss-quickstart-iceberg
cd fluss-quickstart-iceberg
```
-2. Create a `lib` directory and download the required Hadoop jar file:
+2. Create directories and download required jars:
```shell
-mkdir lib
-wget -O lib/hadoop-apache-3.3.5-2.jar https://repo1.maven.org/maven2/io/trino/hadoop/hadoop-apache/3.3.5-2/hadoop-apache-3.3.5-2.jar
-```
+mkdir -p lib opt
-This jar file provides Hadoop 3.3.5 dependencies required for Iceberg's Hadoop catalog integration.
+# Flink connectors
+wget -O lib/flink-faker.jar https://github.com/knaufk/flink-faker/releases/download/v0.5.3/flink-faker-0.5.3.jar
Review Comment:
Can we still keep the version in the file naming so that it's easy for user to track
##########
website/docs/quickstart/lakehouse.md:
##########
@@ -199,9 +222,8 @@ services:
datalake.iceberg.warehouse: /tmp/iceberg
volumes:
- shared-tmpfs:/tmp/iceberg
Review Comment:
from flink 0.9, we will also need mount `/tmp/fluss`:
```
shared-tmpfs:/tmp/fluss
```
##########
website/docs/quickstart/lakehouse.md:
##########
@@ -220,9 +242,11 @@ services:
datalake.iceberg.warehouse: /tmp/iceberg
volumes:
- shared-tmpfs:/tmp/iceberg
Review Comment:
from flink 0.9, we will also need mount `/tmp/fluss`:
```
shared-tmpfs:/tmp/fluss
```
##########
website/docs/quickstart/lakehouse.md:
##########
Review Comment:
I encounter
```
inherit an implementation of the resolved method 'abstract void generate(com.fasterxml.jackson.core.JsonGenerator)' of interface org.apache.iceberg.util.JsonUtil$ToJson.
```
when fire this sql:
The reason is that `iceberg-flink` introduce `JsonUtil` which already shade `com.fasterxml.jackson` to `org.apache.iceberg.shaded.com.fasterxml.jackson`.
`fluss-lake-iceberg` also introduce `JsonUtil` which doesn't shade `com.fasterxml.jackson`, so the class conflict happens.
To solve it, we need to shade `com.fasterxml.jackson` in our `fluss-lake-iceberg` module:
```
<configuration>
<artifactSet>
<includes>
<include>*:*</include>
</includes>
</artifactSet>
<relocations>
<relocation>
<pattern>com.fasterxml.jackson</pattern>
<shadedPattern>org.apache.iceberg.shaded.com.fasterxml.jackson</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.parquet</pattern>
<shadedPattern>org.apache.iceberg.shaded.org.apache.parquet</shadedPattern>
</relocation>
</relocations>
<filters>
<filter>
<artifact>*</artifact>
<excludes>
<exclude>LICENSE</exclude>
<exclude>NOTICE</exclude>
<exclude>META-INF/versions/21/**</exclude>
<exclude>META-INF/versions/17/**</exclude>
</excludes>
</filter>
</filters>
</configuration>
```
##########
website/docs/quickstart/lakehouse.md:
##########
@@ -155,37 +155,60 @@ mkdir fluss-quickstart-iceberg
cd fluss-quickstart-iceberg
```
-2. Create a `lib` directory and download the required Hadoop jar file:
+2. Create directories and download required jars:
```shell
-mkdir lib
-wget -O lib/hadoop-apache-3.3.5-2.jar https://repo1.maven.org/maven2/io/trino/hadoop/hadoop-apache/3.3.5-2/hadoop-apache-3.3.5-2.jar
-```
+mkdir -p lib opt
-This jar file provides Hadoop 3.3.5 dependencies required for Iceberg's Hadoop catalog integration.
+# Flink connectors
+wget -O lib/flink-faker.jar https://github.com/knaufk/flink-faker/releases/download/v0.5.3/flink-faker-0.5.3.jar
+wget -O lib/fluss-flink.jar "https://repo1.maven.org/maven2/org/apache/fluss/fluss-flink-1.20/$FLUSS_DOCKER_VERSION$/fluss-flink-1.20-$FLUSS_DOCKER_VERSION$.jar"
+wget -O lib/iceberg-flink-runtime.jar "https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-flink-runtime-1.20/1.10.1/iceberg-flink-runtime-1.20-1.10.1.jar"
-:::info
-The `lib` directory serves as a staging area for additional jars needed by the Fluss coordinator server. The docker-compose configuration (see step 3) mounts this directory and copies all jars to `/opt/fluss/plugins/iceberg/` inside the coordinator container at startup.
+# Fluss lake plugin
+wget -O lib/fluss-lake-iceberg.jar https://repo1.maven.org/maven2/org/apache/fluss/fluss-lake-iceberg/$FLUSS_DOCKER_VERSION$/fluss-lake-iceberg-$FLUSS_DOCKER_VERSION$.jar
+
+# Hadoop filesystem support
+wget -O lib/hadoop-apache.jar https://repo1.maven.org/maven2/io/trino/hadoop/hadoop-apache/3.3.5-2/hadoop-apache-3.3.5-2.jar
+wget -O lib/failsafe.jar https://repo1.maven.org/maven2/dev/failsafe/failsafe/3.3.2/failsafe-3.3.2.jar
+# Tiering service
+wget -O opt/fluss-flink-tiering.jar https://repo1.maven.org/maven2/org/apache/fluss/fluss-flink-tiering/$FLUSS_DOCKER_VERSION$/fluss-flink-tiering-$FLUSS_DOCKER_VERSION$.jar
+```
+
+:::info
You can add more jars to this `lib` directory based on your requirements:
- **Cloud storage support**: For AWS S3 integration with Iceberg, add the corresponding Iceberg bundle jars (e.g., `iceberg-aws-bundle`)
- **Custom Hadoop configurations**: Add jars for specific HDFS distributions or custom authentication mechanisms
- **Other catalog backends**: Add jars needed for alternative Iceberg catalog implementations (e.g., Rest, Hive, Glue)
-
-Any jar placed in the `lib` directory will be automatically loaded by the Fluss coordinator server, making it available for Iceberg integration.
:::
-3. Create a `docker-compose.yml` file with the following content:
+3. Create a `Dockerfile` for the custom Flink image:
Review Comment:
It'll be hard for user to build flink image by themself. IIUC, the problem is that the flink image use user `flink` which cause the permission. I have solved by the following content:
```
jobmanager:
image: flink:1.20-scala_2.12-java17
ports:
- "8083:8081"
entrypoint: ["/bin/bash", "-c"]
command: >
"sed -i 's/exec $$(drop_privs_cmd)//g' /docker-entrypoint.sh &&
cp /tmp/jars/*.jar /opt/flink/lib/ 2>/dev/null || true;
/docker-entrypoint.sh jobmanager"
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
volumes:
- shared-tmpfs:/tmp/iceberg
- ./lib:/tmp/jars # Mount the JARs directory
taskmanager:
image: flink:1.20-scala_2.12-java17
depends_on:
- jobmanager
entrypoint: ["/bin/bash", "-c"]
command: >
"sed -i 's/exec $$(drop_privs_cmd)//g' /docker-entrypoint.sh &&
cp /tmp/jars/*.jar /opt/flink/lib/ 2>/dev/null || true;
/docker-entrypoint.sh taskmanager"
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
taskmanager.numberOfTaskSlots: 10
taskmanager.memory.process.size: 2048m
taskmanager.memory.framework.off-heap.size: 256m
volumes:
- shared-tmpfs:/tmp/iceberg
- ./lib:/tmp/jars # Mount the JARs directory
```
##########
website/docs/quickstart/lakehouse.md:
##########
Review Comment:
since we use standard flink image, we have no `tree` command in the `taskmanager`. I think we can remove this part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [doc/build] Fix iceberg quickstart to build on flink standard image [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #2555:
URL: https://github.com/apache/fluss/pull/2555#discussion_r2764191625
##########
website/docs/quickstart/lakehouse.md:
##########
@@ -220,33 +224,49 @@ services:
datalake.iceberg.warehouse: /tmp/iceberg
volumes:
- shared-tmpfs:/tmp/iceberg
-
+ - shared-tmpfs:/tmp/fluss
+ zookeeper:
+ restart: always
+ image: zookeeper:3.9.2
jobmanager:
- image: apache/fluss-quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ image: flink:1.20-scala_2.12-java17
ports:
- "8083:8081"
- command: jobmanager
+ entrypoint: ["/bin/bash", "-c"]
+ command: >
+ "sed -i 's/exec $(drop_privs_cmd)//g' /docker-entrypoint.sh &&
+ cp /tmp/jars/*.jar /opt/flink/lib/ 2>/dev/null || true;
+ cp /tmp/opt/*.jar /opt/flink/opt/ 2>/dev/null || true;
+ /docker-entrypoint.sh jobmanager"
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
volumes:
- shared-tmpfs:/tmp/iceberg
-
+ - shared-tmpfs:/tmp/fluss
+ - ./lib:/tmp/jars
+ - ./opt:/tmp/opt
taskmanager:
- image: apache/fluss-quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ image: flink:1.20-scala_2.12-java17
depends_on:
- jobmanager
- command: taskmanager
+ entrypoint: ["/bin/bash", "-c"]
+ command: >
+ "sed -i 's/exec $(drop_privs_cmd)//g' /docker-entrypoint.sh &&
+ cp /tmp/jars/*.jar /opt/flink/lib/ 2>/dev/null || true;
+ cp /tmp/opt/*.jar /opt/flink/opt/ 2>/dev/null || true;
+ /docker-entrypoint.sh taskmanager"
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
taskmanager.numberOfTaskSlots: 10
- taskmanager.memory.process.size: 2048m
Review Comment:
mistakenly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [doc/build] Fix iceberg quickstart to build on flink standard image [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #2555:
URL: https://github.com/apache/fluss/pull/2555#discussion_r2760563334
##########
website/docs/quickstart/lakehouse.md:
##########
Review Comment:
yes it will be better to better dependency mangement and to get rid of shading issues
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [doc/build] Fix iceberg quickstart to build on flink standard image [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra merged PR #2555:
URL: https://github.com/apache/fluss/pull/2555
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [doc/build] Fix iceberg quickstart to build on flink standard image [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #2555:
URL: https://github.com/apache/fluss/pull/2555#discussion_r2761834045
##########
website/docs/quickstart/lakehouse.md:
##########
@@ -199,9 +203,9 @@ services:
datalake.iceberg.warehouse: /tmp/iceberg
volumes:
- shared-tmpfs:/tmp/iceberg
- - ./lib:/tmp/lib
- entrypoint: ["sh", "-c", "cp -v /tmp/lib/*.jar /opt/fluss/plugins/iceberg/ && exec /docker-entrypoint.sh coordinatorServer"]
-
+ - shared-tmpfs:/tmp/fluss
+ - ./lib/fluss-lake-iceberg-$FLUSS_DOCKER_VERSION$.jar:/opt/fluss/plugins/iceberg/fluss-lake-iceberg.jar
+ - ./lib/hadoop-apache-3.3.5-2.jar:/opt/fluss/plugins/iceberg/hadoop-apache.jar
Review Comment:
dito, to add version for the jar mounted in image?
##########
website/docs/quickstart/lakehouse.md:
##########
@@ -220,33 +224,49 @@ services:
datalake.iceberg.warehouse: /tmp/iceberg
volumes:
- shared-tmpfs:/tmp/iceberg
-
+ - shared-tmpfs:/tmp/fluss
+ zookeeper:
+ restart: always
+ image: zookeeper:3.9.2
jobmanager:
- image: apache/fluss-quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ image: flink:1.20-scala_2.12-java17
ports:
- "8083:8081"
- command: jobmanager
+ entrypoint: ["/bin/bash", "-c"]
+ command: >
+ "sed -i 's/exec $(drop_privs_cmd)//g' /docker-entrypoint.sh &&
+ cp /tmp/jars/*.jar /opt/flink/lib/ 2>/dev/null || true;
+ cp /tmp/opt/*.jar /opt/flink/opt/ 2>/dev/null || true;
+ /docker-entrypoint.sh jobmanager"
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
volumes:
- shared-tmpfs:/tmp/iceberg
-
+ - shared-tmpfs:/tmp/fluss
+ - ./lib:/tmp/jars
+ - ./opt:/tmp/opt
taskmanager:
- image: apache/fluss-quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ image: flink:1.20-scala_2.12-java17
depends_on:
- jobmanager
- command: taskmanager
+ entrypoint: ["/bin/bash", "-c"]
+ command: >
+ "sed -i 's/exec $(drop_privs_cmd)//g' /docker-entrypoint.sh &&
+ cp /tmp/jars/*.jar /opt/flink/lib/ 2>/dev/null || true;
+ cp /tmp/opt/*.jar /opt/flink/opt/ 2>/dev/null || true;
+ /docker-entrypoint.sh taskmanager"
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
taskmanager.numberOfTaskSlots: 10
- taskmanager.memory.process.size: 2048m
Review Comment:
why remove this?
##########
website/docs/quickstart/lakehouse.md:
##########
@@ -220,9 +242,11 @@ services:
datalake.iceberg.warehouse: /tmp/iceberg
volumes:
- shared-tmpfs:/tmp/iceberg
Review Comment:
the tablet server will need to access remote file in getLatestLakeSnapshot method
##########
website/docs/quickstart/lakehouse.md:
##########
@@ -199,9 +203,9 @@ services:
datalake.iceberg.warehouse: /tmp/iceberg
volumes:
- shared-tmpfs:/tmp/iceberg
- - ./lib:/tmp/lib
- entrypoint: ["sh", "-c", "cp -v /tmp/lib/*.jar /opt/fluss/plugins/iceberg/ && exec /docker-entrypoint.sh coordinatorServer"]
-
+ - shared-tmpfs:/tmp/fluss
+ - ./lib/fluss-lake-iceberg-$FLUSS_DOCKER_VERSION$.jar:/opt/fluss/plugins/iceberg/fluss-lake-iceberg.jar
Review Comment:
nit:
```suggestion
- ./lib/fluss-lake-iceberg-$FLUSS_DOCKER_VERSION$.jar:/opt/fluss/plugins/iceberg/fluss-lake-iceberg-$FLUSS_DOCKER_VERSION$.jar
```
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [doc/build] Fix iceberg quickstart to build on flink standard image [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #2555:
URL: https://github.com/apache/fluss/pull/2555#discussion_r2760493789
##########
website/docs/quickstart/lakehouse.md:
##########
@@ -155,37 +155,60 @@ mkdir fluss-quickstart-iceberg
cd fluss-quickstart-iceberg
```
-2. Create a `lib` directory and download the required Hadoop jar file:
+2. Create directories and download required jars:
```shell
-mkdir lib
-wget -O lib/hadoop-apache-3.3.5-2.jar https://repo1.maven.org/maven2/io/trino/hadoop/hadoop-apache/3.3.5-2/hadoop-apache-3.3.5-2.jar
-```
+mkdir -p lib opt
-This jar file provides Hadoop 3.3.5 dependencies required for Iceberg's Hadoop catalog integration.
+# Flink connectors
+wget -O lib/flink-faker.jar https://github.com/knaufk/flink-faker/releases/download/v0.5.3/flink-faker-0.5.3.jar
Review Comment:
Agreed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org