You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "larryloi (via GitHub)" <gi...@apache.org> on 2025/01/23 16:06:03 UTC
[I] [Bug] [Connector v2] Sink to iceberg MinIO; but getting org.apache.iceberg.aws.s3.S3FileIO does not implement FileIO [seatunnel]
larryloi opened a new issue, #8585:
URL: https://github.com/apache/seatunnel/issues/8585
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
Iceberg sink Connector got error to spark-iceberg
Cannot initialize FileIO, org.apache.iceberg.aws.s3.S3FileIO does not implement FileIO
=== With below spark iceberg environment
spark iceberg environment from https://github.com/tabular-io/docker-spark-iceberg
ENV SPARK_VERSION=3.5.2
ENV SPARK_MAJOR_VERSION=3.5
ENV ICEBERG_VERSION=1.6.0
seatunnel environment build with below docker compose
image: apache/seatunnel:2.3.9
container_name: seatunnel-master
environment:
- TZ=Asia/Macau
- ST_DOCKER_MEMBER_LIST=172.19.0.11,172.19.0.12,172.19.0.13
entrypoint: >
/bin/sh -c "
/opt/seatunnel/bin/seatunnel-cluster.sh -r master -DJvmOption=\"-Xms2G -Xmx2G\"
"
ports:
- "5701:5701"
volumes:
- ./config:/opt/seatunnel/config
networks:
integration:
ipv4_address: 172.19.0.11
(2 worker nodes are the same)
### SeaTunnel Version
tried 2.3.8 and 2.3.9 docker images
### SeaTunnel Config
```conf
env {
job.name = "mssql_starrocks.inventory.INV.orders__ODS_startup_latest"
parallelism = 1
job.mode = "STREAMING"
checkpoint.interval = 5000
}
source {
SqlServer-CDC {
base-url = "jdbc:sqlserver://db01:1433;databaseName=inventory"
plugin_output = "orders_demo00"
username = "seatunnel_src"
password = ""
startup.mode="latest"
database-names = ["inventory"]
table-names = ["inventory.INV.orders_demo00"]
table-names-config = [
{
table = "inventory.INV.orders_demo00"
primaryKeys = ["id"]
}
]
}
}
sink {
Iceberg {
catalog_name = "ods"
iceberg.catalog.config = {
"type" = "rest"
"uri" = "http://iceberg-rest:8181"
"warehouse" = "s3://warehouse/"
"io-impl" = "org.apache.iceberg.aws.s3.S3FileIO"
"s3.endpoint" = "http://minio:9000"
"s3.access-key-id" = "admin"
"s3.secret-access-key" = "password"
"s3.path-style-access" = true
}
namespace = "ods_namespace"
table = "iceberg_sink_orders_demo00"
iceberg.table.write-props = {
write.format.default = "parquet"
write.target-file-size-bytes = 536870912
}
iceberg.table.primary-keys = ["id"]
iceberg.table.partition-keys = ["f_datetime"]
iceberg.table.upsert-mode-enabled = true
iceberg.table.schema-evolution-enabled = true
case_sensitive = true
}
}
```
### Running Command
```shell
./bin/seatunnel.sh --config ./JOBS/mssql_iceberg/inventory.INV.orders__ODS_startup_latest.yaml
```
### Error Exception
```log
=== But got the below error, and already tried different verisons of iceberg-aws-bundle-<version>.jar and iceberg-spark-runtime-<SPARK_MAJOR_VERSION>_2.12-<ICEBERG_VERSION>.jar. but also get the same problem. is it FileIO did not implement in org.apache.iceberg.aws.s3.S3FileIO or org.apache.iceberg.io.FileIO ????
Caused by: java.lang.IllegalArgumentException: Cannot initialize FileIO, org.apache.iceberg.aws.s3.S3FileIO does not implement FileIO.
at org.apache.iceberg.CatalogUtil.loadFileIO(CatalogUtil.java:320)
at org.apache.iceberg.rest.RESTSessionCatalog.newFileIO(RESTSessionCatalog.java:827)
at org.apache.iceberg.rest.RESTSessionCatalog.initialize(RESTSessionCatalog.java:204)
at org.apache.iceberg.rest.RESTCatalog.initialize(RESTCatalog.java:72)
at org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:239)
at org.apache.iceberg.CatalogUtil.buildIcebergCatalog(CatalogUtil.java:284)
at org.apache.seatunnel.connectors.seatunnel.iceberg.IcebergCatalogLoader.loadCatalog(IcebergCatalogLoader.java:61)
at org.apache.seatunnel.connectors.seatunnel.iceberg.catalog.IcebergCatalog.open(IcebergCatalog.java:91)
at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.open(DefaultSaveModeHandler.java:78)
at org.apache.seatunnel.engine.server.master.JobMaster.handleSaveMode(JobMaster.java:523)
... 21 more
Caused by: java.lang.ClassCastException: org.apache.iceberg.aws.s3.S3FileIO cannot be cast to org.apache.iceberg.io.FileIO
at org.apache.iceberg.CatalogUtil.loadFileIO(CatalogUtil.java:317)
... 30 more
```
### Zeta or Flink or Spark Version
ENV SPARK_VERSION=3.5.2
ENV SPARK_MAJOR_VERSION=3.5
ENV ICEBERG_VERSION=1.6.0
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Connector v2] Sink to iceberg MinIO; but getting org.apache.iceberg.aws.s3.S3FileIO does not implement FileIO [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8585:
URL: https://github.com/apache/seatunnel/issues/8585#issuecomment-2677219168
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Connector v2] Sink to iceberg MinIO; but getting org.apache.iceberg.aws.s3.S3FileIO does not implement FileIO [seatunnel]
Posted by "larryloi (via GitHub)" <gi...@apache.org>.
larryloi commented on issue #8585:
URL: https://github.com/apache/seatunnel/issues/8585#issuecomment-2611505049
means I cannot use "rest" type to connect iceberg-rest and minIO endpoint directly. we need to use spark engine to access MinIO, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Connector v2] Sink to iceberg MinIO; but getting org.apache.iceberg.aws.s3.S3FileIO does not implement FileIO [seatunnel]
Posted by "litiliu (via GitHub)" <gi...@apache.org>.
litiliu commented on issue #8585:
URL: https://github.com/apache/seatunnel/issues/8585#issuecomment-2611458794
I think it's related to classloader or classpath, Seems the org.apache.iceberg.aws.s3.S3FileIO And org.apache.iceberg.io.FileIO were loaded by the different classloader. Have you tried to use the Flink or Spark engine?
<img width="468" alt="Image" src="https://github.com/user-attachments/assets/bcb6870f-9e9a-4802-8151-966a093a7064" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Connector v2] Sink to iceberg MinIO; but getting org.apache.iceberg.aws.s3.S3FileIO does not implement FileIO [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8585:
URL: https://github.com/apache/seatunnel/issues/8585#issuecomment-2693003668
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Connector v2] Sink to iceberg MinIO; but getting org.apache.iceberg.aws.s3.S3FileIO does not implement FileIO [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #8585: [Bug] [Connector v2] Sink to iceberg MinIO; but getting org.apache.iceberg.aws.s3.S3FileIO does not implement FileIO
URL: https://github.com/apache/seatunnel/issues/8585
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org