You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "tnhgiang (via GitHub)" <gi...@apache.org> on 2025/04/22 08:28:42 UTC
[I] Iceberg rest catalog lost all schemas and tables after restarting docker container [iceberg]
tnhgiang opened a new issue, #12866:
URL: https://github.com/apache/iceberg/issues/12866
### Query engine
I'm using Trino
### Question
I can't find any guidance on how to locally persist data in a REST container. After restarting the container, all of my schemas and tables are gone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg rest catalog lost all schemas and tables after restarting docker container [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #12866:
URL: https://github.com/apache/iceberg/issues/12866#issuecomment-2820907701
@tnhgiang this isn't really related to Iceberg itself. In your configuration you basically need to make sure that your storage container (I'm assuming MinIO?) is persistent across restarts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg rest catalog lost all schemas and tables after restarting docker container [iceberg]
Posted by "tnhgiang (via GitHub)" <gi...@apache.org>.
tnhgiang commented on issue #12866:
URL: https://github.com/apache/iceberg/issues/12866#issuecomment-2820925281
@nastra I'm pretty sure that my storage container (MinIO in this case) was configured correctly, since the data created by REST is still persistent across restarts.
I also ran a small test to check whether the issue is related to the REST container. I restarted only the REST container, and all of my schemas and tables were gone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg rest catalog lost all schemas and tables after restarting docker container [iceberg]
Posted by "tnhgiang (via GitHub)" <gi...@apache.org>.
tnhgiang commented on issue #12866:
URL: https://github.com/apache/iceberg/issues/12866#issuecomment-2823384646
Thanks @ebyhr @nastra.
> You can try this docker compose yml (please download `postgresql-42.7.5.jar` beforehand):
>
> volumes:
> data: {}
>
> services:
> postgresql:
> container_name: postgresql
> image: postgres:12
> environment:
> POSTGRES_DB: 'test'
> POSTGRES_USER: 'test'
> POSTGRES_PASSWORD: 'test'
> ports:
> - "5432:5432"
> healthcheck:
> test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
> interval: 5s
> retries: 3
>
> minio:
> container_name: minio
> image: quay.io/minio/minio
> environment:
> MINIO_ROOT_USER: minioadmin
> MINIO_ROOT_PASSWORD: minioadmin
> ports:
> - "9000:9000"
> - "9001:9001"
> volumes:
> - data:/data
> command: server /data --console-address ":9001"
>
> create-bucket:
> image: minio/mc
> depends_on:
> - minio
> volumes:
> - data:/data
> entrypoint: mc mb /data/bucket
>
> irc:
> hostname: irc
> image: apache/iceberg-rest-fixture:1.8.1
> depends_on:
> postgresql:
> condition: service_healthy
> minio:
> condition: service_started
> volumes:
> - ./postgresql-42.7.5.jar:/usr/lib/iceberg-rest/postgresql-42.7.5.jar
> ports:
> - "8181:8181"
> environment:
> CATALOG_URI: jdbc:postgresql://postgresql:5432/test
> CATALOG_JDBC_USER: test
> CATALOG_JDBC_PASSWORD: test
> AWS_REGION: us-east-1
> CATALOG_WAREHOUSE: s3://bucket/warehouse/
> CATALOG_IO__IMPL: org.apache.iceberg.aws.s3.S3FileIO
> CATALOG_S3_ENDPOINT: http://minio:9000
> CATALOG_S3_PATH__STYLE__ACCESS: true
> CATALOG_S3_ACCESS__KEY__ID: minioadmin
> CATALOG_S3_SECRET__ACCESS__KEY: minioadmin
>
> command: java -cp /usr/lib/iceberg-rest/*:iceberg-rest-adapter.jar org.apache.iceberg.rest.RESTCatalogServer
>
> trino:
> container_name: trino
> image: trinodb/trino:474
> environment:
> CATALOG_MANAGEMENT: 'dynamic'
> depends_on:
> - irc
> - minio
> ports:
> - "8080:8080"
It works
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg rest catalog lost all schemas and tables after restarting docker container [iceberg]
Posted by "tnhgiang (via GitHub)" <gi...@apache.org>.
tnhgiang commented on issue #12866:
URL: https://github.com/apache/iceberg/issues/12866#issuecomment-2820983744
@ebyhr It seems to be a bit cumbersome. Is it easier if I switch to use `nessie` or `polaris`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg rest catalog lost all schemas and tables after restarting docker container [iceberg]
Posted by "tnhgiang (via GitHub)" <gi...@apache.org>.
tnhgiang closed issue #12866: Iceberg rest catalog lost all schemas and tables after restarting docker container
URL: https://github.com/apache/iceberg/issues/12866
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg rest catalog lost all schemas and tables after restarting docker container [iceberg]
Posted by "tnhgiang (via GitHub)" <gi...@apache.org>.
tnhgiang commented on issue #12866:
URL: https://github.com/apache/iceberg/issues/12866#issuecomment-2820956388
Thanks @ebyhr.
Could you please give me a docs on how to configure `iceberg-rest-fixture`? I cannot find any available docs related to this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg rest catalog lost all schemas and tables after restarting docker container [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #12866:
URL: https://github.com/apache/iceberg/issues/12866#issuecomment-2821307369
sorry I forgot that the default JDBCCatalog used by the REST image is an InMemory one as indicated [here](https://github.com/apache/iceberg/blob/b18ab74c3c2ecddc8f17c9bdff625996a8ee600c/docker/iceberg-rest-fixture/Dockerfile#L37)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg rest catalog lost all schemas and tables after restarting docker container [iceberg]
Posted by "tnhgiang (via GitHub)" <gi...@apache.org>.
tnhgiang commented on issue #12866:
URL: https://github.com/apache/iceberg/issues/12866#issuecomment-2820934203
Here is my `compose file`, plase have a look at it.
```yaml
services:
trino:
image: trinodb/trino:474
hostname: trino
container_name: trino
environment:
WAREHOUSE: s3a://${MINIO_WAREHOUSE_BUCKET}/
CATALOG_URI: ${REST_CATALOG_ENDPOINT}
S3_AWS_ENDPOINT: ${MINIO_ENDPOINT}
S3_AWS_REGION: ${MINIO_REGION}
S3_AWS_ACCESS_KEY: ${MINIO_ROOT_USER}
S3_AWS_SECRET_KEY: ${MINIO_ROOT_PASSWORD}
ports:
- ${TRINO_PORT}:${TRINO_PORT}
volumes:
- ../src/lakehouse/trino/etc/catalog:/etc/trino/catalog
depends_on:
- rest
- minio
rest:
image: apache/iceberg-rest-fixture:1.8.1
hostname: ${REST_CATALOG_HOST}
container_name: iceberg-rest
ports:
- ${REST_CATALOG_PORT}:${REST_CATALOG_PORT}
environment:
AWS_ACCESS_KEY_ID: ${MINIO_ROOT_USER}
AWS_SECRET_ACCESS_KEY: ${MINIO_ROOT_PASSWORD}
AWS_REGION: ${MINIO_REGION}
CATALOG_WAREHOUSE: s3a://${MINIO_WAREHOUSE_BUCKET}/
CATALOG_IO__IMPL: org.apache.iceberg.aws.s3.S3FileIO
CATALOG_S3_ENDPOINT: ${MINIO_ENDPOINT}
CATALOG_S3_PATH__STYLE__ACCESS: true
depends_on:
- minio
minio:
image: minio/minio:RELEASE.2025-04-08T15-41-24Z
hostname: ${MINIO_HOST}
container_name: minio
command: ["server", "/data", "--console-address", ":${MINIO_CONSOLE_PORT}"]
environment:
MINIO_ROOT_USER: ${MINIO_ROOT_USER}
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
ports:
- "${MINIO_CONSOLE_PORT}:${MINIO_CONSOLE_PORT}"
expose:
- "${MINIO_API_PORT}"
volumes:
- ../data/minio:/data
mc:
image: minio/mc:RELEASE.2025-04-08T15-39-49Z
container_name: mc
hostname: mc
entrypoint: >
/bin/sh -c "
until /usr/bin/mc config host add minio ${MINIO_ENDPOINT} ${MINIO_ROOT_USER} ${MINIO_ROOT_PASSWORD};
do
echo '...waiting...' && sleep 1;
done;
/usr/bin/mc mb minio/${MINIO_WAREHOUSE_BUCKET};
/usr/bin/mc policy set public minio/${MINIO_WAREHOUSE_BUCKET};
exit 0;
"
depends_on:
- minio
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg rest catalog lost all schemas and tables after restarting docker container [iceberg]
Posted by "ebyhr (via GitHub)" <gi...@apache.org>.
ebyhr commented on issue #12866:
URL: https://github.com/apache/iceberg/issues/12866#issuecomment-2820967698
You can try this docker compose yml (please download `postgresql-42.7.5.jar` beforehand):
```yml
volumes:
data: {}
services:
postgresql:
container_name: postgresql
image: postgres:12
environment:
POSTGRES_DB: 'test'
POSTGRES_USER: 'test'
POSTGRES_PASSWORD: 'test'
ports:
- "5432:5432"
healthcheck:
test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
interval: 5s
retries: 3
minio:
container_name: minio
image: quay.io/minio/minio
environment:
MINIO_ROOT_USER: minioadmin
MINIO_ROOT_PASSWORD: minioadmin
ports:
- "9000:9000"
- "9001:9001"
volumes:
- data:/data
command: server /data --console-address ":9001"
create-bucket:
image: minio/mc
depends_on:
- minio
volumes:
- data:/data
entrypoint: mc mb /data/bucket
irc:
hostname: irc
image: apache/iceberg-rest-fixture:1.8.1
depends_on:
postgresql:
condition: service_healthy
minio:
condition: service_started
volumes:
- ./postgresql-42.7.5.jar:/usr/lib/iceberg-rest/postgresql-42.7.5.jar
ports:
- "8181:8181"
environment:
CATALOG_URI: jdbc:postgresql://postgresql:5432/test
CATALOG_JDBC_USER: test
CATALOG_JDBC_PASSWORD: test
AWS_REGION: us-east-1
CATALOG_WAREHOUSE: s3://bucket/warehouse/
CATALOG_IO__IMPL: org.apache.iceberg.aws.s3.S3FileIO
CATALOG_S3_ENDPOINT: http://minio:9000
CATALOG_S3_PATH__STYLE__ACCESS: true
CATALOG_S3_ACCESS__KEY__ID: minioadmin
CATALOG_S3_SECRET__ACCESS__KEY: minioadmin
command: java -cp /usr/lib/iceberg-rest/*:iceberg-rest-adapter.jar org.apache.iceberg.rest.RESTCatalogServer
trino:
container_name: trino
image: trinodb/trino:474
environment:
CATALOG_MANAGEMENT: 'dynamic'
depends_on:
- irc
- minio
ports:
- "8080:8080"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg rest catalog lost all schemas and tables after restarting docker container [iceberg]
Posted by "ebyhr (via GitHub)" <gi...@apache.org>.
ebyhr commented on issue #12866:
URL: https://github.com/apache/iceberg/issues/12866#issuecomment-2820936674
I see you are using the default catalog (in-memory SQLite). Please use PostgreSQL or something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org