You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ming12713 (via GitHub)" <gi...@apache.org> on 2023/05/26 07:52:10 UTC
[GitHub] [iceberg] ming12713 opened a new issue, #7709: org.apache.iceberg.exceptions.ServiceFailureException: Server error: SdkClientException: Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previoorg.apache.iceberg.exceptions.ServiceFailureException: Server error: SdkClientException: Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long.usly worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long.
ming12713 opened a new issue, #7709:
URL: https://github.com/apache/iceberg/issues/7709
### Apache Iceberg version
1.2.1 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
Follow this [bkm](https://iceberg.apache.org/spark-quickstart/) test rest catalog is ok
Attempting to modify spark and restcata to with production MinIO address <los.xxxx.com>, there is an issue with can't resolve dns "bucket.los.xxxx.com"
rest cataserver log
```
org.apache.iceberg.exceptions.RESTException: Unhandled error: ErrorResponse(code=500, type=SdkClientException, message=Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long.)
software.amazon.awssdk.core.exception.SdkClientException: Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long.
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
at software.amazon.awssdk.awscore.interceptor.HelpfulUnknownHostExceptionInterceptor.modifyException(HelpfulUnknownHostExceptionInterceptor.java:59)
at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.modifyException(ExecutionInterceptorChain.java:202)
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.ExceptionReportingUtils.runModifyException(ExceptionReportingUtils.java:54)
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.ExceptionReportingUtils.reportFailureToInterceptors(ExceptionReportingUtils.java:38)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:39)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:171)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:82)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)
at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:9321)
at org.apache.iceberg.aws.s3.S3OutputStream.completeUploads(S3OutputStream.java:435)
at org.apache.iceberg.aws.s3.S3OutputStream.close(S3OutputStream.java:269)
at java.base/sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:347)
at java.base/sun.nio.cs.StreamEncoder.close(StreamEncoder.java:169)
at java.base/java.io.OutputStreamWriter.close(OutputStreamWriter.java:252)
at org.apache.iceberg.TableMetadataParser.$closeResource(TableMetadataParser.java:131)
at org.apache.iceberg.TableMetadataParser.internalWrite(TableMetadataParser.java:131)
at org.apache.iceberg.TableMetadataParser.overwrite(TableMetadataParser.java:114)
at org.apache.iceberg.BaseMetastoreTableOperations.writeNewMetadata(BaseMetastoreTableOperations.java:170)
at org.apache.iceberg.BaseMetastoreTableOperations.writeNewMetadataIfRequired(BaseMetastoreTableOperations.java:160)
at org.apache.iceberg.jdbc.JdbcTableOperations.doCommit(JdbcTableOperations.java:106)
at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:135)
at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:199)
at org.apache.iceberg.rest.CatalogHandlers.createTable(CatalogHandlers.java:214)
at org.apache.iceberg.rest.RESTCatalogAdapter.handleRequest(RESTCatalogAdapter.java:319)
at org.apache.iceberg.rest.RESTCatalogAdapter.execute(RESTCatalogAdapter.java:384)
at org.apache.iceberg.rest.RESTCatalogServlet.execute(RESTCatalogServlet.java:100)
at org.apache.iceberg.rest.RESTCatalogServlet.doPost(RESTCatalogServlet.java:78)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:713)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: xx.los.xxxx.com
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:223)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:83)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
... 59 more
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: xx.los.xxxx.com
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 2 failure: Unable to execute HTTP request: xx.los.xxxx.com
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 3 failure: Unable to execute HTTP request: xx.los.xxxx.com
Caused by: java.net.UnknownHostException: xx.los.xxxx.com
```
**However, using IP resolution instead of the domain name does not encounter any problems.**
spark conf
```
spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.demo org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.demo.catalog-impl org.apache.iceberg.rest.RESTCatalog
spark.sql.catalog.demo.uri http://rest:8181
spark.sql.catalog.demo.io-impl org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.demo.warehouse s3a://xx/
spark.sql.catalog.demo.s3.endpoint http://los.xxxx.com:9000
spark.sql.defaultCatalog demo
spark.eventLog.enabled true
spark.eventLog.dir /home/iceberg/spark-events
spark.history.fs.logDirectory /home/iceberg/spark-events
spark.sql.catalogImplementation in-memory
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ming12713 commented on issue #7709: reset catalog can't reslove s3 domain name
Posted by "ming12713 (via GitHub)" <gi...@apache.org>.
ming12713 commented on issue #7709:
URL: https://github.com/apache/iceberg/issues/7709#issuecomment-1569403840
> Same here! It seems that within the Spark context, a different DNS resolver is used. We currently did an ugly workaround by giving the container a fixed IP and reference this in the config.
>
> Any clean solve is much appreciated!
i also use ip as workaroud solution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] reset catalog can't reslove s3 domain name [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7709:
URL: https://github.com/apache/iceberg/issues/7709#issuecomment-1837652801
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on issue #7709: reset catalog can't reslove s3 domain name
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #7709:
URL: https://github.com/apache/iceberg/issues/7709#issuecomment-1570413628
Can you try setting:
```
s3.path-style-access=true
```
This can be done on a catalog level:
```sh
spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.demo org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.demo.catalog-impl org.apache.iceberg.rest.RESTCatalog
spark.sql.catalog.demo.uri http://rest:8181
spark.sql.catalog.demo.io-impl org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.demo.warehouse s3a://xx/
spark.sql.catalog.demo.s3.endpoint http://los.xxxx.com:9000
spark.sql.catalog.demo.s3.path-style-access. true
spark.sql.defaultCatalog demo
spark.eventLog.enabled true
spark.eventLog.dir /home/iceberg/spark-events
spark.history.fs.logDirectory /home/iceberg/spark-events
spark.sql.catalogImplementation in-memory
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] reset catalog can't reslove s3 domain name [iceberg]
Posted by "sg-c0de (via GitHub)" <gi...@apache.org>.
sg-c0de commented on issue #7709:
URL: https://github.com/apache/iceberg/issues/7709#issuecomment-1993790181
> Is there anyone who solved this issue?
Try to add env variable to the rest container: CATALOG_S3_PATH__STYLE__ACCESS: true
It'll be [converted](https://github.com/tabular-io/iceberg-rest-image/blob/2e4d04184e6db38f23a98498151aa18bb6c148ab/src/main/java/org/apache/iceberg/rest/RESTCatalogServer.java#L54) to s3.path-style-access=true on catalog creation and passed to [S3FileIO ](https://github.com/apache/iceberg/blob/43c3397528101859250160f123a0749bae79fb4d/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java#L178)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] reset catalog can't reslove s3 domain name [iceberg]
Posted by "springbearpro (via GitHub)" <gi...@apache.org>.
springbearpro commented on issue #7709:
URL: https://github.com/apache/iceberg/issues/7709#issuecomment-2028825632
@sg-c0de Thanks. Saved my day. I can't find it in the iceberg document though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] SvenThies commented on issue #7709: reset catalog can't reslove s3 domain name
Posted by "SvenThies (via GitHub)" <gi...@apache.org>.
SvenThies commented on issue #7709:
URL: https://github.com/apache/iceberg/issues/7709#issuecomment-1568783459
Same here! It seems that within the Spark context, a different DNS resolver is used. We currently did an ugly workaround by giving the container a fixed IP and reference this in the config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] reset catalog can't reslove s3 domain name [iceberg]
Posted by "sg-c0de (via GitHub)" <gi...@apache.org>.
sg-c0de commented on issue #7709:
URL: https://github.com/apache/iceberg/issues/7709#issuecomment-2028827829
> > > Is there anyone who solved this issue?
> >
> >
> > Try to add env variable to the rest container: `CATALOG_S3_PATH__STYLE__ACCESS: true`
> > It'll be [converted](https://github.com/tabular-io/iceberg-rest-image/blob/2e4d04184e6db38f23a98498151aa18bb6c148ab/src/main/java/org/apache/iceberg/rest/RESTCatalogServer.java#L54) to `s3.path-style-access=true` when the catalog is created and passed to [S3FileIO ](https://github.com/apache/iceberg/blob/43c3397528101859250160f123a0749bae79fb4d/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java#L178)
> > @sg-c0de
> > Can the jdbc catalog be configured like this?
@springbearpro Sure, you can do it by passing this parameter in the config of ETL tool you use. For spark in spark-defaults.conf it will be like:
spark.sql.catalog.catalof_name.s3.path-style-access true
Full listing
```
spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.demo org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.demo.type jdbc
spark.sql.catalog.demo.uri jdbc:postgresql://192.168.0.16:5432/jdbc_catalog_test
spark.sql.catalog.demo.jdbc.verifyServerCertificate false
spark.sql.catalog.demo.jdbc.useSSL false
spark.sql.catalog.demo.jdbc.user jdbc
spark.sql.catalog.demo.jdbc.password pass
spark.sql.catalog.demo.io-impl org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.demo.warehouse s3://iceberg-test-jdbc/
spark.sql.catalog.demo.s3.endpoint https://minio.example.com
spark.sql.catalog.demo.s3.path-style-access true
spark.sql.defaultCatalog demo
spark.eventLog.enabled true
spark.eventLog.dir /home/iceberg/spark-events
spark.history.fs.logDirectory /home/iceberg/spark-events
spark.sql.catalogImplementation in-memory
```
For trino it will be **s3.path-style-access** in the catalog properties
https://trino.io/docs/current/object-storage/file-system-s3.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] reset catalog can't reslove s3 domain name [iceberg]
Posted by "dongsupkim-onepredict (via GitHub)" <gi...@apache.org>.
dongsupkim-onepredict commented on issue #7709:
URL: https://github.com/apache/iceberg/issues/7709#issuecomment-1965763850
Is there anyone who solved this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] reset catalog can't reslove s3 domain name [iceberg]
Posted by "ming12713 (via GitHub)" <gi...@apache.org>.
ming12713 commented on issue #7709:
URL: https://github.com/apache/iceberg/issues/7709#issuecomment-2008705570
> > Is there anyone who solved this issue?
>
> Try to add env variable to the rest container: `CATALOG_S3_PATH__STYLE__ACCESS: true`
>
> It'll be [converted](https://github.com/tabular-io/iceberg-rest-image/blob/2e4d04184e6db38f23a98498151aa18bb6c148ab/src/main/java/org/apache/iceberg/rest/RESTCatalogServer.java#L54) to `s3.path-style-access=true` when the catalog is created and passed to [S3FileIO ](https://github.com/apache/iceberg/blob/43c3397528101859250160f123a0749bae79fb4d/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java#L178)
@sg-c0de
Can the JDBC type catalog be configured like this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] reset catalog can't reslove s3 domain name [iceberg]
Posted by "sg-c0de (via GitHub)" <gi...@apache.org>.
sg-c0de commented on issue #7709:
URL: https://github.com/apache/iceberg/issues/7709#issuecomment-2029161591
> > > Is there anyone who solved this issue?
> >
> >
> > Try to add env variable to the rest container: `CATALOG_S3_PATH__STYLE__ACCESS: true`
> > It'll be [converted](https://github.com/tabular-io/iceberg-rest-image/blob/2e4d04184e6db38f23a98498151aa18bb6c148ab/src/main/java/org/apache/iceberg/rest/RESTCatalogServer.java#L54) to `s3.path-style-access=true` when the catalog is created and passed to [S3FileIO ](https://github.com/apache/iceberg/blob/43c3397528101859250160f123a0749bae79fb4d/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java#L178)
> > @sg-c0de
> > Can the jdbc catalog be configured like this?
@ming12713 Sure, you can do it by passing this parameter in the config of ETL tool you use. For spark in spark-defaults.conf it will be like:
spark.sql.catalog.catalof_name.s3.path-style-access true
Full listing
```
spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.demo org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.demo.type jdbc
spark.sql.catalog.demo.uri jdbc:postgresql://192.168.0.16:5432/jdbc_catalog_test
spark.sql.catalog.demo.jdbc.verifyServerCertificate false
spark.sql.catalog.demo.jdbc.useSSL false
spark.sql.catalog.demo.jdbc.user jdbc
spark.sql.catalog.demo.jdbc.password pass
spark.sql.catalog.demo.io-impl org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.demo.warehouse s3://iceberg-test-jdbc/
spark.sql.catalog.demo.s3.endpoint https://minio.example.com
spark.sql.catalog.demo.s3.path-style-access true
spark.sql.defaultCatalog demo
spark.eventLog.enabled true
spark.eventLog.dir /home/iceberg/spark-events
spark.history.fs.logDirectory /home/iceberg/spark-events
spark.sql.catalogImplementation in-memory
```
For trino it will be **s3.path-style-access** in the catalog properties
https://trino.io/docs/current/object-storage/file-system-s3.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] reset catalog can't reslove s3 domain name [iceberg]
Posted by "dongsupkim-onepredict (via GitHub)" <gi...@apache.org>.
dongsupkim-onepredict commented on issue #7709:
URL: https://github.com/apache/iceberg/issues/7709#issuecomment-2008658279
It works Thanks! @sg-c0de
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ming12713 commented on issue #7709: reset catalog can't reslove s3 domain name
Posted by "ming12713 (via GitHub)" <gi...@apache.org>.
ming12713 commented on issue #7709:
URL: https://github.com/apache/iceberg/issues/7709#issuecomment-1577843182
> Can you try setting:
>
> ```
> s3.path-style-access=true
> ```
>
> This can be done on a catalog level:
>
> ```shell
> spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
> spark.sql.catalog.demo org.apache.iceberg.spark.SparkCatalog
> spark.sql.catalog.demo.catalog-impl org.apache.iceberg.rest.RESTCatalog
> spark.sql.catalog.demo.uri http://rest:8181
> spark.sql.catalog.demo.io-impl org.apache.iceberg.aws.s3.S3FileIO
> spark.sql.catalog.demo.warehouse s3a://xx/
> spark.sql.catalog.demo.s3.endpoint http://los.xxxx.com:9000
> spark.sql.catalog.demo.s3.path-style-access true
> spark.sql.defaultCatalog demo
> spark.eventLog.enabled true
> spark.eventLog.dir /home/iceberg/spark-events
> spark.history.fs.logDirectory /home/iceberg/spark-events
> spark.sql.catalogImplementation in-memory
>
Thanks @Fokko
But, Failed to modify sql.catalog
```
spark.hadoop.fs.s3a.aws.credentials.provider org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
spark.hadoop.fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.endpoint.region cn-north-1
spark.hadoop.fs.s3a.access.key xm2U3ww
spark.hadoop.fs.s3a.secret.key eDMZQAulRQJm
spark.hadoop.fs.s3a.endpoint http://los.xxx.com
spark.hadoop.fs.s3a.bucket.probe 0
spark.hadoop.fs.s3a.change.detection.version.required false
spark.hadoop.fs.s3a.change.detection.mode none
spark.hadoop.fs.s3a.path.style.access true
# spark jdbc
spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.jdbc org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.jdbc.catalog-impl org.apache.iceberg.jdbc.JdbcCatalog
spark.sql.catalog.jdbc.uri jdbc:postgresql://10.0.162.5:31884/jdbc_gm
spark.sql.catalog.jdbc.jdbc.user postgres
spark.sql.catalog.jdbc.jdbc.password QzBodddd
spark.sql.catalog.jdbc.warehouse s3://minio-jdbc
spark.sql.catalog.jdbc.io-impl org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.jdbc.s3.endpoint http://los.xxx.com
spark.sql.catalog.jdbc.s3.path-style-access true
spark.eventLog.enabled true
```
errors output
AWS profile is configured in os profile
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] SvenThies commented on issue #7709: reset catalog can't reslove s3 domain name
Posted by "SvenThies (via GitHub)" <gi...@apache.org>.
SvenThies commented on issue #7709:
URL: https://github.com/apache/iceberg/issues/7709#issuecomment-1571479183
Hey @Fokko,
thanks for the suggestions. Unfortunately, it doesn't work. We tried the following
```
spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.default org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.default.catalog-impl org.apache.iceberg.rest.RESTCatalog
spark.sql.catalog.default.uri http://rest:8181
spark.sql.catalog.default.io-impl org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.default.warehouse s3://dev-customer-analytics/
spark.sql.catalog.default.s3.endpoint http://minio:9000
spark.sql.catalog.default.s3.path-style-access true
spark.sql.defaultCatalog default
spark.sql.sources.partitionOverwriteMode dynamic
spark.eventLog.enabled true
spark.eventLog.dir /home/iceberg/spark-events
spark.history.fs.logDirectory /home/iceberg/spark-events
spark.sql.catalogImplementation in-memory
spark.jars.packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.3.2,org.apache.kafka:kafka-clients:2.8.1,org.apache.commons:commons-pool2:2.11.1,org.apache.spark:spark-token-provider-kafka-0-10_2.12:3.3.2,com.google.code.findbugs:jsr305:3.0.0,org.apache.spark:spark-tags_2.12:3.3.2
```
Additionally, we tried setting `spark.hadoop.fs.s3a.path.style.access true` in the `spark-defaults.conf`, but didn't work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] reset catalog can't reslove s3 domain name [iceberg]
Posted by "ming12713 (via GitHub)" <gi...@apache.org>.
ming12713 commented on issue #7709:
URL: https://github.com/apache/iceberg/issues/7709#issuecomment-2029217628
> > > > Is there anyone who solved this issue?
> > >
> > >
> > > Try to add env variable to the rest container: `CATALOG_S3_PATH__STYLE__ACCESS: true`
> > > It'll be [converted](https://github.com/tabular-io/iceberg-rest-image/blob/2e4d04184e6db38f23a98498151aa18bb6c148ab/src/main/java/org/apache/iceberg/rest/RESTCatalogServer.java#L54) to `s3.path-style-access=true` when the catalog is created and passed to [S3FileIO ](https://github.com/apache/iceberg/blob/43c3397528101859250160f123a0749bae79fb4d/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java#L178)
> > > @sg-c0de
> > > Can the jdbc catalog be configured like this?
>
> @ming12713 Sure, you can do it by passing this parameter in the config of ETL tool you use. For spark in spark-defaults.conf it will be like: spark.sql.catalog.catalof_name.s3.path-style-access true
>
> Full listing
>
> ```
> spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
> spark.sql.catalog.demo org.apache.iceberg.spark.SparkCatalog
> spark.sql.catalog.demo.type jdbc
> spark.sql.catalog.demo.uri jdbc:postgresql://192.168.0.16:5432/jdbc_catalog_test
> spark.sql.catalog.demo.jdbc.verifyServerCertificate false
> spark.sql.catalog.demo.jdbc.useSSL false
> spark.sql.catalog.demo.jdbc.user jdbc
> spark.sql.catalog.demo.jdbc.password pass
> spark.sql.catalog.demo.io-impl org.apache.iceberg.aws.s3.S3FileIO
> spark.sql.catalog.demo.warehouse s3://iceberg-test-jdbc/
> spark.sql.catalog.demo.s3.endpoint https://minio.example.com
> spark.sql.catalog.demo.s3.path-style-access true
> spark.sql.defaultCatalog demo
> spark.eventLog.enabled true
> spark.eventLog.dir /home/iceberg/spark-events
> spark.history.fs.logDirectory /home/iceberg/spark-events
> spark.sql.catalogImplementation in-memory
> ```
>
> For trino it will be **s3.path-style-access** in the catalog properties https://trino.io/docs/current/object-storage/file-system-s3.html
@sg-c0de it's work thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] reset catalog can't reslove s3 domain name [iceberg]
Posted by "ming12713 (via GitHub)" <gi...@apache.org>.
ming12713 closed issue #7709: reset catalog can't reslove s3 domain name
URL: https://github.com/apache/iceberg/issues/7709
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org