[I] [Bug] Iceberg REST metadata committer cannot create tables on AWS Glue [paimon]

["tejanshrana (via GitHub)" <gi...@apache.org>]

tejanshrana opened a new issue, #8291:
URL: https://github.com/apache/paimon/issues/8291

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
   
   
   ### Paimon version
   
   1.4.1 (also reproduces on master since IcebergRestMetadataCommitter#createTable is unchanged).
   
   ### Compute Engine
   
   Flink (Paimon sink with metadata.iceberg.storage = rest-catalog).
   
   ### Minimal reproduce step
   
   1. Create a Paimon table whose warehouse is on S3 via the s3a:// filesystem with Iceberg REST compatibility pointed at AWS Glue's Iceberg REST endpoint:
   
   ```
   'metadata.iceberg.storage' = 'rest-catalog'
   'metadata.iceberg.rest.uri' = 'https://glue.us-west-2.amazonaws.com/iceberg'
   'metadata.iceberg.rest.warehouse' = '<aws-account-id>'
   'metadata.iceberg.rest.rest.sigv4-enabled' = 'true'
   'metadata.iceberg.rest.rest.signing-region' = 'us-west-2'
   'metadata.iceberg.rest.rest.signing-name' = 'glue'
   ```
   Note: warehouse path uses s3a://, e.g. s3a://<bucket>/ingest/<db>.db/<table>
   
   2. Write a record and commit so the committer attempts to create the Iceberg table in Glue.
   
   ### What doesn't meet your expectations?
   
   The first commit fails in IcebergRestMetadataCommitter#createTable. Two reasons:
    
   1. No table location is sent. createTable calls restCatalog.createTable(identifier, schema[, spec]) without a location. AWS Glue's standard Iceberg REST catalog does not auto-assign a table location, so it rejects the request: `BadRequest: Location information cannot be null` (This is not Paimon-specific. Native Iceberg's own createTable against Glue REST fails the same way. Catalogs that auto-assign a location, e.g. Hadoop/RESTCatalogServer with a HadoopCatalog backend, happen to tolerate the missing location, which is why existing tests don't catch it. Setting the Glue database LocationUri does not help either since the namespace-derived location isn't applied to the table.)
   2. The s3a:// scheme is rejected with `Invalid S3 URI: s3a://...` Paimon's warehouse uses Flink's s3a:// (or legacy s3n://) Hadoop filesystem, but Glue validates the table location and only accepts the s3:// scheme.
   
   
   **Expected:** the REST committer should create the table with the location Paimon already writes its metadata to, with the scheme normalised to s3:// (same physical object). With both fixes, the full Flink to Paimon to Glue create/commit/read should work.
   
   ### Anything else?
   
   Fix is small and contained in createTable: switch to the builder form and pass the (scheme-normalized) location:
   ```
   restCatalog.buildTable(identifier, schema)
       .withPartitionSpec(spec)            // partitioned branch only
       .withLocation(toRestLocation(newMetadata.location()))
       .create();
   ```
   plus a toRestLocation helper that rewrites s3a:///s3n:// to s3:// and leaves other schemes (and null) unchanged. 
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] Iceberg REST metadata committer cannot create tables on AWS Glue [paimon]

["tejanshrana (via GitHub)" <gi...@apache.org>]

tejanshrana commented on issue #8291:
URL: https://github.com/apache/paimon/issues/8291#issuecomment-5251142818

   Closed by https://github.com/apache/paimon/pull/8290 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] Iceberg REST metadata committer cannot create tables on AWS Glue [paimon]

["bube (via GitHub)" <gi...@apache.org>]

bube commented on issue #8291:
URL: https://github.com/apache/paimon/issues/8291#issuecomment-5250881388

   I think this can be closed, right? Looks like the fix landed in master under a different issue id. See https://github.com/apache/paimon/pull/8290


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] Iceberg REST metadata committer cannot create tables on AWS Glue [paimon]

["tejanshrana (via GitHub)" <gi...@apache.org>]

tejanshrana closed issue #8291: [Bug] Iceberg REST metadata committer cannot create tables on AWS Glue
URL: https://github.com/apache/paimon/issues/8291


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org