You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "coded9 (via GitHub)" <gi...@apache.org> on 2023/08/13 17:07:57 UTC
[GitHub] [iceberg] coded9 opened a new pull request, #8315: fix(aws): Unclosed S3FileIO instance in GlueTableOperations
coded9 opened a new pull request, #8315:
URL: https://github.com/apache/iceberg/pull/8315
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] coded9 commented on a diff in pull request #8315: fix(aws): Unclosed S3FileIO instance in GlueTableOperations
Posted by "coded9 (via GitHub)" <gi...@apache.org>.
coded9 commented on code in PR #8315:
URL: https://github.com/apache/iceberg/pull/8315#discussion_r1294477551
##########
aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java:
##########
@@ -238,7 +238,8 @@ protected TableOperations newTableOps(TableIdentifier tableIdentifier) {
awsProperties,
tableSpecificCatalogPropertiesBuilder.buildOrThrow(),
hadoopConf,
- tableIdentifier);
+ tableIdentifier,
+ closeableGroup);
Review Comment:
Changed as suggested
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a diff in pull request #8315: fix(aws): Unclosed S3FileIO instance in GlueTableOperations
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #8315:
URL: https://github.com/apache/iceberg/pull/8315#discussion_r1293236872
##########
aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java:
##########
@@ -238,7 +238,8 @@ protected TableOperations newTableOps(TableIdentifier tableIdentifier) {
awsProperties,
tableSpecificCatalogPropertiesBuilder.buildOrThrow(),
hadoopConf,
- tableIdentifier);
+ tableIdentifier,
+ closeableGroup);
Review Comment:
no need to pass the `closeableGroup`. Why not just
```
--- a/aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java
+++ b/aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java
@@ -217,24 +217,30 @@ public class GlueCatalog extends BaseMetastoreCatalog
// FileIO initialization depends on tableSpecificCatalogProperties, so a new FileIO is
// initialized each time
- return new GlueTableOperations(
- glue,
- lockManager,
- catalogName,
- awsProperties,
- tableSpecificCatalogPropertiesBuilder.buildOrThrow(),
- hadoopConf,
- tableIdentifier);
+ GlueTableOperations tableOperations =
+ new GlueTableOperations(
+ glue,
+ lockManager,
+ catalogName,
+ awsProperties,
+ tableSpecificCatalogPropertiesBuilder.buildOrThrow(),
+ hadoopConf,
+ tableIdentifier);
+ closeableGroup.addCloseable(tableOperations.io());
+ return tableOperations;
}
- return new GlueTableOperations(
- glue,
- lockManager,
- catalogName,
- awsProperties,
- catalogProperties,
- hadoopConf,
- tableIdentifier);
+ GlueTableOperations tableOperations =
+ new GlueTableOperations(
+ glue,
+ lockManager,
+ catalogName,
+ awsProperties,
+ catalogProperties,
+ hadoopConf,
+ tableIdentifier);
+ closeableGroup.addCloseable(tableOperations.io());
+ return tableOperations;
```
But even then, this only solves it for the case where the `GlueCatalog` itself is being actively closed by calling `close()`.
If you're running Spark and using `GlueCatalog` and a `FileIO` instance is being created on an executor node, then there's no backlink anymore and we don't know when/how to call `close()`.
That's why I suggested to fix this issue similarly to how it was done in https://github.com/apache/iceberg/pull/7487
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] coded9 commented on pull request #8315: AWS: Add FileIO tracker/closer to Glue catalog
Posted by "coded9 (via GitHub)" <gi...@apache.org>.
coded9 commented on PR #8315:
URL: https://github.com/apache/iceberg/pull/8315#issuecomment-1679052617
> LGTM, thanks for fixing this @coded9. @jackye1995 could you also take a look please?
Thanks for helping me @nastra!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] coded9 commented on a diff in pull request #8315: fix(aws): Unclosed S3FileIO instance in GlueTableOperations
Posted by "coded9 (via GitHub)" <gi...@apache.org>.
coded9 commented on code in PR #8315:
URL: https://github.com/apache/iceberg/pull/8315#discussion_r1294282908
##########
aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java:
##########
@@ -619,4 +635,21 @@ public void setConf(Configuration conf) {
protected Map<String, String> properties() {
return catalogProperties == null ? ImmutableMap.of() : catalogProperties;
}
+
+ private Cache<TableOperations, FileIO> newFileIOCloser() {
+ return Caffeine.newBuilder()
+ .weakKeys()
+ .removalListener(
+ (RemovalListener<TableOperations, FileIO>)
+ (ops, fileIO, cause) -> {
+ if (null != fileIO) {
+ fileIO.close();
+ }
+ })
+ .build();
+ }
+
+ private void trackFileIO(GlueTableOperations ops) {
Review Comment:
It's used at two places in the above code, otherwise I would have removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] coded9 commented on a diff in pull request #8315: fix(aws): Unclosed S3FileIO instance in GlueTableOperations
Posted by "coded9 (via GitHub)" <gi...@apache.org>.
coded9 commented on code in PR #8315:
URL: https://github.com/apache/iceberg/pull/8315#discussion_r1293481237
##########
aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java:
##########
@@ -197,6 +201,7 @@ void initialize(
closeableGroup.addCloseable(lockManager);
closeableGroup.addCloseable(fileIO);
closeableGroup.setSuppressCloseFailure(true);
+ this.fileIOCloser = newFileIOCloser();
Review Comment:
Thanks, I totally missed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] fivetran-ashokborra commented on a diff in pull request #8315: fix(aws): Unclosed S3FileIO instance in GlueTableOperations
Posted by "fivetran-ashokborra (via GitHub)" <gi...@apache.org>.
fivetran-ashokborra commented on code in PR #8315:
URL: https://github.com/apache/iceberg/pull/8315#discussion_r1293243347
##########
aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java:
##########
@@ -238,7 +238,8 @@ protected TableOperations newTableOps(TableIdentifier tableIdentifier) {
awsProperties,
tableSpecificCatalogPropertiesBuilder.buildOrThrow(),
hadoopConf,
- tableIdentifier);
+ tableIdentifier,
+ closeableGroup);
Review Comment:
`closeableGroup.addCloseable(tableOperations.io());`
This would create fileIO even before when it is actually needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a diff in pull request #8315: fix(aws): Unclosed S3FileIO instance in GlueTableOperations
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #8315:
URL: https://github.com/apache/iceberg/pull/8315#discussion_r1294288109
##########
aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java:
##########
@@ -619,4 +635,21 @@ public void setConf(Configuration conf) {
protected Map<String, String> properties() {
return catalogProperties == null ? ImmutableMap.of() : catalogProperties;
}
+
+ private Cache<TableOperations, FileIO> newFileIOCloser() {
+ return Caffeine.newBuilder()
+ .weakKeys()
+ .removalListener(
+ (RemovalListener<TableOperations, FileIO>)
+ (ops, fileIO, cause) -> {
+ if (null != fileIO) {
+ fileIO.close();
+ }
+ })
+ .build();
+ }
+
+ private void trackFileIO(GlueTableOperations ops) {
Review Comment:
both places can just call `fileIOCloser.put(ops, ops.io());` rather than introducing this method. In `RESTSessionCatalog#trackFileIO(..)` the situation is slightly different, because `trackFileIO()` does some additional checking, which isn't necessary here, meaning that we don't need this method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS: Add FileIO tracker/closer to Glue catalog [iceberg]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on code in PR #8315:
URL: https://github.com/apache/iceberg/pull/8315#discussion_r1650952000
##########
aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java:
##########
@@ -619,4 +635,17 @@ public void setConf(Configuration conf) {
protected Map<String, String> properties() {
return catalogProperties == null ? ImmutableMap.of() : catalogProperties;
}
+
+ private Cache<TableOperations, FileIO> newFileIOCloser() {
+ return Caffeine.newBuilder()
+ .weakKeys()
+ .removalListener(
+ (RemovalListener<TableOperations, FileIO>)
+ (ops, fileIO, cause) -> {
+ if (null != fileIO) {
+ fileIO.close();
Review Comment:
This is not a regular cache, this could be better served with `java.lang.ref.Cleaner`.
However, none of these addresses the main problem that we use Java GC as a proxy to discover and close resource leaks. I understand that fileIO instances can hold on to resources that require closing (like file descriptors or sockets inside AWS S3 client). If i have abundance of memory, the weakly held instances may not get collected, so the app can run out of descriptors. Can you please help me understand why this isn't a concern?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a diff in pull request #8315: fix(aws): Unclosed S3FileIO instance in GlueTableOperations
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #8315:
URL: https://github.com/apache/iceberg/pull/8315#discussion_r1294280634
##########
aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java:
##########
@@ -619,4 +635,21 @@ public void setConf(Configuration conf) {
protected Map<String, String> properties() {
return catalogProperties == null ? ImmutableMap.of() : catalogProperties;
}
+
+ private Cache<TableOperations, FileIO> newFileIOCloser() {
+ return Caffeine.newBuilder()
+ .weakKeys()
+ .removalListener(
+ (RemovalListener<TableOperations, FileIO>)
+ (ops, fileIO, cause) -> {
+ if (null != fileIO) {
+ fileIO.close();
+ }
+ })
+ .build();
+ }
+
+ private void trackFileIO(GlueTableOperations ops) {
Review Comment:
I don't think we need this wrapper method just for a put operation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] coded9 commented on a diff in pull request #8315: fix(aws): Unclosed S3FileIO instance in GlueTableOperations
Posted by "coded9 (via GitHub)" <gi...@apache.org>.
coded9 commented on code in PR #8315:
URL: https://github.com/apache/iceberg/pull/8315#discussion_r1294291330
##########
aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java:
##########
@@ -619,4 +635,21 @@ public void setConf(Configuration conf) {
protected Map<String, String> properties() {
return catalogProperties == null ? ImmutableMap.of() : catalogProperties;
}
+
+ private Cache<TableOperations, FileIO> newFileIOCloser() {
+ return Caffeine.newBuilder()
+ .weakKeys()
+ .removalListener(
+ (RemovalListener<TableOperations, FileIO>)
+ (ops, fileIO, cause) -> {
+ if (null != fileIO) {
+ fileIO.close();
+ }
+ })
+ .build();
+ }
+
+ private void trackFileIO(GlueTableOperations ops) {
Review Comment:
Makes senses. Removed the method.
##########
aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java:
##########
@@ -619,4 +635,21 @@ public void setConf(Configuration conf) {
protected Map<String, String> properties() {
return catalogProperties == null ? ImmutableMap.of() : catalogProperties;
}
+
+ private Cache<TableOperations, FileIO> newFileIOCloser() {
+ return Caffeine.newBuilder()
+ .weakKeys()
+ .removalListener(
+ (RemovalListener<TableOperations, FileIO>)
+ (ops, fileIO, cause) -> {
+ if (null != fileIO) {
+ fileIO.close();
+ }
+ })
+ .build();
+ }
+
+ private void trackFileIO(GlueTableOperations ops) {
Review Comment:
Makes sense. Removed the method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra merged pull request #8315: AWS: Add FileIO tracker/closer to Glue catalog
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra merged PR #8315:
URL: https://github.com/apache/iceberg/pull/8315
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] coded9 commented on a diff in pull request #8315: fix(aws): Unclosed S3FileIO instance in GlueTableOperations
Posted by "coded9 (via GitHub)" <gi...@apache.org>.
coded9 commented on code in PR #8315:
URL: https://github.com/apache/iceberg/pull/8315#discussion_r1293251526
##########
aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java:
##########
@@ -238,7 +238,8 @@ protected TableOperations newTableOps(TableIdentifier tableIdentifier) {
awsProperties,
tableSpecificCatalogPropertiesBuilder.buildOrThrow(),
hadoopConf,
- tableIdentifier);
+ tableIdentifier,
+ closeableGroup);
Review Comment:
`closeableGroup.addCloseable(tableOperations.io());`
This would create fileIO even before when it is actually needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a diff in pull request #8315: fix(aws): Unclosed S3FileIO instance in GlueTableOperations
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #8315:
URL: https://github.com/apache/iceberg/pull/8315#discussion_r1293415685
##########
aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java:
##########
@@ -197,6 +201,7 @@ void initialize(
closeableGroup.addCloseable(lockManager);
closeableGroup.addCloseable(fileIO);
closeableGroup.setSuppressCloseFailure(true);
+ this.fileIOCloser = newFileIOCloser();
Review Comment:
you need to handle proper cleanup in `close()`:
```
if (fileIOCloser != null) {
fileIOCloser.invalidateAll();
fileIOCloser.cleanUp();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] fivetran-ashokborra commented on a diff in pull request #8315: fix(aws): Unclosed S3FileIO instance in GlueTableOperations
Posted by "fivetran-ashokborra (via GitHub)" <gi...@apache.org>.
fivetran-ashokborra commented on code in PR #8315:
URL: https://github.com/apache/iceberg/pull/8315#discussion_r1293243347
##########
aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java:
##########
@@ -238,7 +238,8 @@ protected TableOperations newTableOps(TableIdentifier tableIdentifier) {
awsProperties,
tableSpecificCatalogPropertiesBuilder.buildOrThrow(),
hadoopConf,
- tableIdentifier);
+ tableIdentifier,
+ closeableGroup);
Review Comment:
`closeableGroup.addCloseable(tableOperations.io());`
This would create fileIO even before when it is actually needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org