You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "timmyyao (via GitHub)" <gi...@apache.org> on 2025/07/25 03:03:42 UTC
[PR] [VFS] Enable table cache in PVFS [paimon]
timmyyao opened a new pull request, #5958:
URL: https://github.com/apache/paimon/pull/5958
### Purpose
Enable table cache in virtual filesystem to reduce rest requests to rest server.
Linked issue: https://github.com/apache/paimon/issues/5851
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [VFS] Enable table cache in PVFS [paimon]
Posted by "timmyyao (via GitHub)" <gi...@apache.org>.
timmyyao commented on code in PR #5958:
URL: https://github.com/apache/paimon/pull/5958#discussion_r2230663727
##########
paimon-vfs/pom.xml:
##########
@@ -49,43 +49,43 @@ under the License.
<groupId>org.apache.paimon</groupId>
<artifactId>paimon-core</artifactId>
<version>${project.version}</version>
- <scope>test</scope>
+ <scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.paimon</groupId>
<artifactId>paimon-core</artifactId>
<classifier>tests</classifier>
<version>${project.version}</version>
- <scope>test</scope>
+ <scope>provided</scope>
Review Comment:
It's a misoperation, I will revert this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [VFS] Enable table cache in PVFS [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #5958:
URL: https://github.com/apache/paimon/pull/5958
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [VFS] Enable table cache in PVFS [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #5958:
URL: https://github.com/apache/paimon/pull/5958#discussion_r2230062134
##########
paimon-vfs/paimon-vfs-common/src/main/java/org/apache/paimon/vfs/VFSOperations.java:
##########
@@ -238,4 +288,35 @@ private GetTableResponse loadTableMetadata(Identifier identifier) throws IOExcep
return response;
}
+
+ private VFSTableInfo loadTableInfo(Identifier identifier) throws IOException {
+ // Get table from REST server
+ GetTableResponse table;
+ table = loadTableMetadata(identifier);
+
+ if (table.isExternal()) {
+ throw new IOException("Do not support visiting external table " + identifier);
+ }
+ Path tablePath = new Path(table.getPath());
+ FileIO fileIO = new RESTTokenFileIO(context, api, identifier, tablePath);
Review Comment:
We should cache `FileIO` too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [VFS] Enable table cache in PVFS [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #5958:
URL: https://github.com/apache/paimon/pull/5958#discussion_r2230093352
##########
paimon-vfs/paimon-vfs-common/src/main/java/org/apache/paimon/vfs/VFSOperations.java:
##########
@@ -55,8 +65,40 @@ public class VFSOperations {
private final RESTApi api;
private final CatalogContext context;
+ private Cache<Identifier, VFSTableInfo> tableCache;
Review Comment:
`@Nullable`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [VFS] Enable table cache in PVFS [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #5958:
URL: https://github.com/apache/paimon/pull/5958#discussion_r2230093177
##########
paimon-vfs/pom.xml:
##########
@@ -49,43 +49,43 @@ under the License.
<groupId>org.apache.paimon</groupId>
<artifactId>paimon-core</artifactId>
<version>${project.version}</version>
- <scope>test</scope>
+ <scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.paimon</groupId>
<artifactId>paimon-core</artifactId>
<classifier>tests</classifier>
<version>${project.version}</version>
- <scope>test</scope>
+ <scope>provided</scope>
Review Comment:
Why change to provided?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [VFS] Enable table cache in PVFS [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #5958:
URL: https://github.com/apache/paimon/pull/5958#discussion_r2230062134
##########
paimon-vfs/paimon-vfs-common/src/main/java/org/apache/paimon/vfs/VFSOperations.java:
##########
@@ -238,4 +288,35 @@ private GetTableResponse loadTableMetadata(Identifier identifier) throws IOExcep
return response;
}
+
+ private VFSTableInfo loadTableInfo(Identifier identifier) throws IOException {
+ // Get table from REST server
+ GetTableResponse table;
+ table = loadTableMetadata(identifier);
+
+ if (table.isExternal()) {
+ throw new IOException("Do not support visiting external table " + identifier);
+ }
+ Path tablePath = new Path(table.getPath());
+ FileIO fileIO = new RESTTokenFileIO(context, api, identifier, tablePath);
Review Comment:
We should cache `FileIO` too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [VFS] Enable table cache in PVFS [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #5958:
URL: https://github.com/apache/paimon/pull/5958#discussion_r2230093885
##########
paimon-vfs/paimon-vfs-common/src/main/java/org/apache/paimon/vfs/VFSOperations.java:
##########
@@ -55,8 +65,40 @@ public class VFSOperations {
private final RESTApi api;
private final CatalogContext context;
+ private Cache<Identifier, VFSTableInfo> tableCache;
+ private final boolean cacheEnabled;
Review Comment:
Just use tableCache is null or not null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org