You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "timmyyao (via GitHub)" <gi...@apache.org> on 2026/01/04 07:09:13 UTC
[PR] [wip][core]JindoFileIO support cache using JindoCache [paimon]
timmyyao opened a new pull request, #6949:
URL: https://github.com/apache/paimon/pull/6949
### Purpose
We tend to persist files of Paimon table on object storage such as S3 and OSS. However the throughput or latency of such storage is usually limited, making IO performance become the bottleneck in ETL or OLAP occasions with high concurrency. Thus we introduce cache when FileIO visits files within a Paimon table. The solution is to support JindoCache in JindoFileIO, which is a distributed filesystem cache system. And table-level cache policy is implemented, which can be supported by REST server such as DLF.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core]JindoFileIO support cache using JindoCache [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #6949:
URL: https://github.com/apache/paimon/pull/6949
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core]JindoFileIO support cache using JindoCache [paimon]
Posted by "timmyyao (via GitHub)" <gi...@apache.org>.
timmyyao commented on code in PR #6949:
URL: https://github.com/apache/paimon/pull/6949#discussion_r2690296955
##########
paimon-api/src/main/java/org/apache/paimon/rest/RESTCatalogOptions.java:
##########
@@ -103,4 +103,27 @@ public class RESTCatalogOptions {
.stringType()
.noDefaultValue()
.withDescription("REST Catalog DLF OSS endpoint.");
+
+ public static final ConfigOption<Boolean> DLF_FILE_IO_CACHE_ENABLED =
+ ConfigOptions.key("dlf.io-cache-enabled")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription(
+ "Enable cache for visiting files using file io (currently only JindoFileIO supports cache).");
+ public static final ConfigOption<String> DLF_FILE_IO_CACHE_WHITELIST_PATH =
+ ConfigOptions.key("dlf.io-cache.whitelist-path")
+ .stringType()
+ .defaultValue("bucket-,manifest")
+ .withDescription(
+ "Cache is only applied to paths which contain the specified pattern, and * means all paths.");
+ public static final ConfigOption<String> DLF_FILE_IO_CACHE_POLICY =
+ ConfigOptions.key("dlf.io-cache.policy")
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [wip][core]JindoFileIO support cache using JindoCache [paimon]
Posted by "timmyyao (via GitHub)" <gi...@apache.org>.
timmyyao commented on PR #6949:
URL: https://github.com/apache/paimon/pull/6949#issuecomment-3736780238
> @timmyyao I'm not using DLF, but the data is still stored on OSS. Does this also apply? like this:
>
> ```sql
> CREATE CATALOG paimon_catalog WITH (
> 'type' = 'paimon',
> 'metastore' = 'hive',
> 'warehouse' = 'oss://xxxx.oss-dls.aliyuncs.com/user/hive/warehouse'
> );
> ```
The major purpose of this PR is to provide a table-level cache policy which can be supported by DLF server-side and cache is backed by JindoCache. So related configs are generated in RESTCatalog and RESTTokenFileIO according to the policy defined at server-side. If you just use filesystem or hive catalog using JindoFileIO, you can simply apply the cache configs to JindoSDK (fs.xengine, fs.jindocache.namespace.rpc.address) as long as you have a JindoCache cluster. But there is no server-side support for cache management, you can only adjust at client-side to decide whether or not to enable cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core]JindoFileIO support cache using JindoCache [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #6949:
URL: https://github.com/apache/paimon/pull/6949#issuecomment-3758911757
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core]JindoFileIO support cache using JindoCache [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #6949:
URL: https://github.com/apache/paimon/pull/6949#discussion_r2689794952
##########
paimon-api/src/main/java/org/apache/paimon/rest/RESTCatalogOptions.java:
##########
@@ -103,4 +103,27 @@ public class RESTCatalogOptions {
.stringType()
.noDefaultValue()
.withDescription("REST Catalog DLF OSS endpoint.");
+
+ public static final ConfigOption<Boolean> DLF_FILE_IO_CACHE_ENABLED =
+ ConfigOptions.key("dlf.io-cache-enabled")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription(
+ "Enable cache for visiting files using file io (currently only JindoFileIO supports cache).");
+ public static final ConfigOption<String> DLF_FILE_IO_CACHE_WHITELIST_PATH =
+ ConfigOptions.key("dlf.io-cache.whitelist-path")
Review Comment:
just `io-cache.whitelist-path`
##########
paimon-api/src/main/java/org/apache/paimon/rest/RESTCatalogOptions.java:
##########
@@ -103,4 +103,27 @@ public class RESTCatalogOptions {
.stringType()
.noDefaultValue()
.withDescription("REST Catalog DLF OSS endpoint.");
+
+ public static final ConfigOption<Boolean> DLF_FILE_IO_CACHE_ENABLED =
+ ConfigOptions.key("dlf.io-cache-enabled")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription(
+ "Enable cache for visiting files using file io (currently only JindoFileIO supports cache).");
+ public static final ConfigOption<String> DLF_FILE_IO_CACHE_WHITELIST_PATH =
+ ConfigOptions.key("dlf.io-cache.whitelist-path")
+ .stringType()
+ .defaultValue("bucket-,manifest")
+ .withDescription(
+ "Cache is only applied to paths which contain the specified pattern, and * means all paths.");
+ public static final ConfigOption<String> DLF_FILE_IO_CACHE_POLICY =
+ ConfigOptions.key("dlf.io-cache.policy")
Review Comment:
just `io-cache.policy`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [wip][core]JindoFileIO support cache using JindoCache [paimon]
Posted by "sd4324530 (via GitHub)" <gi...@apache.org>.
sd4324530 commented on PR #6949:
URL: https://github.com/apache/paimon/pull/6949#issuecomment-3717143860
@timmyyao
I'm not using DLF, but the data is still stored on OSS. Does this also apply?
like this:
```sql
CREATE CATALOG paimon_catalog WITH (
'type' = 'paimon',
'metastore' = 'hive',
'warehouse' = 'oss://xxxx.oss-dls.aliyuncs.com/user/hive/warehouse'
);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core]JindoFileIO support cache using JindoCache [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #6949:
URL: https://github.com/apache/paimon/pull/6949#discussion_r2689794109
##########
paimon-api/src/main/java/org/apache/paimon/rest/RESTCatalogOptions.java:
##########
@@ -103,4 +103,27 @@ public class RESTCatalogOptions {
.stringType()
.noDefaultValue()
.withDescription("REST Catalog DLF OSS endpoint.");
+
+ public static final ConfigOption<Boolean> DLF_FILE_IO_CACHE_ENABLED =
+ ConfigOptions.key("dlf.io-cache-enabled")
Review Comment:
just `io-cache.enabled`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core]JindoFileIO support cache using JindoCache [paimon]
Posted by "timmyyao (via GitHub)" <gi...@apache.org>.
timmyyao commented on code in PR #6949:
URL: https://github.com/apache/paimon/pull/6949#discussion_r2690296300
##########
paimon-api/src/main/java/org/apache/paimon/rest/RESTCatalogOptions.java:
##########
@@ -103,4 +103,27 @@ public class RESTCatalogOptions {
.stringType()
.noDefaultValue()
.withDescription("REST Catalog DLF OSS endpoint.");
+
+ public static final ConfigOption<Boolean> DLF_FILE_IO_CACHE_ENABLED =
+ ConfigOptions.key("dlf.io-cache-enabled")
Review Comment:
fixed
##########
paimon-api/src/main/java/org/apache/paimon/rest/RESTCatalogOptions.java:
##########
@@ -103,4 +103,27 @@ public class RESTCatalogOptions {
.stringType()
.noDefaultValue()
.withDescription("REST Catalog DLF OSS endpoint.");
+
+ public static final ConfigOption<Boolean> DLF_FILE_IO_CACHE_ENABLED =
+ ConfigOptions.key("dlf.io-cache-enabled")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription(
+ "Enable cache for visiting files using file io (currently only JindoFileIO supports cache).");
+ public static final ConfigOption<String> DLF_FILE_IO_CACHE_WHITELIST_PATH =
+ ConfigOptions.key("dlf.io-cache.whitelist-path")
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [wip][core]JindoFileIO support cache using JindoCache [paimon]
Posted by "timmyyao (via GitHub)" <gi...@apache.org>.
timmyyao commented on PR #6949:
URL: https://github.com/apache/paimon/pull/6949#issuecomment-3737119080
In addition, this PR also allows to config the cache policy at client-side. So instead of directly config fs.xengine=true, you can config dlf.io-cache-enabled and dlf.io-cache.policy to define more detailed cache policy by this PR (e.g. meta,read,write), and JindoFileIO will apply cache only to specific file to guarantee consistency. Thus it will be more safe to use cache for Paimon table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org