You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@paimon.apache.org by Jingsong Li <ji...@gmail.com> on 2025/07/07 11:17:03 UTC

[DISCUSS] Make Object Table as a virtual table

Hi everyone,

After the object table was developed, almost no one used it. Object
Table is a real Paimon table, it requires a refresh service to refresh
the metadata of files, which is difficult to complete in operation.

In fact, we don't need to refresh it, and accessing the file system
directly during reading is not a bad thing.

This discussion discusses whether it is possible to transform an
object table into a virtual existence. Users can:

1. Use the virtual file system to read and write files.
2. Or use the SQL computing engine to read it as a structured file list.

On this basis, we can develop virtual file systems, such as through
similar approaches vfs://catalog_name/database_name/table_name/ Use
the path to access files in the specific object table.

By using this method, we can integrate file operations on object
tables with the permission system of data catalogs.

What do you think?

Best,
Jingsong

回复： [DISCUSS] Make Object Table as a virtual table

Posted by Timmy <19...@qq.com.INVALID>.
+1. Using refresh tool to update is not so easy to use and may cause&nbsp;
delay in metadata.&nbsp; And filesystem support like vfs://xxx makes it much&nbsp;
easier to use in many cases such as AI traning.&nbsp;


Best regrads,
Shunyang




------------------&nbsp;原始邮件&nbsp;------------------
发件人:                                                                                                                        "dev"                                                                                    <sundapeng2008@gmail.com&gt;;
发送时间:&nbsp;2025年7月7日(星期一) 晚上8:12
收件人:&nbsp;"dev"<dev@paimon.apache.org&gt;;

主题:&nbsp;Re: [DISCUSS] Make Object Table as a virtual table



+1, Object Table provides a virtual file system abstraction that satisfies
flexible usage requirements. For most scenarios, directly reading the
records(filesize, mtime...) of the Object Table from the file system should
also be okay.

Regards,
Dapeng

李佳佳(佳加) <ljj270386@alibaba-inc.com.invalid&gt; 于2025年7月7日周一 19:36写道：

&gt; Using a virtual file system to read and write files, or using the SQL
&gt; computing engine to treat it as a structured file list, seems like a very
&gt; practical solution and making it easy for users to access files in the
&gt; object table.
&gt; Best regards,
&gt; Jiajia
&gt; ------------------------------------------------------------------
&gt; 发件人：Jingsong Li <jingsonglee0@gmail.com&gt;
&gt; 发送时间：2025年7月7日(周一) 19:17
&gt; 收件人：dev<dev@paimon.apache.org&gt;
&gt; 主 题：[DISCUSS] Make Object Table as a virtual table
&gt; Hi everyone,
&gt; After the object table was developed, almost no one used it. Object
&gt; Table is a real Paimon table, it requires a refresh service to refresh
&gt; the metadata of files, which is difficult to complete in operation.
&gt; In fact, we don't need to refresh it, and accessing the file system
&gt; directly during reading is not a bad thing.
&gt; This discussion discusses whether it is possible to transform an
&gt; object table into a virtual existence. Users can:
&gt; 1. Use the virtual file system to read and write files.
&gt; 2. Or use the SQL computing engine to read it as a structured file list.
&gt; On this basis, we can develop virtual file systems, such as through
&gt; similar approaches vfs://catalog_name/database_name/table_name/ Use
&gt; the path to access files in the specific object table.
&gt; By using this method, we can integrate file operations on object
&gt; tables with the permission system of data catalogs.
&gt; What do you think?
&gt; Best,
&gt; Jingsong
&gt;

Re: [DISCUSS] Make Object Table as a virtual table

Posted by Dapeng Sun <su...@gmail.com>.
+1, Object Table provides a virtual file system abstraction that satisfies
flexible usage requirements. For most scenarios, directly reading the
records(filesize, mtime...) of the Object Table from the file system should
also be okay.

Regards,
Dapeng

李佳佳(佳加) <lj...@alibaba-inc.com.invalid> 于2025年7月7日周一 19:36写道：

> Using a virtual file system to read and write files, or using the SQL
> computing engine to treat it as a structured file list, seems like a very
> practical solution and making it easy for users to access files in the
> object table.
> Best regards,
> Jiajia
> ------------------------------------------------------------------
> 发件人：Jingsong Li <ji...@gmail.com>
> 发送时间：2025年7月7日(周一) 19:17
> 收件人：dev<de...@paimon.apache.org>
> 主 题：[DISCUSS] Make Object Table as a virtual table
> Hi everyone,
> After the object table was developed, almost no one used it. Object
> Table is a real Paimon table, it requires a refresh service to refresh
> the metadata of files, which is difficult to complete in operation.
> In fact, we don't need to refresh it, and accessing the file system
> directly during reading is not a bad thing.
> This discussion discusses whether it is possible to transform an
> object table into a virtual existence. Users can:
> 1. Use the virtual file system to read and write files.
> 2. Or use the SQL computing engine to read it as a structured file list.
> On this basis, we can develop virtual file systems, such as through
> similar approaches vfs://catalog_name/database_name/table_name/ Use
> the path to access files in the specific object table.
> By using this method, we can integrate file operations on object
> tables with the permission system of data catalogs.
> What do you think?
> Best,
> Jingsong
>

回复：[DISCUSS] Make Object Table as a virtual table

Posted by "李佳佳(佳加)" <lj...@alibaba-inc.com.INVALID>.
Using a virtual file system to read and write files, or using the SQL computing engine to treat it as a structured file list, seems like a very practical solution and making it easy for users to access files in the object table.
Best regards,
Jiajia
------------------------------------------------------------------
发件人：Jingsong Li <ji...@gmail.com>
发送时间：2025年7月7日(周一) 19:17
收件人：dev<de...@paimon.apache.org>
主　题：[DISCUSS] Make Object Table as a virtual table
Hi everyone,
After the object table was developed, almost no one used it. Object
Table is a real Paimon table, it requires a refresh service to refresh
the metadata of files, which is difficult to complete in operation.
In fact, we don't need to refresh it, and accessing the file system
directly during reading is not a bad thing.
This discussion discusses whether it is possible to transform an
object table into a virtual existence. Users can:
1. Use the virtual file system to read and write files.
2. Or use the SQL computing engine to read it as a structured file list.
On this basis, we can develop virtual file systems, such as through
similar approaches vfs://catalog_name/database_name/table_name/ Use
the path to access files in the specific object table.
By using this method, we can integrate file operations on object
tables with the permission system of data catalogs.
What do you think?
Best,
Jingsong