[jira] [Updated] (ARROW-9820) [C++] Plugin Architecture for Filesystem and File IO

["Neal Richardson (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Neal Richardson updated ARROW-9820:
-----------------------------------
    Summary: [C++] Plugin Architecture for Filesystem and File IO  (was: Plugin Architecture for Filesystem and File IO)

> [C++] Plugin Architecture for Filesystem and File IO
> ----------------------------------------------------
>
>                 Key: ARROW-9820
>                 URL: https://issues.apache.org/jira/browse/ARROW-9820
>             Project: Apache Arrow
>          Issue Type: New Feature
>          Components: C++
>            Reporter: Lawrence Chan
>            Priority: Minor
>
> Adding a new custom filesystem with corresponding file i/o streams is quite a process at the moment.  Looks like HDFS and S3FS are basically hardcoded in many places.  It would be useful to develop a plugin system to allow users to interface with other data stores without maintaining a permanent fork with hardcoded changes.
> We can either do runtime plugins or compile-time plugins.  Runtime is more user-friendly, but with C++, ABI compatibility is fairly delicate.  So we would either want to use a C ABI or accept a youre-on-your-own situation where the user is expected to be very careful with versioning and compiler flags.
> With compile-time plugins, maybe there's a way to have the cmake machinery build third party code and also register those new URI schemes automatically.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)