[I] [Feature] Add an extensible file-format provider contract to PyPaimon [paimon]

["Zerolzj (via GitHub)" <gi...@apache.org>]

Zerolzj opened a new issue, #9656:
URL: https://github.com/apache/paimon/issues/9656

   ### Search before asking
   
   - [x] I searched in the issues and found nothing similar.
   
   ### Motivation
   
   PyPaimon currently selects readers and writers with built-in `if/elif` branches. New formats such
   as Lance and Mosaic have therefore been integrated directly into the PyPaimon package. This works
   well for formats that the project wants to support and release as built-ins, but there is no stable
   extension boundary for a deployment-specific format or for an embedding runtime that supplies its
   own implementation.
   
   Java Paimon already discovers `FileFormatFactory` implementations. I also noticed that Java PR
   #8292 proposed another runtime-side `FileFormatProvider` and was closed because an execution engine
   could provide that SPI in its own integration. PyPaimon is itself the SDK/runtime that performs the
   format dispatch, so an embedding application cannot add a format without patching that dispatch.
   
   Would the community consider a small PyPaimon extension contract, or is built-in-only format
   support the intended boundary?
   
   ### Solution
   
   A possible contract would have these properties:
   
   1. Table metadata stores only an immutable, language-neutral format identifier, never a Python
      module or class name.
   2. A provider declares read and write capabilities independently and receives public context
      objects rather than PyPaimon writer implementation details.
   3. Registration is explicit and deterministic. Python package entry points may be an optional
      discovery mechanism, but duplicate identifiers fail rather than depend on import order.
   4. A missing provider fails during planning with a clear diagnostic. It never falls back to a
      different physical format.
   5. Packaging and distribution to remote workers remain the responsibility of the embedding engine;
      the table does not persist package locations.
   6. The extension API has a small version/capability handshake so a provider developed against one
      PyPaimon release does not silently run with incompatible context objects.
   
   The minimum API could be an explicit registry such as
   `register_file_format(identifier, provider)`. Entry-point discovery can be considered separately
   after the packaging work in #8049 is settled.
   
   Questions:
   
   - Is third-party format discovery in scope for PyPaimon?
   - If so, would maintainers prefer an explicit registry, Python entry points, or an embedding-runtime
     hook?
   - Should the first version expose read-only providers before defining writer and abort semantics?
   - Would this public extension contract require a PIP before a prototype PR?
   
   ### Anything else?
   
   Relevant code and discussions:
   
   - PyPaimon reader dispatch:
     https://github.com/apache/paimon/blob/master/paimon-python/pypaimon/read/split_read.py
   - PyPaimon writer dispatch:
     https://github.com/apache/paimon/blob/master/paimon-python/pypaimon/write/writer/data_writer.py
   - Lance format discussion: https://github.com/apache/paimon/issues/6739
   - PyPaimon packaging discussion: https://github.com/apache/paimon/issues/8049
   - Java runtime-side provider discussion: https://github.com/apache/paimon/pull/8292
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!
   
   I can contribute a minimal read-only prototype with an out-of-tree example provider and tests for
   duplicate registration, missing providers, capability checks, and distributed-worker loading after
   the preferred boundary is clear.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Add an extensible file-format provider contract to PyPaimon [paimon]

["Zerolzj (via GitHub)" <gi...@apache.org>]

Zerolzj commented on issue #9656:
URL: https://github.com/apache/paimon/issues/9656#issuecomment-5563936726

   Related but independently scoped discussions for end-to-end interoperability:
   
   - Custom bucket semantics across runtimes and table layouts: https://github.com/apache/paimon/issues/5436#issuecomment-5550987251
   - A stable API for registering existing immutable files, including file-ownership semantics: https://github.com/apache/paimon/issues/7173#issuecomment-5552907234
   
   These are not prerequisites for the PyPaimon FileFormat provider proposed here. I am linking them only to keep the adjacent contracts traceable while keeping this issue focused on Python format discovery and lifecycle.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org