You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "wuchong (via GitHub)" <gi...@apache.org> on 2025/12/25 11:32:22 UTC

[I] [Docs] Document the Compacted format support for Log and PK Tables [fluss]

wuchong opened a new issue, #2256:
URL: https://github.com/apache/fluss/issues/2256

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
   
   
   ### Description
   
   We’ve now added support for the **`COMPACTED` format** for both Log and KV tables (see [FLUSS-2218](https://github.com/apache/fluss/issues/2218) and [PR #2108](https://github.com/apache/fluss/pull/2108)).  
   
   This feature should be documented on the Fluss website to help users understand how and when to use it.
   
   Moreover, when combined with `table.changelog.image=WAL`, the `COMPACTED` format enables an **efficient, lightweight KV store**—ideal for use cases that don’t require full changelog reads or column projections (e.g., future system tables).  In such scenarios, there’s no need to look up previous values or deserialize records into an intermediate log format, reducing both latency and CPU overhead.
   
   ### Willingness to contribute
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Docs] Document the Compacted format support for Log and PK Tables [fluss]

Posted by "Priyamanjare54 (via GitHub)" <gi...@apache.org>.
Priyamanjare54 commented on issue #2256:
URL: https://github.com/apache/fluss/issues/2256#issuecomment-3692813235

   I’d like to work on documenting the COMPACTED format. Please let me know if that’s okay.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Docs] Document the Compacted format support for Log and PK Tables [fluss]

Posted by "polyzos (via GitHub)" <gi...@apache.org>.
polyzos closed issue #2256: [Docs] Document the Compacted format support for Log and PK Tables
URL: https://github.com/apache/fluss/issues/2256


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Docs] Document the Compacted format support for Log and PK Tables [fluss]

Posted by "Priyamanjare54 (via GitHub)" <gi...@apache.org>.
Priyamanjare54 commented on issue #2256:
URL: https://github.com/apache/fluss/issues/2256#issuecomment-3699329218

   Makes sense!!
   I agree that putting COMPACTED under a broader Data Encodings page would be cleaner, especially with ARROW, INDEXED, and more encodings coming up.
   
   I’m happy to restructure the docs this way if everyone’s aligned.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Docs] Document the Compacted format support for Log and PK Tables [fluss]

Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong commented on issue #2256:
URL: https://github.com/apache/fluss/issues/2256#issuecomment-3699245205

   Hi @Priyamanjare54,  
   
   It seems the content of your pull request is empty. There might have been an issue during the submission process. Could you please double-check and resubmit if needed?
   
   Hi @developer-ravi-03,  
   
   Thank you very much for your contribution! We truly appreciate your enthusiasm. However, since this issue has already been assigned to @Priyamanjare54, we’ll prioritize reviewing his PR first.  
   
   To avoid duplicated effort in the community, we strongly encourage contributors to **first comment on the issue to request assignment** and **discuss the implementation approach with committers** before starting development. Once there’s consensus, you can proceed with coding.  
   
   For more details, please refer to our official contribution guide:  
   https://fluss.apache.org/community/how-to-contribute/contribute-code/#code-contribution-process  
   
   Thanks again for your support!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Docs] Document the Compacted format support for Log and PK Tables [fluss]

Posted by "Priyamanjare54 (via GitHub)" <gi...@apache.org>.
Priyamanjare54 commented on issue #2256:
URL: https://github.com/apache/fluss/issues/2256#issuecomment-3698323393

   Hi @wuchong,
   Just a friendly reminder about PR #2264. Would appreciate your review when you have time.
   
   Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Docs] Document the Compacted format support for Log and PK Tables [fluss]

Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong commented on issue #2256:
URL: https://github.com/apache/fluss/issues/2256#issuecomment-3699296367

   Besides, I’d like to discuss the documentation structure and design here first. 
   
   Directly adding documentation for the `COMPACTED` table format under the current "Table Design" section may not be the best approach, especially since `COMPACTED` is just one of several data encodings we support. We also have `ARROW`, `INDEXED`, and will likely introduce more in the future.
   
   To better organize this information, I suggest creating a dedicated documentation page titled **“Data Encodings”** (between "Data Distribution" and "Data Types"). This page would provide:
   
   - An **overview** of the encoding landscape in Fluss,  
   - A clear distinction between **log data encoding** and **KV data encoding**,  
   - Guidance on **when to choose which encoding**, including key trade-offs.
   
   Under this page, we can then introduce each encoding with consistent subsections covering: main characteristics, primary use cases, design trade-offs, and APIs how to use. 
   
   - **ARROW**: A columnar encoding. Can only be used as a *log encoding*. Ideal for streaming analytical workloads due to columnar efficiency and interoperability with Arrow-based systems. Like projection pushdown for streaming read. 
   
   - **COMPACTED**: A row-oriented encoding that leverages [variable-length](https://lucene.apache.org/java/3_5_0/fileformats.html#VInt) coding to have a smallest binary data size. Can be used as both *log encoding* and *KV encoding*. Introduce when it is combined with `table.changelog.image=WAL`, it enables an **efficient, lightweight KV store**, perfect for system tables or use cases that don’t require full changelog reads or column projections.
   
   - **INDEXED**: A row-oriented encoding but is deprecated. Supports both *log* and *KV encoding*. 
   
   This structure would make the documentation more scalable, discoverable, and aligned with Fluss’s evolving architecture. What do you think?
   
   cc @polyzos , what do you think?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org