You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@horaedb.apache.org by "jiacai2050 (via GitHub)" <gi...@apache.org> on 2024/11/25 07:46:31 UTC

[I] Explore another format for manifest [horaedb]

jiacai2050 opened a new issue, #1600:
URL: https://github.com/apache/horaedb/issues/1600

   ### Describe This Problem
   
   Currently our manifest is defined using protobuf, that's:
   https://github.com/apache/horaedb/blob/9e81c4ed5df1998cbd210dc48fc67b6b7405a553/horaedb/pb_types/protos/sst.proto#L32
   
   Protobuf is useful for schema evolution, but not very efficient in our case:
   - For `Vec<struct>` field, protobuf will serialize metadata of every struct, which is a waste of space.
   - Serialize manifest body using protobuf in a whole make a hard to update incrementally.
   
   ### Proposal
   
   We can serialize manifest all by ourselves, a proposed format:
   
   ```
   | version(u8) | length(u32)|  Record(N) ... |
   
   # Record is a self-descriptive message
   | id(u64) | time_range(i64*2)| size(u32) | 
   ```
   
   ### Additional Context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org
For additional commands, e-mail: commits-help@horaedb.apache.org


Re: [I] Explore another format for manifest [horaedb]

Posted by "zealchen (via GitHub)" <gi...@apache.org>.
zealchen commented on issue #1600:
URL: https://github.com/apache/horaedb/issues/1600#issuecomment-2551725851

   Append 100 new delta sstfiles to snapshot with 1000 records.
   ```sh
   Benchmarking bench_encoding/new_format_encoding/0: Collecting 10 samples in estimated 5.0005 s (494k iterations)
   bench_encoding/new_format_encoding/0
                           time:   [10.232 µs 10.611 µs 10.900 µs]
                           change: [-1.5176% +2.1745% +5.9315%] (p = 0.30 > 0.05)
                           No change in performance detected.
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org
For additional commands, e-mail: commits-help@horaedb.apache.org


Re: [I] Explore another format for manifest [horaedb]

Posted by "zealchen (via GitHub)" <gi...@apache.org>.
zealchen commented on issue #1600:
URL: https://github.com/apache/horaedb/issues/1600#issuecomment-2519265285

   > The general ideas looks GOOD for me, but I wonder if you put length in the middle the manifest, how would you update it when append new record?
   
   Since the Bytes obtained from object storage is a fixed-length object, each time we merge SST files, we need to create a new Vec<u8> with a capacity equal to the original bytes length plus the number of records multiplied by the record length.
   
   Following that, We make Vec<u8> as_mut_slice [u8] and then:
   1.  create the new header struct and write it to slice[0..header_length].
   2. copy original records bytes to slice[header_length..header_length+original_record_length].
   3. for each sstfiles, write record to the rest for slice one by one.
   
   Another way is to:
   1. copy original bytes to slice [0..header_length+original_record_length]
   2. update the slice at length field position with the new length.
   3. for each sstfiles, write record to the rest for slice one by one.
   
   
   I prefer the first way, looks more ergonomics.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org
For additional commands, e-mail: commits-help@horaedb.apache.org


Re: [I] Explore another format for manifest [horaedb]

Posted by "jiacai2050 (via GitHub)" <gi...@apache.org>.
jiacai2050 closed issue #1600: Explore another format for manifest
URL: https://github.com/apache/horaedb/issues/1600


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org
For additional commands, e-mail: commits-help@horaedb.apache.org


Re: [I] Explore another format for manifest [horaedb]

Posted by "jiacai2050 (via GitHub)" <gi...@apache.org>.
jiacai2050 commented on issue #1600:
URL: https://github.com/apache/horaedb/issues/1600#issuecomment-2518971041

   The general ideas looks GOOD for me, but I wonder if you put length in the middle the manifest, how would you update it when append new record?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org
For additional commands, e-mail: commits-help@horaedb.apache.org


Re: [I] Explore another format for manifest [horaedb]

Posted by "jiacai2050 (via GitHub)" <gi...@apache.org>.
jiacai2050 commented on issue #1600:
URL: https://github.com/apache/horaedb/issues/1600#issuecomment-2506900473

   Those are all great questions, 
   1. We use `version` field to deal with schema  evolution, if we want to add some fields to manifest, a new version could be added, and when merge, to convert old manifest to new one.
   2. Object store don't have the append interface, so append here means download the old one into memory, then merge with new delta manifests, then upload it again to overwrite the old one.
   3. There shouldn't many manifest files, we have a hard limit on how many delta files a manifest can have, if there are more than that, the write process will fail, only after the merge process is finished, we can allow creating new manifest deltas.
   
   As for the third question, that why we need to keep metadata of each sst small, so we can hold millions of sst files in one manifest snapshot, whose size is less than 1GB.
   
   ``` shell
   1024*1024*1024 / 28 (size of each sst's metadata) = 38347922
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org
For additional commands, e-mail: commits-help@horaedb.apache.org


Re: [I] Explore another format for manifest [horaedb]

Posted by "zealchen (via GitHub)" <gi...@apache.org>.
zealchen commented on issue #1600:
URL: https://github.com/apache/horaedb/issues/1600#issuecomment-2504322882

   Interesting, I'll get into it. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org
For additional commands, e-mail: commits-help@horaedb.apache.org


Re: [I] Explore another format for manifest [horaedb]

Posted by "jiacai2050 (via GitHub)" <gi...@apache.org>.
jiacai2050 closed issue #1600: Explore another format for manifest
URL: https://github.com/apache/horaedb/issues/1600


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org
For additional commands, e-mail: commits-help@horaedb.apache.org


Re: [I] Explore another format for manifest [horaedb]

Posted by "zealchen (via GitHub)" <gi...@apache.org>.
zealchen commented on issue #1600:
URL: https://github.com/apache/horaedb/issues/1600#issuecomment-2515509429

   Agreed, your proposal works for me. Specifically, I can implement a MetaSerializer to convert Vec<SstFile> into bytes and merge them with the delta metadata bytes. Additionally, we'll need a deserialization function to parse the bytes back into Vec<SstFile>. To account for future evolution, we'll include version verification and handle data migration when the versions differ during startup. However the upgrade process can be tackled separately in another issue.
   
   Old data flow in do_merge:
                                                                              delta_sstmetas
                                                                                  | (extend vec)
                                                                                 V                                
   object_store -> org_bytes -> org_pb -> Vec<sstmeta> -> dst_pb -> dst_bytes -> object_store
   
   New data flow in do_merge:
                           delta_sstmetas -> bytes
                                                            | (append)
                                                           V                                
   object_store -> org_bytes -> dst_bytes -> object_store


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org
For additional commands, e-mail: commits-help@horaedb.apache.org


Re: [I] Explore another format for manifest [horaedb]

Posted by "zealchen (via GitHub)" <gi...@apache.org>.
zealchen commented on issue #1600:
URL: https://github.com/apache/horaedb/issues/1600#issuecomment-2506305676

   Here are some key design considerations to clarify in advance:
   
   1. Since the sstmeta schema may evolve over time, we need to ensure backward compatibility for each of the self-descriptive record. If so, I'm wondering how and when these manifest files will be utilized.
   2. The object storage crate (e.g., LocalFileSystem) does not appear to have an append interface. This implies that during the do_merge operation, we would need to load the entire file at snapshot_path into memory, append the new data, and then write the entire file back.
   3. What is the expected order of magnitude for the number of sstmeta files? For example, are we dealing with millions?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org
For additional commands, e-mail: commits-help@horaedb.apache.org


Re: [I] Explore another format for manifest [horaedb]

Posted by "jiacai2050 (via GitHub)" <gi...@apache.org>.
jiacai2050 commented on issue #1600:
URL: https://github.com/apache/horaedb/issues/1600#issuecomment-2516328481

   The new flow looks GOOD for me, before you get started, you need to design the format manifest will be using.
   
   Or using the format I proposed in description?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org
For additional commands, e-mail: commits-help@horaedb.apache.org


Re: [I] Explore another format for manifest [horaedb]

Posted by "zealchen (via GitHub)" <gi...@apache.org>.
zealchen commented on issue #1600:
URL: https://github.com/apache/horaedb/issues/1600#issuecomment-2517977676

   > The new flow looks GOOD for me, before you get started, you need to design the format manifest will be using.
   > 
   > Or using the format I proposed in description?
   
   This is format manifest of my design
   ```text
   | magic(u32) | version(u8) |  flags(u8) |Record(N) ... |
   
   Magic(u32) is used for data source validity.
   Flags(u8) is reserved for future extendability, like compression or other stuff.
   
   # Record is a self-descriptive message
   | id(u64) | time_range(i64*2)| size(u32) |  num_rows(u32)|
   ```
   I add num_rows as the struct FileMeta has that field
   ```rust
   pub struct FileMeta {
       pub max_sequence: u64,
       pub num_rows: u32,
       pub size: u32,
       pub time_range: TimeRange,
   }
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org
For additional commands, e-mail: commits-help@horaedb.apache.org