[GitHub] [incubator-doris] vagetablechicken commented on issue #3469: [Memory Engine] Write Pipeline: internal data processing and storage

[GitBox <gi...@apache.org>]

vagetablechicken commented on issue #3469:
URL: https://github.com/apache/incubator-doris/issues/3469#issuecomment-626553115


   ### Data Flow
   1.  User data(json format) -> PartialRow
       1. PartialRow(Type,...), Type: INSERT/UPSERT/DELETE/UPDATE
       1. In POC: PartialRow(cells[0],...), cells[0].isset == 1 means DELETE. Don't distinguish between INSERT and UPDATE.
   1. PartialRowBatch ---transport---> ... --> DeltaWriter::write(PartialRow row)
   
   We focus on DeltaWriter.
   
   #### DeltaWriter
   About DeltaWriter, needs a new class, effects similar to MemTable, temporarily named WALMemTable. Flush WAL when DeltaWriter closed.
   1. DeltaWriter prepare works
   1. DeltaWriter::write(PartialRow row)
   1. DeltaWriter::close()
       1. WALMemTable::flush()
       1. WAL file format(the simplest): one line is a partial row, saved in {data_dir}/wal/{shard_id}/{tablet_id}/{schema_hash}/, filename: {txn_id}.wal
   1. DeltaWriter::close_wait()
       1. Apply rows to SubTablet (If meet errors, e.g. can't find key for DELETE/UPDATE, return error)
       1. commit_txn()


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org