[GitHub] [incubator-doris] vagetablechicken commented on issue #3469: [Memory Engine] Write Pipeline: internal data processing and storage

[GitBox <gi...@apache.org>]

vagetablechicken commented on issue #3469:
URL: https://github.com/apache/incubator-doris/issues/3469#issuecomment-639358916


   ### Prepare Phase Detail Design
   #### Sender BE:
   ```
   Get batch from json_scanner
   For partialrow in batch:
       // may missing key cols(partition cols or distributed cols), should fillin default value.
       // If we want to support default values for each write txn, those default values need to 
       // be pass to receiver BE(e.g. PTabletWriterOpenRequest adds a field). And no need to 
       // modify rows here, coz we only need key cols here. Filling all key & value cols 
       // could exec in commit phase. 
       calc_partition_and_tablet(partialrow, &channel, &tablet_id)
       channel.add_row(partialrow, tablet_id)
   ```
   
   #### Receiver BE:
   ```
   Get batch from sender BE
   For partialrow, tablet_id of batch:
       get delta_writer by tablet_id
       delta_writer.write(partialrow)
   ```
   
    ##### DeltaWriter(DeltaWriter has two mode, one for disk engine, one for memory engine. Sets up the mode upon initialization)
   ```
   // change arg Tuple* to void*
   // disk engine MemTable will convert it to Tuple
   // memory engine MemTable will convert it to PartialRow
   DeltaWriter::insert(void* row):
       mem_table.insert(row)
       if memory_usage > config:
           _flush_memtable_async()
           _reset_mem_table()
   ```
   ##### WTDMemTable(WriteTxnData MemTable, flush WriteTxnData files)
   Member var: _buffer, ...
   ```
   WTDMemTable::insert(void* row):
       append(_buffer, row, sizeof(uint64_t)+*(uint64_t*)row)
       num_rows++;
   
   WTDMemTable::flush():
       append(_buffer, footer)
       write_to_disk(_buffer)
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org