[GitHub] [incubator-doris] wangbo opened a new issue #1943: Segment v2 use string real length

[GitBox <gi...@apache.org>]

wangbo opened a new issue #1943: Segment v2 use string real length
URL: https://github.com/apache/incubator-doris/issues/1943
 
 
   Current segment v2 use fixed length to allocate string field,using real length of string field is better
   
   # major process
   1. init a Field by type in SegmentWriter.init,then ColumnWriter and ColumnZoneMapBuilder holds it instead of holding a typeinfo,this is because field contains whole info of a specific type Field,while typeInfo only contains method
   2. using polymorphism to realize string type allocate value with length while other type allocate value with default method.
   Char/VarcharField override allocate_value_from_arena,and Char/VarcharField contains string real length
   
   # Other question
   1. bug of set_to_max;
   	current set_to_max of varchar,hard code setting size of varchar length to 1,but it should use user define varchar length.
   	we can use field.length(come from tabletColumn.length) to set_to_max
   2. problem of using field.length in set_to_max
   	segment v1 think field.length=sizeof(StringLengthType) + slice.size
   	segment v2 think field.length=slice.size
   	so confict happends
   	I defined a new method signature (set_to_max_v1)to let segment v1 code using it.
   	the logic of set_to_max_v1 not changed,it is still hard code.
   	segment v2 using set_to_max method, and it set string using string'real length
   3. current code refers field.set_to_max
   	wrapper_field.cpp, change to set_to_max_v1
   	stream_index_common.cpp,no change,string and hll skipped
   	row_cursor.cpp, change to set_to_max_v1

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org