[GitHub] [incubator-doris] imay commented on a change in pull request #1238: Reduce memory RowBlock needed

[GitBox <gi...@apache.org>]

imay commented on a change in pull request #1238: Reduce memory RowBlock needed
URL: https://github.com/apache/incubator-doris/pull/1238#discussion_r290099274
 
 

 ##########
 File path: be/src/olap/row_block.cpp
 ##########
 @@ -314,51 +115,29 @@ void RowBlock::clear() {
 }
 
 void RowBlock::_compute_layout() {
+    std::unordered_set<uint32_t> column_set(_info.column_ids.begin(), _info.column_ids.end());
     size_t memory_size = 0;
-    size_t storage_fixed_bytes = 0;
-    size_t storage_variable_bytes = 0;
-    for (auto& field : _tablet_schema) {
+    for (int i = 0; i < _tablet_schema.size(); ++i) {
+        auto& field = _tablet_schema[i];
+        if (!column_set.empty() && column_set.find(i) == std::end(column_set)) {
+            // which may lead BE crash
+            _field_offset_in_memory.push_back(std::numeric_limits<std::size_t>::max());
+            continue;
+        }
+
         _field_offset_in_memory.push_back(memory_size);
 
         // All field has a nullbyte in memory
-        if (field.type == OLAP_FIELD_TYPE_VARCHAR || field.type == OLAP_FIELD_TYPE_HLL) {
+        if (field.type == OLAP_FIELD_TYPE_VARCHAR || field.type == OLAP_FIELD_TYPE_HLL
+                || field.type == OLAP_FIELD_TYPE_CHAR) {
             // 变长部分额外计算下实际最大的字符串长度（此处length已经包括记录Length的2个字节）
-            storage_fixed_bytes += sizeof(StringOffsetType);
-            storage_variable_bytes += field.length;
-            if (has_nullbyte()) {
-                storage_variable_bytes += sizeof(char);
-            }
             memory_size += sizeof(Slice) + sizeof(char);
         } else {
-            storage_fixed_bytes += field.length;
-            if (has_nullbyte()) {
-                storage_fixed_bytes += sizeof(char);
-            }
-            if (field.type == OLAP_FIELD_TYPE_CHAR) {
-                memory_size += sizeof(Slice) + sizeof(char);
-            } else {
-                memory_size += field.length + sizeof(char);
-            }
+            memory_size += field.length + sizeof(char);
 
 Review comment:
   of course yes, but not in this patch

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@doris.apache.org
For additional commands, e-mail: dev-help@doris.apache.org