[GitHub] [incubator-doris] yiguolei commented on a diff in pull request #9842: [feature] global dict optimization for low cardinality string column

[GitBox <gi...@apache.org>]

yiguolei commented on code in PR #9842:
URL: https://github.com/apache/incubator-doris/pull/9842#discussion_r889959785


##########
be/src/olap/delta_writer.cpp:
##########
@@ -320,6 +321,41 @@ Status DeltaWriter::close_wait() {
         return res;
     }
 
+    if (invalid_dict_column_names) {
+        invalid_dict_column_names->clear();
+        std::vector<const TabletColumn*> dict_columns;
+        for (const auto& column : _tablet_schema->columns()) {
+            FieldType column_type = column.type();
+            if (_req.dicts != nullptr && (column_type == FieldType::OLAP_FIELD_TYPE_CHAR ||
+                                          column_type == FieldType::OLAP_FIELD_TYPE_VARCHAR ||
+                                          column_type == FieldType::OLAP_FIELD_TYPE_STRING)) {
+                auto iter = _req.dicts->find(column.name().data());
+                if (iter != _req.dicts->end()) {
+                    dict_columns.push_back(&column);
+                }
+            }
+        }
+
+        auto beta_rowset = dynamic_cast<BetaRowset*>(_cur_rowset.get());
+        assert(beta_rowset);
+        std::set<std::string> dict_words;
+        for (const TabletColumn* column : dict_columns) {
+            auto iter = _req.dicts->find(column->name().data());
+            assert(iter != _req.dicts->end());
+            dict_words.clear();
+            Status status = beta_rowset->get_dict_data(dict_words, column->unique_id());

Review Comment:
   If there are too many new dict words in column, get dict data may failed to get dict and return error status. If you return error status here, the load task is failed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org