You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@orc.apache.org by "taiyang-li (via GitHub)" <gi...@apache.org> on 2025/07/08 13:41:27 UTC

[PR] ORC-1950: [C++] Replace std::unorder_map with google dense_hash_map in SortedStringDictionary and remove reorder to improve write performance of dict-encoding columns [orc]

taiyang-li opened a new pull request, #2321:
URL: https://github.com/apache/orc/pull/2321

   
   
   ### What changes were proposed in this pull request?
   Replace std::unorder_map with google dense_hash_map in SortedStringDictionary and remove reorder to improve write performance of dict-encoding columns 
   
   
   ### Why are the changes needed?
   
   
   ### How was this patch tested?
   
   
   ### Was this patch authored or co-authored using generative AI tooling?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1950: [C++] Replace std::unorder_map with google dense_hash_map in SortedStringDictionary and remove reorder to improve write performance of dict-encoding columns [orc]

Posted by "taiyang-li (via GitHub)" <gi...@apache.org>.
taiyang-li commented on PR #2321:
URL: https://github.com/apache/orc/pull/2321#issuecomment-3072482631

   @dongjoon-hyun Now all the tests had been passed.  Hope for your reviews, thanks very much! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1950: [C++] Replace std::unorder_map with google dense_hash_map in SortedStringDictionary and remove reorder to improve write performance of dict-encoding columns [orc]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #2321:
URL: https://github.com/apache/orc/pull/2321#discussion_r2219569602


##########
c++/src/ColumnWriter.cc:
##########
@@ -991,61 +977,33 @@ namespace orc {
   // insert a new string into dictionary, return its insertion order
   size_t SortedStringDictionary::insert(const char* str, size_t len) {
     size_t index = flatDict_.size();
-    auto ret = keyToIndex_.emplace(std::string(str, len), index);
-    if (ret.second) {
-      flatDict_.emplace_back(ret.first->first.data(), ret.first->first.size(), index);
+
+    auto it = keyToIndex_.find(std::string_view{str, len});
+    if (it != keyToIndex_.end()) {
+      return it->second;
+    } else {
+      flatDict_.emplace_back(str, len);
       totalLength_ += len;
+
+      const auto& lastEntry = flatDict_.back();
+      keyToIndex_.emplace(std::string_view{lastEntry.data->data(), lastEntry.data->size()}, index);
+      return index;
     }
-    return ret.first->second;
   }
 
   // write dictionary data & length to output buffer
   void SortedStringDictionary::flush(AppendOnlyBufferedStream* dataStream,
                                      RleEncoder* lengthEncoder) const {
-    std::sort(flatDict_.begin(), flatDict_.end(), LessThan());

Review Comment:
   `For dictionary encodings the dictionary is sorted (in lexicographical order of bytes in the UTF-8 encodings) and UTF-8 bytes of each unique value are placed into DICTIONARY_DATA.`
   
   Sorry for chiming in late. According to the [spec](https://orc.apache.org/specification/ORCv1/), entries in the dictionary must be sorted. I think we need to revert this to observe the spec.
   
   cc @ffacs 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1950: [C++] Replace std::unorder_map with google dense_hash_map in SortedStringDictionary and remove reorder to improve write performance of dict-encoding columns [orc]

Posted by "taiyang-li (via GitHub)" <gi...@apache.org>.
taiyang-li commented on PR #2321:
URL: https://github.com/apache/orc/pull/2321#issuecomment-3072311973

   > Thank you for making a PR, @taiyang-li . It seems to fail on `Debian 12` with CMake. Could you fix that first?
   > 
   > * https://github.com/apache/orc/actions/runs/16162231691/job/45615974033?pr=2321
   > 
   > ```
   > [==========] 744 tests from 47 test suites ran. (12344 ms total)
   > [  PASSED  ] 742 tests.
   > [  FAILED  ] 2 tests, listed below:
   > [  FAILED  ] DictionaryEncoding.stringDictionaryWithNullWithoutIndex
   > [  FAILED  ] DictionaryEncoding.multipleStripesWithIndex
   > 
   >  2 FAILED TESTS
   > ...
   > 
   > The following tests FAILED:
   > 	  1 - orc-test (Failed)
   > make[3]: *** [CMakeFiles/test-out.dir/build.make:70: CMakeFiles/test-out] Error 8
   > make[2]: *** [CMakeFiles/Makefile2:459: CMakeFiles/test-out.dir/all] Error 2
   > make[1]: *** [CMakeFiles/Makefile2:466: CMakeFiles/test-out.dir/rule] Error 2
   > make: *** [Makefile:303: test-out] Error 2
   > FAILED debian12
   > ```
   
   Already fixed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1950: [C++] Replace std::unorder_map with google dense_hash_map in SortedStringDictionary and remove reorder to improve write performance of dict-encoding columns [orc]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #2321: ORC-1950: [C++] Replace std::unorder_map with google dense_hash_map in SortedStringDictionary and remove reorder to improve write performance of dict-encoding columns
URL: https://github.com/apache/orc/pull/2321


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1950: [C++] Replace std::unorder_map with google dense_hash_map in SortedStringDictionary and remove reorder to improve write performance of dict-encoding columns [orc]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #2321:
URL: https://github.com/apache/orc/pull/2321#issuecomment-3074302727

   Merged to main/2.2 for Apache ORC 2.2.0.
   
   This will be released at the end of July (this month), @taiyang-li .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1950: [C++] Replace std::unorder_map with google dense_hash_map in SortedStringDictionary and remove reorder to improve write performance of dict-encoding columns [orc]

Posted by "taiyang-li (via GitHub)" <gi...@apache.org>.
taiyang-li commented on PR #2321:
URL: https://github.com/apache/orc/pull/2321#issuecomment-3051393862

   I don't know how to fix meson build. Do you have any example about it? @wgtmac Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1950: [C++] Replace std::unorder_map with google dense_hash_map in SortedStringDictionary and remove reorder to improve write performance of dict-encoding columns [orc]

Posted by "taiyang-li (via GitHub)" <gi...@apache.org>.
taiyang-li commented on PR #2321:
URL: https://github.com/apache/orc/pull/2321#issuecomment-3072725680

   I tested it in clickhouse. Here are the results:
   
   ``` sql
   Query: select concat('gluten ', cast(rand()%1000 as String)) from numbers(10000000) into outfile 'dict.orc' truncate;
   
   -- Enable dictionary encoding before optimization
   set max_threads =1 ;
   set output_format_orc_dictionary_key_size_threshold = 1;
   
   10000000 rows in set. Elapsed: 0.760 sec. Processed 10.00 million rows, 80.00 MB (13.16 million rows/s., 105.31 MB/s.)
   10000000 rows in set. Elapsed: 0.796 sec. Processed 10.00 million rows, 80.00 MB (12.56 million rows/s., 100.51 MB/s.)
   10000000 rows in set. Elapsed: 0.803 sec. Processed 10.00 million rows, 80.00 MB (12.45 million rows/s., 99.63 MB/s.)
   
   
   -- Enable dictionary encoding after optimization
   set max_threads =1 ;
   set output_format_orc_dictionary_key_size_threshold = 1;
   
   10000000 rows in set. Elapsed: 0.645 sec. Processed 10.00 million rows, 80.00 MB (15.50 million rows/s., 124.00 MB/s.)
   10000000 rows in set. Elapsed: 0.622 sec. Processed 10.00 million rows, 80.00 MB (16.08 million rows/s., 128.64 MB/s.)
   10000000 rows in set. Elapsed: 0.631 sec. Processed 10.00 million rows, 80.00 MB (15.85 million rows/s., 126.83 MB/s.)  
   
   
   
   -- Disable dictionary encoding
   set max_threads =1 ;
   set output_format_orc_dictionary_key_size_threshold = 0;
   
   10000000 rows in set. Elapsed: 0.707 sec. Processed 10.00 million rows, 80.00 MB (14.14 million rows/s., 113.15 MB/s.)
   10000000 rows in set. Elapsed: 0.671 sec. Processed 10.00 million rows, 80.00 MB (14.90 million rows/s., 119.17 MB/s.)
   10000000 rows in set. Elapsed: 0.727 sec. Processed 10.00 million rows, 80.00 MB (13.75 million rows/s., 109.98 MB/s.)
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1950: [C++] Replace std::unorder_map with google dense_hash_map in SortedStringDictionary and remove reorder to improve write performance of dict-encoding columns [orc]

Posted by "taiyang-li (via GitHub)" <gi...@apache.org>.
taiyang-li commented on code in PR #2321:
URL: https://github.com/apache/orc/pull/2321#discussion_r2220989989


##########
c++/src/ColumnWriter.cc:
##########
@@ -991,61 +977,33 @@ namespace orc {
   // insert a new string into dictionary, return its insertion order
   size_t SortedStringDictionary::insert(const char* str, size_t len) {
     size_t index = flatDict_.size();
-    auto ret = keyToIndex_.emplace(std::string(str, len), index);
-    if (ret.second) {
-      flatDict_.emplace_back(ret.first->first.data(), ret.first->first.size(), index);
+
+    auto it = keyToIndex_.find(std::string_view{str, len});
+    if (it != keyToIndex_.end()) {
+      return it->second;
+    } else {
+      flatDict_.emplace_back(str, len);
       totalLength_ += len;
+
+      const auto& lastEntry = flatDict_.back();
+      keyToIndex_.emplace(std::string_view{lastEntry.data->data(), lastEntry.data->size()}, index);
+      return index;
     }
-    return ret.first->second;
   }
 
   // write dictionary data & length to output buffer
   void SortedStringDictionary::flush(AppendOnlyBufferedStream* dataStream,
                                      RleEncoder* lengthEncoder) const {
-    std::sort(flatDict_.begin(), flatDict_.end(), LessThan());

Review Comment:
   @wgtmac  @ffacs I will fix it in another PR. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org