You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "llama90 (via GitHub)" <gi...@apache.org> on 2023/10/09 08:02:16 UTC

[PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

llama90 opened a new pull request, #38147:
URL: https://github.com/apache/arrow/pull/38147

   <!--
   Thanks for opening a pull request!
   If this is your first pull request you can find detailed information on how 
   to contribute here:
     * [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
     * [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
   
   
   If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
   
   Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
   
   Then could you also rename the pull request title in the following format?
   
       GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
   
   or
   
       MINOR: [${COMPONENT}] ${SUMMARY}
   
   In the case of PARQUET issues on JIRA the title also supports:
   
       PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
   
   -->
   
   ### Rationale for this change
   
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
   -->
   
   * #37729 
   
   According to this issue, it has been identified that incorrect matching results are being produced when performing an Inner Join. I took an interest in this issue and proceeded with debugging, during which I discovered that incorrect hash values are being generated when executing the BloomFilter operation.
   
   * #38074
   
   Essentially, when executing the `HashBatch` function, data for the minibatch size is retrieved through the `Slice` function. However, I found that within the `Slice` function, offset calculation is performed as `uint32_t`, regardless of the type value. Consequently, I have modified this to process the value according to its type.
   
   The error likely wouldn't have occurred if users had configured the column with values corresponding to uint32_t. (I have actually tested this aspect as well.)
   
   ### What changes are included in this PR?
   
   I have modified the `Slice` function to reflect the type size.
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   ### Are these changes tested?
   
   I have added unit tests and also ran the full test suite.
   
   * TEST(KeyColumnArray, SliceBinary)
   * TEST(KeyColumnArray, SliceLargeBinary)
   
   I also wrote test code for Inner Join, but did not include it, as it was not directly related to the bug (If you think adding those tests is beneficial, please let me know. Personally, I considered it an unnecessary load on the unit tests).
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   
   ### Are there any user-facing changes?
   
   Acero might not have a large user base as it is an experimental feature, but I deemed the issue of incorrect join results as critical and have addressed the bug.
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!--
   If there are any breaking changes to public APIs, please uncomment the line below and explain which changes are breaking.
   -->
   <!-- **This PR includes breaking changes to public APIs.** -->
   
   <!--
   Please uncomment the line below (and provide explanation) if the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld). We use this to highlight fixes to issues that may affect users without their knowledge. For this reason, fixing bugs that cause errors don't count, since those are usually obvious.
   -->
   <!-- **This PR contains a "Critical Fix".** -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38147:
URL: https://github.com/apache/arrow/pull/38147#issuecomment-1752518152

   :warning: GitHub issue #38074 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38147:
URL: https://github.com/apache/arrow/pull/38147#discussion_r1356409631


##########
cpp/src/arrow/acero/hash_join_node_test.cc:
##########
@@ -2124,5 +2124,266 @@ TEST(HashJoin, ChainedIntegerHashJoins) {
   }
 }
 
+// This test case is related to GH-38147
+// To verify that the issue with offset handling has been fixed, the number of matching
+// records needs to be larger than the mini-batch size (1024).
+constexpr uint64_t NUM_MATCH_RECORDS = 1234;
+constexpr uint64_t NUM_LEFT_RECORDS = 2000;
+constexpr uint64_t NUM_RIGHT_RECORDS = 1500;
+
+std::string GenerateTimestamp() {

Review Comment:
   Are the custom generation functions required to surface the bug, or can we just build on existing generation functions? We already have random generation functions, for example.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38147:
URL: https://github.com/apache/arrow/pull/38147#discussion_r1356414226


##########
cpp/src/arrow/compute/light_array.cc:
##########
@@ -80,16 +80,15 @@ KeyColumnArray KeyColumnArray::Slice(int64_t offset, int64_t length) const {
   KeyColumnArray sliced;
   sliced.metadata_ = metadata_;
   sliced.length_ = length;
-  uint32_t fixed_size =
-      !metadata_.is_fixed_length ? sizeof(uint32_t) : metadata_.fixed_length;
+  uint32_t fixed_size = metadata_.fixed_length;
 
   sliced.buffers_[0] =
       buffers_[0] ? buffers_[0] + (bit_offset_[0] + offset) / 8 : nullptr;
   sliced.mutable_buffers_[0] =
       mutable_buffers_[0] ? mutable_buffers_[0] + (bit_offset_[0] + offset) / 8 : nullptr;
   sliced.bit_offset_[0] = (bit_offset_[0] + offset) % 8;
 
-  if (fixed_size == 0 && !metadata_.is_null_type) {
+  if (metadata_.fixed_length == 0 && !metadata_.is_null_type) {
     sliced.buffers_[1] =
         buffers_[1] ? buffers_[1] + (bit_offset_[1] + offset) / 8 : nullptr;

Review Comment:
   I'm not sure I understand your explanation, sorry :-) Can we add checks (or debug assertions) against the actual datatype in this function? Right now it's difficult to figure out what happens exactly, and to make sure we don't break it again.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #38147:
URL: https://github.com/apache/arrow/pull/38147#discussion_r1358489977


##########
cpp/src/arrow/compute/light_array.h:
##########
@@ -183,6 +183,36 @@ class ARROW_EXPORT KeyColumnArray {
   // Starting bit offset within the first byte (between 0 and 7)
   // to be used when accessing buffers that store bit vectors.
   int bit_offset_[kMaxBuffers - 1];
+
+  bool is_dictionary_type() const {

Review Comment:
   You are right.
   
   My understanding is that the code is added, but probably unsupported or not being tested properly. 
   
   So, I took a hint from what you said, I removed the code for the dictionary type within `KeyColumnArray` and ran the entire test and found that none of the tests covered that code :O 
   (If that's okay, can we add a **new issue** for `KeyColumns` to have complex values like dictionary types?)
   
   Looking at the data I was randomly generating in the `hash_join_node_test.cc` file, I was only generating data for the following types.
   
   In the code below, I have added support for `LargeString` type.
   
   https://github.com/apache/arrow/blob/e9aac8a4ce4b8833667efe52957fb0881a488c4b/cpp/src/arrow/acero/hash_join_node_test.cc#L284-L291



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on PR #38147:
URL: https://github.com/apache/arrow/pull/38147#issuecomment-1755743706

   > Here are some comments.
   > 
   > Also, can you add join tests with large_binary or large_utf8?
   
   I added the unit test about inner join for `large_binary` or `large_utf8`
   
   * https://github.com/apache/arrow/pull/38147/commits/3894147f8273facf551f5070c084c2513a6d048d


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #38147:
URL: https://github.com/apache/arrow/pull/38147#discussion_r1352836497


##########
cpp/src/arrow/compute/light_array_test.cc:
##########
@@ -226,6 +226,104 @@ TEST(KeyColumnArray, SliceBool) {
   }
 }
 
+TEST(KeyColumnArray, SliceBinary) {

Review Comment:
   I enhanced the unit tests for readability



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #38147:
URL: https://github.com/apache/arrow/pull/38147#issuecomment-1757738451

   FTR, I still need to take a look at the fix and see if we can make things more maintainable and more understandable in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on code in PR #38147:
URL: https://github.com/apache/arrow/pull/38147#discussion_r1359467543


##########
cpp/src/arrow/compute/light_array.cc:
##########
@@ -80,27 +80,32 @@ KeyColumnArray KeyColumnArray::Slice(int64_t offset, int64_t length) const {
   KeyColumnArray sliced;
   sliced.metadata_ = metadata_;
   sliced.length_ = length;
-  uint32_t fixed_size =
-      !metadata_.is_fixed_length ? sizeof(uint32_t) : metadata_.fixed_length;
+  uint32_t fixed_size = metadata_.fixed_length;
 
   sliced.buffers_[0] =
       buffers_[0] ? buffers_[0] + (bit_offset_[0] + offset) / 8 : nullptr;
   sliced.mutable_buffers_[0] =
       mutable_buffers_[0] ? mutable_buffers_[0] + (bit_offset_[0] + offset) / 8 : nullptr;
   sliced.bit_offset_[0] = (bit_offset_[0] + offset) % 8;
 
-  if (fixed_size == 0 && !metadata_.is_null_type) {
+  if (metadata_.fixed_length == 0 && !metadata_.is_null_type) {
+    ARROW_DCHECK(is_bool_type() || is_other_fixed_width_types())
+        << "Expected BOOL type or OTHER FIXED WIDTH type but got a different type.";
     sliced.buffers_[1] =
         buffers_[1] ? buffers_[1] + (bit_offset_[1] + offset) / 8 : nullptr;
     sliced.mutable_buffers_[1] = mutable_buffers_[1]
                                      ? mutable_buffers_[1] + (bit_offset_[1] + offset) / 8
                                      : nullptr;
     sliced.bit_offset_[1] = (bit_offset_[1] + offset) % 8;
-  } else {
+  } else if (metadata_.fixed_length > 0) {
+    ARROW_DCHECK(is_dictionary_type() || is_binary_type() || is_large_binary_type())

Review Comment:
   > If I'm misunderstanding, I'd be grateful to hear your concerns, I'm still a newbie to Arrow.
   
   I do not think this is a misunderstanding.  The key column array marks variable data as `!is_fixed_length` and then `fixed_length` should store the width of the offset (4 for string/binary and 8 for large_string/large_binary)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #38147:
URL: https://github.com/apache/arrow/pull/38147#issuecomment-1763033249

   Also, big +1 to what @westonpace said [above](https://github.com/apache/arrow/pull/38147#issuecomment-1763032429). You definitely didn't choose the easiest part of Arrow to contribute to :-)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on PR #38147:
URL: https://github.com/apache/arrow/pull/38147#issuecomment-1760185003

   I have removed the unrelated commits and forced a push. 
   
   If this isn't the appropriate way to resolve the issue, please advise. I appreciate any guidance.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on PR #38147:
URL: https://github.com/apache/arrow/pull/38147#issuecomment-1753293165

   @pitrou You are right. I will incorporate your feedback and add the Commit as soon as possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #38147:
URL: https://github.com/apache/arrow/pull/38147#issuecomment-1753221234

   @llama90 I don't understand why you've changed your fix, while I was asking you to explain the underlying bug.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38147:
URL: https://github.com/apache/arrow/pull/38147#discussion_r1350490256


##########
cpp/src/arrow/compute/light_array_test.cc:
##########
@@ -226,6 +226,104 @@ TEST(KeyColumnArray, SliceBool) {
   }
 }
 
+TEST(KeyColumnArray, SliceBinary) {

Review Comment:
   These tests are honestly difficult to understand.
   
   Can you perhaps change them to use `ColumnArrayFromArrayData` and `ArrayFromJSON`?
   



##########
cpp/src/arrow/compute/light_array.cc:
##########
@@ -80,16 +80,15 @@ KeyColumnArray KeyColumnArray::Slice(int64_t offset, int64_t length) const {
   KeyColumnArray sliced;
   sliced.metadata_ = metadata_;
   sliced.length_ = length;
-  uint32_t fixed_size =
-      !metadata_.is_fixed_length ? sizeof(uint32_t) : metadata_.fixed_length;
+  uint32_t fixed_size = metadata_.fixed_length;
 
   sliced.buffers_[0] =
       buffers_[0] ? buffers_[0] + (bit_offset_[0] + offset) / 8 : nullptr;
   sliced.mutable_buffers_[0] =
       mutable_buffers_[0] ? mutable_buffers_[0] + (bit_offset_[0] + offset) / 8 : nullptr;
   sliced.bit_offset_[0] = (bit_offset_[0] + offset) % 8;
 
-  if (fixed_size == 0 && !metadata_.is_null_type) {
+  if (metadata_.fixed_length == 0 && !metadata_.is_null_type) {
     sliced.buffers_[1] =
         buffers_[1] ? buffers_[1] + (bit_offset_[1] + offset) / 8 : nullptr;

Review Comment:
   Is this correct? You probably need to multiply the offset by 4 (for binary) or 8 (for large_binary) to point to the sliced offsets.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on PR #38147:
URL: https://github.com/apache/arrow/pull/38147#issuecomment-1757728598

   @llama90 could you please merge/rebase this with the latest changes on the main branch? That should fix the remaining CI failure.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on PR #38147:
URL: https://github.com/apache/arrow/pull/38147#issuecomment-1756155632

   @llama90 could you please run the linter? Instructions at https://arrow.apache.org/docs/developers/cpp/development.html#code-style-linting-and-ci


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #38147:
URL: https://github.com/apache/arrow/pull/38147#discussion_r1352836497


##########
cpp/src/arrow/compute/light_array_test.cc:
##########
@@ -226,6 +226,104 @@ TEST(KeyColumnArray, SliceBool) {
   }
 }
 
+TEST(KeyColumnArray, SliceBinary) {

Review Comment:
   I enhanced the unit tests for readability.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on PR #38147:
URL: https://github.com/apache/arrow/pull/38147#issuecomment-1753252441

   @pitrou I thought you were pointing out a part in the code where a bug could occur due to implicit type conversion. 
   
   So that's why I made the changes, and I didn't realize that there should be a discussion first when such reviews are given. I apologize for the confusion.
   
   Is it right to fundamentally ask why the code was changed?
   
   The initial issue raised was regarding incorrect return values of the Inner Join. Upon analyzing the code, it was found that during the execution of the `BuildBloomFilter_exec_task` function, incorrect offset calculations were made when calling the `HashBatch` function, leading to incorrect hash values being generated.
   
   `HashBatch` is responsible for copying ColumnArrays within the Key Batch using offset and length, and it calls the `Slice` function during this process. 
   
   In the issue, a `large_utf8` type key column was being used, and the original code was set to always calculate the offset for such binary types as `uint32_t` size, which resulted in incorrect Inner Join outcomes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #38147:
URL: https://github.com/apache/arrow/pull/38147#discussion_r1350430899


##########
cpp/src/arrow/compute/light_array.cc:
##########
@@ -80,26 +80,24 @@ KeyColumnArray KeyColumnArray::Slice(int64_t offset, int64_t length) const {
   KeyColumnArray sliced;
   sliced.metadata_ = metadata_;
   sliced.length_ = length;
-  uint32_t fixed_size =
-      !metadata_.is_fixed_length ? sizeof(uint32_t) : metadata_.fixed_length;
 
   sliced.buffers_[0] =
       buffers_[0] ? buffers_[0] + (bit_offset_[0] + offset) / 8 : nullptr;
   sliced.mutable_buffers_[0] =
       mutable_buffers_[0] ? mutable_buffers_[0] + (bit_offset_[0] + offset) / 8 : nullptr;
   sliced.bit_offset_[0] = (bit_offset_[0] + offset) % 8;
 
-  if (fixed_size == 0 && !metadata_.is_null_type) {
+  if (metadata_.fixed_length == 0 && !metadata_.is_null_type) {
     sliced.buffers_[1] =
         buffers_[1] ? buffers_[1] + (bit_offset_[1] + offset) / 8 : nullptr;
     sliced.mutable_buffers_[1] = mutable_buffers_[1]
                                      ? mutable_buffers_[1] + (bit_offset_[1] + offset) / 8
                                      : nullptr;
     sliced.bit_offset_[1] = (bit_offset_[1] + offset) % 8;
   } else {
-    sliced.buffers_[1] = buffers_[1] ? buffers_[1] + offset * fixed_size : nullptr;
+    sliced.buffers_[1] = buffers_[1] ? buffers_[1] + offset * metadata_.fixed_length : nullptr;

Review Comment:
   Indeed. Thank you for the review.
   
   I realized that there's a potential for error due to implicit type conversion. I modified the code to handle it explicitly and ran the unit tests again.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #38147:
URL: https://github.com/apache/arrow/pull/38147#issuecomment-1753269411

   > In the issue, a `large_utf8` type key column was being used, and the original code was set to always calculate the offset for such binary types as `uint32_t` size, which resulted in incorrect Inner Join outcomes.
   
   Ahah, ok, thanks for the explanation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on PR #38147:
URL: https://github.com/apache/arrow/pull/38147#issuecomment-1757458461

   > > @llama90 could you please run the linter? Instructions at https://arrow.apache.org/docs/developers/cpp/development.html#code-style-linting-and-ci
   > 
   > Did I apply the lint correctly as you intended?
   
   Yes, the "Dev / Lint C++, Python, R, Docker, RAT" test is passing now


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on PR #38147:
URL: https://github.com/apache/arrow/pull/38147#issuecomment-1761832444

   @westonpace I've refined the code, removing the dictionary type as it doesn't seem to be added in any test.
   
   Also, I truly appreciate all the reviews.
   
   As a beginner, I feel both overwhelmed and excited to handle an issue that requires a complex understanding. While I am aware of my limitations, I am committed to giving my best.
   
   I humbly ask for your generous advice and guidance. Thank you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #38147:
URL: https://github.com/apache/arrow/pull/38147#discussion_r1357754499


##########
cpp/src/arrow/compute/light_array.cc:
##########
@@ -80,16 +80,15 @@ KeyColumnArray KeyColumnArray::Slice(int64_t offset, int64_t length) const {
   KeyColumnArray sliced;
   sliced.metadata_ = metadata_;
   sliced.length_ = length;
-  uint32_t fixed_size =
-      !metadata_.is_fixed_length ? sizeof(uint32_t) : metadata_.fixed_length;
+  uint32_t fixed_size = metadata_.fixed_length;
 
   sliced.buffers_[0] =
       buffers_[0] ? buffers_[0] + (bit_offset_[0] + offset) / 8 : nullptr;
   sliced.mutable_buffers_[0] =
       mutable_buffers_[0] ? mutable_buffers_[0] + (bit_offset_[0] + offset) / 8 : nullptr;
   sliced.bit_offset_[0] = (bit_offset_[0] + offset) % 8;
 
-  if (fixed_size == 0 && !metadata_.is_null_type) {
+  if (metadata_.fixed_length == 0 && !metadata_.is_null_type) {
     sliced.buffers_[1] =
         buffers_[1] ? buffers_[1] + (bit_offset_[1] + offset) / 8 : nullptr;

Review Comment:
   The `Slice` function processes the offset according to the types of `KeyColumnMetadata` it handles. With this in consideration, I've added the assertions using `ARROW_DCHECK`. Will this contribute to enhancing the understandability and maintenance of the code?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #38147:
URL: https://github.com/apache/arrow/pull/38147#issuecomment-1763032831

   Yes, I agree the PR as is is a good improvement now.
   
   What I would suggest is to update the PR title and description to better explain the problem. Specifically, it is about slicing large string and large binary types, with the problem being the offset size not correctly computed, IIUC.
   
   ("uint64_t Types" in the title is really confusing as this PR has nothing to do with 64-bit integer columns)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on code in PR #38147:
URL: https://github.com/apache/arrow/pull/38147#discussion_r1359474026


##########
cpp/src/arrow/compute/light_array.cc:
##########
@@ -80,16 +80,15 @@ KeyColumnArray KeyColumnArray::Slice(int64_t offset, int64_t length) const {
   KeyColumnArray sliced;
   sliced.metadata_ = metadata_;
   sliced.length_ = length;
-  uint32_t fixed_size =
-      !metadata_.is_fixed_length ? sizeof(uint32_t) : metadata_.fixed_length;
+  uint32_t fixed_size = metadata_.fixed_length;
 
   sliced.buffers_[0] =
       buffers_[0] ? buffers_[0] + (bit_offset_[0] + offset) / 8 : nullptr;
   sliced.mutable_buffers_[0] =
       mutable_buffers_[0] ? mutable_buffers_[0] + (bit_offset_[0] + offset) / 8 : nullptr;
   sliced.bit_offset_[0] = (bit_offset_[0] + offset) % 8;
 
-  if (fixed_size == 0 && !metadata_.is_null_type) {
+  if (metadata_.fixed_length == 0 && !metadata_.is_null_type) {
     sliced.buffers_[1] =
         buffers_[1] ? buffers_[1] + (bit_offset_[1] + offset) / 8 : nullptr;

Review Comment:
   I think this is correct now.  `buffers_[0]` is the validity buffer and is handled consistently across all types.  This if/else if/else branch is handling the values (for fixed-width types) / offsets (for variable-width types) buffer.
   
   So it's roughly...
   
   ```
   if (is_bool) {
     slice by bit, update bit offset
   } else if (!is_null) {
     slice by word (`fixed_size`), no bit offset needed
   } else // is null {
     do nothing, there is no values buffer
   }
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #38147:
URL: https://github.com/apache/arrow/pull/38147#discussion_r1358505770


##########
cpp/src/arrow/compute/light_array.cc:
##########
@@ -80,27 +80,32 @@ KeyColumnArray KeyColumnArray::Slice(int64_t offset, int64_t length) const {
   KeyColumnArray sliced;
   sliced.metadata_ = metadata_;
   sliced.length_ = length;
-  uint32_t fixed_size =
-      !metadata_.is_fixed_length ? sizeof(uint32_t) : metadata_.fixed_length;
+  uint32_t fixed_size = metadata_.fixed_length;
 
   sliced.buffers_[0] =
       buffers_[0] ? buffers_[0] + (bit_offset_[0] + offset) / 8 : nullptr;
   sliced.mutable_buffers_[0] =
       mutable_buffers_[0] ? mutable_buffers_[0] + (bit_offset_[0] + offset) / 8 : nullptr;
   sliced.bit_offset_[0] = (bit_offset_[0] + offset) % 8;
 
-  if (fixed_size == 0 && !metadata_.is_null_type) {
+  if (metadata_.fixed_length == 0 && !metadata_.is_null_type) {
+    ARROW_DCHECK(is_bool_type() || is_other_fixed_width_types())
+        << "Expected BOOL type or OTHER FIXED WIDTH type but got a different type.";
     sliced.buffers_[1] =
         buffers_[1] ? buffers_[1] + (bit_offset_[1] + offset) / 8 : nullptr;
     sliced.mutable_buffers_[1] = mutable_buffers_[1]
                                      ? mutable_buffers_[1] + (bit_offset_[1] + offset) / 8
                                      : nullptr;
     sliced.bit_offset_[1] = (bit_offset_[1] + offset) % 8;
-  } else {
+  } else if (metadata_.fixed_length > 0) {
+    ARROW_DCHECK(is_dictionary_type() || is_binary_type() || is_large_binary_type())

Review Comment:
   That's right. 
   
   If you're worried about why variable and fixed lengths are handled with the same logic, then the answer is as follows.
   
   As I understand it, Arrow maintains a separate buffer for storing values for variable lengths. 
   
   Because of this, we only need to calculate the offset well for the address pointing to the value, so I don't think fixed or variable length is an important consideration when calculating the offset. This is why I didn't think it was a problem to calculate it with the same logic.
   
   If I'm misunderstanding, I'd be grateful to hear your concerns, I'm still a newbie to Arrow.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on PR #38147:
URL: https://github.com/apache/arrow/pull/38147#issuecomment-1756769643

   > @llama90 could you please run the linter? Instructions at https://arrow.apache.org/docs/developers/cpp/development.html#code-style-linting-and-ci
   
   Did I apply the lint correctly as you intended?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38147:
URL: https://github.com/apache/arrow/pull/38147#discussion_r1356411588


##########
cpp/src/arrow/acero/hash_join_node_test.cc:
##########
@@ -2124,5 +2124,266 @@ TEST(HashJoin, ChainedIntegerHashJoins) {
   }
 }
 
+// This test case is related to GH-38147
+// To verify that the issue with offset handling has been fixed, the number of matching
+// records needs to be larger than the mini-batch size (1024).
+constexpr uint64_t NUM_MATCH_RECORDS = 1234;
+constexpr uint64_t NUM_LEFT_RECORDS = 2000;
+constexpr uint64_t NUM_RIGHT_RECORDS = 1500;
+
+std::string GenerateTimestamp() {
+  auto now = std::chrono::system_clock::now();
+  auto now_time_t = std::chrono::system_clock::to_time_t(now);
+  auto now_ms =
+      std::chrono::duration_cast<std::chrono::milliseconds>(now.time_since_epoch()) %
+      1000;
+
+  std::stringstream ss;
+  ss << std::put_time(std::localtime(&now_time_t), "%Y-%m-%d %H:%M:%S");
+  ss << '.' << std::setw(6) << std::setfill('0') << now_ms.count();
+  return ss.str();
+}
+
+std::string GenerateRecord(
+    uint64_t index, const std::string& timestamp, bool useNegativeOne,
+    std::function<std::vector<std::string>(uint64_t)> getExtraValues) {
+  std::stringstream record;
+  record << R"([")" << timestamp << R"(",)";
+  if (useNegativeOne && index >= NUM_MATCH_RECORDS) {
+    record << "\"Not Matched\"";
+  } else {
+    record << "\"" << index + 1 << "\"";
+  }
+  auto extraValues = getExtraValues(index);
+  for (const auto& value : extraValues) {
+    record << R"(, ")" << value << R"(")";
+  }
+  record << R"(])";
+  return record.str();
+}
+
+std::string GenerateLeftRecords(
+    uint64_t recordCount, const std::string& timestamp,
+    std::function<std::vector<std::string>(uint64_t)> getExtraValues) {
+  std::stringstream ss;
+  ss << R"([)";
+  for (uint64_t i = 0; i < recordCount; ++i) {
+    if (i != 0) ss << ",";
+    ss << GenerateRecord(i, timestamp, true, getExtraValues);
+  }
+  ss << R"(])";
+  return ss.str();
+}
+
+std::string GenerateRightRecords(
+    uint64_t recordCount, const std::string& timestamp,
+    std::function<std::vector<std::string>(uint64_t)> getExtraValues) {
+  std::stringstream ss;
+  ss << R"([)";
+  for (uint64_t i = 0; i < recordCount; ++i) {
+    if (i != 0) ss << ",";
+    ss << GenerateRecord(i, timestamp, false, getExtraValues);
+  }
+  ss << R"(])";
+  return ss.str();
+}
+
+std::string GenerateExpectedRecords(
+    uint64_t recordCount, const std::string& timestamp,
+    std::function<std::vector<std::string>(uint64_t)> getLeftExtraValues,
+    std::function<std::vector<std::string>(uint64_t)> getRightExtraValues) {
+  std::stringstream ss;
+  ss << R"([)";
+  for (uint64_t i = 0; i < recordCount; ++i) {
+    if (i >= NUM_MATCH_RECORDS) break;  // Consider only matched records
+
+    if (i != 0) ss << ",";
+
+    auto leftExtraValues = getLeftExtraValues(i);
+    auto rightExtraValues = getRightExtraValues(i);
+
+    // Construct a matched record (inner join result)
+    ss << R"([")" << timestamp << R"(",)"
+       << "\"" << i + 1 << "\"";
+    for (const auto& value : leftExtraValues) {
+      ss << R"(,")" << value << R"(")";
+    }
+    ss << R"(,")" << timestamp << R"(",)"
+       << "\"" << i + 1 << "\"";
+    for (const auto& value : rightExtraValues) {
+      ss << R"(,")" << value << R"(")";
+    }
+    ss << R"(])";
+  }
+  ss << R"(])";
+  return ss.str();
+}
+
+// This test case is related to GH-38147
+// In the previous code, only 1030 matching records are returned.
+TEST(HashJoin, InnerJoinTestLargerThanMiniBatchSize) {
+  // Just as an example, returning "foo" for every record
+  auto leftExtraValues = [](uint64_t index) -> std::vector<std::string> {
+    return {"foo"};
+  };
+
+  // Just as an example, returning "foo" and "bar" for every record
+  auto rightExtraValues = [](uint64_t index) -> std::vector<std::string> {
+    return {"foo", "bar"};
+  };
+
+  std::string currTS = GenerateTimestamp();
+
+  std::string leftRecords =
+      GenerateLeftRecords(NUM_LEFT_RECORDS, currTS, leftExtraValues);
+  std::string rightRecords =
+      GenerateRightRecords(NUM_RIGHT_RECORDS, currTS, rightExtraValues);
+  std::string expectedRecords = GenerateExpectedRecords(
+      NUM_MATCH_RECORDS, currTS, leftExtraValues, rightExtraValues);
+
+  // Initialize left input data with the specific timestamp and large utf8 data
+  BatchesWithSchema input_left;
+  input_left.batches = {ExecBatchFromJSON(
+      {timestamp(TimeUnit::MICRO, "UTC"), large_utf8(), large_utf8()}, leftRecords)};

Review Comment:
   Similar question here. Is using timestamps necessary to surface the bug or would it also be reproduced with e.g. plain integers? Trying to reduce the test case into a minimal reproducer would be useful for understanding and maintenance.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #38147:
URL: https://github.com/apache/arrow/pull/38147#issuecomment-1760199940

   > If this isn't the appropriate way to resolve the issue, please advise. I appreciate any guidance. 😭
   
   It seems like that did the trick :-)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on code in PR #38147:
URL: https://github.com/apache/arrow/pull/38147#discussion_r1359465060


##########
cpp/src/arrow/compute/light_array.h:
##########
@@ -183,6 +183,36 @@ class ARROW_EXPORT KeyColumnArray {
   // Starting bit offset within the first byte (between 0 and 7)
   // to be used when accessing buffers that store bit vectors.
   int bit_offset_[kMaxBuffers - 1];
+
+  bool is_dictionary_type() const {

Review Comment:
   >  (If that's okay, can we add a new issue for KeyColumns to have complex values like dictionary types?)
   
   Is this request maybe covered by https://github.com/apache/arrow/issues/31622 , https://github.com/apache/arrow/issues/32494 , or https://github.com/apache/arrow/issues/32504 ?
   
   Adding dictionary support for keys will be trickier than supporting large strings unfortunately.  I can give an overview of some of the steps but I don't know that anyone has been spending a lot of time on the hash join code lately.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on code in PR #38147:
URL: https://github.com/apache/arrow/pull/38147#discussion_r1358388917


##########
cpp/src/arrow/compute/light_array.cc:
##########
@@ -80,27 +80,32 @@ KeyColumnArray KeyColumnArray::Slice(int64_t offset, int64_t length) const {
   KeyColumnArray sliced;
   sliced.metadata_ = metadata_;
   sliced.length_ = length;
-  uint32_t fixed_size =
-      !metadata_.is_fixed_length ? sizeof(uint32_t) : metadata_.fixed_length;
+  uint32_t fixed_size = metadata_.fixed_length;
 
   sliced.buffers_[0] =
       buffers_[0] ? buffers_[0] + (bit_offset_[0] + offset) / 8 : nullptr;
   sliced.mutable_buffers_[0] =
       mutable_buffers_[0] ? mutable_buffers_[0] + (bit_offset_[0] + offset) / 8 : nullptr;
   sliced.bit_offset_[0] = (bit_offset_[0] + offset) % 8;
 
-  if (fixed_size == 0 && !metadata_.is_null_type) {
+  if (metadata_.fixed_length == 0 && !metadata_.is_null_type) {
+    ARROW_DCHECK(is_bool_type() || is_other_fixed_width_types())

Review Comment:
   How would `is_other_fixed_width_types` be true here if `metadata_.fixed_length == 0`?  It seems we can only be here if bool.



##########
cpp/src/arrow/compute/light_array.h:
##########
@@ -183,6 +183,36 @@ class ARROW_EXPORT KeyColumnArray {
   // Starting bit offset within the first byte (between 0 and 7)
   // to be used when accessing buffers that store bit vectors.
   int bit_offset_[kMaxBuffers - 1];
+
+  bool is_dictionary_type() const {

Review Comment:
   Does the `KeyColumnArray` even support storing dictionary encoded data?  I didn't think it did but I could be remembering incorrectly.



##########
cpp/src/arrow/compute/light_array.h:
##########
@@ -183,6 +183,36 @@ class ARROW_EXPORT KeyColumnArray {
   // Starting bit offset within the first byte (between 0 and 7)
   // to be used when accessing buffers that store bit vectors.
   int bit_offset_[kMaxBuffers - 1];
+
+  bool is_dictionary_type() const {
+    return metadata_.is_fixed_length && metadata_.fixed_length != 0 &&
+           !metadata_.is_null_type;
+  }
+
+  bool is_bool_type() const {
+    return metadata_.is_fixed_length && metadata_.fixed_length == 0 &&
+           !metadata_.is_null_type;
+  }
+
+  bool is_other_fixed_width_types() const {
+    return metadata_.is_fixed_length && metadata_.fixed_length != 0 &&
+           !metadata_.is_null_type;
+  }

Review Comment:
   Isn't this checking the exact same things as `is_dictionary_type`?



##########
cpp/src/arrow/compute/light_array.h:
##########
@@ -183,6 +183,36 @@ class ARROW_EXPORT KeyColumnArray {
   // Starting bit offset within the first byte (between 0 and 7)
   // to be used when accessing buffers that store bit vectors.
   int bit_offset_[kMaxBuffers - 1];
+
+  bool is_dictionary_type() const {
+    return metadata_.is_fixed_length && metadata_.fixed_length != 0 &&
+           !metadata_.is_null_type;

Review Comment:
   I'm not sure these conditions imply dictionary.  This could be any regular fixed-width column (e.g. int32) could it?



##########
cpp/src/arrow/compute/light_array.cc:
##########
@@ -80,27 +80,32 @@ KeyColumnArray KeyColumnArray::Slice(int64_t offset, int64_t length) const {
   KeyColumnArray sliced;
   sliced.metadata_ = metadata_;
   sliced.length_ = length;
-  uint32_t fixed_size =
-      !metadata_.is_fixed_length ? sizeof(uint32_t) : metadata_.fixed_length;
+  uint32_t fixed_size = metadata_.fixed_length;
 
   sliced.buffers_[0] =
       buffers_[0] ? buffers_[0] + (bit_offset_[0] + offset) / 8 : nullptr;
   sliced.mutable_buffers_[0] =
       mutable_buffers_[0] ? mutable_buffers_[0] + (bit_offset_[0] + offset) / 8 : nullptr;
   sliced.bit_offset_[0] = (bit_offset_[0] + offset) % 8;
 
-  if (fixed_size == 0 && !metadata_.is_null_type) {
+  if (metadata_.fixed_length == 0 && !metadata_.is_null_type) {
+    ARROW_DCHECK(is_bool_type() || is_other_fixed_width_types())
+        << "Expected BOOL type or OTHER FIXED WIDTH type but got a different type.";
     sliced.buffers_[1] =
         buffers_[1] ? buffers_[1] + (bit_offset_[1] + offset) / 8 : nullptr;
     sliced.mutable_buffers_[1] = mutable_buffers_[1]
                                      ? mutable_buffers_[1] + (bit_offset_[1] + offset) / 8
                                      : nullptr;
     sliced.bit_offset_[1] = (bit_offset_[1] + offset) % 8;
-  } else {
+  } else if (metadata_.fixed_length > 0) {
+    ARROW_DCHECK(is_dictionary_type() || is_binary_type() || is_large_binary_type())

Review Comment:
   I am confused.  If the type is `int32` then it seems we would reach this branch.  `metadata_.fixed_length` would be `4` and `metadata_.is_null_type` would be `false` and `metadata._is_fixed_length` would be `true`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Fix Offset Size Calculation for Slicing Large String and Binary Types in Hash Join [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #38147:
URL: https://github.com/apache/arrow/pull/38147#issuecomment-1767250275

   After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit fb2617833200d9c90b6d79a85ae9e840b41938da.
   
   There were no benchmark performance regressions. 🎉
   
   The [full Conbench report](https://github.com/apache/arrow/runs/17798242760) has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on PR #38147:
URL: https://github.com/apache/arrow/pull/38147#issuecomment-1760187899

   @pitrou I have enhanced the hash join tests as you mentioned, by modifying the existing unit test code. 
   
   I confirmed that the random generator does not create data types corresponding to `uint64_t`, which is considered 'Large'. Accordingly, I have updated it to generate LargeString type data.
   
   This appears to adequately generate data for offset calculations. An error occurs when testing based on the previously written `Slice` function code.
   
   I'm still contemplating for the `Slice` function and need more time to consider the best approach.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Fix Offset Size Calculation for Slicing Large String and Binary Types in Hash Join [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #38147:
URL: https://github.com/apache/arrow/pull/38147#issuecomment-1764029516

   Thanks a lot for this fix @llama90 !
   
   @raulcd This should probably be a candidate for 14.0.0.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #38147:
URL: https://github.com/apache/arrow/pull/38147#discussion_r1356752774


##########
cpp/src/arrow/compute/light_array_test.cc:
##########
@@ -226,6 +226,120 @@ TEST(KeyColumnArray, SliceBool) {
   }
 }
 
+TEST(KeyColumnArray, SliceBinary) {
+  auto type = binary();
+  // Sample binary data using ArrayFromJSON
+  std::shared_ptr<Array> array =
+      ArrayFromJSON(type, R"(["Hello", "World", "Slice", "Binary", "Test"])");
+
+  // Create KeyColumnArray from the ArrayData
+  KeyColumnArray kc_array =
+      ColumnArrayFromArrayData(array->data(), 0, array->length()).ValueOrDie();
+
+  // Define test cases
+  struct {
+    int offset;
+    int length;
+    std::vector<std::string> expected;
+  } testCases[] = {
+      {0, 1, {"Hello"}},
+      {1, 1, {"World"}},
+      {2, 1, {"Slice"}},
+      {3, 1, {"Binary"}},
+      {4, 1, {"Test"}},
+      {0, 2, {"Hello", "World"}},
+      {1, 2, {"World", "Slice"}},
+      {2, 2, {"Slice", "Binary"}},
+      {3, 2, {"Binary", "Test"}},
+      {0, 3, {"Hello", "World", "Slice"}},
+      {1, 3, {"World", "Slice", "Binary"}},
+      {2, 3, {"Slice", "Binary", "Test"}},
+      {0, 4, {"Hello", "World", "Slice", "Binary"}},
+      {1, 4, {"World", "Slice", "Binary", "Test"}},
+      {0, 5, {"Hello", "World", "Slice", "Binary", "Test"}},
+  };
+
+  for (const auto& testCase : testCases) {
+    ARROW_SCOPED_TRACE("Offset: ", testCase.offset, " Length: ", testCase.length);
+    KeyColumnArray sliced = kc_array.Slice(testCase.offset, testCase.length);
+
+    // Extract binary data from the sliced KeyColumnArray
+    std::vector<std::string> sliced_data;
+
+    const auto* offset_data = reinterpret_cast<const int32_t*>(sliced.data(1));
+    const auto* string_data = reinterpret_cast<const char*>(sliced.data(2));
+
+    for (auto i = 0; i < testCase.length; ++i) {
+      auto start = offset_data[i];
+      auto end = offset_data[i + 1];
+      sliced_data.push_back(std::string(string_data + start, string_data + end));
+    }
+
+    // Compare the sliced values to the expected string
+    ASSERT_EQ(testCase.expected, sliced_data);
+  }
+}
+
+TEST(KeyColumnArray, SliceLargeBinary) {
+  auto type = large_binary();
+  // Sample binary data using ArrayFromJSON
+  std::shared_ptr<Array> array =
+      ArrayFromJSON(type, R"(["Hello", "World", "Slice", "Large", "Binary", "Test"])");
+
+  // Create KeyColumnArray from the ArrayData
+  KeyColumnArray kc_array =
+      ColumnArrayFromArrayData(array->data(), 0, array->length()).ValueOrDie();
+
+  // Define test cases
+  struct {
+    int offset;
+    int length;
+    std::vector<std::string> expected;
+  } testCases[] = {
+      {0, 1, {"Hello"}},
+      {1, 1, {"World"}},
+      {2, 1, {"Slice"}},
+      {3, 1, {"Large"}},
+      {4, 1, {"Binary"}},
+      {5, 1, {"Test"}},
+      {0, 2, {"Hello", "World"}},
+      {1, 2, {"World", "Slice"}},
+      {2, 2, {"Slice", "Large"}},
+      {3, 2, {"Large", "Binary"}},
+      {4, 2, {"Binary", "Test"}},
+      {0, 3, {"Hello", "World", "Slice"}},
+      {1, 3, {"World", "Slice", "Large"}},
+      {2, 3, {"Slice", "Large", "Binary"}},
+      {3, 3, {"Large", "Binary", "Test"}},
+      {0, 4, {"Hello", "World", "Slice", "Large"}},
+      {1, 4, {"World", "Slice", "Large", "Binary"}},
+      {2, 4, {"Slice", "Large", "Binary", "Test"}},
+      {0, 5, {"Hello", "World", "Slice", "Large", "Binary"}},
+      {1, 5, {"World", "Slice", "Large", "Binary", "Test"}},
+      {0, 6, {"Hello", "World", "Slice", "Large", "Binary", "Test"}},
+  };

Review Comment:
   I've tried refactoring the Slice test function. Is this refactoring what you intended?
   
   I'm also making progress on two other items, the join test and the assertion statements for the Slice function.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #38147:
URL: https://github.com/apache/arrow/pull/38147#discussion_r1358489977


##########
cpp/src/arrow/compute/light_array.h:
##########
@@ -183,6 +183,36 @@ class ARROW_EXPORT KeyColumnArray {
   // Starting bit offset within the first byte (between 0 and 7)
   // to be used when accessing buffers that store bit vectors.
   int bit_offset_[kMaxBuffers - 1];
+
+  bool is_dictionary_type() const {

Review Comment:
   Maybe, you are right.
   
   My understanding is that the code is added, but probably unsupported or not being tested properly. 
   
   So, I took a hint from what you said, I removed the code for the dictionary type within `KeyColumnArray` and ran the entire test and found that none of the tests covered that code :O 
   (If that's okay, can we add a **new issue** for `KeyColumns` to have complex values like dictionary types?)
   
   Looking at the data I was randomly generating in the `hash_join_node_test.cc` file, I was only generating data for the following types.
   
   In the code below, I have added support for `LargeString` type.
   
   https://github.com/apache/arrow/blob/e9aac8a4ce4b8833667efe52957fb0881a488c4b/cpp/src/arrow/acero/hash_join_node_test.cc#L284-L291



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #38147:
URL: https://github.com/apache/arrow/pull/38147#discussion_r1358500583


##########
cpp/src/arrow/compute/light_array.cc:
##########
@@ -80,27 +80,32 @@ KeyColumnArray KeyColumnArray::Slice(int64_t offset, int64_t length) const {
   KeyColumnArray sliced;
   sliced.metadata_ = metadata_;
   sliced.length_ = length;
-  uint32_t fixed_size =
-      !metadata_.is_fixed_length ? sizeof(uint32_t) : metadata_.fixed_length;
+  uint32_t fixed_size = metadata_.fixed_length;
 
   sliced.buffers_[0] =
       buffers_[0] ? buffers_[0] + (bit_offset_[0] + offset) / 8 : nullptr;
   sliced.mutable_buffers_[0] =
       mutable_buffers_[0] ? mutable_buffers_[0] + (bit_offset_[0] + offset) / 8 : nullptr;
   sliced.bit_offset_[0] = (bit_offset_[0] + offset) % 8;
 
-  if (fixed_size == 0 && !metadata_.is_null_type) {
+  if (metadata_.fixed_length == 0 && !metadata_.is_null_type) {
+    ARROW_DCHECK(is_bool_type() || is_other_fixed_width_types())

Review Comment:
   You are right. The only thing that qualifies is the Bool type.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on PR #38147:
URL: https://github.com/apache/arrow/pull/38147#issuecomment-1763032429

   > As a beginner, I feel both overwhelmed and excited to handle an issue that requires a complex understanding. While I am aware of my limitations, I am committed to giving my best.
   
   The hash join code is complex and quite different from the rest of the arrow code base.  It unfortunately reinvents things that we have elsewhere.  Don't worry about feeling overwhelmed here, I think many of us are.  Do you have any long term goals for this feature?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Fix Offset Size Calculation for Slicing Large String and Binary Types in Hash Join [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou merged PR #38147:
URL: https://github.com/apache/arrow/pull/38147


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on PR #38147:
URL: https://github.com/apache/arrow/pull/38147#issuecomment-1757780551

   > @llama90 could you please merge/rebase this with the latest changes on the main branch? That should fix the remaining CI failure.
   
   I have rebased and force pushed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on PR #38147:
URL: https://github.com/apache/arrow/pull/38147#issuecomment-1757778227

   > FTR, I still need to take a look at the fix and see if we can make things more maintainable and more understandable in the future.
   
   If possible, could you provide specific guidelines?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38147:
URL: https://github.com/apache/arrow/pull/38147#discussion_r1356415958


##########
cpp/src/arrow/compute/light_array_test.cc:
##########
@@ -226,6 +226,120 @@ TEST(KeyColumnArray, SliceBool) {
   }
 }
 
+TEST(KeyColumnArray, SliceBinary) {
+  auto type = binary();
+  // Sample binary data using ArrayFromJSON
+  std::shared_ptr<Array> array =
+      ArrayFromJSON(type, R"(["Hello", "World", "Slice", "Binary", "Test"])");
+
+  // Create KeyColumnArray from the ArrayData
+  KeyColumnArray kc_array =
+      ColumnArrayFromArrayData(array->data(), 0, array->length()).ValueOrDie();
+
+  // Define test cases
+  struct {
+    int offset;
+    int length;
+    std::vector<std::string> expected;
+  } testCases[] = {
+      {0, 1, {"Hello"}},
+      {1, 1, {"World"}},
+      {2, 1, {"Slice"}},
+      {3, 1, {"Binary"}},
+      {4, 1, {"Test"}},
+      {0, 2, {"Hello", "World"}},
+      {1, 2, {"World", "Slice"}},
+      {2, 2, {"Slice", "Binary"}},
+      {3, 2, {"Binary", "Test"}},
+      {0, 3, {"Hello", "World", "Slice"}},
+      {1, 3, {"World", "Slice", "Binary"}},
+      {2, 3, {"Slice", "Binary", "Test"}},
+      {0, 4, {"Hello", "World", "Slice", "Binary"}},
+      {1, 4, {"World", "Slice", "Binary", "Test"}},
+      {0, 5, {"Hello", "World", "Slice", "Binary", "Test"}},
+  };
+
+  for (const auto& testCase : testCases) {
+    ARROW_SCOPED_TRACE("Offset: ", testCase.offset, " Length: ", testCase.length);
+    KeyColumnArray sliced = kc_array.Slice(testCase.offset, testCase.length);
+
+    // Extract binary data from the sliced KeyColumnArray
+    std::vector<std::string> sliced_data;
+
+    const auto* offset_data = reinterpret_cast<const int32_t*>(sliced.data(1));
+    const auto* string_data = reinterpret_cast<const char*>(sliced.data(2));
+
+    for (auto i = 0; i < testCase.length; ++i) {
+      auto start = offset_data[i];
+      auto end = offset_data[i + 1];
+      sliced_data.push_back(std::string(string_data + start, string_data + end));
+    }
+
+    // Compare the sliced values to the expected string
+    ASSERT_EQ(testCase.expected, sliced_data);
+  }
+}
+
+TEST(KeyColumnArray, SliceLargeBinary) {
+  auto type = large_binary();
+  // Sample binary data using ArrayFromJSON
+  std::shared_ptr<Array> array =
+      ArrayFromJSON(type, R"(["Hello", "World", "Slice", "Large", "Binary", "Test"])");
+
+  // Create KeyColumnArray from the ArrayData
+  KeyColumnArray kc_array =
+      ColumnArrayFromArrayData(array->data(), 0, array->length()).ValueOrDie();
+
+  // Define test cases
+  struct {
+    int offset;
+    int length;
+    std::vector<std::string> expected;
+  } testCases[] = {
+      {0, 1, {"Hello"}},
+      {1, 1, {"World"}},
+      {2, 1, {"Slice"}},
+      {3, 1, {"Large"}},
+      {4, 1, {"Binary"}},
+      {5, 1, {"Test"}},
+      {0, 2, {"Hello", "World"}},
+      {1, 2, {"World", "Slice"}},
+      {2, 2, {"Slice", "Large"}},
+      {3, 2, {"Large", "Binary"}},
+      {4, 2, {"Binary", "Test"}},
+      {0, 3, {"Hello", "World", "Slice"}},
+      {1, 3, {"World", "Slice", "Large"}},
+      {2, 3, {"Slice", "Large", "Binary"}},
+      {3, 3, {"Large", "Binary", "Test"}},
+      {0, 4, {"Hello", "World", "Slice", "Large"}},
+      {1, 4, {"World", "Slice", "Large", "Binary"}},
+      {2, 4, {"Slice", "Large", "Binary", "Test"}},
+      {0, 5, {"Hello", "World", "Slice", "Large", "Binary"}},
+      {1, 5, {"World", "Slice", "Large", "Binary", "Test"}},
+      {0, 6, {"Hello", "World", "Slice", "Large", "Binary", "Test"}},
+  };

Review Comment:
   The two test cases (large and non-large) are almost identical except for the slight and non-useful variation in example data. I think it should be easy to refactor them into a common codebase so as to share most the code here, can you do that?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38147:
URL: https://github.com/apache/arrow/pull/38147#discussion_r1357258668


##########
cpp/src/arrow/compute/light_array_test.cc:
##########
@@ -226,6 +226,120 @@ TEST(KeyColumnArray, SliceBool) {
   }
 }
 
+TEST(KeyColumnArray, SliceBinary) {
+  auto type = binary();
+  // Sample binary data using ArrayFromJSON
+  std::shared_ptr<Array> array =
+      ArrayFromJSON(type, R"(["Hello", "World", "Slice", "Binary", "Test"])");
+
+  // Create KeyColumnArray from the ArrayData
+  KeyColumnArray kc_array =
+      ColumnArrayFromArrayData(array->data(), 0, array->length()).ValueOrDie();
+
+  // Define test cases
+  struct {
+    int offset;
+    int length;
+    std::vector<std::string> expected;
+  } testCases[] = {
+      {0, 1, {"Hello"}},
+      {1, 1, {"World"}},
+      {2, 1, {"Slice"}},
+      {3, 1, {"Binary"}},
+      {4, 1, {"Test"}},
+      {0, 2, {"Hello", "World"}},
+      {1, 2, {"World", "Slice"}},
+      {2, 2, {"Slice", "Binary"}},
+      {3, 2, {"Binary", "Test"}},
+      {0, 3, {"Hello", "World", "Slice"}},
+      {1, 3, {"World", "Slice", "Binary"}},
+      {2, 3, {"Slice", "Binary", "Test"}},
+      {0, 4, {"Hello", "World", "Slice", "Binary"}},
+      {1, 4, {"World", "Slice", "Binary", "Test"}},
+      {0, 5, {"Hello", "World", "Slice", "Binary", "Test"}},
+  };
+
+  for (const auto& testCase : testCases) {
+    ARROW_SCOPED_TRACE("Offset: ", testCase.offset, " Length: ", testCase.length);
+    KeyColumnArray sliced = kc_array.Slice(testCase.offset, testCase.length);
+
+    // Extract binary data from the sliced KeyColumnArray
+    std::vector<std::string> sliced_data;
+
+    const auto* offset_data = reinterpret_cast<const int32_t*>(sliced.data(1));
+    const auto* string_data = reinterpret_cast<const char*>(sliced.data(2));
+
+    for (auto i = 0; i < testCase.length; ++i) {
+      auto start = offset_data[i];
+      auto end = offset_data[i + 1];
+      sliced_data.push_back(std::string(string_data + start, string_data + end));
+    }
+
+    // Compare the sliced values to the expected string
+    ASSERT_EQ(testCase.expected, sliced_data);
+  }
+}
+
+TEST(KeyColumnArray, SliceLargeBinary) {
+  auto type = large_binary();
+  // Sample binary data using ArrayFromJSON
+  std::shared_ptr<Array> array =
+      ArrayFromJSON(type, R"(["Hello", "World", "Slice", "Large", "Binary", "Test"])");
+
+  // Create KeyColumnArray from the ArrayData
+  KeyColumnArray kc_array =
+      ColumnArrayFromArrayData(array->data(), 0, array->length()).ValueOrDie();
+
+  // Define test cases
+  struct {
+    int offset;
+    int length;
+    std::vector<std::string> expected;
+  } testCases[] = {
+      {0, 1, {"Hello"}},
+      {1, 1, {"World"}},
+      {2, 1, {"Slice"}},
+      {3, 1, {"Large"}},
+      {4, 1, {"Binary"}},
+      {5, 1, {"Test"}},
+      {0, 2, {"Hello", "World"}},
+      {1, 2, {"World", "Slice"}},
+      {2, 2, {"Slice", "Large"}},
+      {3, 2, {"Large", "Binary"}},
+      {4, 2, {"Binary", "Test"}},
+      {0, 3, {"Hello", "World", "Slice"}},
+      {1, 3, {"World", "Slice", "Large"}},
+      {2, 3, {"Slice", "Large", "Binary"}},
+      {3, 3, {"Large", "Binary", "Test"}},
+      {0, 4, {"Hello", "World", "Slice", "Large"}},
+      {1, 4, {"World", "Slice", "Large", "Binary"}},
+      {2, 4, {"Slice", "Large", "Binary", "Test"}},
+      {0, 5, {"Hello", "World", "Slice", "Large", "Binary"}},
+      {1, 5, {"World", "Slice", "Large", "Binary", "Test"}},
+      {0, 6, {"Hello", "World", "Slice", "Large", "Binary", "Test"}},
+  };

Review Comment:
   > I've tried refactoring the Slice test function. Is this refactoring what you intended?
   
   Yes, perfect. Thank you!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on PR #38147:
URL: https://github.com/apache/arrow/pull/38147#issuecomment-1760150156

   Apologies everyone, I performed a rebase on my personal repository's main branch and after rebasing the development branch, I pushed the changes. 
   
   Unfortunately, commits that were already merged into main are now showing up in the history again. 
   
   This was not intentional, and I'm looking for ways to correct this. If anyone has insights or suggestions on how to fix this, your assistance would be greatly appreciated.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #38147:
URL: https://github.com/apache/arrow/pull/38147#discussion_r1358499371


##########
cpp/src/arrow/compute/light_array.h:
##########
@@ -183,6 +183,36 @@ class ARROW_EXPORT KeyColumnArray {
   // Starting bit offset within the first byte (between 0 and 7)
   // to be used when accessing buffers that store bit vectors.
   int bit_offset_[kMaxBuffers - 1];
+
+  bool is_dictionary_type() const {
+    return metadata_.is_fixed_length && metadata_.fixed_length != 0 &&
+           !metadata_.is_null_type;

Review Comment:
   You are correct. I overlooked that.



##########
cpp/src/arrow/compute/light_array.h:
##########
@@ -183,6 +183,36 @@ class ARROW_EXPORT KeyColumnArray {
   // Starting bit offset within the first byte (between 0 and 7)
   // to be used when accessing buffers that store bit vectors.
   int bit_offset_[kMaxBuffers - 1];
+
+  bool is_dictionary_type() const {
+    return metadata_.is_fixed_length && metadata_.fixed_length != 0 &&
+           !metadata_.is_null_type;

Review Comment:
   You are correct. I overlooked that.



##########
cpp/src/arrow/compute/light_array.h:
##########
@@ -183,6 +183,36 @@ class ARROW_EXPORT KeyColumnArray {
   // Starting bit offset within the first byte (between 0 and 7)
   // to be used when accessing buffers that store bit vectors.
   int bit_offset_[kMaxBuffers - 1];
+
+  bool is_dictionary_type() const {
+    return metadata_.is_fixed_length && metadata_.fixed_length != 0 &&
+           !metadata_.is_null_type;
+  }
+
+  bool is_bool_type() const {
+    return metadata_.is_fixed_length && metadata_.fixed_length == 0 &&
+           !metadata_.is_null_type;
+  }
+
+  bool is_other_fixed_width_types() const {
+    return metadata_.is_fixed_length && metadata_.fixed_length != 0 &&
+           !metadata_.is_null_type;
+  }

Review Comment:
   You are correct. I overlooked that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #38147:
URL: https://github.com/apache/arrow/pull/38147#discussion_r1358498562


##########
cpp/src/arrow/compute/light_array.h:
##########
@@ -183,6 +183,36 @@ class ARROW_EXPORT KeyColumnArray {
   // Starting bit offset within the first byte (between 0 and 7)
   // to be used when accessing buffers that store bit vectors.
   int bit_offset_[kMaxBuffers - 1];
+
+  bool is_dictionary_type() const {
+    return metadata_.is_fixed_length && metadata_.fixed_length != 0 &&
+           !metadata_.is_null_type;

Review Comment:
   Answer excluding dictionary types. 
   
   Unless bit types are covered by a custom type (with the exception of bool), it seems that data of type byte would be included in the condition to calculate the offset.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on code in PR #38147:
URL: https://github.com/apache/arrow/pull/38147#discussion_r1352878832


##########
cpp/src/arrow/compute/light_array.cc:
##########
@@ -80,16 +80,15 @@ KeyColumnArray KeyColumnArray::Slice(int64_t offset, int64_t length) const {
   KeyColumnArray sliced;
   sliced.metadata_ = metadata_;
   sliced.length_ = length;
-  uint32_t fixed_size =
-      !metadata_.is_fixed_length ? sizeof(uint32_t) : metadata_.fixed_length;
+  uint32_t fixed_size = metadata_.fixed_length;
 
   sliced.buffers_[0] =
       buffers_[0] ? buffers_[0] + (bit_offset_[0] + offset) / 8 : nullptr;
   sliced.mutable_buffers_[0] =
       mutable_buffers_[0] ? mutable_buffers_[0] + (bit_offset_[0] + offset) / 8 : nullptr;
   sliced.bit_offset_[0] = (bit_offset_[0] + offset) % 8;
 
-  if (fixed_size == 0 && !metadata_.is_null_type) {
+  if (metadata_.fixed_length == 0 && !metadata_.is_null_type) {
     sliced.buffers_[1] =
         buffers_[1] ? buffers_[1] + (bit_offset_[1] + offset) / 8 : nullptr;

Review Comment:
   The code I modified is handled in the section below (`else` statement), reflecting the type size as `fixed_size`. 
   
   Upon reviewing the implementation of the `ColumnMetadataFromDataType` function, it is understood that the type applicable to the mentioned case is `BOOL` and `is_fixed_width()`. (In the case of `DICTIONARY`, it ensures that ARROW_DCHECK(bit_width % 8 == 0) is checked to guarantee that it is a multiple of 8.)
   
   https://github.com/apache/arrow/blob/907723663e6e20033d0c2fef4eb0f880daacdb09/cpp/src/arrow/compute/light_array.cc#L111-L145
   
   * [Data Types](https://arrow.apache.org/docs/cpp/api/datatype.html)
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Support uint64_t Types in Slice Function to Address Specific Inner Join Bug [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38147:
URL: https://github.com/apache/arrow/pull/38147#discussion_r1350379177


##########
cpp/src/arrow/compute/light_array.cc:
##########
@@ -80,26 +80,24 @@ KeyColumnArray KeyColumnArray::Slice(int64_t offset, int64_t length) const {
   KeyColumnArray sliced;
   sliced.metadata_ = metadata_;
   sliced.length_ = length;
-  uint32_t fixed_size =
-      !metadata_.is_fixed_length ? sizeof(uint32_t) : metadata_.fixed_length;
 
   sliced.buffers_[0] =
       buffers_[0] ? buffers_[0] + (bit_offset_[0] + offset) / 8 : nullptr;
   sliced.mutable_buffers_[0] =
       mutable_buffers_[0] ? mutable_buffers_[0] + (bit_offset_[0] + offset) / 8 : nullptr;
   sliced.bit_offset_[0] = (bit_offset_[0] + offset) % 8;
 
-  if (fixed_size == 0 && !metadata_.is_null_type) {
+  if (metadata_.fixed_length == 0 && !metadata_.is_null_type) {
     sliced.buffers_[1] =
         buffers_[1] ? buffers_[1] + (bit_offset_[1] + offset) / 8 : nullptr;
     sliced.mutable_buffers_[1] = mutable_buffers_[1]
                                      ? mutable_buffers_[1] + (bit_offset_[1] + offset) / 8
                                      : nullptr;
     sliced.bit_offset_[1] = (bit_offset_[1] + offset) % 8;
   } else {
-    sliced.buffers_[1] = buffers_[1] ? buffers_[1] + offset * fixed_size : nullptr;
+    sliced.buffers_[1] = buffers_[1] ? buffers_[1] + offset * metadata_.fixed_length : nullptr;

Review Comment:
   So, IIUC, `uint32_t * int64_t` would result in a `uint32_t` result?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38074: [C++] Fix Offset Size Calculation for Slicing Large String and Binary Types in Hash Join [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on PR #38147:
URL: https://github.com/apache/arrow/pull/38147#issuecomment-1763059449

   @pitrou Hello, I have revised and updated the PR title and content.
   
   @westonpace It seems like the issues you mentioned include the following items:
   
   * Support for Dictionary types in key columns during Hash Join
   * Handling of Dictionary types in Hash Join (Swiss)
   * Join support when some columns contain lists
   
   All are related to joins and seem to be interesting areas. I am also interested in the issues you've highlighted and would like to attempt improvements when I have some spare time.
   
   I feel proud to have made a meaningful contribution.
   
   @pitrou @westonpace @ianmcook Thank you again for your review, and I hope to engage with you more frequently with new contributions.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org