You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "clintropolis (via GitHub)" <gi...@apache.org> on 2023/01/27 06:02:27 UTC

[GitHub] [druid] clintropolis opened a new pull request, #13714: fix nested column handling of null and "null"

clintropolis opened a new pull request, #13714:
URL: https://github.com/apache/druid/pull/13714

   ### Description
   Fixes an issue with nested columns that can occur when both actual `null` and the string `"null"` are present in any nested path that results in the `null` values incorrectly becoming associated with the `"null"` values.
   
   The bug was caused by a usage of `String.valueOf` in `StringFieldColumnWriter` that was not checking for null values when writing out the column. This still worked by dumb luck because the fastutils 2Int maps that were backing the globalId lookup when writing out segments had a default value of 0, which happens to be `null` global id, so even though `"null"` wasn't present in the global dictionary it ended up with the correct id. However, if `"null"` was present, `null` would incorrectly be written out as the `"null"` global id and associated to that value instead.
   
   As a safety measure, I've changed the 2int maps to have a default value of -1, and check that the globalid is in range before writing it to the column to ensure mistakes like this don't happen in the future.
   
   The added test data in `NestedDataColumnSupplierTest` would fail prior to this PR.
   
   Unrelated, this PR also moves some column selector tests that had nothing to do with scan queries out of `NestedDataScanQueryTest` into their own file.
   
   This PR has:
   
   - [x] been self-reviewed.
   - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [x] been tested in a test Druid cluster.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] clintropolis merged pull request #13714: fix nested column handling of null and "null"

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis merged PR #13714:
URL: https://github.com/apache/druid/pull/13714


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] github-code-scanning[bot] commented on a diff in pull request #13714: fix nested column handling of null and "null"

Posted by github-code-scanning.
github-code-scanning[bot] commented on code in PR #13714:
URL: https://github.com/apache/druid/pull/13714#discussion_r1088959516


##########
processing/src/test/java/org/apache/druid/query/NestedDataTestUtils.java:
##########
@@ -223,6 +231,95 @@
     return segments;
   }
 
+  public static List<Segment> createSegmentsWithConcatenatedInput(
+      AggregationTestHelper helper,
+      TemporaryFolder tempFolder,
+      Closer closer,
+      Granularity granularity,
+      boolean rollup,
+      int maxRowCount,
+      int numCopies,
+      int numSegments
+  ) throws Exception
+  {
+    return createSegmentsWithConcatenatedInput(
+        helper,
+        tempFolder,
+        closer,
+        SIMPLE_DATA_FILE,
+        SIMPLE_PARSER_FILE,
+        null,
+        SIMPLE_AGG_FILE,
+        granularity,
+        rollup,
+        maxRowCount,
+        numCopies,
+        numSegments
+    );
+  }
+
+  /**
+   * turn small test data into bigger test data by duplicating itself into a bigger stream
+   */
+  public static List<Segment> createSegmentsWithConcatenatedInput(
+      AggregationTestHelper helper,
+      TemporaryFolder tempFolder,
+      Closer closer,
+      String inputFileName,
+      String parserJsonFileName,
+      String transformSpecJsonFileName,
+      String aggJsonFileName,
+      Granularity granularity,
+      boolean rollup,
+      int maxRowCount,
+      int numCopies,
+      int numSegments
+  ) throws Exception
+  {
+    String parserJson = readFileFromClasspathAsString(parserJsonFileName);
+    String transformSpecJson = transformSpecJsonFileName != null ? readFileFromClasspathAsString(transformSpecJsonFileName) : null;
+    String aggJson = readFileFromClasspathAsString(aggJsonFileName);
+
+    List<File> segmentDirs = Lists.newArrayListWithCapacity(numSegments);
+    for (int i = 0; i < numSegments; i++) {
+      List<InputStream> inputStreams = Lists.newArrayListWithCapacity(numCopies);
+      for (int j = 0; j < numCopies; j++) {
+        inputStreams.add(new FileInputStream(readFileFromClasspath(inputFileName)));

Review Comment:
   ## Potential input resource leak
   
   This FileInputStream is not always closed on method exit.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/4202)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] clintropolis commented on a diff in pull request #13714: fix nested column handling of null and "null"

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #13714:
URL: https://github.com/apache/druid/pull/13714#discussion_r1092765300


##########
processing/src/main/java/org/apache/druid/segment/nested/GlobalDictionaryEncodedFieldColumnWriter.java:
##########
@@ -133,8 +134,15 @@ public void addValue(int row, Object val) throws IOException
       fillNull(row);
     }
     final T value = processValue(val);
-    final int globalId = lookupGlobalId(value);
-    final int localId = localDictionary.add(globalId);
+    final int localId;
+    // null is always 0
+    if (value == null) {
+      localId = localDictionary.add(0);
+    } else {
+      final int globalId = lookupGlobalId(value);
+      Preconditions.checkArgument(globalId >= 0, "Value [%s] is not present in global dictionary", value);
+      localId = localDictionary.add(globalId);
+    }

Review Comment:
   this is the shared function, `lookupGlobalId` has different implementations depending on the type of writer so would need to duplicate there



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] imply-cheddar commented on a diff in pull request #13714: fix nested column handling of null and "null"

Posted by "imply-cheddar (via GitHub)" <gi...@apache.org>.
imply-cheddar commented on code in PR #13714:
URL: https://github.com/apache/druid/pull/13714#discussion_r1092759252


##########
processing/src/main/java/org/apache/druid/segment/nested/GlobalDictionaryEncodedFieldColumnWriter.java:
##########
@@ -133,8 +134,15 @@ public void addValue(int row, Object val) throws IOException
       fillNull(row);
     }
     final T value = processValue(val);
-    final int globalId = lookupGlobalId(value);
-    final int localId = localDictionary.add(globalId);
+    final int localId;
+    // null is always 0
+    if (value == null) {
+      localId = localDictionary.add(0);
+    } else {
+      final int globalId = lookupGlobalId(value);
+      Preconditions.checkArgument(globalId >= 0, "Value [%s] is not present in global dictionary", value);
+      localId = localDictionary.add(globalId);
+    }

Review Comment:
   Why is it not okay to do this check inside of `lookupGlobalId`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] github-code-scanning[bot] commented on a diff in pull request #13714: fix nested column handling of null and "null"

Posted by github-code-scanning.
github-code-scanning[bot] commented on code in PR #13714:
URL: https://github.com/apache/druid/pull/13714#discussion_r1089574049


##########
processing/src/test/java/org/apache/druid/query/NestedDataTestUtils.java:
##########
@@ -146,7 +154,7 @@
   {
     File segmentDir = tempFolder.newFolder();
     File inputFile = readFileFromClasspath(inputFileName);
-    FileInputStream inputDataStream = new FileInputStream(inputFile);
+    FileInputStream inputDataStream = closer.register(new FileInputStream(inputFile));

Review Comment:
   ## Potential input resource leak
   
   This FileInputStream is not always closed on method exit.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/4204)



##########
processing/src/test/java/org/apache/druid/query/NestedDataTestUtils.java:
##########
@@ -191,7 +199,7 @@
   {
     File segmentDir = tempFolder.newFolder();
     File inputFile = readFileFromClasspath(inputFileName);
-    FileInputStream inputDataStream = new FileInputStream(inputFile);
+    FileInputStream inputDataStream = closer.register(new FileInputStream(inputFile));

Review Comment:
   ## Potential input resource leak
   
   This FileInputStream is not always closed on method exit.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/4205)



##########
processing/src/test/java/org/apache/druid/query/NestedDataTestUtils.java:
##########
@@ -223,6 +231,95 @@
     return segments;
   }
 
+  public static List<Segment> createSegmentsWithConcatenatedInput(
+      AggregationTestHelper helper,
+      TemporaryFolder tempFolder,
+      Closer closer,
+      Granularity granularity,
+      boolean rollup,
+      int maxRowCount,
+      int numCopies,
+      int numSegments
+  ) throws Exception
+  {
+    return createSegmentsWithConcatenatedInput(
+        helper,
+        tempFolder,
+        closer,
+        SIMPLE_DATA_FILE,
+        SIMPLE_PARSER_FILE,
+        null,
+        SIMPLE_AGG_FILE,
+        granularity,
+        rollup,
+        maxRowCount,
+        numCopies,
+        numSegments
+    );
+  }
+
+  /**
+   * turn small test data into bigger test data by duplicating itself into a bigger stream
+   */
+  public static List<Segment> createSegmentsWithConcatenatedInput(
+      AggregationTestHelper helper,
+      TemporaryFolder tempFolder,
+      Closer closer,
+      String inputFileName,
+      String parserJsonFileName,
+      String transformSpecJsonFileName,
+      String aggJsonFileName,
+      Granularity granularity,
+      boolean rollup,
+      int maxRowCount,
+      int numCopies,
+      int numSegments
+  ) throws Exception
+  {
+    String parserJson = readFileFromClasspathAsString(parserJsonFileName);
+    String transformSpecJson = transformSpecJsonFileName != null ? readFileFromClasspathAsString(transformSpecJsonFileName) : null;
+    String aggJson = readFileFromClasspathAsString(aggJsonFileName);
+
+    List<File> segmentDirs = Lists.newArrayListWithCapacity(numSegments);
+    for (int i = 0; i < numSegments; i++) {
+      List<InputStream> inputStreams = Lists.newArrayListWithCapacity(numCopies);
+      for (int j = 0; j < numCopies; j++) {
+        inputStreams.add(closer.register(new FileInputStream(readFileFromClasspath(inputFileName))));

Review Comment:
   ## Potential input resource leak
   
   This FileInputStream is not always closed on method exit.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/4206)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org