You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/11/15 05:58:22 UTC

[GitHub] [druid] LakshSingla opened a new pull request, #13366: Convert errors based on implicit type conversion in multi value arrays to parse exception in MSQ

LakshSingla opened a new pull request, #13366:
URL: https://github.com/apache/druid/pull/13366

   ### Description
   
   Ingesting a multi-value column which was originally expecting a string array but received an object array throws a cryptic error message due to a failed type conversion that happens in the FrameWriters. This PR aims to convert those catch those error messages and throw them as ParseExceptions.
   
   <hr>
   
   <hr>
   
   This PR has:
   
   - [ ] been self-reviewed.
      - [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
   - [ ] added documentation for new or modified features or behaviors.
   - [ ] a release note entry in the PR description.
   - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
   - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [ ] added integration tests.
   - [ ] been tested in a test Druid cluster.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe merged pull request #13366: Convert errors based on implicit type conversion in multi value arrays to parse exception in MSQ

Posted by GitBox <gi...@apache.org>.
cryptoe merged PR #13366:
URL: https://github.com/apache/druid/pull/13366


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] LakshSingla commented on a diff in pull request #13366: Convert errors based on implicit type conversion in multi value arrays to parse exception in MSQ

Posted by GitBox <gi...@apache.org>.
LakshSingla commented on code in PR #13366:
URL: https://github.com/apache/druid/pull/13366#discussion_r1030038121


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/scan/ScanQueryFrameProcessor.java:
##########
@@ -223,27 +224,32 @@ protected ReturnOrAwait<Long> runWithInputChannel(
     }
   }
 
-  private void populateFrameWriterAndFlushIfNeeded() throws IOException
+  private void populateFrameWriterAndFlushIfNeeded()
   {
     createFrameWriterIfNeeded();
 
-    while (!cursor.isDone()) {
-      if (!frameWriter.addSelection()) {
-        if (frameWriter.getNumRows() > 0) {
-          final long numRowsWritten = flushFrameWriter();
+    try {
+      while (!cursor.isDone()) {
+        if (!frameWriter.addSelection()) {
+          if (frameWriter.getNumRows() > 0) {
+            final long numRowsWritten = flushFrameWriter();
 
-          if (runningCountForLimit != null) {
-            runningCountForLimit.addAndGet(numRowsWritten);
-          }
+            if (runningCountForLimit != null) {
+              runningCountForLimit.addAndGet(numRowsWritten);
+            }
 
-          return;
-        } else {
-          throw new FrameRowTooLargeException(currentAllocatorCapacity);
+            return;
+          } else {
+            throw new FrameRowTooLargeException(currentAllocatorCapacity);
+          }
         }
-      }
 
-      cursor.advance();
-      partitionBoostVirtualColumn.setValue(partitionBoostVirtualColumn.getValue() + 1);
+        cursor.advance();
+        partitionBoostVirtualColumn.setValue(partitionBoostVirtualColumn.getValue() + 1);
+      }
+    }
+    catch (Exception e) {

Review Comment:
   Thanks for catching this. I have reduced the surface of the try..catch block so as to not obstruct any other exceptions



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #13366: Convert errors based on implicit type conversion in multi value arrays to parse exception in MSQ

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #13366:
URL: https://github.com/apache/druid/pull/13366#discussion_r1029256663


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/scan/ScanQueryFrameProcessor.java:
##########
@@ -223,27 +224,32 @@ protected ReturnOrAwait<Long> runWithInputChannel(
     }
   }
 
-  private void populateFrameWriterAndFlushIfNeeded() throws IOException
+  private void populateFrameWriterAndFlushIfNeeded()
   {
     createFrameWriterIfNeeded();
 
-    while (!cursor.isDone()) {
-      if (!frameWriter.addSelection()) {
-        if (frameWriter.getNumRows() > 0) {
-          final long numRowsWritten = flushFrameWriter();
+    try {
+      while (!cursor.isDone()) {
+        if (!frameWriter.addSelection()) {
+          if (frameWriter.getNumRows() > 0) {
+            final long numRowsWritten = flushFrameWriter();
 
-          if (runningCountForLimit != null) {
-            runningCountForLimit.addAndGet(numRowsWritten);
-          }
+            if (runningCountForLimit != null) {
+              runningCountForLimit.addAndGet(numRowsWritten);
+            }
 
-          return;
-        } else {
-          throw new FrameRowTooLargeException(currentAllocatorCapacity);
+            return;
+          } else {
+            throw new FrameRowTooLargeException(currentAllocatorCapacity);
+          }
         }
-      }
 
-      cursor.advance();
-      partitionBoostVirtualColumn.setValue(partitionBoostVirtualColumn.getValue() + 1);
+        cursor.advance();
+        partitionBoostVirtualColumn.setValue(partitionBoostVirtualColumn.getValue() + 1);
+      }
+    }
+    catch (Exception e) {

Review Comment:
   Woudn't FrameRowTooLargeException be caught here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] LakshSingla commented on a diff in pull request #13366: Convert errors based on implicit type conversion in multi value arrays to parse exception in MSQ

Posted by GitBox <gi...@apache.org>.
LakshSingla commented on code in PR #13366:
URL: https://github.com/apache/druid/pull/13366#discussion_r1034323954


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/scan/ScanQueryFrameProcessor.java:
##########
@@ -228,7 +229,14 @@ private void populateFrameWriterAndFlushIfNeeded() throws IOException
     createFrameWriterIfNeeded();
 
     while (!cursor.isDone()) {
-      if (!frameWriter.addSelection()) {
+      boolean addedSelection;

Review Comment:
   Updated the test case and made the change. Instead of the individual processors, I have added the handling in the individual frame writers so that we won't have to add a block in every processor. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #13366: Convert errors based on implicit type conversion in multi value arrays to parse exception in MSQ

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #13366:
URL: https://github.com/apache/druid/pull/13366#discussion_r1033152602


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/scan/ScanQueryFrameProcessor.java:
##########
@@ -228,7 +229,14 @@ private void populateFrameWriterAndFlushIfNeeded() throws IOException
     createFrameWriterIfNeeded();
 
     while (!cursor.isDone()) {
-      if (!frameWriter.addSelection()) {
+      boolean addedSelection;

Review Comment:
   We should also make this change in all the leaf processors. 
   So it can be the scan processor and the `GroupByPreShuffleFrameProcessor`. So this change might be required there as well. 
   Or you can send the common code to the superclass `BaseLeafFrameProcessor`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org