You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/11/16 10:33:22 UTC

[GitHub] [pinot] vmarchaud opened a new issue, #9812: bug(broker): fail to group by after groovy extraction when upgrading to 0.11

vmarchaud opened a new issue, #9812:
URL: https://github.com/apache/pinot/issues/9812

   Slack ref: https://apache-pinot.slack.com/archives/C011C9JHN7R/p1668446311267249
   
   We just upgraded to 0.11 from 0.10 and some of previously working queries started to fail:
   ```sql
   SELECT groovy('{"returnType":"STRING","isSingleValue":true}', 'arg0.toList().join('';'')', JSONEXTRACTKEY("labels", '$.*')) FROM datasource_609bc534f46c000300b29dcf_REALTIME WHERE (("timestamp" >= 1667833781565)) AND "labels" != '{}' GROUP BY 1 LIMIT 0,100
   ```
   Yield:
   ```
   [
     {
       "errorCode": 200,
       "message": "QueryExecutionError:\njava.lang.NullPointerException"
     },
     {
       "errorCode": 200,
       "message": "QueryExecutionError:\njava.lang.NullPointerException"
     }
   ]
   ```
   In the broker log:
   ```
   2022/11/14 17:06:15.764 INFO [BaseBrokerRequestHandler] [jersey-server-managed-async-executor-76] requestId=64701,table=datasource_609bc534f46c000300b29dcf_REALTIME,timeMs=50,docs=60000/374868,entries=204226/60000,segments(queried/processed/matched/consumingQueried/consumingProcessed/consumingMatched/unavailable):4/4/4/1/1/1/0,consumingFreshnessTimeMs=1668393656448,servers=2/2,groupLimitReached=false,brokerReduceTimeMs=0,exceptions=2,serverStats=(Server=SubmitDelayMs,ResponseDelayMs,ResponseSize,DeserializationTimeMs,RequestSentDelayMs);10.1.1.13_R=0,48,307,0,-1;10.1.1.14_R=0,42,299,0,-1,offlineThreadCpuTimeNs(total/thread/sysActivity/resSer):0/0/0/0,realtimeThreadCpuTimeNs(total/thread/sysActivity/resSer):0/0/0/0,clientIp=unknown,query=SELECT groovy('{"returnType":"STRING","isSingleValue":true}', 'arg0.toList().join('';'')', JSONEXTRACTKEY("labels", '$.*')) FROM datasource_609bc534f46c000300b29dcf_REALTIME WHERE (("timestamp" >= 1667833781565)) AND "labels" != '{}' GROUP BY 1 LIM
 IT 0,100
   ```
   
   However the query works if we remove the `GROUP BY 1` but thats doesn't get the correct result anyway. I suspect that comes from the groovy part but again this was working previously in 0.10


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] snleee commented on issue #9812: bug(broker): fail to group by after groovy extraction when upgrading to 0.11

Posted by GitBox <gi...@apache.org>.
snleee commented on issue #9812:
URL: https://github.com/apache/pinot/issues/9812#issuecomment-1319010990

   @vmarchaud Can you post the server side log to check where we get the NullPointerException? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang closed issue #9812: bug(broker): fail to group by after groovy extraction when upgrading to 0.11

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang closed issue #9812: bug(broker): fail to group by after groovy extraction when upgrading to 0.11
URL: https://github.com/apache/pinot/issues/9812


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] vmarchaud commented on issue #9812: bug(broker): fail to group by after groovy extraction when upgrading to 0.11

Posted by GitBox <gi...@apache.org>.
vmarchaud commented on issue #9812:
URL: https://github.com/apache/pinot/issues/9812#issuecomment-1319124296

   @snleee I only get the following line:
   ```
   nov. 17 19:49:36 pinot-hosts-3 bash[13646]: 2022/11/17 19:49:36.307 ERROR [BaseCombineOperator] [pqw-10] Caught exception while processing query: QueryContext{_tableName='datasource_609bc534f46c000300b29dcf_REALTIME', _subquery=null, _selectExpressions=[distinct(groovy('{"returnType":"STRING","isSingleValue":true}','arg0.toList().join(';')',jsonextractkey(labels,'$.*')))], _aliasList=[null], _filter=(timestamp >= '1667833781565' AND labels != '{}'), _groupByExpressions=null, _havingFilter=null, _orderByExpressions=null, _limit=100, _offset=0, _queryOptions={responseFormat=sql, groupByMode=sql, timeoutMs=10000}, _expressionOverrideHints={}, _explain=false}
   nov. 17 19:49:36 pinot-hosts-3 bash[13646]: java.lang.NullPointerException: null
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] vmarchaud commented on issue #9812: bug(broker): fail to group by after groovy extraction when upgrading to 0.11

Posted by GitBox <gi...@apache.org>.
vmarchaud commented on issue #9812:
URL: https://github.com/apache/pinot/issues/9812#issuecomment-1323886488

   @Jackie-Jiang I'm using the default (https://github.com/apache/pinot/blob/master/pinot-tools/src/main/resources/conf/log4j2.xml) log4j and i can't find any stacktrace in the log (only the errorr message), i guess this isnt normal ?
   
   I'm not very familiar with log4j but can the server specific config log stacktraces ? https://github.com/apache/pinot/blob/master/pinot-tools/src/main/resources/conf/pinot-server-log4j2.xml


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on issue #9812: bug(broker): fail to group by after groovy extraction when upgrading to 0.11

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #9812:
URL: https://github.com/apache/pinot/issues/9812#issuecomment-1322895848

   @vmarchaud Can you check if there is stack trace for this NPE? Usually the stack trace is logged when the exception occur for the first time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] vmarchaud commented on issue #9812: bug(broker): fail to group by after groovy extraction when upgrading to 0.11

Posted by GitBox <gi...@apache.org>.
vmarchaud commented on issue #9812:
URL: https://github.com/apache/pinot/issues/9812#issuecomment-1324750883

   > Is the intention to get the distinct values?
   
   Yes it is.
   
   > It didn't fail in 0.10.0 probably because the GROUP BY is ignored?
   
   Well we do get distinct values with 0.10, if i remove the group by on 0.11 i do get a result but not distinct.
   
   > To get the stack trace, can you try restarting the server and then issue the same query again?
   
   Thanks for the tip to restart it, here's the error:
   ```
   QueryExecutionError:\njava.lang.NullPointerException
   org.apache.pinot.core.operator.transform.function.JsonExtractKeyTransformFunction.transformToStringValuesMV(JsonExtractKeyTransformFunction.java:90)
   org.apache.pinot.core.operator.transform.function.GroovyTransformFunction.transformToStringValuesSV(GroovyTransformFunction.java:430)
   org.apache.pinot.core.operator.docvalsets.TransformBlockValSet.getStringValuesSV(TransformBlockValSet.java:161)
   org.apache.pinot.core.query.distinct.raw.RawStringSingleColumnDistinctOnlyExecutor.process(RawStringSingleColumnDistinctOnlyExecutor.java:44)
   ```
   
   NPE is [here](https://github.com/apache/pinot/blob/release-0.11.0/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/JsonExtractKeyTransformFunction.java#L90) but i'm not sure to understand how this can throw a NPE ...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on issue #9812: bug(broker): fail to group by after groovy extraction when upgrading to 0.11

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #9812:
URL: https://github.com/apache/pinot/issues/9812#issuecomment-1324396500

   Actually this is not an aggregation group-by. Is the intention to get the distinct values? It didn't fail in `0.10.0` probably because the `GROUP BY` is ignored?
   To get the stack trace, can you try restarting the server and then issue the same query again? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] vmarchaud commented on issue #9812: bug(broker): fail to group by after groovy extraction when upgrading to 0.11

Posted by GitBox <gi...@apache.org>.
vmarchaud commented on issue #9812:
URL: https://github.com/apache/pinot/issues/9812#issuecomment-1318250667

   @snleee Yeah i saw that in the changelog and i've re-enabled groovy, it actually gives another error if it's not enabled.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on issue #9812: bug(broker): fail to group by after groovy extraction when upgrading to 0.11

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #9812:
URL: https://github.com/apache/pinot/issues/9812#issuecomment-1329841751

   @vmarchaud We'll probably cut the next release late Dec. or early Jan.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] snleee commented on issue #9812: bug(broker): fail to group by after groovy extraction when upgrading to 0.11

Posted by GitBox <gi...@apache.org>.
snleee commented on issue #9812:
URL: https://github.com/apache/pinot/issues/9812#issuecomment-1317433808

   @vmarchaud Can you try to add the Pinot controller/server config and check if it works after that? From 0.11.0, I believe that the default behavior is to disable Groovy functionality due to the security concern.
   
   ```
   # Enable Groovy globally on Pinot broker
   pinot.broker.disable.query.groovy=false
   
   # Enable Groovy globally on Pinot controller
   controller.disable.ingestion.groovy=false
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on issue #9812: bug(broker): fail to group by after groovy extraction when upgrading to 0.11

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #9812:
URL: https://github.com/apache/pinot/issues/9812#issuecomment-1325738834

   Wow, it is a bug.. Notice the second one is SV instead of MV..
   ```
       if (_stringValuesMV == null || _stringValuesSV.length < length) {
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] vmarchaud commented on issue #9812: bug(broker): fail to group by after groovy extraction when upgrading to 0.11

Posted by GitBox <gi...@apache.org>.
vmarchaud commented on issue #9812:
URL: https://github.com/apache/pinot/issues/9812#issuecomment-1327503220

   @Jackie-Jiang Thanks for the quick fix ! Wondering if you have a idea for the next release ? I don't expect it to be in the next two week but asking anyway, i'll do my own build if thats too far


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org