You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "saketbairoliya2 (via GitHub)" <gi...@apache.org> on 2023/02/26 17:33:53 UTC
[GitHub] [druid] saketbairoliya2 opened a new issue, #13851: Not able to access column from a Join datasource inside virtualColumns fields of scan queryType
saketbairoliya2 opened a new issue, #13851:
URL: https://github.com/apache/druid/issues/13851
Please provide a detailed title (e.g. "Broker crashes when using TopN query with Bound filter" instead of just "Broker crashes").
### Affected Version
24.0.1
### Description
Please include as much detailed information about the problem as possible.
- Cluster size
Not sure if this information will be so relevant for this bug so skipping the details.
- Configurations in use
```
{
"millisToWaitBeforeDeleting": 900000,
"mergeBytesLimit": 524288000,
"mergeSegmentsLimit": 100,
"maxSegmentsToMove": 5,
"percentOfSegmentsToConsiderPerMove": 100,
"useBatchedSegmentSampler": false,
"replicantLifetime": 15,
"replicationThrottleLimit": 10,
"balancerComputeThreads": 1,
"emitBalancingStats": false,
"killDataSourceWhitelist": [],
"killPendingSegmentsSkipList": [],
"maxSegmentsInNodeLoadingQueue": 100,
"decommissioningNodes": [],
"decommissioningMaxPercentOfMaxSegmentsToMove": 70,
"pauseCoordination": false,
"replicateAfterLoadTimeout": false,
"maxNonPrimaryReplicantsToLoad": 2147483647
}
```
- Steps to reproduce the problem
We were trying out a Join query - with left being queried at `granularity` of `day` and right side of query is set to `all` `granularity`. On the `Join` datasource, we wanted to add a virtualColumn and access one of the column.
Below is the sample query:
```
{
"queryType": "scan",
"dataSource": {
"type": "join",
"left": {
"type": "query",
"query": {
"queryType": "timeseries",
"dataSource": "multi_tenant_validated_output",
"granularity": "day",
"aggregations": [
{
"type": "doubleSum",
"fieldName": "double_sum",
"name": "sum_actual"
},
{
"type": "doubleSum",
"fieldName": "count",
"name": "count_actual"
}
],
"postAggregations": [
{
"type": "arithmetic",
"name": "mean_actual",
"fn": "/",
"fields": [
{
"type": "fieldAccess",
"fieldName": "sum_actual"
},
{
"type": "fieldAccess",
"fieldName": "count_actual"
}
]
}
],
"intervals": "2015-08-28/2025-10-20"
}
},
"right": {
"type": "query",
"query": {
"queryType": "timeseries",
"dataSource": "multi_tenant_validated_output",
"granularity": "all",
"aggregations": [
{
"type": "doubleSum",
"fieldName": "double_sum",
"name": "sum_expected"
},
{
"type": "doubleSum",
"fieldName": "count",
"name": "count_expected"
}
],
"postAggregations": [
{
"type": "arithmetic",
"name": "mean_expected",
"fn": "/",
"fields": [
{
"type": "fieldAccess",
"fieldName": "sum_expected"
},
{
"type": "fieldAccess",
"fieldName": "count_expected"
}
]
}
],
"intervals": "2015-08-28/2025-10-20"
}
},
"rightPrefix": "basesplit.",
"condition": "1",
"joinType": "INNER"
},
"intervals": {
"type": "intervals",
"intervals": [
"2015-08-28/2025-10-20"
]
},
"filter": {
"type": "and",
"fields": [
{
"type": "bound",
"dimension": "count_actual",
"lower": "1",
"ordering": "numeric"
}
]
},
"virtualColumns": [
{
"type": "expression",
"name": "mean_expected",
"expression": "basesplit.mean_expected",
"outputType": "DOUBLE"
}
]
}
```
- The error message or stack traces encountered. Providing more context, such as nearby log messages or even entire logs, can be helpful.
<img width="1454" alt="Screenshot 2023-02-26 at 11 01 07 PM" src="https://user-images.githubusercontent.com/5949441/221426474-96e494ee-3817-4050-a4eb-b5b3300759d6.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] saketbairoliya2 closed issue #13851: Not able to access column from a Join datasource inside virtualColumns fields of scan queryType
Posted by "saketbairoliya2 (via GitHub)" <gi...@apache.org>.
saketbairoliya2 closed issue #13851: Not able to access column from a Join datasource inside virtualColumns fields of scan queryType
URL: https://github.com/apache/druid/issues/13851
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] ApoorvGuptaAi commented on issue #13851: Not able to access column from a Join datasource inside virtualColumns fields of scan queryType
Posted by "ApoorvGuptaAi (via GitHub)" <gi...@apache.org>.
ApoorvGuptaAi commented on issue #13851:
URL: https://github.com/apache/druid/issues/13851#issuecomment-1445431527
Attaching results from a slightly modified query, it can be clearly seen that accessing the mean_expected column in virtual expression in the top most query returns null.
```
{
...
"virtualColumns": [
{
"type": "expression",
"name": "diff_of_mean",
"expression": "basesplit.mean_expected - mean_actual",
"outputType": "DOUBLE"
},
{
"type": "expression",
"name": "diff_of_mean_debug_1",
"expression": "basesplit.mean_expected",
"outputType": "DOUBLE"
},
{
"type": "expression",
"name": "diff_of_mean_debug_2",
"expression": "mean_expected",
"outputType": "DOUBLE"
},
],
}
```
```csv
"__time","diff_of_mean","diff_of_mean_debug_1","diff_of_mean_debug_2","sum_actual","count_actual","mean_actual","basesplit.__time","basesplit.sum_expected","basesplit.count_expected","basesplit.mean_expected"
"1664409600000","-643165.1670692185","null","null","1851672515.99228","2879","643165.1670692185","1440720000000","24080990493.740643","40319","597261.6010749434"
"1664496000000","-529102.7366475407","null","null","1523815881.5449173","2880","529102.7366475407","1440720000000","24080990493.740643","40319","597261.6010749434"
"1664582400000","-569473.1718008721","null","null","1640082734.7865114","2880","569473.1718008721","1440720000000","24080990493.740643","40319","597261.6010749434"
"1664668800000","-552417.1615504303","null","null","1590961425.2652395","2880","552417.1615504303","1440720000000","24080990493.740643","40319","597261.6010749434"
"1664755200000","-565688.6536937789","null","null","1629183322.6380832","2880","565688.6536937789","1440720000000","24080990493.740643","40319","597261.6010749434"
"1664841600000","-666414.1523327589","null","null","1919272758.7183456","2880","666414.1523327589","1440720000000","24080990493.740643","40319","597261.6010749434"
"1664928000000","-465871.1106727756","null","null","1341708798.7375937","2880","465871.1106727756","1440720000000","24080990493.740643","40319","597261.6010749434"
"1665014400000","-482558.17478761915","null","null","1389767543.388343","2880","482558.17478761915","1440720000000","24080990493.740643","40319","597261.6010749434"
"1665100800000","-441239.2982682393","null","null","1270769179.0125291","2880","441239.2982682393","1440720000000","24080990493.740643","40319","597261.6010749434"
"1665187200000","-543284.1701650404","null","null","1564658410.0753164","2880","543284.1701650404","1440720000000","24080990493.740643","40319","597261.6010749434"
"1665273600000","-557229.5186876628","null","null","1604821013.8204687","2880","557229.5186876628","1440720000000","24080990493.740643","40319","597261.6010749434"
"1665360000000","-495219.3813783375","null","null","1426231818.369612","2880","495219.3813783375","1440720000000","24080990493.740643","40319","597261.6010749434"
"1665446400000","-471617.48787544563","null","null","1358258365.0812833","2880","471617.48787544563","1440720000000","24080990493.740643","40319","597261.6010749434"
"1665532800000","-605353.4029983195","null","null","1743417800.63516","2880","605353.4029983195","1440720000000","24080990493.740643","40319","597261.6010749434"
"1665619200000","-685076.1327278281","null","null","1973019262.256145","2880","685076.1327278281","1440720000000","24080990493.740643","40319","597261.6010749434"
"1665705600000","-616306.0573581769","null","null","640341993.5951458","1039","616306.0573581769","1440720000000","24080990493.740643","40319","597261.6010749434"
"1676937600000","-608602.4824432421","null","null","49077095581.7406","80639","608602.4824432421","1440720000000","24080990493.740643","40319","597261.6010749434"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] saketbairoliya2 commented on issue #13851: Not able to access column from a Join datasource inside virtualColumns fields of scan queryType
Posted by "saketbairoliya2 (via GitHub)" <gi...@apache.org>.
saketbairoliya2 commented on issue #13851:
URL: https://github.com/apache/druid/issues/13851#issuecomment-1446697290
Got this working by using the expression like this: `"expression": "\"basesplit.mean_expected\" - mean_actual",`
Might have given error due to quotations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org