You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/10/14 16:11:06 UTC
[GitHub] [incubator-pinot] lgo opened a new issue #6143: Invalid query results when querying with non-existent column
lgo opened a new issue #6143:
URL: https://github.com/apache/incubator-pinot/issues/6143
While testing some results, I accidentally used the wrong name to refer to a column. When querying, rather than raising an error the query actually returned but with `0` results, such as the following
**Query**
_Note: `amount` does not exist on the table. This was meant to be `balance`!_
```sql
select count(*), sum(amount) from adjustments where type = 'foobar'
```
**Result**
```json
{
"resultTable": {
"dataSchema": {
"columnNames": [
"count(*)",
"sum(amount)"
],
"columnDataTypes": [
"LONG",
"DOUBLE"
]
},
"rows": [
[
0,
0
]
]
},
"exceptions": [],
"numServersQueried": ...,
"numServersResponded": ...,
"numSegmentsQueried": ...,
"numSegmentsProcessed": 0,
"numSegmentsMatched": 0,
"numConsumingSegmentsQueried": 0,
"numDocsScanned": 0,
"numEntriesScannedInFilter": 0,
"numEntriesScannedPostFilter": 0,
"numGroupsLimitReached": false,
"totalDocs": ...,
"timeUsedMs": 10,
"segmentStatistics": [],
"traceInfo": {},
"minConsumingFreshnessTimeMs": 0
}
```
Here is a simplified example configuration we were using, in case this ends up being particular about the table config. (irrelevant columns were prunted from the example).
## Schema
```json
{
"schemaName": "adjustment",
"dimensionFieldSpecs": [
{
"name": "type",
"dataType": "STRING"
},
],
"metricFieldSpecs": [
{
"name": "balance",
"dataType": "DOUBLE"
},
],
"dateTimeFieldSpecs": [
{
"name": "created_at",
"dataType": "LONG",
"format": "1:SECONDS:EPOCH",
"granularity": "1:HOURS"
}
]
}
```
## Table config
```json
{
"OFFLINE": {
"tableName": "adjustment_OFFLINE",
"tableType": "OFFLINE",
"segmentsConfig": {
"timeType": "SECONDS",
"schemaName": "adjustment",
"segmentPushFrequency": "HOURLY",
"segmentPushType": "APPEND",
"timeColumnName": "created_at",
"replication": "1"
},
"tenants": {
"broker": "DefaultTenant",
"server": "DefaultTenant"
},
"tableIndexConfig": {
"bloomFilterColumns": [
"type"
],
"loadMode": "MMAP",
"noDictionaryColumns": [],
"enableDefaultStarTree": false,
"starTreeIndexConfigs": [
{
"dimensionsSplitOrder": [
"type",
],
"functionColumnPairs": [
"SUM__balance",
"COUNT"
],
"maxLeafRecords": 1
}
],
"enableDynamicStarTreeCreation": false,
"segmentPartitionConfig": {
"columnPartitionMap": {
"type": {
"functionName": "Murmur",
"numPartitions": 100
}
}
},
"aggregateMetrics": false,
"nullHandlingEnabled": false,
"invertedIndexColumns": [
"type"
],
},
"metadata": {},
"routing": {
"segmentPrunerTypes": [
"partition"
]
}
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #6143: Invalid query results when querying with non-existent column
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6143:
URL: https://github.com/apache/incubator-pinot/issues/6143#issuecomment-708548825
I feel we should by default do column validation add an flag to loose this behavior for legacy offline tables which doesn't enforce this behavior. Thoughts? @mayankshriv @mcvsubbu
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] jackjlli commented on issue #6143: Invalid query results when querying with non-existent column
Posted by GitBox <gi...@apache.org>.
jackjlli commented on issue #6143:
URL: https://github.com/apache/incubator-pinot/issues/6143#issuecomment-708562143
Yes, I have a PR to detect the column mismatch in the query:
https://github.com/apache/incubator-pinot/pull/6066
We'll first monitor how many existing use cases are still firing queries with non-existing columns. After adjusting their behavior, we can restrict this in SQL.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on issue #6143: Invalid query results when querying with non-existent column
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #6143:
URL: https://github.com/apache/incubator-pinot/issues/6143#issuecomment-708559618
I think there was a PR that @jackjlli had sometime before? It is best if we enforce it as a part of SQL, and let PQL be. We are in the process of migrating from PQL to SQL, and can add this as an enforcement instead of retro fitting queries of a ton of tables that somehow get by today.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org