You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/03/31 06:12:39 UTC
[GitHub] [druid] suneet-s opened a new issue #9592: Re-indexing a
DruidInputSource with explicit dimensions includes all dimensions from
original datasource
suneet-s opened a new issue #9592: Re-indexing a DruidInputSource with explicit dimensions includes all dimensions from original datasource
URL: https://github.com/apache/druid/issues/9592
### Affected Version
0.18
### Description
If you attempt to re-index a druid datasource using a Druid InputSource, and you explicitly set the columns in the dimension spec, the re-index job will ingest all columns. To exclude columns, they need to be manually added to the dimensionsExclusion field
This ingestion spec used in integration tests re-produces the issue. The field "robot" is included in the re-indexed datasource even though it's not explicitly specified in the ingestionSpec
```
{
"type": "index",
"spec": {
"ioConfig": {
"type": "index",
"inputSource": {
"type": "druid",
"dataSource": "%%DATASOURCE%%",
"interval": "2013-08-31/2013-09-01"
}
},
"tuningConfig": {
"type": "index",
"partitionsSpec": {
"type": "dynamic"
}
},
"dataSchema": {
"dataSource": "%%REINDEX_DATASOURCE%%",
"granularitySpec": {
"type": "uniform",
"queryGranularity": "SECOND",
"segmentGranularity": "DAY"
},
"timestampSpec": {
"column": "__time",
"format": "iso"
},
"dimensionsSpec": {
"dimensions": [
"page",
{"type": "string", "name": "language", "createBitmapIndex": false},
"user",
"unpatrolled",
"newPage",
"anonymous",
"namespace",
"country",
"region",
"city"
]
},
"transformSpec": {
"transforms": [
{
"type": "expression",
"name": "newPage",
"expression": "page"
},
{
"type": "expression",
"name": "one-plus-triple-added",
"expression": "\"triple-added\" + 1"
},
{
"type": "expression",
"name": "double-deleted",
"expression": "deleted * 2"
}
]
},
"metricsSpec": [
{
"type": "doubleSum",
"name": "added",
"fieldName": "added"
},
{
"type": "doubleSum",
"name": "triple-added",
"fieldName": "triple-added"
},
{
"type": "doubleSum",
"name": "one-plus-triple-added",
"fieldName": "one-plus-triple-added"
},
{
"type": "doubleSum",
"name": "deleted",
"fieldName": "deleted"
},
{
"type": "doubleSum",
"name": "double-deleted",
"fieldName": "double-deleted"
},
{
"type": "doubleSum",
"name": "delta",
"fieldName": "delta"
}
]
}
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] suneet-s commented on issue #9592: Re-indexing a
DruidInputSource with explicit dimensions includes all dimensions from
original datasource
Posted by GitBox <gi...@apache.org>.
suneet-s commented on issue #9592: Re-indexing a DruidInputSource with explicit dimensions includes all dimensions from original datasource
URL: https://github.com/apache/druid/issues/9592#issuecomment-611563647
Fixed by #9590
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] suneet-s closed issue #9592: Re-indexing a DruidInputSource
with explicit dimensions includes all dimensions from original datasource
Posted by GitBox <gi...@apache.org>.
suneet-s closed issue #9592: Re-indexing a DruidInputSource with explicit dimensions includes all dimensions from original datasource
URL: https://github.com/apache/druid/issues/9592
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org