You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/03/31 01:20:00 UTC
[GitHub] [druid] suneet-s opened a new pull request #9590: Transform reindex
suneet-s opened a new pull request #9590: Transform reindex
URL: https://github.com/apache/druid/pull/9590
This test will fail till #9589 is fixed
### Description
Add integration tests for using transformSpec during ingestion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] ccaominh merged pull request #9590: DruidInputSource can
add new dimensions during re-ingestion
Posted by GitBox <gi...@apache.org>.
ccaominh merged pull request #9590: DruidInputSource can add new dimensions during re-ingestion
URL: https://github.com/apache/druid/pull/9590
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #9590: Add
integration tests for ingestion with transformSpec
Posted by GitBox <gi...@apache.org>.
suneet-s commented on a change in pull request #9590: Add integration tests for ingestion with transformSpec
URL: https://github.com/apache/druid/pull/9590#discussion_r400637506
##########
File path: integration-tests/src/test/resources/indexer/wikipedia_index_queries_with_transform.json
##########
@@ -0,0 +1,54 @@
+[
+ {
+ "description":"having spec on post aggregation",
+ "query":{
+ "queryType":"groupBy",
+ "dataSource":"%%DATASOURCE%%",
+ "granularity":"day",
+ "dimensions":[
+ "page"
Review comment:
Using a filter on "newPage" (the renamed column) fails the test locally. But if I apply the transformation from another datasource it appears to work. Need to verify if this is actually a bug or not
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #9590: Add
integration test for ingestion with transformSpec
Posted by GitBox <gi...@apache.org>.
suneet-s commented on a change in pull request #9590: Add integration test for ingestion with transformSpec
URL: https://github.com/apache/druid/pull/9590#discussion_r400593315
##########
File path: integration-tests/src/test/resources/indexer/wikipedia_index_queries_with_transform.json
##########
@@ -0,0 +1,54 @@
+[
+ {
+ "description":"having spec on post aggregation",
+ "query":{
+ "queryType":"groupBy",
+ "dataSource":"%%DATASOURCE%%",
+ "granularity":"day",
+ "dimensions":[
+ "page"
+ ],
+ "filter":{
+ "type":"selector",
+ "dimension":"language",
+ "value":"language-zh"
+ },
+ "aggregations":[
+ {
+ "type":"count",
+ "name":"rows"
+ },
+ {
+ "type":"longSum",
+ "fieldName":"triple-added",
+ "name":"added_count"
+ }
+ ],
+ "postAggregations": [
+ {
+ "type":"arithmetic",
+ "name":"added_count_times_ten",
+ "fn":"*",
+ "fields":[
+ {"type":"fieldAccess", "name":"added_count", "fieldName":"added_count"},
+ {"type":"constant", "name":"const", "value":10}
+ ]
+ }
+ ],
+ "having":{"type":"greaterThan", "aggregation":"added_count_times_ten", "value":9000},
+ "intervals":[
+ "2013-08-31T00:00/2013-09-01T00:00"
+ ]
+ },
+ "expectedResults":[ {
+ "version" : "v1",
+ "timestamp" : "2013-08-31T00:00:00.000Z",
+ "event" : {
+ "added_count_times_ten" : 9050.0,
Review comment:
typo in expected results
```suggestion
"added_count_times_ten" : 27150.0,
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] suneet-s commented on issue #9590: Add integration tests
for ingestion with transformSpec
Posted by GitBox <gi...@apache.org>.
suneet-s commented on issue #9590: Add integration tests for ingestion with transformSpec
URL: https://github.com/apache/druid/pull/9590#issuecomment-606414811
> Can you parameterize all the query and index json files? I think it's much harder to keep track of many many many different query and index json files we have for IT. Especially when they are 80-90% the same.
Across all `AbstractITBatchIndexTest` or within the new tests I've added in `ITTransformTest`?
It's hard to parameterize `ITTransformTest` because it's broken for most combinations. And there are 2 categories of tests. I think properly parameterizing these tests requires a re-think of what the test environment should look like because all the re-indexing tests start by indexing a datasource which slows down the test execution.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] suneet-s commented on issue #9590: Add integration tests
for ingestion with transformSpec
Posted by GitBox <gi...@apache.org>.
suneet-s commented on issue #9590: Add integration tests for ingestion with transformSpec
URL: https://github.com/apache/druid/pull/9590#issuecomment-606676972
>
> I am thinking of parameterizing the json files themself across the new json files you added
I think parameterizing the json files is a great idea, but there's a lot of tight coupling between the json files and the base IT classes. I'm not sure where the best place to do this is because of that.
I'd rather do the parameterization in a separate change - when #9591 #9592 or #9589 are fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] maytasm3 commented on issue #9590: Add integration tests
for ingestion with transformSpec
Posted by GitBox <gi...@apache.org>.
maytasm3 commented on issue #9590: Add integration tests for ingestion with transformSpec
URL: https://github.com/apache/druid/pull/9590#issuecomment-606433727
> > Can you parameterize all the query and index json files? I think it's much harder to keep track of many many many different query and index json files we have for IT. Especially when they are 80-90% the same.
>
> Across all `AbstractITBatchIndexTest` or within the new tests I've added in `ITTransformTest`?
>
> It's hard to parameterize `ITTransformTest` because it's broken for most combinations. And there are 2 categories of tests. I think properly parameterizing these tests requires a re-think of what the test environment should look like because all the re-indexing tests start by indexing a datasource which slows down the test execution.
I am thinking of parameterizing the json files themself across the new json files you added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #9590:
DruidInputSource can add new dimensions during re-ingestion
Posted by GitBox <gi...@apache.org>.
jihoonson commented on a change in pull request #9590: DruidInputSource can add new dimensions during re-ingestion
URL: https://github.com/apache/druid/pull/9590#discussion_r401914402
##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/ReingestionTimelineUtils.java
##########
@@ -101,4 +103,34 @@
.mapToObj(orderedMetrics::get)
.collect(Collectors.toList());
}
+
+ /**
+ * Utility function to get dimensions that should be ingested. The preferred order is
+ * - Explicit dimensions if they are provided.
+ * - Custom dimensions are provided in the inputSpec.
+ * - Calculate dimensions from the timeline but exclude any dimension exclusions.
+ *
+ * @param explicitDimensions sent as part of the re-ingestion InputSource.
+ * @param dimensionsSpec from the provided ingestion spec.
+ * @param timeLineSegments for the datasource that is being read.
+ * @return
Review comment:
nit: please add description for the return value or remove this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #9590:
DruidInputSource can add new dimensions during re-ingestion
Posted by GitBox <gi...@apache.org>.
jihoonson commented on a change in pull request #9590: DruidInputSource can add new dimensions during re-ingestion
URL: https://github.com/apache/druid/pull/9590#discussion_r401913530
##########
File path: examples/quickstart/tutorial/transform-index.json
##########
@@ -55,7 +55,7 @@
"baseDir" : "quickstart/tutorial",
"filter" : "transform-data.json"
},
- "inpuFormat" : {
+ "inputFormat" : {
Review comment:
Good catch!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #9590: Add
integration tests for ingestion with transformSpec
Posted by GitBox <gi...@apache.org>.
suneet-s commented on a change in pull request #9590: Add integration tests for ingestion with transformSpec
URL: https://github.com/apache/druid/pull/9590#discussion_r400637506
##########
File path: integration-tests/src/test/resources/indexer/wikipedia_index_queries_with_transform.json
##########
@@ -0,0 +1,54 @@
+[
+ {
+ "description":"having spec on post aggregation",
+ "query":{
+ "queryType":"groupBy",
+ "dataSource":"%%DATASOURCE%%",
+ "granularity":"day",
+ "dimensions":[
+ "page"
Review comment:
Using a filter on "newPage" fails the test locally. But if I apply the transformation from another datasource it appears to work. Need to verify if this is actually a bug or not
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] maytasm3 commented on issue #9590: Add integration tests
for ingestion with transformSpec
Posted by GitBox <gi...@apache.org>.
maytasm3 commented on issue #9590: Add integration tests for ingestion with transformSpec
URL: https://github.com/apache/druid/pull/9590#issuecomment-606395028
Can you parameterize all the query and index json files? I think it's much harder to keep track of many many many different query and index json files we have for IT. Especially when they are 80-90% the same.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #9590: Add
integration tests for ingestion with transformSpec
Posted by GitBox <gi...@apache.org>.
suneet-s commented on a change in pull request #9590: Add integration tests for ingestion with transformSpec
URL: https://github.com/apache/druid/pull/9590#discussion_r401215630
##########
File path: integration-tests/src/test/resources/indexer/wikipedia_index_queries_with_transform.json
##########
@@ -0,0 +1,54 @@
+[
+ {
+ "description":"having spec on post aggregation",
+ "query":{
+ "queryType":"groupBy",
+ "dataSource":"%%DATASOURCE%%",
+ "granularity":"day",
+ "dimensions":[
+ "page"
Review comment:
This is a bug filed #9593 for this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org