You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/02/27 00:00:06 UTC
[GitHub] [druid] samarthjain opened a new issue #9421: Unknown complex types
break Druid SQL
samarthjain opened a new issue #9421: Unknown complex types break Druid SQL
URL: https://github.com/apache/druid/issues/9421
### Affected Version
0.16.0
### Description
After upgrading our clusters to 0.16.0, we noticed that the Druid SQL wasn't working because the metadata updates in DruidSchema were failing because of unknown aggregators in the segment metadata.
Turns out, in 0.16.0, I made a backward incompatible change in t-digest aggregator extension where I removed the aggregator types `buildTDigestSketch` and `mergeTDigestSketch` and replaced it with a single aggregator `tDigestSketch`.
We had some users ingest their data using the `buildTDigestSketch` aggregator. However, when the 0.16.0 cluster was deployed, it wasn't able to deserialize the aggregator metadata as they were removed. This caused problems for the Druid SQL metadata cache as SegmentMetadataRequests started failing which in turn completely broke the Druid SQL functionality on the cluster.
I think we should fix this and prevent future cases of someone making incompatible changes, especially in extensions-contrib modules, from breaking a core Druid functionality.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] samarthjain commented on issue #9421: Unknown complex types
break Druid SQL
Posted by GitBox <gi...@apache.org>.
samarthjain commented on issue #9421: Unknown complex types break Druid SQL
URL: https://github.com/apache/druid/issues/9421#issuecomment-610040732
I think you are right. I had backported #7331 to our internal fork based off of 0.12.2 and users started using it. Then when the cluster was updated to 0.16, things broke because 0.16 had #8100 which removed the aggregators that were added in #7331.
Having said that, I think this PR is still worth while. Aggregator modules, especially in the extensions-contrib module, can make some changes which may or may not be compatible with each other. I am actually not even sure if we have any such guidelines around backward and forward compatibility when it comes to extension modules. This change is making sure that an incompatibility like the one I inadvertently introduced doesn't end up breaking a major Druid functionality (Druid SQL in this case).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] samarthjain edited a comment on issue #9421: Unknown
complex types break Druid SQL
Posted by GitBox <gi...@apache.org>.
samarthjain edited a comment on issue #9421: Unknown complex types break Druid SQL
URL: https://github.com/apache/druid/issues/9421#issuecomment-591709254
I was seeing SegmentMetadata queries fail on the historical nodes with an NPE.
```
Caused by: java.lang.NullPointerException
@400000005de84b7507eae4c4 at org.apache.druid.segment.column.SimpleColumnHolder.getColumn(SimpleColumnHolder.java:68) ~[druid-processing-0.16.0.jar:0.16.0]
@400000005de84b7507eb0404 at org.apache.druid.segment.QueryableIndexColumnSelectorFactory.lambda$getCachedColumn$2(QueryableIndexColumnSelectorFactory.java:175) ~[druid-processing-0.16.0.jar:0.16.0]
@400000005de84b7507eb07ec at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_231]
@400000005de84b7507eb07ec at org.apache.druid.segment.QueryableIndexColumnSelectorFactory.getCachedColumn(QueryableIndexColumnSelectorFactory.java:171) ~[druid-processing-0.16.0.jar:0.16.0]
@400000005de84b7507eb0bd4 at org.apache.druid.segment.QueryableIndexColumnSelectorFactory.lambda$makeColumnValueSelector$1(QueryableIndexColumnSelectorFactory.java:146) ~[druid-processing-0.16.0.jar:0.16.0]
@400000005de84b7507eb0bd4 at org.apache.druid.segment.QueryableIndexColumnSelectorFactory.makeColumnValueSelector(QueryableIndexColumnSelectorFactory.java:160) ~[druid-processing-0.16.0.jar:0.16.0]
@400000005de84b7507eb0fbc at org.apache.druid.query.aggregation.tdigestsketch.TDigestSketchAggregatorFactory.factorizeBuffered(TDigestSketchAggregatorFactory.java:117) ~[?:?]
@400000005de84b7507eb178c at org.apache.druid.query.aggregation.AggregatorAdapters.factorizeBuffered(AggregatorAdapters.java:103) ~[druid-processing-0.16.0.jar:0.16.0]
@400000005de84b7507eb178c at org.apache.druid.query.groupby.epinephelinae.GroupByQueryEngineV2$HashAggregateIterator.newGrouper(GroupByQueryEngineV2.java:550) ~[druid-processing-0.16.0.jar:0.16.0]
@400000005de84b7507eb1b74
```
These metadata queries were generated by DruidSchema over here:
https://github.com/apache/druid/blob/master/sql/src/main/java/org/apache/druid/sql/calcite/schema/DruidSchema.java#L508
The reason behind NPE was `columnSupplier` being null in `SimpleColumnHolder`. This was happening because the metadata deserialization was failing here:
https://github.com/apache/druid/blob/master/processing/src/main/java/org/apache/druid/segment/IndexIO.java#L578
which was "silently" setting the `columnSupplier` to null.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #9421: Unknown complex types
break Druid SQL
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #9421: Unknown complex types break Druid SQL
URL: https://github.com/apache/druid/issues/9421#issuecomment-610548834
AFAIK, we have different compatibility policies in different areas. There are two areas I'm aware of.
- For APIs, we have `PublicApi` and `ExtensionPoint` annotations. For those APIs annotated with one of them, there could be incompatible changes in those APIs between major releases. For non-annotated APIs, they can change in breaking ways in any releases.
- For segment read, we should be able to read any format of segments no matter how old Druid created them (I didn't know this policy when I raised https://github.com/apache/druid/pull/9246). This applies to segment format, all shardSpecs, and all loadSpecs.
These policies apply to across all Druid modules including contrib extensions.
Based on these policies, if the aggregator type has changed inadvertently, that's a compatibility bug and should be fixed by supporting both old and new types.
However, I think your PR might be still useful since people sometimes might want to run Druid without some extensions (for example, they just want to stop using those extensions). Even in that case, as you mentioned, it would be nice that Druid could be still functioning. Your PR could address this use case. I'll review it soon.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #9421: Unknown complex types
break Druid SQL
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #9421: Unknown complex types break Druid SQL
URL: https://github.com/apache/druid/issues/9421#issuecomment-610036429
Hi @samarthjain, thank you for the report.
> Turns out, in 0.16.0, I made a backward incompatible change in t-digest aggregator extension where I removed the aggregator types buildTDigestSketch and mergeTDigestSketch and replaced it with a single aggregator tDigestSketch.
I'm wondering this incompatibility issue exists with Apache Druid extension. AFAIT, the TDigest extension was first added in #7331 and those aggregators were merged in #8100, but both PRs are in 0.16.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] samarthjain commented on issue #9421: Unknown complex types
break Druid SQL
Posted by GitBox <gi...@apache.org>.
samarthjain commented on issue #9421: Unknown complex types break Druid SQL
URL: https://github.com/apache/druid/issues/9421#issuecomment-591709254
I was seeing SegmentMetadata queries fail on the historical nodes with an NPE.
```
Caused by: java.lang.NullPointerException
@400000005de84b7507eae4c4 at org.apache.druid.segment.column.SimpleColumnHolder.getColumn(SimpleColumnHolder.java:68) ~[druid-processing-0.16.0.jar:0.16.0]
@400000005de84b7507eb0404 at org.apache.druid.segment.QueryableIndexColumnSelectorFactory.lambda$getCachedColumn$2(QueryableIndexColumnSelectorFactory.java:175) ~[druid-processing-0.16.0.jar:0.16.0]
@400000005de84b7507eb07ec at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_231]
@400000005de84b7507eb07ec at org.apache.druid.segment.QueryableIndexColumnSelectorFactory.getCachedColumn(QueryableIndexColumnSelectorFactory.java:171) ~[druid-processing-0.16.0.jar:0.16.0]
@400000005de84b7507eb0bd4 at org.apache.druid.segment.QueryableIndexColumnSelectorFactory.lambda$makeColumnValueSelector$1(QueryableIndexColumnSelectorFactory.java:146) ~[druid-processing-0.16.0.jar:0.16.0]
@400000005de84b7507eb0bd4 at org.apache.druid.segment.QueryableIndexColumnSelectorFactory.makeColumnValueSelector(QueryableIndexColumnSelectorFactory.java:160) ~[druid-processing-0.16.0.jar:0.16.0]
@400000005de84b7507eb0fbc at org.apache.druid.query.aggregation.tdigestsketch.TDigestSketchAggregatorFactory.factorizeBuffered(TDigestSketchAggregatorFactory.java:117) ~[?:?]
@400000005de84b7507eb178c at org.apache.druid.query.aggregation.AggregatorAdapters.factorizeBuffered(AggregatorAdapters.java:103) ~[druid-processing-0.16.0.jar:0.16.0]
@400000005de84b7507eb178c at org.apache.druid.query.groupby.epinephelinae.GroupByQueryEngineV2$HashAggregateIterator.newGrouper(GroupByQueryEngineV2.java:550) ~[druid-processing-0.16.0.jar:0.16.0]
@400000005de84b7507eb1b74
```
These metadata queries were generated by DruidCache over here:
https://github.com/apache/druid/blob/master/sql/src/main/java/org/apache/druid/sql/calcite/schema/DruidSchema.java#L508
The reason behind NPE was `columnSupplier` being null in `SimpleColumnHolder`. This was happening because the metadata deserialization was failing here:
https://github.com/apache/druid/blob/master/processing/src/main/java/org/apache/druid/segment/IndexIO.java#L578
which was "silently" setting the `columnSupplier` to null.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] samarthjain commented on issue #9421: Unknown complex types
break Druid SQL
Posted by GitBox <gi...@apache.org>.
samarthjain commented on issue #9421: Unknown complex types break Druid SQL
URL: https://github.com/apache/druid/issues/9421#issuecomment-591713556
The above NPE is unhandled in DruidSchema which causes the metadata refresh to fail and restart and fail at the same point over and over again. Maybe, DruidSchema should also be fixed to report only and swallow any such failures and proceed ahead with refreshing other datasources.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] gianm commented on issue #9421: Unknown complex types break
Druid SQL
Posted by GitBox <gi...@apache.org>.
gianm commented on issue #9421: Unknown complex types break Druid SQL
URL: https://github.com/apache/druid/issues/9421#issuecomment-591707379
I suppose in this case it would be best to ignore the column rather than the entire table. What were you seeing? Where'd the failure occur?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org