You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2019/02/13 13:50:38 UTC
[GitHub] scrawfor commented on issue #5727: Lookup module not registered
with overlord
scrawfor commented on issue #5727: Lookup module not registered with overlord
URL: https://github.com/apache/incubator-druid/issues/5727#issuecomment-463205987
I'm taking a stab at this. In my first attempt I added the `new LookupModule()` to the `CliOverlord.java` file as @2PacIsAlive suggested. This got me further, but the middle manager wouldn't run any tasks and threw this error.
```
com.fasterxml.jackson.databind.JsonMappingException: Instantiation of [simple type, class org.apache.druid.indexing.kafka.KafkaIndexTask] value failed: function 'lookup' is not defined.
```
So I added `new LookupModule()` to `CliMiddleManager.java` as well. This worked as the tasks get created and executed, and lookups are mapping data.
However, it also attempts to load lookup values on the middle manager, which throws out of memory errors
```
2019-02-13T13:24:56,227 ERROR [NamespaceExtractionCacheManager-1] org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Failed to update namespace [JdbcExtractionNamespace{connectorConfig=DbConnectorConfig{cr
eateTables=true, connectURI='Redacted', user='Redacted', passwordProvider=org.apache.druid.metadata.DefaultPasswordProvider}, table='Redacted', keyC
olumn='Redacted', valueColumn='Redacted', tsColumn='null', filter='null', pollPeriod=P1D}] : org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@6fc24ef7
org.skife.jdbi.v2.exceptions.CallbackFailedException: org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException: org.postgresql.util.PSQLException: Ran out of memory retrieving query results. [statement:"Redacted", located:"Redacted", rewritten:"Redacted", arguments:{ positional:{}, named:{}, finder:[]}]
at org.apache.druid.server.lookup.namespace.JdbcCacheGenerator.generateCache(JdbcCacheGenerator.java:77) ~[druid-lookups-cached-global-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.server.lookup.namespace.JdbcCacheGenerator.generateCache(JdbcCacheGenerator.java:49) ~[druid-lookups-cached-global-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl.tryUpdateCache(CacheScheduler.java:228) [druid-lookups-cached-global-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl.updateCache(CacheScheduler.java:207) [druid-lookups-cached-global-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl.access$600(CacheScheduler.java:143) [druid-lookups-cached-global-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl$2.run(CacheScheduler.java:189) [druid-lookups-cached-global-0.13.0-incubating.jar:0.13.0-incubating]
Caused by: org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException: org.postgresql.util.PSQLException: Ran out of memory retrieving query results. [statement:"Redacted", located:"Redacted", rewritten:"Redacted", arguments:{ posi
tional:{}, named:{}, finder:[]}]
at org.apache.druid.server.lookup.namespace.JdbcCacheGenerator$1.withHandle(JdbcCacheGenerator.java:99) ~[?:?]
at org.apache.druid.server.lookup.namespace.JdbcCacheGenerator$1.withHandle(JdbcCacheGenerator.java:79) ~[?:?]
at org.apache.druid.server.lookup.namespace.JdbcCacheGenerator$1.withHandle(JdbcCacheGenerator.java:99) ~[?:?]
at org.apache.druid.server.lookup.namespace.JdbcCacheGenerator$1.withHandle(JdbcCacheGenerator.java:79) ~[?:?]
```
Does anyone with a better understanding of the architecture have any ideas on how we could get the MiddleManager to recognize the lookup syntax without attempting to load them? We can't set `druid.lookup.enableLookupSyncOnStartup=false` because the middle manager will pass that property to the peon right?
For reference, my Indexing Task Spec:
```
{
"type": "kafka",
"dataSchema": {
"dataSource": "REDACTED",
"parser": {
"type": "string",
"parseSpec": {
"format": "json",
"timestampSpec": {
"column": "timestamp",
"format": "iso"
},
"dimensionsSpec": {
"dimensions": [
"user_type",
"client_id",
]
}
}
},
"metricsSpec": [
{
"type": "hyperUnique",
"name": "unique_views",
"fieldName": "view_id"
},
],
"transformSpec": {
"transforms": [
{
"type": "expression",
"name": "user_type",
"expression": "lookup(client_id, 'user_type_lookup')"
},
],
"filter": {
"type": "and",
"fields": [
{
"type": "not",
"field": {
"type": "in",
"dimension": "profile",
"values": ["REDACTED"]
}
}
]
}
},
"granularitySpec": {
"type": "uniform",
"segmentGranularity": "DAY",
"queryGranularity": "NONE",
"rollup": true
}
},
"tuningConfig": {
"type": "kafka",
"maxRowsInMemory": 5000,
"maxRowsPerSegment": 5000000,
"intermediatePersistPeriod": "PT2M",
"resetOffsetAutomatically": true
},
"ioConfig": {
"topic": "REDACTED",
"consumerProperties": {
"bootstrap.servers": "REDACTED"
},
"useEarliestOffset": false,
"skipOffsetGaps": true
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org