You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "BozhanStoimenov-smartico (via GitHub)" <gi...@apache.org> on 2023/10/02 09:41:40 UTC
[I] Web UI limited to 5000 display results only when doing lookups (druid)
BozhanStoimenov-smartico opened a new issue, #15065:
URL: https://github.com/apache/druid/issues/15065
Hello we are facing a problem when doing lookups via a jdbc connection to postgres. The displayed results are limited to 5000 while the postgresql source table has 40 000+ .
Is there any property which controls such a limit ?
### Affected Version
24.0.0
### Description
Please include as much detailed information about the problem as possible.
- Configuration reference used : https://druid.apache.org/docs/24.0.0/configuration/index.html
- Cluster size: nano-quickstart
- Configurations in use: /opt/druid/apache-druid-27.0.0/conf/druid/single-server/nano-quickstart/_common/common.runtime.properties
- Output of global config file in use :
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
# Extensions specified in the load list will be loaded by Druid
# We are using local fs for deep storage - not recommended for production - use S3, HDFS, or NFS instead
# We are using local derby for the metadata store - not recommended for production - use MySQL or Postgres instead
# If you specify `druid.extensions.loadList=[]`, Druid won't load any extension from file system.
# If you don't specify `druid.extensions.loadList`, Druid will load all the extensions under root extension directory.
# More info: https://druid.apache.org/docs/latest/operations/including-extensions.html
druid.extensions.loadList=["druid-kafka-indexing-service", "druid-datasketches", "druid-multi-stage-query","druid-s3-extensions", "druid-lookups-cached-global", "postgresql-metadata-storage", "druid-lookups-cached-single", "dropwizard-emitter"]
# If you have a different version of Hadoop, place your Hadoop client jar files in your hadoop-dependencies directory
# and uncomment the line below to point to your directory.
#druid.extensions.hadoopDependenciesDir=/my/dir/hadoop-dependencies
#
# Hostname
#
druid.host=some.hostname
#
# Logging
#
# Log all runtime properties on startup. Disable to avoid logging properties on startup:
druid.startup.logging.logProperties=true
#
# Zookeeper
#
druid.zk.service.host=localhost
druid.zk.paths.base=/druid
#
# Metadata storage
#
# For Derby server on your Druid Coordinator (only viable in a cluster with a single Coordinator, no fail-over):
#druid.metadata.storage.type=derby
#druid.metadata.storage.connector.connectURI=jdbc:derby://localhost:1527/var/druid/metadata.db;create=true
#druid.metadata.storage.connector.host=localhost
#druid.metadata.storage.connector.port=1527
# For MySQL (make sure to include the MySQL JDBC driver on the classpath):
#druid.metadata.storage.type=mysql
#druid.metadata.storage.connector.connectURI=jdbc:mysql://db.example.com:3306/druid
#druid.metadata.storage.connector.user=...
#druid.metadata.storage.connector.password=...
# For PostgreSQL:
druid.metadata.storage.type=postgresql
druid.metadata.storage.connector.connectURI=jdbc:postgresql://rds-some-host.rds.amazonaws.com:5432/postgres?currentSchema=druid
druid.metadata.storage.connector.user=postgres
druid.metadata.postgres.dbTableSchema=druid
druid.metadata.storage.connector.password=some-password
druid.metadata.storage.connector.createTables=false
#
# Deep storage
#
# For local disk (only viable in a cluster if this is a network mount):
#druid.storage.type=local
#druid.storage.storageDirectory=var/druid/segments
# For HDFS:
#druid.storage.type=hdfs
#druid.storage.storageDirectory=/druid/segments
# For S3:
druid.storage.type=s3
druid.storage.bucket=druid-bucket
druid.storage.baseKey=segments
druid.s3.accessKey=key
druid.s3.secretKey=secret-key
#
# Indexing service logs
#
# For local disk (only viable in a cluster if this is a network mount):
#druid.indexer.logs.type=file
#druid.indexer.logs.directory=var/druid/indexing-logs
# For HDFS:
#druid.indexer.logs.type=hdfs
#druid.indexer.logs.directory=/druid/indexing-logs
# For S3:
druid.indexer.logs.type=s3
druid.indexer.logs.s3Bucket=druid-bucket
druid.indexer.logs.s3Prefix=indexing-logs
#
# Service discovery
#
druid.selectors.indexing.serviceName=druid/overlord
druid.selectors.coordinator.serviceName=druid/coordinator
### ### ### ### ### Monitoring ### ### ### ### ###
#druid.monitoring.monitors=["org.apache.druid.java.util.metrics.JvmMonitor"]
#Frequency that Druid emits metrics. (Default: PT1M - 1 min)
#druid.monitoring.emissionPeriod=PT1M
#Setting this value initializes one of the emitter modules.
druid.emitter=dropwizard
#druid.emitter.composing.emitters=["dropwizard"]
#druid.emitter=noop
#Choices: debug, info, warn, error. The log level at which message are logged.
druid.emitter.logging.logLevel=debug
#Used to report druid metrics via JMX.
druid.emitter.dropwizard.reporters=[{"type":"jmx"}]
#Maximum size of metrics registry to be cached at any time. (Default: 100 mb)
druid.emitter.dropwizard.maxMetricsRegistrySize=250
### ### ### ### ### End Monitoring block ### ### ### ### ###
# Storage type of double columns
# ommiting this will lead to index double as float at the storage layer
druid.indexing.doubleStorage=double
#
# Security
#
druid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password", "password", "key", "token", "pwd"]
#
# SQL
#
druid.sql.enable=true
# Planning SQL query when there is aggregate distinct in the statement
druid.sql.planner.useGroupingSetForExactDistinct=true
#
# Lookups
druid.lookup.enableLookupSyncOnStartup=true
druid.lookup.snapshotWorkingDir=/opt/druid/var
druid.javascript.enabled=true
druid.manager.lookups.period=1200000
druid.manager.lookups.hostUpdateTimeout=PT120s
druid.manager.lookups.updateAllTimeout=PT360s
druid.lookup.namespace.cache.type=offHeap
druid.lookup.namespace.numBufferedEntries=200000
#
# Expression processing config
#
druid.expressions.useStrictBooleans=true
#
# Http client
#
druid.global.http.eagerInitialization=false
druid.server.http.defaultQueryTimeout=120000
#1gb
druid.query.groupBy.maxOnDiskStorage=1073741824
#500mb
druid.processing.dictionary.maxSize=524288000
- Picture with the limit in question
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Druid limited to 5000 display results only when doing lookups (druid)
Posted by "BozhanStoimenov-smartico (via GitHub)" <gi...@apache.org>.
BozhanStoimenov-smartico closed issue #15065: Druid limited to 5000 display results only when doing lookups
URL: https://github.com/apache/druid/issues/15065
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org