You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/05/19 16:47:17 UTC
[GitHub] [druid] ofirnn opened a new issue #11275: New Unified Console Unusable for Large Clusters (1000+ datasources, 10k+ segments)
ofirnn opened a new issue #11275:
URL: https://github.com/apache/druid/issues/11275
After upgrading to 0.21.0 we noticed that the legacy UI (old-console) endpoints are not working anymore. our cluster is pretty big (thousands of data sources, a-lot of segments) - making the unified console in the router mode to not work.
### Affected Version
0.21.0 (and all versions when legacy UI isn't working anymore)
The Druid version where the problem was encountered.
### Description
When having:
- ~3500 data-sources
- ~20,000 segments
Router mode:
- Datasources page isn't working in router mode - the SQL query endpoint hangs
- Segments page isn't working in router mode - SQL hangs
Coordinator mode:
- Datasources page working well in coordinator mode- but without the segment timeline view
- Segments page crashing in coordinator mode - making thousands of requests - one per datasource - crashing chrome with insufficient resources
More:
- no way to filter prior query in both segments and datasources page
- no way to save a preset of filters with the responsive UI
- Ingestion page seems to work well even under high pressure of tasks (20-30k daily tasks)
All of the above worked well in the old-console - which we didn't find a way to enable in 0.21.0 - we saw a commit completely remove the binding so I believe there's no way to use it anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] ofirnn commented on issue #11275: New Unified Console Unusable for Large Clusters (1000+ datasources, 10k+ segments)
Posted by GitBox <gi...@apache.org>.
ofirnn commented on issue #11275:
URL: https://github.com/apache/druid/issues/11275#issuecomment-845340324
Hi @vogievetsky - Thanks for the quick reply!
re #10909 - sounds like it could help - we'll wait or try to play with our own built druid
Re the filtering/behavior change - the old console wasn't responsive so you could use add the datasource in the URL - to get the specific needed data like segments map - so it's a behavior that helped in large-scale environments -.
in the current behavior - we must wait for all segments to load (even after the optimizations branch) before having the ability to filter. another thing is the request per datasource to the backend in the segments page - which in the old console was only after digging into specific datasource - so not sure if its a feature request/behavior change caused the new approach to hang on large scale environments. I didn't mean for the "saving bookmarks" feature but just letting control the filters before querying the backend endpoints.
I'll take your suggestion to try the master-branch web-console.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vogievetsky commented on issue #11275: New Unified Console Unusable for Large Clusters (1000+ datasources, 10k+ segments)
Posted by GitBox <gi...@apache.org>.
vogievetsky commented on issue #11275:
URL: https://github.com/apache/druid/issues/11275#issuecomment-844317444
👋 Hi, I am the one you can blame for the new console. I want to understand your issues and make sure they are addressed.
This change https://github.com/apache/druid/pull/10909 I think will address most of your perf concerns. Sadly it did not make it into Apache Druid 0.21.0. It is available in the latest Imply distribution of Druid you could use that or build your own Druid.
Just to clarify `no way to filter prior query in both segments and datasources page` and `no way to save a preset of filters with the responsive UI` these are feature requests right? Not something that worked in the old console. Also can you clarify both point?
The legacy console was removed from the Druid codebase because it had security alerts for its dependancies that no one knew how to fix. It is still available here: https://github.com/druid-io/druid-console you can run it as a standalone server but I personally never could get it to work as it is using many npm dependancies that no longer exist etc.
If I was in your position I would either grab the latest Imply distro Druid. Or check out the master branch on this repo and run the dev mode of the web console (follow the readme here: https://github.com/apache/druid/tree/master/web-console#getting-started) pointing at your druid cluster like so: `druid_host=your.router.node:8888 npm start` to verify that the fix above works for you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] TheRum commented on issue #11275: New Unified Console Unusable for Large Clusters (1000+ datasources, 10k+ segments)
Posted by GitBox <gi...@apache.org>.
TheRum commented on issue #11275:
URL: https://github.com/apache/druid/issues/11275#issuecomment-882014545
Hi @vogievetsky , I'm also seeing the same issue of very very slow loading of 'Datasources' tab. (Druid - 0.21.1)
Cluster has close to 30,000+ segments and 10+ datasources.
To my experience, not just 'DataSources', actually all the tabs are very slow to load compared to previous versions. And Sometimes it is just slow enough to give up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org