You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/01/20 08:30:34 UTC
[GitHub] [druid] jbonofre opened a new issue #10782: User queries storage in zookeeper
jbonofre opened a new issue #10782:
URL: https://github.com/apache/druid/issues/10782
### Motivation
Today, the user writes and executes queries in the web console or the query API.
These queries can be scheduled via an external tool (like Decanter) to periodically execute queries.
However, it would be great (for maintenance purpose) to give an ID to each query and be able to store the queries in zookeeper.
The broker would be able to "load" a query by ID.
### Proposed changes
The idea is to update:
- historical to store query in zookeeper
- broker to execute query with ID instead of the "concrete" query
### Rationale
With this change, it would be:
- easier to find and maintain analytic queries
- easier to have multiple scheduler/client executing queries (the queries code itself is not on the client side, but rather on the Druid side)
Thoughts ?
### Operational impact
The current behavior stays as it is: we don't change anything on the existing behavior. We "just" add a new layer for queries storage and execution of stored queries.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] Gabriel39 commented on issue #10782: User queries storage in zookeeper
Posted by GitBox <gi...@apache.org>.
Gabriel39 commented on issue #10782:
URL: https://github.com/apache/druid/issues/10782#issuecomment-766659991
i think it's a meaningful idea. however i think zk is not the best choice since it's not a pure storage system. i think mysql is a better choice
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] paul-rogers commented on issue #10782: User queries storage in zookeeper
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on issue #10782:
URL: https://github.com/apache/druid/issues/10782#issuecomment-970639554
From experience on other Apache projects (Drill, Impala), ZK is not good at this sort of thing: ZK is not a distributed DB. It will struggle to hold very many queries. The SQL DB solution is good. In an HDFS or S3 environment, it may also be possible to store queries as files.
Security is often another consideration with this kind of thing: should user A be able to see user B's queries? How are queries kept private or shared? Should user A have the ability to alter or replace a shared query? Or, only the admin can do that?
If someone accidentally replaces a query with the wrong one, is there a way to get the old one back?
Pretty soon, we are building a DB application and the MySQL idea looks pretty good.
Suggestion: consider doing this as an extension point: if someone wants to implement it in ZK (because it fits their use case), go for it. Maybe another works with MySQL, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] Gabriel39 commented on issue #10782: User queries storage in zookeeper
Posted by GitBox <gi...@apache.org>.
Gabriel39 commented on issue #10782:
URL: https://github.com/apache/druid/issues/10782#issuecomment-766659991
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] Gabriel39 commented on issue #10782: User queries storage in zookeeper
Posted by GitBox <gi...@apache.org>.
Gabriel39 commented on issue #10782:
URL: https://github.com/apache/druid/issues/10782#issuecomment-766949882
> i think it's a meaningful idea. however i think zk is not the best choice since it's not a pure storage system. i think mysql is a better choice
@jbonofre how do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org