[jira] [Commented] (LENS-124) Add scheduler service on lens

["Srikanth Sundarrajan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LENS-124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14304771#comment-14304771 ] 

Srikanth Sundarrajan commented on LENS-124:
-------------------------------------------

Adding comments from LENS-268 for consolidating the discussions.

It would be quite handy to support scheduling as a capability in the Lens system to allow users to submit a query with a repeatable schedule. For ex. I should be able to submit a query and expect that to be running the first monday of every month or first day of every month over a moving data window.

The list of features that need to be supported with this are

1. Parameterization of queries to allow fixed, moving time windows
2. Allow Cron style scheduling & specific weekday or specific day of month etc
3. Allow scheduled queries to be paced in such a way that regular queries that users are submitting point in time don't suffer
4. Allow gating of the scheduled queries based on partition availability. It shouldn't happen that ideally query is run in low cost engine and due to data availability delays, the system chooses a higher cost engine because of availability of data earlier in that system. In some cases that may be acceptable as well. But the choice should be conscious.
5.Handling query failures in the schedule and being able to run them again through administrative levers
6. Administrative levers to repeating the entire schedule due to errors in data, requiring re-publishing the reports.
7. Support query stats analysis over scheduled queries

Also see [comments|https://issues.apache.org/jira/browse/LENS-268?focusedCommentId=14304642&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14304642] from [~rohitchattar] and [comments|https://issues.apache.org/jira/browse/LENS-268?focusedCommentId=14304672&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14304672] from [~jaideepdhok]

> Add scheduler service on lens
> -----------------------------
>
>                 Key: LENS-124
>                 URL: https://issues.apache.org/jira/browse/LENS-124
>             Project: Apache Lens
>          Issue Type: Improvement
>          Components: server
>            Reporter: Amareshwari Sriramadasu
>
> Query scheduler service should provide following services
>  * Schedule a query
>  ** Periodically
>  ** On data availability, for ex: whenever a day's data is available
>  * Look at status of scheduled query
>  ** Mostly the current run status
>  * Look at the output of scheduled query - through email?
>  * Update scheduled query
>  ** Update the query
>  ** Update the frequency
>  ** Update the configuration
>   * Look at stats of scheduled query
>   **  number of failures
>   **  number of reruns
>   **  Get handles for completed queries
>  * Cancel a scheduled query
>  * Pause and resume a scheduled query
>  *  Look at all scheduled queries



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)