[jira] [Commented] (LENS-124) Add scheduler service on lens

["Bala Nathan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LENS-124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14642632#comment-14642632 ] 

Bala Nathan commented on LENS-124:
----------------------------------

Merging Comments from two threads:

1. Parameterization of queries to allow fixed, moving time windows 

- Pre-requisite is that the queries are already saved and prepared in Lens and can be executed later. Not sure if bind arg support is available

2. Allow Cron style scheduling & specific weekday or specific day of month etc

- Similar to the quartz scheduler format (supports cron)


3. Allow scheduled queries to be paced in such a way that regular queries that users are submitting point in time don't suffer  

- Should be possible to solve this by having a dedicated scheduler instance for running scheduled queries

4. Allow gating of the scheduled queries based on partition availability. It shouldn't happen that ideally query is run in low cost engine and due to data availability delays, the system chooses a higher cost engine because of availability of data earlier in that system. In some cases that may be acceptable as well. But the choice should be conscious

- Need more inputs on this. Can you give me an example please?

5.Handling query failures in the schedule and being able to run them again through administrative levers

- How is this different from re-running failed queries ? 

6. Administrative levers to repeating the entire schedule due to errors in data, requiring re-publishing the reports 

- Does this mean ability to selectively choose schedules that can be run again?

7. Support query stats analysis over scheduled queries

 - Need more inputs on What kind of analysis can be run on scheduled queries ? 

8. Limit number of schedules per user/app/client 

9. Schedule should be aware of data availability 

- Can this not be handled as part of a pre-condition to running a schedule. Assuming there is a proper expression syntax supported for specifying a pre-condition (query/script/set of rules), the schedule instance itself can either be executed/rejected based on the output of a pre-condition check.

10. Schedule should get retried on non data availability and failures and send appropriate alerts to an end point 

- I'm assuming this means "Schedule should not run" and an alert should be sent. 

11. User will set schedule start time in his/her time zone 

12. Ability to expire a schedule

13. Pause and resume of schedule 

14. Cancel a running instance

15. Upper limit on number of scheduled jobs which will be submitted to the grid at a time. Otherwise it might end up overloading the cluster 

- Somewhat similar to #8 ?

> Add scheduler service on lens
> -----------------------------
>
>                 Key: LENS-124
>                 URL: https://issues.apache.org/jira/browse/LENS-124
>             Project: Apache Lens
>          Issue Type: New Feature
>          Components: server
>            Reporter: Amareshwari Sriramadasu
>              Labels: Hackathon-July
>
> Query scheduler service should provide following services
>  * Schedule a query
>  ** Periodically
>  ** On data availability, for ex: whenever a day's data is available
>  * Look at status of scheduled query
>  ** Mostly the current run status
>  * Look at the output of scheduled query - through email?
>  * Update scheduled query
>  ** Update the query
>  ** Update the frequency
>  ** Update the configuration
>   * Look at stats of scheduled query
>   **  number of failures
>   **  number of reruns
>   **  Get handles for completed queries
>  * Cancel a scheduled query
>  * Pause and resume a scheduled query
>  *  Look at all scheduled queries



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)