[jira] [Comment Edited] (YARN-6027) Support fromId for flows API

["Vrushali C (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15817288#comment-15817288 ] 

Vrushali C edited comment on YARN-6027 at 1/11/17 5:55 AM:
-----------------------------------------------------------

bq. How many do we expect typically ? Can it run into thousands ? 
So, let's see. Say, something is running every 5 mins on the cluster and let's say it completes in less than 5 mins, meaning we have at least one hadoop job every 5 mins, then that is 60 * 24 / 5 = 288. So much less than a thousand. I think it's reasonable to think it won't be a thousand runs in a day, unless someone is being malicious (triggering a flow run every min is 1440 runs). 

That said, what do we think might break or be a problem if we do end up having >1000 runs in one flow in a day? Over time, we would anyways have thousands of runs for that flow. 

bq. I had raised a JIRA to limit flow runs within a flow. We should probably have that support then.
Hmm. Is that YARN-4489? In general, it is good idea to limit the number of records being returned, regardless of what is being returned, be it flows, entities, applications, anything really. Unless the user explicitly asks for more or disables the limit by setting it to say -1, we should consider having a limit on payload being returned. 




was (Author: vrushalic):

bq. How many do we expect typically ? Can it run into thousands ? 
So, let's see. Say, something is running every 5 mins on the cluster and let's say it completes in less than 5 mins, meaning we have at least one hadoop job every 5 mins, then that is 60 * 24 / 5 = 288. So much less than a thousand. I think it's reasonable to think it won't be a thousand runs in a day, unless someone is being malicious (triggering a hadoop job every min is 1440 runs). 

That said, what do we think might break or be a problem if we do end up having >1000 runs in one flow in a day? Over time, we would have thousands of runs for that flow. 

bq. I had raised a JIRA to limit flow runs within a flow. We should probably have that support then.
Hmm. Is that YARN-4489? In general, it is good idea to limit the number of records being returned, regardless of what is being returned, be it flows, entities, applications, anything really. Unless the user explicitly asks for more or disables the limit by setting it to say -1, we should consider having a limit on payload being returned. 



> Support fromId for flows API 
> -----------------------------
>
>                 Key: YARN-6027
>                 URL: https://issues.apache.org/jira/browse/YARN-6027
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Rohith Sharma K S
>            Assignee: Rohith Sharma K S
>              Labels: yarn-5355-merge-blocker
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar filter for flows/flowRun apps and flow run and flow as well. 
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org