[jira] [Updated] (YARN-5585) [Atsv2] Reader side changes for entity prefix and support for pagination via additional filters

["Rohith Sharma K S (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YARN-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rohith Sharma K S updated YARN-5585:
------------------------------------
    Attachment: YARN-5585-YARN-5355.0002.patch

It is quite long time, apologies for this. I worked on this JIRA and tested for pagination, initial testing for pagination mode looks fine in real cluster. Attaching incremental patch(but it does not contain REST API changes), but it still required to modify few code patch based on the discussion.

There are couple of things to discuss and get a consensus
# *REST API definition* : There are set of pre-defined REST API for retrieval of entities/entity. 
*Entities* : I would like to keep same API's for pagination mode. FromId can be passed as filter where in *fromId=idPrefix!entityId*. User need to input combination idPrefix and entityId separated by *\!* or *+* symbol in filters. Reader server parses this combination and start scan from given idPrefix!entityId row key. Separator to be decided whether *\!* or any other symbols. 
*Entity* : Current entity API's are defined without idPrefix. Should we provide new API's for entityIdPrefix ? If we plan  to reuse same API's, then we need to handle one scenario where same entityId is published with 2 entityIdPrefix.  entityIdPrefix is mandatorily written even though user do not provide any idPrefix while publishing entities. So, if case of idPrefix is not known, should we use default idPrefix to get a row?
# For single entity retrieval, when IdPrefix is not known, need to match column value for entityType by doing range scan. Any other way this can achieve this?

Thoughts? cc:/ [~gtCarrera9] [~varun_saxena] [~sjlee0] [~vrushalic]

> [Atsv2] Reader side changes for entity prefix and support for pagination via additional filters
> -----------------------------------------------------------------------------------------------
>
>                 Key: YARN-5585
>                 URL: https://issues.apache.org/jira/browse/YARN-5585
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelinereader
>            Reporter: Rohith Sharma K S
>            Assignee: Rohith Sharma K S
>            Priority: Critical
>              Labels: oct16-hard
>         Attachments: 0001-YARN-5585.patch, YARN-5585-YARN-5355.0001.patch, YARN-5585-YARN-5355.0002.patch, YARN-5585-workaround.patch, YARN-5585.v0.patch
>
>
> TimelineReader REST API's provides lot of filters to retrieve the applications. Along with those, it would be good to add new filter i.e fromId so that entities can be retrieved after the fromId. 
> Current Behavior : Default limit is set to 100. If there are 1000 entities then REST call gives first/last 100 entities. How to retrieve next set of 100 entities i.e 101 to 200 OR 900 to 801?
> Example : If applications are stored database, app-1 app-2 ... app-10.
> *getApps?limit=5* gives app-1 to app-5. But to retrieve next 5 apps, there is no way to achieve this. 
> So proposal is to have fromId in the filter like *getApps?limit=5&&fromId=app-5* which gives list of apps from app-6 to app-10. 
> Since ATS is targeting large number of entities storage, it is very common use case to get next set of entities using fromId rather than querying all the entites. This is very useful for pagination in web UI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org