[jira] [Comment Edited] (YARN-5585) [Atsv2] Add a new filter fromId in REST endpoints

["Varun Saxena (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15545818#comment-15545818 ] 

Varun Saxena edited comment on YARN-5585 at 10/4/16 4:06 PM:
-------------------------------------------------------------

Thanks [~rohithsharma] for the patch. Few comments.

# Intention behind having ID_PREFIX in EntityColumn ? According to me, we need not store prefix in the column. Is it because we want to read it back and send it to client ?
# No need of GenericEntityReader#calculateTheClosestNextRowKeyForPrefix. Scan#setRowPrefixFilter will do it for you. We should call it the same way as was done previously.
# As entity ID prefix is a long, EntityRowKeyConverter#SEGMENT_SIZES should have new segment as Bytes.SIZEOF_LONG. It is currently given as VARIABLE_SIZE. Same change in TestRowKeys.
# In EntityRowKeyConverter#encode, no need to invert entity id prefix. We will take prefix as-is. Sender can publish the entity with inverted prefix if he wants contents in descending order (say). We can probably add something to TimelineUtils to invert it, if required, which then clients can use.
# In GenericEntityReader#parseEntity we should fetch id prefix from result set and setIdPrefix in TimelineEntity to be returned back. This will be useful for clients when they want to set fromPrefix (will be useful in Tez UI use case).
# Javadoc in TimelineReader should be changed. It currently says entities would be sorted by created time which is no longer true.
{code}
   * @return A set of <cite>TimelineEntity</cite> instances of the given entity
   *    type in the given context scope which matches the given predicates
   *    ordered by created time, descending. Each entity will only contain the
   *    metadata(id, type and created time) plus the given fields to retrieve.
{code}
# We should also update documentation to reflect id prefix.





was (Author: varun_saxena):
Thanks [~rohithsharma] for the patch. Few comments.

# Intention behind having ID_PREFIX in EntityColumn ? According to me, we need not store prefix in the column. Is it because we want to read it back and send it to client ?
# No need of GenericEntityReader#calculateTheClosestNextRowKeyForPrefix. Scan#setRowPrefixFilter will do it for you. We should call it the same way as was done previously.
# As entity ID prefix is a long, EntityRowKeyConverter#SEGMENT_SIZES should have new segment as Bytes.SIZEOF_LONG. Same change in TestRowKeys.
# In EntityRowKeyConverter#encode, no need to invert entity id prefix. We will take prefix as-is. Sender can publish the entity with inverted prefix if he wants contents in descending order (say). We can probably add something to TimelineUtils to invert it, if required, which then clients can use.
# In GenericEntityReader#parseEntity we should fetch id prefix from result set and setIdPrefix in TimelineEntity to be returned back. This will be useful for clients when they want to set fromPrefix (will be useful in Tez UI use case).
# Javadoc in TimelineReader should be changed. It currently says entities would be sorted by created time which is no longer true.
{code}
   * @return A set of <cite>TimelineEntity</cite> instances of the given entity
   *    type in the given context scope which matches the given predicates
   *    ordered by created time, descending. Each entity will only contain the
   *    metadata(id, type and created time) plus the given fields to retrieve.
{code}
# We should also update documentation to reflect id prefix.




> [Atsv2] Add a new filter fromId in REST endpoints
> -------------------------------------------------
>
>                 Key: YARN-5585
>                 URL: https://issues.apache.org/jira/browse/YARN-5585
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelinereader
>            Reporter: Rohith Sharma K S
>            Assignee: Rohith Sharma K S
>            Priority: Critical
>         Attachments: 0001-YARN-5585.patch, YARN-5585-workaround.patch, YARN-5585.v0.patch
>
>
> TimelineReader REST API's provides lot of filters to retrieve the applications. Along with those, it would be good to add new filter i.e fromId so that entities can be retrieved after the fromId. 
> Current Behavior : Default limit is set to 100. If there are 1000 entities then REST call gives first/last 100 entities. How to retrieve next set of 100 entities i.e 101 to 200 OR 900 to 801?
> Example : If applications are stored database, app-1 app-2 ... app-10.
> *getApps?limit=5* gives app-1 to app-5. But to retrieve next 5 apps, there is no way to achieve this. 
> So proposal is to have fromId in the filter like *getApps?limit=5&&fromId=app-5* which gives list of apps from app-6 to app-10. 
> Since ATS is targeting large number of entities storage, it is very common use case to get next set of entities using fromId rather than querying all the entites. This is very useful for pagination in web UI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org