[jira] [Commented] (YARN-4062) Add the flush and compaction functionality via coprocessors and scanners for flow run table

["Joep Rottinghuis (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15143909#comment-15143909 ] 

Joep Rottinghuis commented on YARN-4062:
----------------------------------------

Been slow to make progress on review mainly due to other work taking away attention.
I think that in general the patch will work as written.
While going through the design again from the top down I noticed (and discussed with [~vrushalic] the following things:
- SUM is an aggregation operation that sums the latest value of each app in a flow(run) (or the latest value of each aggregation dimension in the higher level aggregation).
- The current MIN and MAX are actually different things. They are global mins and global maxes in the sense that they keep only the lowest (or highest) value we've ever seen by any app in the flow(run). While this is a totally valid thing to do, there is actually something like a MIN and MAX value for each app in a flow as well. What we currently call MIN and MAX should probably be called GLOBAL_MIN and GLOBAL_MAX (or something similar). We can then also have a min and max that work similar in keeping the latest value for each app (aggregation dimension in general) and then computes the MIN and MAX at read-time. The flush compaction then works the same for MIN, MAX, and SUM, while for GLOBAL_MIN, and GLOBAL_MAX we can keep the current code behavior of shedding values as we go.

The GLOBAL_MIN and max are appropriate for the existing use-case of min start time, but also for gauges. The new MIN and MAX would be appropriate to answer questions such as, what is the app with smallest number of mappers in this flow, or rather what is that #?

With this realization also came the awareness that SUM_FINAL is really a different thing then SUM, MIN, MAX, GLOBAL_MIN and GLOBAL_MAX, despite what I had earlier thought (and suggested). The former "this is the final value" is something that has to happen at write time. It has to come from the writer itself as an argument. Ideally the latter set of aggregation dimensions SUM, MIN, MAX, etc. are really set of a per-column level and shouldn't be passed from the client, but be instrumented by the ColumnHelper infrastructure instead. We should probably use a different tag value for that.
Both aggregation dimension and this "FINAL_VALUE" or whatever abbreviation we use are needed to determine the right thing to do for compaction. Only one value needs to have this final value bit / tag set.

I'll continue to try to document all of these things so that it is a bit easier to see visually what is going on.

> Add the flush and compaction functionality via coprocessors and scanners for flow run table
> -------------------------------------------------------------------------------------------
>
>                 Key: YARN-4062
>                 URL: https://issues.apache.org/jira/browse/YARN-4062
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Vrushali C
>            Assignee: Vrushali C
>              Labels: yarn-2928-1st-milestone
>         Attachments: YARN-4062-YARN-2928.1.patch, YARN-4062-feature-YARN-2928.01.patch, YARN-4062-feature-YARN-2928.02.patch
>
>
> As part of YARN-3901, coprocessor and scanner is being added for storing into the flow_run table. It also needs a flush & compaction processing in the coprocessor and perhaps a new scanner to deal with the data during flushing and compaction stages. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)