[jira] [Assigned] (GOBBLIN-128) Reporting metrics on Gobblin Yarn causes App_Master to crash.

["Abhishek Tiwari (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/GOBBLIN-128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Abhishek Tiwari reassigned GOBBLIN-128:
---------------------------------------

       Assignee: Abhishek Tiwari
    Component/s: gobblin-yarn

> Reporting metrics on Gobblin Yarn causes App_Master to crash.
> -------------------------------------------------------------
>
>                 Key: GOBBLIN-128
>                 URL: https://issues.apache.org/jira/browse/GOBBLIN-128
>             Project: Apache Gobblin
>          Issue Type: Bug
>          Components: gobblin-yarn
>            Reporter: Joel Baranick
>            Assignee: Abhishek Tiwari
>              Labels: Bug:Generic, LaunchType:Yarn
>
> @zliu41 @sahilTakiar @liyinan926 As far as I can tell, there are major issues around metrics and running gobblin under yarn:
> ### 1. Reporter Leak - SEVERE
> _Metric reporters are constantly added to the system and never removed._
> On the app_master when the Quartz timer fires of a job, `AbstractJobLauncher.launchJob(JobListener)` is called.  Right away, this calls `GobblinMetrics.startMetricReporting(Properties)`.  This creates new file, jmx, kafka, and custom reporters.  When these reporters are created, they are added to the collection of reporters stored in the `RootMetricContext`.  Later, when the job finished, it calls `JobMetrics.stopMetricsReporting()`.  This ends up calling `RootMetricContext.stopReporting()` which stops, but **does not remove** all reporters.  Additionally, `GobblinMetrics.metricsReportingStarted` is set to `false`.  This means that a subsequent call to `GobblinMetrics.startMetricReporting(Properties)` will cause the new reporters to be created.
> The net result is that the reporter collection in RootMetricContext has unbounded growth, causes excessive numbers of threads to be created, and depending on the reporter can cause the app_master to crash.
> ### 2. Metric Gaps - MODERATE
> _The way reporting is started and stopped can result in gaps in metric reporting._
> On the app_master when the Quartz timer fires of a job, `AbstractJobLauncher.launchJob(JobListener)` is called.  Right away, this calls `GobblinMetrics.startMetricReporting(Properties)`.  This calls `RootMetricContext.startReporting()`, which starts are associated reporters.  Later, when the job is done it calls `GobblinMetrics.stopMetricReporting()`, which calls `RootMetricContext.stopReporting()` method that stops all associated reporters.  This means that if the app_master is running two jobs, A & B, that the following scenario will cause a metric gap for Job B:
> | Time | Job | Action |
> | --- | --- | --- |
> | T | A | Start |
> | T + 1 | B | Start |
> | T + 2 | A | Finish |
> | T + 3 | B | **No Metrics Reported** |
> | T + N | B | Finish |
> ### 3. Invalid Data - MILD
> _All file reporters are assigned all contexts, which will result in the wrong data in the files._
> File metric reporters write to files with filenames which are specific to the job.  Unfortunately, the list of contexts to which the reporters listen is not scoped to a given job.  This means that files may contain metrics from many different jobs.
>  
> *Github Url* : https://github.com/linkedin/gobblin/issues/792 
> *Github Reporter* : [~jbaranick] 
> *Github Assignee* : [~stakiar] 
> *Github Created At* : 2016-03-03T18:19:18Z 
> *Github Updated At* : 2016-03-14T17:01:45Z 
> h3. Comments 
> ----
> [~stakiar] wrote on 2016-03-03T20:02:29Z : @kadaan thanks a ton for reporting and writing all this up. From what I can tell these all seem to be pretty serious issues.
> It seems there may have a disconnect between the job execution model on YARN and how gobblin-metrics starts, stops, and manages metrics.
> I will bring this up with the team today and see if we can create a plan of action.
>  
>  
> *Github Url* : https://github.com/linkedin/gobblin/issues/792#issuecomment-191939044 
> ----
> [~liyinan926] wrote on 2016-03-03T20:06:30Z : Thanks, @sahilTakiar for calling this out. Yes, I agree that the Yarn execution model and even the MR execution model have a disconnect with gobblin-metrics, which is really more designed and implemented for single JVM usage because of its base on the Dropwizard metrics lib. 
>  
>  
> *Github Url* : https://github.com/linkedin/gobblin/issues/792#issuecomment-191940470 
> ----
> [~stakiar] wrote on 2016-03-07T17:40:13Z : I am actively working on this. High level design is to introduce a `ApplicationLauncher` that can `start` and `stop` an application. A `BaseApplicationMaster` will control starting and stopping of metrics reporting as well as a set of core `Service`s (`JMXReporter`, etc.) and pluggable `Service`s (`JobScheduler`, `YarnService`).
>  
>  
> *Github Url* : https://github.com/linkedin/gobblin/issues/792#issuecomment-193364481 
> ----
> [~jbaranick] wrote on 2016-03-07T17:46:16Z : @sahilTakiar Great!  That is kinda how I was thinking it should work.  BTW, at least one of the reporters is job specific, FileReporter.  The reason it is job specific is because it writes to a different file per job.  To me this seems to indicate that there are either:
> 1. Two different kinds of reporters, global and local.  Globals can be added at application startup, while local can be added at job startup.
> 2. The file reporter is global and keeps track of all open files.  The file names are determined by context.
> How are you thinking about this problem?
> How are you taking care of item 3?
>  
>  
> *Github Url* : https://github.com/linkedin/gobblin/issues/792#issuecomment-193366501 
> ----
> [~stakiar] wrote on 2016-03-07T19:31:57Z : Thanks @kadaan! I need to think about item number 3 a little more. I think the work will be orthogonal to the changes for introducing the `ApplicationLauncher` so I am going to address them in different PRs.
> Yes, the notes you listed sounds reasonable. I will circle back with this Issue once I have a better idea of what to do.
>  
>  
> *Github Url* : https://github.com/linkedin/gobblin/issues/792#issuecomment-193412937 
> ----
> [~stakiar] wrote on 2016-03-14T17:01:44Z : Issues 1 and 2 are resolved, issue 3 is still open.
>  
>  
> *Github Url* : https://github.com/linkedin/gobblin/issues/792#issuecomment-196412229



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)