You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "Zhang, Xiuzhu(AWF)" <xi...@paypal.com.INVALID> on 2017/12/28 09:26:06 UTC

Gobblin metrics

Hi guys,

Gobblin metrics in table gobblin_job_metrics will produce metrics info. as follows:
Metric_name                             metric_type                         metric_value
JOB.job_xxx_1514443993201.bytes         COUNTER                             8459988
JOB.job_xxx_1514443993201.records       COUNTER                             6
TASK.task_xxx.bytes                     COUNTER                             8459988
TASK.task_xxx.records                   COUNTER                             6

Same data structure in gobblin_task_metrics.

Who know what's the read info. and what's the write info, why 'JOB.job_xxx_1514443993201.bytes' 'JOB.job_xxx_1514443993201.records' placed in the gobblin_task_metrics(Does it focus tasks)?
Could I get the task record count of reading and writing?

I find code segment to insert it but don't know where to get the original values.
// Insert or update job metrics
      if (jobExecutionInfo.hasMetrics()) {
        for (Metric metric : jobExecutionInfo.getMetrics()) {
          boolean insert =
              !existsMetric(connection, JOB_METRIC_EXIST_QUERY_STATEMENT_TEMPLATE,  jobExecutionInfo.getJobId(), metric);
          updateMetric(connection, insert ? JOB_METRIC_INSERT_STATEMENT_TEMPLATE : JOB_METRIC_UPDATE_STATEMENT_TEMPLATE,
              jobExecutionInfo.getJobId(), metric, insert);
        }
      }

Thanks,
Ethan

Re: Gobblin metrics

Posted by Joel Baranick <jb...@apache.org>.
Ethan,

The metric insertion code isn’t quite right.  They get placed into the properties table, rather than the metrics table in the Gobblin history DB.

--
Joel Baranick
jbaranick@apache.org



Sent from my iPhone
> On Dec 28, 2017, at 4:26 AM, Zhang, Xiuzhu(AWF) <xi...@paypal.com> wrote:
> 
> Hi guys,
>  
> Gobblin metrics in table gobblin_job_metrics will produce metrics info. as follows:
> Metric_name                             metric_type                         metric_value                              
> JOB.job_xxx_1514443993201.bytes         COUNTER                             8459988
> JOB.job_xxx_1514443993201.records       COUNTER                             6
> TASK.task_xxx.bytes                     COUNTER                             8459988
> TASK.task_xxx.records                   COUNTER                             6
>  
> Same data structure in gobblin_task_metrics.
>  
> Who know what’s the read info. and what’s the write info, why ‘JOB.job_xxx_1514443993201.bytes’ ‘JOB.job_xxx_1514443993201.records’ placed in the gobblin_task_metrics(Does it focus tasks)?     
> Could I get the task record count of reading and writing?
>  
> I find code segment to insert it but don’t know where to get the original values.
> // Insert or update job metrics
>       if (jobExecutionInfo.hasMetrics()) {
>         for (Metric metric : jobExecutionInfo.getMetrics()) {
>           boolean insert =
>               !existsMetric(connection, JOB_METRIC_EXIST_QUERY_STATEMENT_TEMPLATE,  jobExecutionInfo.getJobId(), metric);
>           updateMetric(connection, insert ? JOB_METRIC_INSERT_STATEMENT_TEMPLATE : JOB_METRIC_UPDATE_STATEMENT_TEMPLATE,
>               jobExecutionInfo.getJobId(), metric, insert);
>         }
>       }
>  
> Thanks,
> Ethan

Re: Gobblin metrics

Posted by Joel Baranick <jb...@apache.org>.
Ethan,

The metric insertion code isn’t quite right.  They get placed into the properties table, rather than the metrics table in the Gobblin history DB.

--
Joel Baranick
jbaranick@apache.org



Sent from my iPhone
> On Dec 28, 2017, at 4:26 AM, Zhang, Xiuzhu(AWF) <xi...@paypal.com> wrote:
> 
> Hi guys,
>  
> Gobblin metrics in table gobblin_job_metrics will produce metrics info. as follows:
> Metric_name                             metric_type                         metric_value                              
> JOB.job_xxx_1514443993201.bytes         COUNTER                             8459988
> JOB.job_xxx_1514443993201.records       COUNTER                             6
> TASK.task_xxx.bytes                     COUNTER                             8459988
> TASK.task_xxx.records                   COUNTER                             6
>  
> Same data structure in gobblin_task_metrics.
>  
> Who know what’s the read info. and what’s the write info, why ‘JOB.job_xxx_1514443993201.bytes’ ‘JOB.job_xxx_1514443993201.records’ placed in the gobblin_task_metrics(Does it focus tasks)?     
> Could I get the task record count of reading and writing?
>  
> I find code segment to insert it but don’t know where to get the original values.
> // Insert or update job metrics
>       if (jobExecutionInfo.hasMetrics()) {
>         for (Metric metric : jobExecutionInfo.getMetrics()) {
>           boolean insert =
>               !existsMetric(connection, JOB_METRIC_EXIST_QUERY_STATEMENT_TEMPLATE,  jobExecutionInfo.getJobId(), metric);
>           updateMetric(connection, insert ? JOB_METRIC_INSERT_STATEMENT_TEMPLATE : JOB_METRIC_UPDATE_STATEMENT_TEMPLATE,
>               jobExecutionInfo.getJobId(), metric, insert);
>         }
>       }
>  
> Thanks,
> Ethan