[jira] [Commented] (KYLIN-2170) Mapper/Reducer cleanup() exception handling

["liyang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KYLIN-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15650446#comment-15650446 ] 

liyang commented on KYLIN-2170:
-------------------------------

Mapper/Reducer is doomed by a classic exception handling problem. E.g. the below Reducer code, if both reduce() and cleanup() throws exceptions, A will be discarded and only B is reported, losing a root cause. Some suggest never throw exception in cleanup() (in the finally block), then job can falsely succeed when there is no A and only B occurs.

Seems the only solution is log-and-throw in both map()/reduce() and cleanup(), then in all cases, all exceptions are logged and job will fail on error.

We are refactoring {{KylinMapper}} and {{KylinReducer}} to enforce this design.

{code}
  public void run(Context context) throws IOException, InterruptedException {
    setup(context);
    try {
      while (context.nextKey()) {
        reduce(context.getCurrentKey(), context.getValues(), context);  // <======= Exception A
        // If a back up store is used, reset it
        Iterator<VALUEIN> iter = context.getValues().iterator();
        if(iter instanceof ReduceContext.ValueIterator) {
          ((ReduceContext.ValueIterator<VALUEIN>)iter).resetBackupStore();        
        }
      }
    } finally {
      cleanup(context); // <====== Exception B
    }
  }
{code}

> Mapper/Reducer cleanup() exception handling
> -------------------------------------------
>
>                 Key: KYLIN-2170
>                 URL: https://issues.apache.org/jira/browse/KYLIN-2170
>             Project: Kylin
>          Issue Type: Improvement
>            Reporter: liyang
>
> Or it could override the real exception happened in mapper() or reducer()



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)