[jira] [Commented] (FLINK-7221) JDBCOutputFormat swallows errors on last batch

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110625#comment-16110625 ] 

ASF GitHub Bot commented on FLINK-7221:
---------------------------------------

GitHub user fhueske opened a pull request:

    https://github.com/apache/flink/pull/4459

    [FLINK-7221] [jdbc] Throw exception if execution of last JDBC batch fails.

    ## What is the purpose of the change
    
    Ensure that that an exception is thrown if the execution of the last JDBC batch fails.
    At the moment, only an INFO log is emitted which does not inform the user that data was lost.
    
    ## Brief change log
    
    * Throw exception in `JDBCOutputFormat.close()` if execution of last JDBC fails.
    * Split try-catch block in `JDBCOutputFormat.writeRecord()` into statement generation and batch execution.
    * Exception are changed from `IllegalArgumentException` to more generic `RuntimeException` because they are not caused by illegal arguments.
    * Add and fix test cases.
    
    ## Verifying this change
    
    * The fix is verify by `JDBCOutputFormatTest.testExceptionOnClose()` 
    ## Does this pull request potentially affect one of the following parts:
    
      - Dependencies (does it add or upgrade a dependency): **no**
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no**
      - The serializers: **no**
      - The runtime per-record code paths (performance sensitive): **yes**
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: **no**
    
    ## Documentation
    
      - Does this pull request introduce a new feature? **no**
      - If yes, how is the feature documented? **not applicable**
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/fhueske/flink jdbcCloseError

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/4459.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #4459
    
----
commit b32cd13fa61cc03b83b117f2be79d93cac0ad1ba
Author: Fabian Hueske <fh...@apache.org>
Date:   2017-08-01T16:08:53Z

    [FLINK-7221] [jdbc] Throw exception if execution of last JDBC batch fails.

----


> JDBCOutputFormat swallows errors on last batch
> ----------------------------------------------
>
>                 Key: FLINK-7221
>                 URL: https://issues.apache.org/jira/browse/FLINK-7221
>             Project: Flink
>          Issue Type: Bug
>          Components: Batch Connectors and Input/Output Formats
>    Affects Versions: 1.3.1
>         Environment: Java 1.8.0_131, PostgreSQL driver 42.1.3
>            Reporter: Ken Geis
>            Assignee: Fabian Hueske
>
> I have a data set with ~17000 rows that I was trying to write to a PostgreSQL table that I did not (yet) have permission on. No data was loaded, and Flink did not report any problem outputting the data set. The only indication I found of my problem was in the PostgreSQL log.
> With the default parallelism (8) and the default batch interval (5000), my batches were ~2000 rows each, so they were never executed in {{JDBCOutputFormat.writeRecord(..)}}. {{JDBCOutputFormat.close()}} does a final call on {{upload.executeBatch()}}, but if there is a problem, it is logged at INFO level and not rethrown. 
> If I decrease the batch interval to 100 or 1000, then an error is properly reported.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)