You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/12/14 01:12:58 UTC
[jira] [Commented] (BEAM-1151) BigQueryIO.Write has no way of handling failures

    [ https://issues.apache.org/jira/browse/BEAM-1151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15746824#comment-15746824 ] 

ASF GitHub Bot commented on BEAM-1151:
--------------------------------------

GitHub user reuvenlax opened a pull request:

    https://github.com/apache/incubator-beam/pull/1609

    BEAM-1151 Add failure handling to BigQueryIO.Write

    Allow a pipeline to handle streaming insert failures when using BigQueryIO.Write in streaming mode. TableRow objects that fail to insert are returned to the user as a PCollection.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/reuvenlax/incubator-beam bq_dead_letter

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-beam/pull/1609.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1609
    
----
commit 379af6129f91ffb1b1b3aa917f3e0f13cb412132
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date:   2016-11-17T18:57:41Z

    Add dead-letter support to BigQueryIO
    
    Add dead-letter support to BigQueryIO
    
    Strip out non-testing changes.

commit 8b2f3010058674a5ed4359148397296b7aad31de
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date:   2016-11-19T23:09:10Z

    Remove dead code.

commit 4cf5edf61d29466811cb3487561d02f51bbdfd5a
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date:   2016-11-21T20:30:29Z

    Add testing coverage for BigQuery streaming writes.

commit fa33d7621907f9c7e37a769b530514eb44a4c10a
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date:   2016-11-21T21:52:01Z

    Fix checkstyle errors and remove remaining tests out of BigQueryTableInserterTest.

commit a91ac51f3f127921e02c64c4f7c8e2c1c4828eaf
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date:   2016-11-21T22:32:03Z

    Fix more checkstyle errors.

commit e31b593bd2f8d007efeeb14fd30145bc56fff01a
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date:   2016-11-26T12:55:28Z

    Make sure to clear static cache of created tables between tests.

commit 2011433e201c983575208d1353f4221e31b44b0e
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date:   2016-12-09T20:01:10Z

    Refactor tryCreateTable method. Removes unused create and write disposition code on this path.

commit d5b549c24870bcd6593dc6616360d6c9565381d2
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date:   2016-12-10T03:39:02Z

    Fix test failure. Remove no-longer applicable test.

commit 98febf780981ec978fef9a5a9392a2e605e74326
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date:   2016-12-13T00:35:38Z

    Apply review comments.

commit 9e8cadbff8ac7fe5f5fc3d4a6c07221316144ef4
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date:   2016-12-13T05:28:42Z

    Fix failures

commit 1f840efe0a71469fde3db37fba910c296ce0d76c
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date:   2016-12-13T08:27:12Z

    Fix broken test

commit 2a446a508822da167162a022448d3d5f347287f9
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date:   2016-12-13T21:52:44Z

    Template code broke windowed BigQuery output - fix.
    Fix invalid GCS access in BigQueryIOTest

commit 14da9adec51a43e6f02672677d640a88fd42894b
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date:   2016-11-17T18:57:41Z

    Add dead-letter support to BigQueryIO

commit dfced01604d7dff5c46c99389f1f945ac5ab8995
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date:   2016-11-19T22:28:10Z

    Add dead-letter support to BigQueryIO

commit 9e66dffabaa71c7e01f3cc46736df40b163e9e82
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date:   2016-11-26T12:53:55Z

    Add some tests of dead letter policy, and default retry policies.

commit 3e3cfcf311c6db0ac97ae360c1b403d1a00cae2c
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date:   2016-12-01T08:29:58Z

    Add testing.

commit 2af0de1c84ac48bfcf4ec17af8e9d80012ff90c2
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date:   2016-12-14T00:57:48Z

    Fixes to test

----


> BigQueryIO.Write has no way of handling failures
> ------------------------------------------------
>
>                 Key: BEAM-1151
>                 URL: https://issues.apache.org/jira/browse/BEAM-1151
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-core
>    Affects Versions: 0.4.0-incubating
>            Reporter: Reuven Lax
>            Assignee: Reuven Lax
>             Fix For: 0.3.0-incubating
>
>
> BigQueryIO.Write in streaming mode uses BigQuery's streaming insert API. Inserts can fail for a variety of reasons, and the current connector provides no way of managing those failures. This PR allows better failure handling.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)