You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Chamikara Jayalath (JIRA)" <ji...@apache.org> on 2017/09/07 18:03:00 UTC
[jira] [Commented] (BEAM-2858) temp file garbage collection in
BigQuery sink should be in a separate DoFn
[ https://issues.apache.org/jira/browse/BEAM-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16157354#comment-16157354 ]
Chamikara Jayalath commented on BEAM-2858:
------------------------------------------
I tried intentionally deleting one of the files before running a load job.
Dataflow job: 2017-09-07_10_47_31-9640870329732038724
BigQuery load job: f0953eede88e454bb3b01d2cdba3c10d_1a2e1a938a117f8e3ff2af5ee4f58b45_00001_00000-0
This Dataflow job succeeded. So looks like this issue will manifest as a dataloss.
Reuven, feel free to grab this issue if you are hoping to produce a fix otherwise I'll look into this early next week.
> temp file garbage collection in BigQuery sink should be in a separate DoFn
> --------------------------------------------------------------------------
>
> Key: BEAM-2858
> URL: https://issues.apache.org/jira/browse/BEAM-2858
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-gcp
> Affects Versions: 2.1.0
> Reporter: Reuven Lax
> Assignee: Chamikara Jayalath
> Fix For: 2.2.0
>
>
> Currently the WriteTables transform deletes the set of input files as soon as the load() job completes. However this is incorrect - if the task fails partially through deleting files (e.g. if the worker crashes), the task will be retried. If the write disposition is WRITE_TRUNCATE, bad things could result.
> The resulting behavior will depend on what BQ does if one of input files is missing (because we had previously deleted it). In the best case, BQ will fail the load. In this case the step will keep failing until the runner finally fails the entire job. If however BQ ignores the missing file, the load will overwrite the previously-written table with the smaller set of files and the job will succeed. This is the worst-case scenario, as it will result in data loss.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)