[jira] [Updated] (CRUNCH-479) Writing to target with WriteMode.APPEND merges values into PCollection

["Micah Whitacre (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CRUNCH-479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Micah Whitacre updated CRUNCH-479:
----------------------------------
    Attachment: CRUNCH-479.patch

Here is a test showing the behavior...

I believe the issue is that when we "write" the "materializedAt" value is set on the PCollectionImpl[1] instance that points to the directory.  So it therefore reads all of the existing values.  Not sure of the best way to fix this yet.

[1] - https://github.com/apache/crunch/blob/master/crunch-core/src/main/java/org/apache/crunch/impl/dist/collect/PCollectionImpl.java#L100

> Writing to target with WriteMode.APPEND merges values into PCollection
> ----------------------------------------------------------------------
>
>                 Key: CRUNCH-479
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-479
>             Project: Crunch
>          Issue Type: Bug
>          Components: Core
>            Reporter: Micah Whitacre
>            Assignee: Josh Wills
>         Attachments: CRUNCH-479.patch
>
>
> This was mentioned as part of CDK-617[1].  A PCollection that contains a set of values, is written to a target with WriteMode.APPEND, and then that PCollection is materialized, when you iterate over that PCollection it contains not only the new values that were appended but also the existing values.  This is surprising as most would expect that collection to only contain the original collection of values.  A use case for this might be if the solution is looking to only process the new values instead of dealing with all of the existing data.
> [1] - https://issues.cloudera.org/browse/CDK-671



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)