You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crunch.apache.org by "Ben Roling (JIRA)" <ji...@apache.org> on 2018/01/30 20:21:00 UTC
[jira] [Comment Edited] (CRUNCH-663) Expose Record-level File Path
to Processing Functions
[ https://issues.apache.org/jira/browse/CRUNCH-663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16345736#comment-16345736 ]
Ben Roling edited comment on CRUNCH-663 at 1/30/18 8:20 PM:
------------------------------------------------------------
The attached patch is a quick proof-of-concept. I wouldn't expect it to be merged directly. The patch has a modified WordCount examples that demonstrates leveraging this property. I should have just added a unit test to show it, but haven't done that yet. If I get feedback that the general approach is acceptable, I would certainly be happy to add one or more tests.
was (Author: ben.roling):
The attached patch is a quick proof-of-concept. I wouldn't expect it to be merged directly. The patch has a modified WordCount examples that demonstrates leveraging this property. I should have just added a unit test, to show it, but haven't done that yet. If I get feedback that the general approach is acceptable, I would certainly be happy to add one or more tests.
> Expose Record-level File Path to Processing Functions
> -----------------------------------------------------
>
> Key: CRUNCH-663
> URL: https://issues.apache.org/jira/browse/CRUNCH-663
> Project: Crunch
> Issue Type: Improvement
> Components: Core
> Reporter: Ben Roling
> Assignee: Josh Wills
> Priority: Major
> Attachments: CRUNCH-663.patch
>
>
> We have some processing pipelines where we want to know the file path that each record being processed came from. It would be nice if this could be exposed to the DoFns in our pipelines.
>
> This same desire was expressed a little over 1 year ago on the mailing list:
> [http://mail-archives.apache.org/mod_mbox/crunch-user/201611.mbox/%3CCAG-tO+Y42KRFiocg1RJT4qFcyvkPjFSfZa4z=wk34AriP4weTw@mail.gmail.com%3E]
>
> Unfortunately, that thread dead-ended.
>
> I will use the comments section and a patch to propose a simple, albeit slightly hacky solution. Another alternative would be to create a new Source that provides a PCollection<Pair<Path, Record>>, but I'm not sure of the effort it would take to create that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)