You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crunch.apache.org by "Gabriel Reid (JIRA)" <ji...@apache.org> on 2012/10/07 23:56:03 UTC
[jira] [Created] (CRUNCH-91) Enable custom output file naming
Gabriel Reid created CRUNCH-91:
----------------------------------
Summary: Enable custom output file naming
Key: CRUNCH-91
URL: https://issues.apache.org/jira/browse/CRUNCH-91
Project: Crunch
Issue Type: Improvement
Reporter: Gabriel Reid
The current output file naming behavior in Crunch is to use the classic Hadoop-style file naming (i.e. part-m-00001, part-r-00002), with the numerical part of the filename being set based on the number of existing files in the output directory to avoid naming collisions.
The intention of this issue is to allow developers to define their own output file names for Crunch output files.
The original underlying motivation for this issue is having a custom partitioner in a job which routes records to a specific partition (and therefore reducer) based on content of the record, and then needing to perform file renaming operations on the output files to allow their names to include specific information about the partition they contain. The partition number of files currently gets discarded by Crunch, making this renaming impossible. The approach proposed here (custom file naming within Crunch) goes one step further, giving developers a hook to actually define their own output file naming scheme.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-91) Enable custom output file naming
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472093#comment-13472093 ]
Josh Wills commented on CRUNCH-91:
----------------------------------
Patch looks good, I don't have any objections to the idea, although it's not a use case I've hit yet. +1.
> Enable custom output file naming
> --------------------------------
>
> Key: CRUNCH-91
> URL: https://issues.apache.org/jira/browse/CRUNCH-91
> Project: Crunch
> Issue Type: Improvement
> Reporter: Gabriel Reid
> Attachments: CRUNCH-91.patch
>
>
> The current output file naming behavior in Crunch is to use the classic Hadoop-style file naming (i.e. part-m-00001, part-r-00002), with the numerical part of the filename being set based on the number of existing files in the output directory to avoid naming collisions.
> The intention of this issue is to allow developers to define their own output file names for Crunch output files.
> The original underlying motivation for this issue is having a custom partitioner in a job which routes records to a specific partition (and therefore reducer) based on content of the record, and then needing to perform file renaming operations on the output files to allow their names to include specific information about the partition they contain. The partition number of files currently gets discarded by Crunch, making this renaming impossible. The approach proposed here (custom file naming within Crunch) goes one step further, giving developers a hook to actually define their own output file naming scheme.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CRUNCH-91) Enable custom output file naming
Posted by "Gabriel Reid (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-91?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gabriel Reid resolved CRUNCH-91.
--------------------------------
Resolution: Fixed
Fix Version/s: 0.4.0
Assignee: Gabriel Reid
Pushed to master. Next step is to make a more user-friendly wrapper around this, to allow what I described in a previous comment.
> Enable custom output file naming
> --------------------------------
>
> Key: CRUNCH-91
> URL: https://issues.apache.org/jira/browse/CRUNCH-91
> Project: Crunch
> Issue Type: Improvement
> Reporter: Gabriel Reid
> Assignee: Gabriel Reid
> Fix For: 0.4.0
>
> Attachments: CRUNCH-91.patch
>
>
> The current output file naming behavior in Crunch is to use the classic Hadoop-style file naming (i.e. part-m-00001, part-r-00002), with the numerical part of the filename being set based on the number of existing files in the output directory to avoid naming collisions.
> The intention of this issue is to allow developers to define their own output file names for Crunch output files.
> The original underlying motivation for this issue is having a custom partitioner in a job which routes records to a specific partition (and therefore reducer) based on content of the record, and then needing to perform file renaming operations on the output files to allow their names to include specific information about the partition they contain. The partition number of files currently gets discarded by Crunch, making this renaming impossible. The approach proposed here (custom file naming within Crunch) goes one step further, giving developers a hook to actually define their own output file naming scheme.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-91) Enable custom output file naming
Posted by "Gabriel Reid (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-91?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gabriel Reid updated CRUNCH-91:
-------------------------------
Attachment: CRUNCH-91.patch
Patch introduces a FileNamingScheme interface which can be provided to Path-based Targets to create custom file names. The current behavior is maintained by default. Any thoughts on this anyone?
> Enable custom output file naming
> --------------------------------
>
> Key: CRUNCH-91
> URL: https://issues.apache.org/jira/browse/CRUNCH-91
> Project: Crunch
> Issue Type: Improvement
> Reporter: Gabriel Reid
> Attachments: CRUNCH-91.patch
>
>
> The current output file naming behavior in Crunch is to use the classic Hadoop-style file naming (i.e. part-m-00001, part-r-00002), with the numerical part of the filename being set based on the number of existing files in the output directory to avoid naming collisions.
> The intention of this issue is to allow developers to define their own output file names for Crunch output files.
> The original underlying motivation for this issue is having a custom partitioner in a job which routes records to a specific partition (and therefore reducer) based on content of the record, and then needing to perform file renaming operations on the output files to allow their names to include specific information about the partition they contain. The partition number of files currently gets discarded by Crunch, making this renaming impossible. The approach proposed here (custom file naming within Crunch) goes one step further, giving developers a hook to actually define their own output file naming scheme.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-91) Enable custom output file naming
Posted by "Gabriel Reid (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472196#comment-13472196 ]
Gabriel Reid commented on CRUNCH-91:
------------------------------------
Thanks for taking a look at it Josh. I'm still feeling a bit torn on this one -- on the one hand, this (the ability to give output files meaningful names) is definitely a use case that is needed in my day-to-day work. On the other hand, I'm a bit concerned about this being a step towards putting too many bells and whistles into Crunch, as we alternatively just have a config option that allows you to keep the default output names provided by Hadoop, and leave file renaming operations up to the developer.
The really cool feature (well, I think it's cool) that I can see us being able to provide if we do go for this is to be able to have an API something like this:
// Some kind of aggregation per product
PTable<Product, PurchaseSummary> productsAndPurchaseSummaries = ...;
// Writes out the products and purchase summary, with one file per product manufacturer, and the file name
// is the name of the product manufacturer which is extracted from the Product value
pipeline.write(productsAndPurchaseSummaries, At.fanOut(outputDir, new ManufacturerExtractionFn());
Does that sway you (or anyone else) any more in one direction or the other? Obviously I want to try to do something that is useful for general use cases, and not just mine (which is currently mostly based around processing geographical data and outputting it into named files).
> Enable custom output file naming
> --------------------------------
>
> Key: CRUNCH-91
> URL: https://issues.apache.org/jira/browse/CRUNCH-91
> Project: Crunch
> Issue Type: Improvement
> Reporter: Gabriel Reid
> Attachments: CRUNCH-91.patch
>
>
> The current output file naming behavior in Crunch is to use the classic Hadoop-style file naming (i.e. part-m-00001, part-r-00002), with the numerical part of the filename being set based on the number of existing files in the output directory to avoid naming collisions.
> The intention of this issue is to allow developers to define their own output file names for Crunch output files.
> The original underlying motivation for this issue is having a custom partitioner in a job which routes records to a specific partition (and therefore reducer) based on content of the record, and then needing to perform file renaming operations on the output files to allow their names to include specific information about the partition they contain. The partition number of files currently gets discarded by Crunch, making this renaming impossible. The approach proposed here (custom file naming within Crunch) goes one step further, giving developers a hook to actually define their own output file naming scheme.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-91) Enable custom output file naming
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472204#comment-13472204 ]
Matthias Friedrich commented on CRUNCH-91:
------------------------------------------
I'm all for having better control over output file names, I need it at work occasionally, too.
> Enable custom output file naming
> --------------------------------
>
> Key: CRUNCH-91
> URL: https://issues.apache.org/jira/browse/CRUNCH-91
> Project: Crunch
> Issue Type: Improvement
> Reporter: Gabriel Reid
> Attachments: CRUNCH-91.patch
>
>
> The current output file naming behavior in Crunch is to use the classic Hadoop-style file naming (i.e. part-m-00001, part-r-00002), with the numerical part of the filename being set based on the number of existing files in the output directory to avoid naming collisions.
> The intention of this issue is to allow developers to define their own output file names for Crunch output files.
> The original underlying motivation for this issue is having a custom partitioner in a job which routes records to a specific partition (and therefore reducer) based on content of the record, and then needing to perform file renaming operations on the output files to allow their names to include specific information about the partition they contain. The partition number of files currently gets discarded by Crunch, making this renaming impossible. The approach proposed here (custom file naming within Crunch) goes one step further, giving developers a hook to actually define their own output file naming scheme.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira