You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crunch.apache.org by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org> on 2012/09/10 00:25:07 UTC
[jira] [Created] (CRUNCH-59) DoFn doesnt' need both configure and
initialize methods
Vinod Kumar Vavilapalli created CRUNCH-59:
---------------------------------------------
Summary: DoFn doesnt' need both configure and initialize methods
Key: CRUNCH-59
URL: https://issues.apache.org/jira/browse/CRUNCH-59
Project: Crunch
Issue Type: Bug
Reporter: Vinod Kumar Vavilapalli
DoFN doesn't seem to need both {{public void configure(Configuration conf)}} and {{public void initialize()}}. We can do with a single API like {{initialize(Configuration)}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-59) DoFn doesnt' need both configure and
initialize methods
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456576#comment-13456576 ]
Josh Wills commented on CRUNCH-59:
----------------------------------
+1, please commit it, if only to be sure that your karma is setup properly. :)
> DoFn doesnt' need both configure and initialize methods
> -------------------------------------------------------
>
> Key: CRUNCH-59
> URL: https://issues.apache.org/jira/browse/CRUNCH-59
> Project: Crunch
> Issue Type: Bug
> Reporter: Vinod Kumar Vavilapalli
> Attachments: CRUNCH-59.txt
>
>
> DoFN doesn't seem to need both {{public void configure(Configuration conf)}} and {{public void initialize()}}. We can do with a single API like {{initialize(Configuration)}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-59) DoFn doesnt' need both configure and
initialize methods
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451711#comment-13451711 ]
Vinod Kumar Vavilapalli commented on CRUNCH-59:
-----------------------------------------------
Oh, yes, I missed the documentation. And one of the code paths too.
bq. configure() is called during the job construction process on the client side, and provides a mechanism for a DoFn to alter the configuration of an MR job before it is submitted.
Understand what the documentation refers to now. But shouldn't users instead set up the configuration via {{Pipeline.setConfiguration()}}. Any use-case that explicitly needs to overwrite what the rest of crunch-client side does?
> DoFn doesnt' need both configure and initialize methods
> -------------------------------------------------------
>
> Key: CRUNCH-59
> URL: https://issues.apache.org/jira/browse/CRUNCH-59
> Project: Crunch
> Issue Type: Bug
> Reporter: Vinod Kumar Vavilapalli
>
> DoFN doesn't seem to need both {{public void configure(Configuration conf)}} and {{public void initialize()}}. We can do with a single API like {{initialize(Configuration)}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CRUNCH-59) DoFn doesnt' need both configure and
initialize methods
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-59?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli resolved CRUNCH-59.
-------------------------------------------
Resolution: Fixed
Fix Version/s: 0.4.0
Assignee: Vinod Kumar Vavilapalli
Got the access. Just committed this to master branch.
> DoFn doesnt' need both configure and initialize methods
> -------------------------------------------------------
>
> Key: CRUNCH-59
> URL: https://issues.apache.org/jira/browse/CRUNCH-59
> Project: Crunch
> Issue Type: Bug
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.4.0
>
> Attachments: CRUNCH-59.txt
>
>
> DoFN doesn't seem to need both {{public void configure(Configuration conf)}} and {{public void initialize()}}. We can do with a single API like {{initialize(Configuration)}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-59) DoFn doesnt' need both configure and
initialize methods
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453718#comment-13453718 ]
Vinod Kumar Vavilapalli commented on CRUNCH-59:
-----------------------------------------------
Okay, tx for the explanation, Josh!
We can close this as invalid or we can rename the methods if you have any suggestions or just modify the java-docs to reflect what I learnt here.
> DoFn doesnt' need both configure and initialize methods
> -------------------------------------------------------
>
> Key: CRUNCH-59
> URL: https://issues.apache.org/jira/browse/CRUNCH-59
> Project: Crunch
> Issue Type: Bug
> Reporter: Vinod Kumar Vavilapalli
>
> DoFN doesn't seem to need both {{public void configure(Configuration conf)}} and {{public void initialize()}}. We can do with a single API like {{initialize(Configuration)}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-59) Document DoFn to clarify the usage of
configure and initialize methods
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-59?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated CRUNCH-59:
------------------------------------------
Affects Version/s: 0.3.0
Summary: Document DoFn to clarify the usage of configure and initialize methods (was: DoFn doesnt' need both configure and initialize methods)
> Document DoFn to clarify the usage of configure and initialize methods
> ----------------------------------------------------------------------
>
> Key: CRUNCH-59
> URL: https://issues.apache.org/jira/browse/CRUNCH-59
> Project: Crunch
> Issue Type: Bug
> Affects Versions: 0.3.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.4.0
>
> Attachments: CRUNCH-59.txt
>
>
> DoFN doesn't seem to need both {{public void configure(Configuration conf)}} and {{public void initialize()}}. We can do with a single API like {{initialize(Configuration)}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-59) DoFn doesnt' need both configure and
initialize methods
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456607#comment-13456607 ]
Vinod Kumar Vavilapalli commented on CRUNCH-59:
-----------------------------------------------
bq. Vinod, did you push via http or https?
https.
{code}
$ git config remote.origin.url
https://git-wip-us.apache.org/repos/asf/incubator-crunch.git
{code]
It is clearly a case of karma setup. Because if I put in a wrong password, it throws a "authentication failed" error.
> DoFn doesnt' need both configure and initialize methods
> -------------------------------------------------------
>
> Key: CRUNCH-59
> URL: https://issues.apache.org/jira/browse/CRUNCH-59
> Project: Crunch
> Issue Type: Bug
> Reporter: Vinod Kumar Vavilapalli
> Attachments: CRUNCH-59.txt
>
>
> DoFN doesn't seem to need both {{public void configure(Configuration conf)}} and {{public void initialize()}}. We can do with a single API like {{initialize(Configuration)}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-59) DoFn doesnt' need both configure and
initialize methods
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456623#comment-13456623 ]
Vinod Kumar Vavilapalli commented on CRUNCH-59:
-----------------------------------------------
bq. vinodkv, right?
Yes.
My mistake to start it here, but we should take this thread off JIRA :)
> DoFn doesnt' need both configure and initialize methods
> -------------------------------------------------------
>
> Key: CRUNCH-59
> URL: https://issues.apache.org/jira/browse/CRUNCH-59
> Project: Crunch
> Issue Type: Bug
> Reporter: Vinod Kumar Vavilapalli
> Attachments: CRUNCH-59.txt
>
>
> DoFN doesn't seem to need both {{public void configure(Configuration conf)}} and {{public void initialize()}}. We can do with a single API like {{initialize(Configuration)}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-59) DoFn doesnt' need both configure and
initialize methods
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456598#comment-13456598 ]
Vinod Kumar Vavilapalli commented on CRUNCH-59:
-----------------------------------------------
Tx. Matthias, I am fine with git.
I am getting a "remote: You are not authorized to edit this repository." on a push. Josh, can you please take a look?
Will create a Jenkins build for CRUNCH to integrate with JIRA.
> DoFn doesnt' need both configure and initialize methods
> -------------------------------------------------------
>
> Key: CRUNCH-59
> URL: https://issues.apache.org/jira/browse/CRUNCH-59
> Project: Crunch
> Issue Type: Bug
> Reporter: Vinod Kumar Vavilapalli
> Attachments: CRUNCH-59.txt
>
>
> DoFN doesn't seem to need both {{public void configure(Configuration conf)}} and {{public void initialize()}}. We can do with a single API like {{initialize(Configuration)}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-59) DoFn doesnt' need both configure and
initialize methods
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453909#comment-13453909 ]
Josh Wills commented on CRUNCH-59:
----------------------------------
My preference would be for a javadoc update. Thanks!
> DoFn doesnt' need both configure and initialize methods
> -------------------------------------------------------
>
> Key: CRUNCH-59
> URL: https://issues.apache.org/jira/browse/CRUNCH-59
> Project: Crunch
> Issue Type: Bug
> Reporter: Vinod Kumar Vavilapalli
>
> DoFN doesn't seem to need both {{public void configure(Configuration conf)}} and {{public void initialize()}}. We can do with a single API like {{initialize(Configuration)}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-59) DoFn doesnt' need both configure and
initialize methods
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456605#comment-13456605 ]
Matthias Friedrich commented on CRUNCH-59:
------------------------------------------
Vinod, did you push via http or https?
Re Jenkins: Awesome, that'll be a great help, thank you!
> DoFn doesnt' need both configure and initialize methods
> -------------------------------------------------------
>
> Key: CRUNCH-59
> URL: https://issues.apache.org/jira/browse/CRUNCH-59
> Project: Crunch
> Issue Type: Bug
> Reporter: Vinod Kumar Vavilapalli
> Attachments: CRUNCH-59.txt
>
>
> DoFN doesn't seem to need both {{public void configure(Configuration conf)}} and {{public void initialize()}}. We can do with a single API like {{initialize(Configuration)}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-59) DoFn doesnt' need both configure and
initialize methods
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456614#comment-13456614 ]
Josh Wills commented on CRUNCH-59:
----------------------------------
vinodkv, right? You were on the initial ACL list on INFRA-4883, I'll figure out what's going on.
> DoFn doesnt' need both configure and initialize methods
> -------------------------------------------------------
>
> Key: CRUNCH-59
> URL: https://issues.apache.org/jira/browse/CRUNCH-59
> Project: Crunch
> Issue Type: Bug
> Reporter: Vinod Kumar Vavilapalli
> Attachments: CRUNCH-59.txt
>
>
> DoFN doesn't seem to need both {{public void configure(Configuration conf)}} and {{public void initialize()}}. We can do with a single API like {{initialize(Configuration)}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-59) DoFn doesnt' need both configure and
initialize methods
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451687#comment-13451687 ]
Vinod Kumar Vavilapalli commented on CRUNCH-59:
-----------------------------------------------
Just to be sure, I verified all the traces of configure() method, but don't find any real uses of it.
Can work on it if you agree too, otherwise please let me know if I am missing something. Tx.
> DoFn doesnt' need both configure and initialize methods
> -------------------------------------------------------
>
> Key: CRUNCH-59
> URL: https://issues.apache.org/jira/browse/CRUNCH-59
> Project: Crunch
> Issue Type: Bug
> Reporter: Vinod Kumar Vavilapalli
>
> DoFN doesn't seem to need both {{public void configure(Configuration conf)}} and {{public void initialize()}}. We can do with a single API like {{initialize(Configuration)}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-59) DoFn doesnt' need both configure and
initialize methods
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451698#comment-13451698 ]
Josh Wills commented on CRUNCH-59:
----------------------------------
The methods are designed to be called in different places during the flow.
configure() is called during the job construction process on the client side, and provides a mechanism for a DoFn to alter the configuration of an MR job before it is submitted.
initialize() is called on the DoFn when it is executed on a Hadoop at the start of a map or reduce task.
I can see the naming being confusing, since it sounds like the DoFn is the thing that is being configured, when in actuality it is the Configuration object that is being modified by the DoFn, and am certainly open to a clearer name.
> DoFn doesnt' need both configure and initialize methods
> -------------------------------------------------------
>
> Key: CRUNCH-59
> URL: https://issues.apache.org/jira/browse/CRUNCH-59
> Project: Crunch
> Issue Type: Bug
> Reporter: Vinod Kumar Vavilapalli
>
> DoFN doesn't seem to need both {{public void configure(Configuration conf)}} and {{public void initialize()}}. We can do with a single API like {{initialize(Configuration)}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-59) DoFn doesnt' need both configure and
initialize methods
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451814#comment-13451814 ]
Josh Wills commented on CRUNCH-59:
----------------------------------
@Vinod not necessarily. Since a single Pipeline could map to multiple MR jobs, we might want to have some settings apply to only one of those jobs but not others. An example would be a DoFn that wrote records to an external data store, and thus might want to disable speculative task execution on the MR job that it was a part of, but we wouldn't necessarily want to disable speculative task execution in other MR jobs that ran as part of the overall pipeline and did not involve that DoFn.
> DoFn doesnt' need both configure and initialize methods
> -------------------------------------------------------
>
> Key: CRUNCH-59
> URL: https://issues.apache.org/jira/browse/CRUNCH-59
> Project: Crunch
> Issue Type: Bug
> Reporter: Vinod Kumar Vavilapalli
>
> DoFN doesn't seem to need both {{public void configure(Configuration conf)}} and {{public void initialize()}}. We can do with a single API like {{initialize(Configuration)}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-59) DoFn doesnt' need both configure and
initialize methods
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456546#comment-13456546 ]
Matthias Friedrich commented on CRUNCH-59:
------------------------------------------
+1. Josh or I can commit it if you want (don't know how comfortable you are with git), but I think you should have the karma to commit it yourself. BTW, there's documentation on the wiki for committer and non-committer workflows that may be helpful.
Re. CI: We have post-commit builds that run unit tests; integration tests run daily. Anybody to implement JIRA integration with Jenkins would be my personal hero, but it's not on the agenda right now (at least not on mine).
> DoFn doesnt' need both configure and initialize methods
> -------------------------------------------------------
>
> Key: CRUNCH-59
> URL: https://issues.apache.org/jira/browse/CRUNCH-59
> Project: Crunch
> Issue Type: Bug
> Reporter: Vinod Kumar Vavilapalli
> Attachments: CRUNCH-59.txt
>
>
> DoFN doesn't seem to need both {{public void configure(Configuration conf)}} and {{public void initialize()}}. We can do with a single API like {{initialize(Configuration)}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-59) DoFn doesnt' need both configure and
initialize methods
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451776#comment-13451776 ]
Matthias Friedrich commented on CRUNCH-59:
------------------------------------------
I think in the long run, we should move from Hadoop-style Template Method APIs to Java-EE-style Strategy interfaces where Configuration and other infrastructure could be passed in via dependency injection. Startup/finish shouldn't be part of the interface at all, instead, we could make the interface flexible and evolvable like Java EE does via method annotations (@Startup, @Finish, etc.).
In the end, DoFn (and MapFn, FilterFn) would only have one method which is, as far as I know, a requirement for using Lambda expressions from Java 8. In the end, client code could look as concise as Scala code.
> DoFn doesnt' need both configure and initialize methods
> -------------------------------------------------------
>
> Key: CRUNCH-59
> URL: https://issues.apache.org/jira/browse/CRUNCH-59
> Project: Crunch
> Issue Type: Bug
> Reporter: Vinod Kumar Vavilapalli
>
> DoFN doesn't seem to need both {{public void configure(Configuration conf)}} and {{public void initialize()}}. We can do with a single API like {{initialize(Configuration)}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-59) DoFn doesnt' need both configure and
initialize methods
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-59?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated CRUNCH-59:
------------------------------------------
Attachment: CRUNCH-59.txt
Simple patch - fixed javadoc and moved initalize method up so that the two methods are together. Please review/commit.
Are there any automated builds happening? And is JIRA integrated with such builds?
> DoFn doesnt' need both configure and initialize methods
> -------------------------------------------------------
>
> Key: CRUNCH-59
> URL: https://issues.apache.org/jira/browse/CRUNCH-59
> Project: Crunch
> Issue Type: Bug
> Reporter: Vinod Kumar Vavilapalli
> Attachments: CRUNCH-59.txt
>
>
> DoFN doesn't seem to need both {{public void configure(Configuration conf)}} and {{public void initialize()}}. We can do with a single API like {{initialize(Configuration)}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira