You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@oodt.apache.org by "Brian Foster (JIRA)" <ji...@apache.org> on 2011/03/01 20:21:36 UTC
[jira] Created: (OODT-150) ToggleAction addition to crawler
ToggleAction addition to crawler
--------------------------------
Key: OODT-150
URL: https://issues.apache.org/jira/browse/OODT-150
Project: OODT
Issue Type: New Feature
Components: crawler
Affects Versions: 0.2
Environment: none
Reporter: Brian Foster
Assignee: Brian Foster
Priority: Minor
Fix For: 0.3
Allows for cases where there are one of two (or more) actions which needs to run. For example: sometimes the same file types have different checksum files associated with it, this allows for either case.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-150) ToggleAction addition to crawler
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001602#comment-13001602 ]
Chris A. Mattmann commented on OODT-150:
----------------------------------------
bq. the mime-type is based on the data-file, i need mime-type based on the checksum file (i sometimes get different checksum files for the same data-type from the same provider – so attaching a checksum to a data-type mime-type is not gonna cut it in my case)
I'm sorry I don't get what this means. Can you provide an example?
bq. an xml-rpc connect takes just as long as or longer md5 checksum . . . aside from waiting for resources and all the other complications this would introduce
Again rather than make claims I'd like to see some true benchmarks.
bq. changing names and paths takes like a few secs of grep replacements (and the names of the actions are pretty generic – PGPChecksum and MD5Checksum . . . and they expect the checksum to exist in a file which is created by MD5 or PGP with a given configurable extension)
Great, if you would have done that back then, we could probably have kept them around, but it didn't happen.
bq. This would be in another JIRA issue i presume?
Yeah maybe after some more discussion and/or thought on it...
> ToggleAction addition to crawler
> --------------------------------
>
> Key: OODT-150
> URL: https://issues.apache.org/jira/browse/OODT-150
> Project: OODT
> Issue Type: New Feature
> Components: crawler
> Affects Versions: 0.2
> Environment: none
> Reporter: Brian Foster
> Assignee: Brian Foster
> Priority: Minor
> Fix For: 0.3
>
> Attachments: crawler.OODT-150.2011-03-01.patch.txt
>
>
> Allows for cases where there are one of two (or more) actions which needs to run. For example: sometimes the same file types have different checksum files associated with it, this allows for either case.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-150) ToggleAction addition to crawler
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001182#comment-13001182 ]
Chris A. Mattmann commented on OODT-150:
----------------------------------------
OK that's what I thought you were proposing. I'm slightly worried about this because it sounds like you are building a mini-workflow system into the crawler, which I would not be in support of...
> ToggleAction addition to crawler
> --------------------------------
>
> Key: OODT-150
> URL: https://issues.apache.org/jira/browse/OODT-150
> Project: OODT
> Issue Type: New Feature
> Components: crawler
> Affects Versions: 0.2
> Environment: none
> Reporter: Brian Foster
> Assignee: Brian Foster
> Priority: Minor
> Fix For: 0.3
>
> Attachments: crawler.OODT-150.2011-03-01.patch.txt
>
>
> Allows for cases where there are one of two (or more) actions which needs to run. For example: sometimes the same file types have different checksum files associated with it, this allows for either case.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-150) ToggleAction addition to crawler
Posted by "Brian Foster (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001506#comment-13001506 ]
Brian Foster commented on OODT-150:
-----------------------------------
ya, but what i've got in mind is a little more involved than what crawler1 had . . . more like an auto-detect system for checksumming . . . just register a checksummer to a mime-type or whatnot and the crawler would automatically determine the checksum for the file . . . i need the crawler stuff to be instant (milliseconds) -- using the workflow manager to do checksumming is going to slow down the crawler . . . and i had a bunch of checksum stuff in the crawler2 initially (MD5, PGP, . . .) -- you make me remove them because of you wanted to keep the dependency tree small . . . i still have them on PEATE if you want them . . . i also have an example on PEATE which uses the new ExternAction to perform a checksum.
> ToggleAction addition to crawler
> --------------------------------
>
> Key: OODT-150
> URL: https://issues.apache.org/jira/browse/OODT-150
> Project: OODT
> Issue Type: New Feature
> Components: crawler
> Affects Versions: 0.2
> Environment: none
> Reporter: Brian Foster
> Assignee: Brian Foster
> Priority: Minor
> Fix For: 0.3
>
> Attachments: crawler.OODT-150.2011-03-01.patch.txt
>
>
> Allows for cases where there are one of two (or more) actions which needs to run. For example: sometimes the same file types have different checksum files associated with it, this allows for either case.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (OODT-150) ToggleAction addition to crawler
Posted by "Chris A. Mattmann (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris A. Mattmann reassigned OODT-150:
--------------------------------------
Assignee: Chris A. Mattmann (was: Brian Foster)
> ToggleAction addition to crawler
> --------------------------------
>
> Key: OODT-150
> URL: https://issues.apache.org/jira/browse/OODT-150
> Project: OODT
> Issue Type: New Feature
> Components: crawler
> Affects Versions: 0.2
> Environment: none
> Reporter: Brian Foster
> Assignee: Chris A. Mattmann
> Priority: Minor
> Fix For: 0.4
>
> Attachments: crawler.OODT-150.2011-03-01.patch.txt
>
>
> Allows for cases where there are one of two (or more) actions which needs to run. For example: sometimes the same file types have different checksum files associated with it, this allows for either case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (OODT-150) ToggleAction addition to crawler
Posted by "Brian Foster (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brian Foster updated OODT-150:
------------------------------
Attachment: crawler.OODT-150.2011-03-01.patch.txt
- attached patch
> ToggleAction addition to crawler
> --------------------------------
>
> Key: OODT-150
> URL: https://issues.apache.org/jira/browse/OODT-150
> Project: OODT
> Issue Type: New Feature
> Components: crawler
> Affects Versions: 0.2
> Environment: none
> Reporter: Brian Foster
> Assignee: Brian Foster
> Priority: Minor
> Fix For: 0.3
>
> Attachments: crawler.OODT-150.2011-03-01.patch.txt
>
>
> Allows for cases where there are one of two (or more) actions which needs to run. For example: sometimes the same file types have different checksum files associated with it, this allows for either case.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-150) ToggleAction addition to crawler
Posted by "Brian Foster (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001187#comment-13001187 ]
Brian Foster commented on OODT-150:
-----------------------------------
i hear you on that . . . if you've got a better idea for my given checksum example, i'm open for suggestions (i see it as being a pretty common case) . . . pretty much you've got a bunch of checksum actions which you want stand-alone and sometime you want some auto-detection for which checksummer to run . . . however, checksumming is a pretty standard crawling feature, maybe it is not an action at but instead should be an actual pluggin location in the crawler? ? ? just thinking out loud . . .
> ToggleAction addition to crawler
> --------------------------------
>
> Key: OODT-150
> URL: https://issues.apache.org/jira/browse/OODT-150
> Project: OODT
> Issue Type: New Feature
> Components: crawler
> Affects Versions: 0.2
> Environment: none
> Reporter: Brian Foster
> Assignee: Brian Foster
> Priority: Minor
> Fix For: 0.3
>
> Attachments: crawler.OODT-150.2011-03-01.patch.txt
>
>
> Allows for cases where there are one of two (or more) actions which needs to run. For example: sometimes the same file types have different checksum files associated with it, this allows for either case.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-150) ToggleAction addition to crawler
Posted by "Brian Foster (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001168#comment-13001168 ]
Brian Foster commented on OODT-150:
-----------------------------------
OODT-36 says: If a given action completes successfully then run a given action, if it fails then run another given action.
OODT-37 says: Run a group of actions in a given order (all must complete)
I'm proposing: Given a group of actions, that basically have their own preconditions attached to them, make sure at least one of them run. For example: i could have an action which wraps both MD5 and SHA checksuming such that depending on the file extension only one will get run, but the one which runs must complete successfully
> ToggleAction addition to crawler
> --------------------------------
>
> Key: OODT-150
> URL: https://issues.apache.org/jira/browse/OODT-150
> Project: OODT
> Issue Type: New Feature
> Components: crawler
> Affects Versions: 0.2
> Environment: none
> Reporter: Brian Foster
> Assignee: Brian Foster
> Priority: Minor
> Fix For: 0.3
>
> Attachments: crawler.OODT-150.2011-03-01.patch.txt
>
>
> Allows for cases where there are one of two (or more) actions which needs to run. For example: sometimes the same file types have different checksum files associated with it, this allows for either case.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-150) ToggleAction addition to crawler
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001133#comment-13001133 ]
Chris A. Mattmann commented on OODT-150:
----------------------------------------
Hi Brian,
How is this different from OODT-36 and OODT-37?
> ToggleAction addition to crawler
> --------------------------------
>
> Key: OODT-150
> URL: https://issues.apache.org/jira/browse/OODT-150
> Project: OODT
> Issue Type: New Feature
> Components: crawler
> Affects Versions: 0.2
> Environment: none
> Reporter: Brian Foster
> Assignee: Brian Foster
> Priority: Minor
> Fix For: 0.3
>
> Attachments: crawler.OODT-150.2011-03-01.patch.txt
>
>
> Allows for cases where there are one of two (or more) actions which needs to run. For example: sometimes the same file types have different checksum files associated with it, this allows for either case.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-150) ToggleAction addition to crawler
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001516#comment-13001516 ]
Chris A. Mattmann commented on OODT-150:
----------------------------------------
Hi Brian:
bq. just register a checksummer to a mime-type or whatnot and the crawler would automatically determine the checksum for the file
Can't you already do this with the MimeType action, that calls a checksummer?
bq. i need the crawler stuff to be instant (milliseconds) – using the workflow manager to do checksumming is going to slow down the crawler
Maybe we could actually benchmark this before we claim that it will unequivocally slow it down without having any hard # to say how much or why.
bq. and i had a bunch of checksum stuff in the crawler2 initially (MD5, PGP, . . .) – you make me remove them because of you wanted to keep the dependency tree small
Keeping the dependency tree small was one part of it. The main part was that all of your actions had a bunch of PEATE specific nomenclature in them (e.g., the policy to configure them, including environment variable refs, etc.). That was the main thing I objected to.
bq. i still have them on PEATE if you want them . . . i also have an example on PEATE which uses the new ExternAction to perform a checksum.
Yah, probably some baked in example config with the ExternAction would probably make a lot of sense. Let's think about doing that and what a patch would look like.
> ToggleAction addition to crawler
> --------------------------------
>
> Key: OODT-150
> URL: https://issues.apache.org/jira/browse/OODT-150
> Project: OODT
> Issue Type: New Feature
> Components: crawler
> Affects Versions: 0.2
> Environment: none
> Reporter: Brian Foster
> Assignee: Brian Foster
> Priority: Minor
> Fix For: 0.3
>
> Attachments: crawler.OODT-150.2011-03-01.patch.txt
>
>
> Allows for cases where there are one of two (or more) actions which needs to run. For example: sometimes the same file types have different checksum files associated with it, this allows for either case.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-150) ToggleAction addition to crawler
Posted by "Brian Foster (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001524#comment-13001524 ]
Brian Foster commented on OODT-150:
-----------------------------------
| Can't you already do this with the MimeType action, that calls a checksummer?
the mime-type is based on the data-file, i need mime-type based on the checksum file (i sometimes get different checksum files for the same data-type from the same provider -- so attaching a checksum to a data-type mime-type is not gonna cut it in my case)
| Maybe we could actually benchmark this before we claim that it will unequivocally slow it down without having any hard # to say how much or why.
an xml-rpc connect takes just as long as or longer md5 checksum . . . aside from waiting for resources and all the other complications this would introduce
| Keeping the dependency tree small was one part of it. The main part was that all of your actions had a bunch of PEATE specific nomenclature in them (e.g., the policy to configure them, including environment variable refs, etc.). That was the main thing I objected to.
changing names and paths takes like a few secs of grep replacements (and the names of the actions are pretty generic -- PGPChecksum and MD5Checksum . . . and they expect the checksum to exist in a file which is created by MD5 or PGP with a given configurable extension)
| Yah, probably some baked in example config with the ExternAction would probably make a lot of sense. Let's think about doing that and what a patch would look like.
This would be in another JIRA issue i presume?
> ToggleAction addition to crawler
> --------------------------------
>
> Key: OODT-150
> URL: https://issues.apache.org/jira/browse/OODT-150
> Project: OODT
> Issue Type: New Feature
> Components: crawler
> Affects Versions: 0.2
> Environment: none
> Reporter: Brian Foster
> Assignee: Brian Foster
> Priority: Minor
> Fix For: 0.3
>
> Attachments: crawler.OODT-150.2011-03-01.patch.txt
>
>
> Allows for cases where there are one of two (or more) actions which needs to run. For example: sometimes the same file types have different checksum files associated with it, this allows for either case.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (OODT-150) ToggleAction addition to crawler
Posted by "Chris A. Mattmann (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris A. Mattmann resolved OODT-150.
------------------------------------
Resolution: Fixed
- after all the discussion, I just ended up committing this in r1179294. Brian contributed the patch. It compiles. If someone wants to add the associated bean policy, by all means :-)
Thanks Brian!
> ToggleAction addition to crawler
> --------------------------------
>
> Key: OODT-150
> URL: https://issues.apache.org/jira/browse/OODT-150
> Project: OODT
> Issue Type: New Feature
> Components: crawler
> Affects Versions: 0.2
> Environment: none
> Reporter: Brian Foster
> Assignee: Chris A. Mattmann
> Priority: Minor
> Fix For: 0.4
>
> Attachments: crawler.OODT-150.2011-03-01.patch.txt
>
>
> Allows for cases where there are one of two (or more) actions which needs to run. For example: sometimes the same file types have different checksum files associated with it, this allows for either case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-150) ToggleAction addition to crawler
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001271#comment-13001271 ]
Chris A. Mattmann commented on OODT-150:
----------------------------------------
No worries, thanks Brian.
Checksumming used to be a plugin in Crawler1 :)
How about we build some Checksum workflows, and then have the crawler kick them off? That way we take advantage of workflow managers, put the complexity and branching/etc there, and keep the crawler simple.
I think we should probably also add some crawler checksum actions and ship them with the crawler...let's chat more about this...
> ToggleAction addition to crawler
> --------------------------------
>
> Key: OODT-150
> URL: https://issues.apache.org/jira/browse/OODT-150
> Project: OODT
> Issue Type: New Feature
> Components: crawler
> Affects Versions: 0.2
> Environment: none
> Reporter: Brian Foster
> Assignee: Brian Foster
> Priority: Minor
> Fix For: 0.3
>
> Attachments: crawler.OODT-150.2011-03-01.patch.txt
>
>
> Allows for cases where there are one of two (or more) actions which needs to run. For example: sometimes the same file types have different checksum files associated with it, this allows for either case.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OODT-150) ToggleAction addition to crawler
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris A. Mattmann updated OODT-150:
-----------------------------------
Fix Version/s: (was: 0.3)
0.4
> ToggleAction addition to crawler
> --------------------------------
>
> Key: OODT-150
> URL: https://issues.apache.org/jira/browse/OODT-150
> Project: OODT
> Issue Type: New Feature
> Components: crawler
> Affects Versions: 0.2
> Environment: none
> Reporter: Brian Foster
> Assignee: Brian Foster
> Priority: Minor
> Fix For: 0.4
>
> Attachments: crawler.OODT-150.2011-03-01.patch.txt
>
>
> Allows for cases where there are one of two (or more) actions which needs to run. For example: sometimes the same file types have different checksum files associated with it, this allows for either case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira