You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by Thomas Bennett <lm...@gmail.com> on 2012/03/22 15:02:28 UTC
crawler actionIds
Hi,
I can't seem to add multiple actionIds when I launch a crawler. Specifying
a single actionId seems to work.
I'm just wondering if I need to specify the actionId's in a different way.
Note I'm not using the script + abs paths to configuration files.
Cheers,
Tom
Command executed:
~$ java -Dorg.apache.oodt.cas.cli.debug=true
-Djava.util.logging.config.file=/var/kat/katconfig/static/oodt/etc/logging.properties
-Djava.ext.dirs=/usr/local/oodt/cas-crawler/lib
-Dorg.apache.oodt.cas.crawl.bean.repo=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/crawler-config.xml
-Dorg.apache.oodt.cas.cli.action.spring.config=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/cmd-line-actions.xml
-Dorg.apache.oodt.cas.cli.option.spring.config=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/cmd-line-options.xml
org.apache.oodt.cas.crawl.CrawlerLauncher --operation --launchAutoCrawler
--productPath /var/kat/data --filemgrUrl
http://127.0.0.1:9101--mimeExtractorRepo
/var/kat/katconfig/static/oodt/cas-crawler/policy/mime-extractor-map.xml
--clientTransferer
org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory
--failureDir /var/kat/failed/ --actionIds DeleteDataFile DeleteMetadataFile
Error message:
Setting property 'AutoDetectProductCrawler.mimeExtractorRepo'
Setting property 'StdProductCrawler.clientTransferer'
Setting property 'MetExtractorProductCrawler.clientTransferer'
Setting property 'AutoDetectProductCrawler.clientTransferer'
Setting property 'StdProductCrawler.filemgrUrl'
Setting property 'MetExtractorProductCrawler.filemgrUrl'
Setting property 'AutoDetectProductCrawler.filemgrUrl'
Setting property 'StdProductCrawler.actionIds'
Setting property 'MetExtractorProductCrawler.actionIds'
Setting property 'AutoDetectProductCrawler.actionIds'
Setting property 'StdProductCrawler.productPath'
Setting property 'MetExtractorProductCrawler.productPath'
Setting property 'AutoDetectProductCrawler.productPath'
Setting property 'MoveMetadataFileToFailureDir.toDir'
Setting property 'MovePushpullMetFileToFailureDir.toDir'
Setting property 'MoveDataFileToFailureDir.toDir'
Exception in thread "main" java.lang.RuntimeException:
org.apache.oodt.cas.cli.exception.CmdLineActionException: Failed to launch
crawler : No bean named 'DeleteDataFile,DeleteMetadataFile' is defined
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:191)
at org.apache.oodt.cas.crawl.CrawlerLauncher.main(CrawlerLauncher.java:36)
Caused by: org.apache.oodt.cas.cli.exception.CmdLineActionException: Failed
to launch crawler : No bean named 'DeleteDataFile,DeleteMetadataFile' is
defined
at
org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(CrawlerLauncherCliAction.java:61)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
... 1 more
Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException:
No bean named 'DeleteDataFile,DeleteMetadataFile' is defined
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:387)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getMergedLocalBeanDefinition(AbstractBeanFactory.java:968)
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:246)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:168)
at
org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:885)
at
org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFactory(CrawlerActionRepo.java:71)
at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:84)
at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:76)
at
org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(CrawlerLauncherCliAction.java:58)
... 3 more
Re: crawler actionIds
Posted by Thomas Bennett <lm...@gmail.com>.
Sorry forgot to mention that I'm using 0.4. ;-)
On 22 March 2012 16:02, Thomas Bennett <lm...@gmail.com> wrote:
> Hi,
>
> I can't seem to add multiple actionIds when I launch a crawler. Specifying
> a single actionId seems to work.
>
> I'm just wondering if I need to specify the actionId's in a different way.
> Note I'm not using the script + abs paths to configuration files.
>
> Cheers,
> Tom
>
> Command executed:
>
> ~$ java -Dorg.apache.oodt.cas.cli.debug=true
> -Djava.util.logging.config.file=/var/kat/katconfig/static/oodt/etc/logging.properties
> -Djava.ext.dirs=/usr/local/oodt/cas-crawler/lib
> -Dorg.apache.oodt.cas.crawl.bean.repo=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/crawler-config.xml
> -Dorg.apache.oodt.cas.cli.action.spring.config=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/cmd-line-actions.xml
> -Dorg.apache.oodt.cas.cli.option.spring.config=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/cmd-line-options.xml
> org.apache.oodt.cas.crawl.CrawlerLauncher --operation --launchAutoCrawler
> --productPath /var/kat/data --filemgrUrl http://127.0.0.1:9101--mimeExtractorRepo
> /var/kat/katconfig/static/oodt/cas-crawler/policy/mime-extractor-map.xml
> --clientTransferer
> org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory
> --failureDir /var/kat/failed/ --actionIds DeleteDataFile DeleteMetadataFile
>
> Error message:
> Setting property 'AutoDetectProductCrawler.mimeExtractorRepo'
> Setting property 'StdProductCrawler.clientTransferer'
> Setting property 'MetExtractorProductCrawler.clientTransferer'
> Setting property 'AutoDetectProductCrawler.clientTransferer'
> Setting property 'StdProductCrawler.filemgrUrl'
> Setting property 'MetExtractorProductCrawler.filemgrUrl'
> Setting property 'AutoDetectProductCrawler.filemgrUrl'
> Setting property 'StdProductCrawler.actionIds'
> Setting property 'MetExtractorProductCrawler.actionIds'
> Setting property 'AutoDetectProductCrawler.actionIds'
> Setting property 'StdProductCrawler.productPath'
> Setting property 'MetExtractorProductCrawler.productPath'
> Setting property 'AutoDetectProductCrawler.productPath'
> Setting property 'MoveMetadataFileToFailureDir.toDir'
> Setting property 'MovePushpullMetFileToFailureDir.toDir'
> Setting property 'MoveDataFileToFailureDir.toDir'
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.oodt.cas.cli.exception.CmdLineActionException: Failed to launch
> crawler : No bean named 'DeleteDataFile,DeleteMetadataFile' is defined
> at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:191)
> at org.apache.oodt.cas.crawl.CrawlerLauncher.main(CrawlerLauncher.java:36)
> Caused by: org.apache.oodt.cas.cli.exception.CmdLineActionException:
> Failed to launch crawler : No bean named
> 'DeleteDataFile,DeleteMetadataFile' is defined
> at
> org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(CrawlerLauncherCliAction.java:61)
> at
> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
> at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
> ... 1 more
> Caused by:
> org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean
> named 'DeleteDataFile,DeleteMetadataFile' is defined
> at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:387)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getMergedLocalBeanDefinition(AbstractBeanFactory.java:968)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:246)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:168)
> at
> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:885)
> at
> org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFactory(CrawlerActionRepo.java:71)
> at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:84)
> at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:76)
> at
> org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(CrawlerLauncherCliAction.java:58)
> ... 3 more
>
Re: crawler actionIds
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Tom,
Yep that was my question but it looks like you two had this one under control,
so no need for my help! Great work!
Cheers,
Chris
On Mar 26, 2012, at 12:32 AM, Thomas Bennett wrote:
> Hi Chris,
>
> I'm wondering if the beans are named DeleteDataFile and DeleteMetadataFile:
>> have you checked the action-beans.xml file?
>>
>
> If I understand you're question correctly, yes, the beans are
> named DeleteDataFile and DeleteMetadataFile in the action-beans.xml.
>
> Since there have been a few issues that have been resolved relating to the
> crawler cli (thanks to Brian's efforts), here's a quick summary:
>
> * MoveMetadataToFailureDir renamed MoveMetadataFileToFailureDir. see JIRA
> issue OODT-412 <https://issues.apache.org/jira/browse/OODT-412>
> * Specifying multiple actionIds on the command line. Brian has posted the
> fix (pending review) - see JIRA issue
> OODT-428<https://issues.apache.org/jira/browse/OODT-428>
> .
> * Executing boolean beans on the command line. Brian has supplied a fix
> (which I committed). See JIRA issue
> OODT-430<https://issues.apache.org/jira/browse/OODT-430>
>
> Cheers,
> Tom
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: crawler actionIds
Posted by Thomas Bennett <lm...@gmail.com>.
Hi Chris,
I'm wondering if the beans are named DeleteDataFile and DeleteMetadataFile:
> have you checked the action-beans.xml file?
>
If I understand you're question correctly, yes, the beans are
named DeleteDataFile and DeleteMetadataFile in the action-beans.xml.
Since there have been a few issues that have been resolved relating to the
crawler cli (thanks to Brian's efforts), here's a quick summary:
* MoveMetadataToFailureDir renamed MoveMetadataFileToFailureDir. see JIRA
issue OODT-412 <https://issues.apache.org/jira/browse/OODT-412>
* Specifying multiple actionIds on the command line. Brian has posted the
fix (pending review) - see JIRA issue
OODT-428<https://issues.apache.org/jira/browse/OODT-428>
.
* Executing boolean beans on the command line. Brian has supplied a fix
(which I committed). See JIRA issue
OODT-430<https://issues.apache.org/jira/browse/OODT-430>
Cheers,
Tom
Re: crawler actionIds
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Tom,
I'm wondering if the beans are named DeleteDataFile and DeleteMetadataFile:
have you checked the action-beans.xml file?
Cheers,
Chris
On Mar 22, 2012, at 7:02 AM, Thomas Bennett wrote:
> Hi,
>
> I can't seem to add multiple actionIds when I launch a crawler. Specifying
> a single actionId seems to work.
>
> I'm just wondering if I need to specify the actionId's in a different way.
> Note I'm not using the script + abs paths to configuration files.
>
> Cheers,
> Tom
>
> Command executed:
>
> ~$ java -Dorg.apache.oodt.cas.cli.debug=true
> -Djava.util.logging.config.file=/var/kat/katconfig/static/oodt/etc/logging.properties
> -Djava.ext.dirs=/usr/local/oodt/cas-crawler/lib
> -Dorg.apache.oodt.cas.crawl.bean.repo=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/crawler-config.xml
> -Dorg.apache.oodt.cas.cli.action.spring.config=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/cmd-line-actions.xml
> -Dorg.apache.oodt.cas.cli.option.spring.config=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/cmd-line-options.xml
> org.apache.oodt.cas.crawl.CrawlerLauncher --operation --launchAutoCrawler
> --productPath /var/kat/data --filemgrUrl
> http://127.0.0.1:9101--mimeExtractorRepo
> /var/kat/katconfig/static/oodt/cas-crawler/policy/mime-extractor-map.xml
> --clientTransferer
> org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory
> --failureDir /var/kat/failed/ --actionIds DeleteDataFile DeleteMetadataFile
>
> Error message:
> Setting property 'AutoDetectProductCrawler.mimeExtractorRepo'
> Setting property 'StdProductCrawler.clientTransferer'
> Setting property 'MetExtractorProductCrawler.clientTransferer'
> Setting property 'AutoDetectProductCrawler.clientTransferer'
> Setting property 'StdProductCrawler.filemgrUrl'
> Setting property 'MetExtractorProductCrawler.filemgrUrl'
> Setting property 'AutoDetectProductCrawler.filemgrUrl'
> Setting property 'StdProductCrawler.actionIds'
> Setting property 'MetExtractorProductCrawler.actionIds'
> Setting property 'AutoDetectProductCrawler.actionIds'
> Setting property 'StdProductCrawler.productPath'
> Setting property 'MetExtractorProductCrawler.productPath'
> Setting property 'AutoDetectProductCrawler.productPath'
> Setting property 'MoveMetadataFileToFailureDir.toDir'
> Setting property 'MovePushpullMetFileToFailureDir.toDir'
> Setting property 'MoveDataFileToFailureDir.toDir'
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.oodt.cas.cli.exception.CmdLineActionException: Failed to launch
> crawler : No bean named 'DeleteDataFile,DeleteMetadataFile' is defined
> at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:191)
> at org.apache.oodt.cas.crawl.CrawlerLauncher.main(CrawlerLauncher.java:36)
> Caused by: org.apache.oodt.cas.cli.exception.CmdLineActionException: Failed
> to launch crawler : No bean named 'DeleteDataFile,DeleteMetadataFile' is
> defined
> at
> org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(CrawlerLauncherCliAction.java:61)
> at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
> at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
> ... 1 more
> Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException:
> No bean named 'DeleteDataFile,DeleteMetadataFile' is defined
> at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:387)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getMergedLocalBeanDefinition(AbstractBeanFactory.java:968)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:246)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:168)
> at
> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:885)
> at
> org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFactory(CrawlerActionRepo.java:71)
> at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:84)
> at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:76)
> at
> org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(CrawlerLauncherCliAction.java:58)
> ... 3 more
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++