You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airavata.apache.org by Sanjaya Medonsa <sa...@gmail.com> on 2013/03/25 02:43:25 UTC
Integrate Apache OODT file manager as an extension to GFac
Hi Dev,
I have implemented a new module named as oodt-integration in Airavata
which currently contains implementation which extends Apache OODT PGE task
instance to stage files into a local directory. Still I am figuring out
best way to integrate configurations. My main concern is to how to set up a
test environment to test the implementation. My plan is to follow the
following approach.
1. Set up CAS File Manager using Airavata with the help of
https://cwiki.apache.org/OODT/oodt-filemgr-user-guide.html
2. On Airavata is there any sample workflow that uses file
as an input to GFac. Please let me know what is the best sample workflow
that can be used to test my implementation.
Best Regards,
Sanjaya
Re: Integrate Apache OODT file manager as an extension to GFac
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Great to hear, Sanjaya!
Cheers,
Chris
On 3/25/13 8:15 PM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>Thanks Chris! That works. I could able to successfully ingest a file.
>
>Best Regards,
>Sanjaya
>
>On Tue, Mar 26, 2013 at 4:29 AM, Mattmann, Chris A (388J) <
>chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> Hi Sanjaya,
>>
>> Thanks. Did you create a catalog directory? If so, delete it and restart
>> your File Manager.
>> It will create this directory for you.
>>
>> Let me know and thanks!
>>
>> Cheers,
>> Chris
>>
>>
>> On 3/25/13 1:11 PM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>>
>> >Thanks Chris!
>> >
>> >I have followed the OODT File Manager user guide to start file manager
>> >server to ingest file. I couldn't able to ingest a sample file due to
>> >following exception.
>> >
>> >I couldn't able to figure out the issue by googling on the issue. I
>>have
>> >also noticed that file named 'segments' should be available in catalog
>> >directory. Any idea on possible root cause for the issue?
>> >
>> >INFO: Running Met Extractor:
>>
>>>[org.apache.oodt.cas.filemgr.metadata.extractors.examples.FinalFileLocat
>>>io
>> >nExtractor]
>> >for product type: [GenericFile]
>> >Mar 26, 2013 12:40:19 AM
>>org.apache.oodt.cas.filemgr.catalog.LuceneCatalog
>> >addCompleteProductToIndex
>> >WARNING: Unable to index product: [blah.txt]: Message: read past EOF
>> >java.io.IOException: read past EOF
>> >at
>>
>>>org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.jav
>>>a:
>> >60)
>> >at
>>
>>>org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.j
>>>av
>> >a:33)
>> >at org.apache.lucene.store.IndexInput.readInt(IndexInput.java:46)
>> >at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:47)
>> >at org.apache.lucene.index.IndexWriter$1.doBody(IndexWriter.java:264)
>> >at org.apache.lucene.store.Lock$With.run(Lock.java:99)
>> >at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:259)
>> >at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:204)
>> >at
>>
>>>org.apache.oodt.cas.filemgr.catalog.LuceneCatalog.addCompleteProductToIn
>>>de
>> >x(LuceneCatalog.java:993)
>> >at
>>
>>>org.apache.oodt.cas.filemgr.catalog.LuceneCatalog.addProductReferences(L
>>>uc
>> >eneCatalog.java:363)
>> >at
>>
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.addProductReference
>>>s(
>> >XmlRpcFileManager.java:1122)
>> >at
>>
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRp
>>>cF
>> >ileManager.java:740)
>> >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >at
>>
>>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
>>>a:
>> >39)
>> >at
>>
>>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>Im
>> >pl.java:25)
>> >at java.lang.reflect.Method.invoke(Method.java:597)
>> >at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
>> >at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
>> >at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
>> >at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
>> >at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
>> >at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
>> >at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
>> >at java.lang.Thread.run(Thread.java:662)
>> >
>> >
>> >
>> >Best Regards,
>> >Sanjaya
>> >
>> >
>> >
>> >On Mon, Mar 25, 2013 at 8:38 AM, Mattmann, Chris A (388J) <
>> >chris.a.mattmann@jpl.nasa.gov> wrote:
>> >
>> >> Hi Sanjaya,
>> >>
>> >> Great work!
>> >>
>> >> I would also have a look at:
>> >>
>> >> https://cwiki.apache.org/OODT/cas-pge-learn-by-example.html
>> >>
>> >>
>> https://cwiki.apache.org/confluence/display/OODT/OODT+Filemgr+User+Guide
>> >>
>> >>
>> >> That should help learn how to use CAS-PGE (and file manager).
>> >>
>> >> Cheers,
>> >> Chris
>> >>
>> >> On 3/24/13 6:43 PM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>> >>
>> >> >Hi Dev,
>> >> > I have implemented a new module named as oodt-integration in
>> >>Airavata
>> >> >which currently contains implementation which extends Apache OODT
>>PGE
>> >>task
>> >> >instance to stage files into a local directory. Still I am figuring
>>out
>> >> >best way to integrate configurations. My main concern is to how to
>>set
>> >>up
>> >> >a
>> >> >test environment to test the implementation. My plan is to follow
>>the
>> >> >following approach.
>> >> > 1. Set up CAS File Manager using Airavata with the
>> >>help of
>> >> >https://cwiki.apache.org/OODT/oodt-filemgr-user-guide.html
>> >> > 2. On Airavata is there any sample workflow that
>>uses
>> >>file
>> >> >as an input to GFac. Please let me know what is the best sample
>> >>workflow
>> >> >that can be used to test my implementation.
>> >> >
>> >> >Best Regards,
>> >> >Sanjaya
>> >>
>> >>
>>
>>
Re: Integrate Apache OODT file manager as an extension to GFac
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Great to hear, Sanjaya!
Cheers,
Chris
On 3/25/13 8:15 PM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>Thanks Chris! That works. I could able to successfully ingest a file.
>
>Best Regards,
>Sanjaya
>
>On Tue, Mar 26, 2013 at 4:29 AM, Mattmann, Chris A (388J) <
>chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> Hi Sanjaya,
>>
>> Thanks. Did you create a catalog directory? If so, delete it and restart
>> your File Manager.
>> It will create this directory for you.
>>
>> Let me know and thanks!
>>
>> Cheers,
>> Chris
>>
>>
>> On 3/25/13 1:11 PM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>>
>> >Thanks Chris!
>> >
>> >I have followed the OODT File Manager user guide to start file manager
>> >server to ingest file. I couldn't able to ingest a sample file due to
>> >following exception.
>> >
>> >I couldn't able to figure out the issue by googling on the issue. I
>>have
>> >also noticed that file named 'segments' should be available in catalog
>> >directory. Any idea on possible root cause for the issue?
>> >
>> >INFO: Running Met Extractor:
>>
>>>[org.apache.oodt.cas.filemgr.metadata.extractors.examples.FinalFileLocat
>>>io
>> >nExtractor]
>> >for product type: [GenericFile]
>> >Mar 26, 2013 12:40:19 AM
>>org.apache.oodt.cas.filemgr.catalog.LuceneCatalog
>> >addCompleteProductToIndex
>> >WARNING: Unable to index product: [blah.txt]: Message: read past EOF
>> >java.io.IOException: read past EOF
>> >at
>>
>>>org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.jav
>>>a:
>> >60)
>> >at
>>
>>>org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.j
>>>av
>> >a:33)
>> >at org.apache.lucene.store.IndexInput.readInt(IndexInput.java:46)
>> >at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:47)
>> >at org.apache.lucene.index.IndexWriter$1.doBody(IndexWriter.java:264)
>> >at org.apache.lucene.store.Lock$With.run(Lock.java:99)
>> >at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:259)
>> >at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:204)
>> >at
>>
>>>org.apache.oodt.cas.filemgr.catalog.LuceneCatalog.addCompleteProductToIn
>>>de
>> >x(LuceneCatalog.java:993)
>> >at
>>
>>>org.apache.oodt.cas.filemgr.catalog.LuceneCatalog.addProductReferences(L
>>>uc
>> >eneCatalog.java:363)
>> >at
>>
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.addProductReference
>>>s(
>> >XmlRpcFileManager.java:1122)
>> >at
>>
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRp
>>>cF
>> >ileManager.java:740)
>> >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >at
>>
>>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
>>>a:
>> >39)
>> >at
>>
>>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>Im
>> >pl.java:25)
>> >at java.lang.reflect.Method.invoke(Method.java:597)
>> >at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
>> >at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
>> >at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
>> >at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
>> >at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
>> >at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
>> >at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
>> >at java.lang.Thread.run(Thread.java:662)
>> >
>> >
>> >
>> >Best Regards,
>> >Sanjaya
>> >
>> >
>> >
>> >On Mon, Mar 25, 2013 at 8:38 AM, Mattmann, Chris A (388J) <
>> >chris.a.mattmann@jpl.nasa.gov> wrote:
>> >
>> >> Hi Sanjaya,
>> >>
>> >> Great work!
>> >>
>> >> I would also have a look at:
>> >>
>> >> https://cwiki.apache.org/OODT/cas-pge-learn-by-example.html
>> >>
>> >>
>> https://cwiki.apache.org/confluence/display/OODT/OODT+Filemgr+User+Guide
>> >>
>> >>
>> >> That should help learn how to use CAS-PGE (and file manager).
>> >>
>> >> Cheers,
>> >> Chris
>> >>
>> >> On 3/24/13 6:43 PM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>> >>
>> >> >Hi Dev,
>> >> > I have implemented a new module named as oodt-integration in
>> >>Airavata
>> >> >which currently contains implementation which extends Apache OODT
>>PGE
>> >>task
>> >> >instance to stage files into a local directory. Still I am figuring
>>out
>> >> >best way to integrate configurations. My main concern is to how to
>>set
>> >>up
>> >> >a
>> >> >test environment to test the implementation. My plan is to follow
>>the
>> >> >following approach.
>> >> > 1. Set up CAS File Manager using Airavata with the
>> >>help of
>> >> >https://cwiki.apache.org/OODT/oodt-filemgr-user-guide.html
>> >> > 2. On Airavata is there any sample workflow that
>>uses
>> >>file
>> >> >as an input to GFac. Please let me know what is the best sample
>> >>workflow
>> >> >that can be used to test my implementation.
>> >> >
>> >> >Best Regards,
>> >> >Sanjaya
>> >>
>> >>
>>
>>
Re: Integrate Apache OODT file manager as an extension to GFac
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Thanks Chris! That works. I could able to successfully ingest a file.
Best Regards,
Sanjaya
On Tue, Mar 26, 2013 at 4:29 AM, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Hi Sanjaya,
>
> Thanks. Did you create a catalog directory? If so, delete it and restart
> your File Manager.
> It will create this directory for you.
>
> Let me know and thanks!
>
> Cheers,
> Chris
>
>
> On 3/25/13 1:11 PM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>
> >Thanks Chris!
> >
> >I have followed the OODT File Manager user guide to start file manager
> >server to ingest file. I couldn't able to ingest a sample file due to
> >following exception.
> >
> >I couldn't able to figure out the issue by googling on the issue. I have
> >also noticed that file named 'segments' should be available in catalog
> >directory. Any idea on possible root cause for the issue?
> >
> >INFO: Running Met Extractor:
> >[org.apache.oodt.cas.filemgr.metadata.extractors.examples.FinalFileLocatio
> >nExtractor]
> >for product type: [GenericFile]
> >Mar 26, 2013 12:40:19 AM org.apache.oodt.cas.filemgr.catalog.LuceneCatalog
> >addCompleteProductToIndex
> >WARNING: Unable to index product: [blah.txt]: Message: read past EOF
> >java.io.IOException: read past EOF
> >at
> >org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:
> >60)
> >at
> >org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.jav
> >a:33)
> >at org.apache.lucene.store.IndexInput.readInt(IndexInput.java:46)
> >at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:47)
> >at org.apache.lucene.index.IndexWriter$1.doBody(IndexWriter.java:264)
> >at org.apache.lucene.store.Lock$With.run(Lock.java:99)
> >at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:259)
> >at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:204)
> >at
> >org.apache.oodt.cas.filemgr.catalog.LuceneCatalog.addCompleteProductToInde
> >x(LuceneCatalog.java:993)
> >at
> >org.apache.oodt.cas.filemgr.catalog.LuceneCatalog.addProductReferences(Luc
> >eneCatalog.java:363)
> >at
> >org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.addProductReferences(
> >XmlRpcFileManager.java:1122)
> >at
> >org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcF
> >ileManager.java:740)
> >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >at
> >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> >39)
> >at
> >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
> >pl.java:25)
> >at java.lang.reflect.Method.invoke(Method.java:597)
> >at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
> >at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
> >at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
> >at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
> >at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
> >at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
> >at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
> >at java.lang.Thread.run(Thread.java:662)
> >
> >
> >
> >Best Regards,
> >Sanjaya
> >
> >
> >
> >On Mon, Mar 25, 2013 at 8:38 AM, Mattmann, Chris A (388J) <
> >chris.a.mattmann@jpl.nasa.gov> wrote:
> >
> >> Hi Sanjaya,
> >>
> >> Great work!
> >>
> >> I would also have a look at:
> >>
> >> https://cwiki.apache.org/OODT/cas-pge-learn-by-example.html
> >>
> >>
> https://cwiki.apache.org/confluence/display/OODT/OODT+Filemgr+User+Guide
> >>
> >>
> >> That should help learn how to use CAS-PGE (and file manager).
> >>
> >> Cheers,
> >> Chris
> >>
> >> On 3/24/13 6:43 PM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
> >>
> >> >Hi Dev,
> >> > I have implemented a new module named as oodt-integration in
> >>Airavata
> >> >which currently contains implementation which extends Apache OODT PGE
> >>task
> >> >instance to stage files into a local directory. Still I am figuring out
> >> >best way to integrate configurations. My main concern is to how to set
> >>up
> >> >a
> >> >test environment to test the implementation. My plan is to follow the
> >> >following approach.
> >> > 1. Set up CAS File Manager using Airavata with the
> >>help of
> >> >https://cwiki.apache.org/OODT/oodt-filemgr-user-guide.html
> >> > 2. On Airavata is there any sample workflow that uses
> >>file
> >> >as an input to GFac. Please let me know what is the best sample
> >>workflow
> >> >that can be used to test my implementation.
> >> >
> >> >Best Regards,
> >> >Sanjaya
> >>
> >>
>
>
Re: Integrate Apache OODT file manager as an extension to GFac
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Thanks Chris! That works. I could able to successfully ingest a file.
Best Regards,
Sanjaya
On Tue, Mar 26, 2013 at 4:29 AM, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Hi Sanjaya,
>
> Thanks. Did you create a catalog directory? If so, delete it and restart
> your File Manager.
> It will create this directory for you.
>
> Let me know and thanks!
>
> Cheers,
> Chris
>
>
> On 3/25/13 1:11 PM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>
> >Thanks Chris!
> >
> >I have followed the OODT File Manager user guide to start file manager
> >server to ingest file. I couldn't able to ingest a sample file due to
> >following exception.
> >
> >I couldn't able to figure out the issue by googling on the issue. I have
> >also noticed that file named 'segments' should be available in catalog
> >directory. Any idea on possible root cause for the issue?
> >
> >INFO: Running Met Extractor:
> >[org.apache.oodt.cas.filemgr.metadata.extractors.examples.FinalFileLocatio
> >nExtractor]
> >for product type: [GenericFile]
> >Mar 26, 2013 12:40:19 AM org.apache.oodt.cas.filemgr.catalog.LuceneCatalog
> >addCompleteProductToIndex
> >WARNING: Unable to index product: [blah.txt]: Message: read past EOF
> >java.io.IOException: read past EOF
> >at
> >org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:
> >60)
> >at
> >org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.jav
> >a:33)
> >at org.apache.lucene.store.IndexInput.readInt(IndexInput.java:46)
> >at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:47)
> >at org.apache.lucene.index.IndexWriter$1.doBody(IndexWriter.java:264)
> >at org.apache.lucene.store.Lock$With.run(Lock.java:99)
> >at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:259)
> >at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:204)
> >at
> >org.apache.oodt.cas.filemgr.catalog.LuceneCatalog.addCompleteProductToInde
> >x(LuceneCatalog.java:993)
> >at
> >org.apache.oodt.cas.filemgr.catalog.LuceneCatalog.addProductReferences(Luc
> >eneCatalog.java:363)
> >at
> >org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.addProductReferences(
> >XmlRpcFileManager.java:1122)
> >at
> >org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcF
> >ileManager.java:740)
> >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >at
> >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> >39)
> >at
> >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
> >pl.java:25)
> >at java.lang.reflect.Method.invoke(Method.java:597)
> >at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
> >at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
> >at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
> >at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
> >at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
> >at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
> >at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
> >at java.lang.Thread.run(Thread.java:662)
> >
> >
> >
> >Best Regards,
> >Sanjaya
> >
> >
> >
> >On Mon, Mar 25, 2013 at 8:38 AM, Mattmann, Chris A (388J) <
> >chris.a.mattmann@jpl.nasa.gov> wrote:
> >
> >> Hi Sanjaya,
> >>
> >> Great work!
> >>
> >> I would also have a look at:
> >>
> >> https://cwiki.apache.org/OODT/cas-pge-learn-by-example.html
> >>
> >>
> https://cwiki.apache.org/confluence/display/OODT/OODT+Filemgr+User+Guide
> >>
> >>
> >> That should help learn how to use CAS-PGE (and file manager).
> >>
> >> Cheers,
> >> Chris
> >>
> >> On 3/24/13 6:43 PM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
> >>
> >> >Hi Dev,
> >> > I have implemented a new module named as oodt-integration in
> >>Airavata
> >> >which currently contains implementation which extends Apache OODT PGE
> >>task
> >> >instance to stage files into a local directory. Still I am figuring out
> >> >best way to integrate configurations. My main concern is to how to set
> >>up
> >> >a
> >> >test environment to test the implementation. My plan is to follow the
> >> >following approach.
> >> > 1. Set up CAS File Manager using Airavata with the
> >>help of
> >> >https://cwiki.apache.org/OODT/oodt-filemgr-user-guide.html
> >> > 2. On Airavata is there any sample workflow that uses
> >>file
> >> >as an input to GFac. Please let me know what is the best sample
> >>workflow
> >> >that can be used to test my implementation.
> >> >
> >> >Best Regards,
> >> >Sanjaya
> >>
> >>
>
>
Re: Integrate Apache OODT file manager as an extension to GFac
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Sanjaya,
Thanks. Did you create a catalog directory? If so, delete it and restart
your File Manager.
It will create this directory for you.
Let me know and thanks!
Cheers,
Chris
On 3/25/13 1:11 PM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>Thanks Chris!
>
>I have followed the OODT File Manager user guide to start file manager
>server to ingest file. I couldn't able to ingest a sample file due to
>following exception.
>
>I couldn't able to figure out the issue by googling on the issue. I have
>also noticed that file named 'segments' should be available in catalog
>directory. Any idea on possible root cause for the issue?
>
>INFO: Running Met Extractor:
>[org.apache.oodt.cas.filemgr.metadata.extractors.examples.FinalFileLocatio
>nExtractor]
>for product type: [GenericFile]
>Mar 26, 2013 12:40:19 AM org.apache.oodt.cas.filemgr.catalog.LuceneCatalog
>addCompleteProductToIndex
>WARNING: Unable to index product: [blah.txt]: Message: read past EOF
>java.io.IOException: read past EOF
>at
>org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:
>60)
>at
>org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.jav
>a:33)
>at org.apache.lucene.store.IndexInput.readInt(IndexInput.java:46)
>at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:47)
>at org.apache.lucene.index.IndexWriter$1.doBody(IndexWriter.java:264)
>at org.apache.lucene.store.Lock$With.run(Lock.java:99)
>at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:259)
>at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:204)
>at
>org.apache.oodt.cas.filemgr.catalog.LuceneCatalog.addCompleteProductToInde
>x(LuceneCatalog.java:993)
>at
>org.apache.oodt.cas.filemgr.catalog.LuceneCatalog.addProductReferences(Luc
>eneCatalog.java:363)
>at
>org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.addProductReferences(
>XmlRpcFileManager.java:1122)
>at
>org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcF
>ileManager.java:740)
>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>at
>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>39)
>at
>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>pl.java:25)
>at java.lang.reflect.Method.invoke(Method.java:597)
>at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
>at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
>at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
>at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
>at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
>at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
>at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
>at java.lang.Thread.run(Thread.java:662)
>
>
>
>Best Regards,
>Sanjaya
>
>
>
>On Mon, Mar 25, 2013 at 8:38 AM, Mattmann, Chris A (388J) <
>chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> Hi Sanjaya,
>>
>> Great work!
>>
>> I would also have a look at:
>>
>> https://cwiki.apache.org/OODT/cas-pge-learn-by-example.html
>>
>> https://cwiki.apache.org/confluence/display/OODT/OODT+Filemgr+User+Guide
>>
>>
>> That should help learn how to use CAS-PGE (and file manager).
>>
>> Cheers,
>> Chris
>>
>> On 3/24/13 6:43 PM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>>
>> >Hi Dev,
>> > I have implemented a new module named as oodt-integration in
>>Airavata
>> >which currently contains implementation which extends Apache OODT PGE
>>task
>> >instance to stage files into a local directory. Still I am figuring out
>> >best way to integrate configurations. My main concern is to how to set
>>up
>> >a
>> >test environment to test the implementation. My plan is to follow the
>> >following approach.
>> > 1. Set up CAS File Manager using Airavata with the
>>help of
>> >https://cwiki.apache.org/OODT/oodt-filemgr-user-guide.html
>> > 2. On Airavata is there any sample workflow that uses
>>file
>> >as an input to GFac. Please let me know what is the best sample
>>workflow
>> >that can be used to test my implementation.
>> >
>> >Best Regards,
>> >Sanjaya
>>
>>
Re: Integrate Apache OODT file manager as an extension to GFac
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Sanjaya,
Thanks. Did you create a catalog directory? If so, delete it and restart
your File Manager.
It will create this directory for you.
Let me know and thanks!
Cheers,
Chris
On 3/25/13 1:11 PM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>Thanks Chris!
>
>I have followed the OODT File Manager user guide to start file manager
>server to ingest file. I couldn't able to ingest a sample file due to
>following exception.
>
>I couldn't able to figure out the issue by googling on the issue. I have
>also noticed that file named 'segments' should be available in catalog
>directory. Any idea on possible root cause for the issue?
>
>INFO: Running Met Extractor:
>[org.apache.oodt.cas.filemgr.metadata.extractors.examples.FinalFileLocatio
>nExtractor]
>for product type: [GenericFile]
>Mar 26, 2013 12:40:19 AM org.apache.oodt.cas.filemgr.catalog.LuceneCatalog
>addCompleteProductToIndex
>WARNING: Unable to index product: [blah.txt]: Message: read past EOF
>java.io.IOException: read past EOF
>at
>org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:
>60)
>at
>org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.jav
>a:33)
>at org.apache.lucene.store.IndexInput.readInt(IndexInput.java:46)
>at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:47)
>at org.apache.lucene.index.IndexWriter$1.doBody(IndexWriter.java:264)
>at org.apache.lucene.store.Lock$With.run(Lock.java:99)
>at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:259)
>at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:204)
>at
>org.apache.oodt.cas.filemgr.catalog.LuceneCatalog.addCompleteProductToInde
>x(LuceneCatalog.java:993)
>at
>org.apache.oodt.cas.filemgr.catalog.LuceneCatalog.addProductReferences(Luc
>eneCatalog.java:363)
>at
>org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.addProductReferences(
>XmlRpcFileManager.java:1122)
>at
>org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcF
>ileManager.java:740)
>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>at
>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>39)
>at
>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>pl.java:25)
>at java.lang.reflect.Method.invoke(Method.java:597)
>at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
>at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
>at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
>at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
>at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
>at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
>at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
>at java.lang.Thread.run(Thread.java:662)
>
>
>
>Best Regards,
>Sanjaya
>
>
>
>On Mon, Mar 25, 2013 at 8:38 AM, Mattmann, Chris A (388J) <
>chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> Hi Sanjaya,
>>
>> Great work!
>>
>> I would also have a look at:
>>
>> https://cwiki.apache.org/OODT/cas-pge-learn-by-example.html
>>
>> https://cwiki.apache.org/confluence/display/OODT/OODT+Filemgr+User+Guide
>>
>>
>> That should help learn how to use CAS-PGE (and file manager).
>>
>> Cheers,
>> Chris
>>
>> On 3/24/13 6:43 PM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>>
>> >Hi Dev,
>> > I have implemented a new module named as oodt-integration in
>>Airavata
>> >which currently contains implementation which extends Apache OODT PGE
>>task
>> >instance to stage files into a local directory. Still I am figuring out
>> >best way to integrate configurations. My main concern is to how to set
>>up
>> >a
>> >test environment to test the implementation. My plan is to follow the
>> >following approach.
>> > 1. Set up CAS File Manager using Airavata with the
>>help of
>> >https://cwiki.apache.org/OODT/oodt-filemgr-user-guide.html
>> > 2. On Airavata is there any sample workflow that uses
>>file
>> >as an input to GFac. Please let me know what is the best sample
>>workflow
>> >that can be used to test my implementation.
>> >
>> >Best Regards,
>> >Sanjaya
>>
>>
Re: Integrate Apache OODT file manager as an extension to GFac
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Thanks Chris!
I have followed the OODT File Manager user guide to start file manager
server to ingest file. I couldn't able to ingest a sample file due to
following exception.
I couldn't able to figure out the issue by googling on the issue. I have
also noticed that file named 'segments' should be available in catalog
directory. Any idea on possible root cause for the issue?
INFO: Running Met Extractor:
[org.apache.oodt.cas.filemgr.metadata.extractors.examples.FinalFileLocationExtractor]
for product type: [GenericFile]
Mar 26, 2013 12:40:19 AM org.apache.oodt.cas.filemgr.catalog.LuceneCatalog
addCompleteProductToIndex
WARNING: Unable to index product: [blah.txt]: Message: read past EOF
java.io.IOException: read past EOF
at
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:60)
at
org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:33)
at org.apache.lucene.store.IndexInput.readInt(IndexInput.java:46)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:47)
at org.apache.lucene.index.IndexWriter$1.doBody(IndexWriter.java:264)
at org.apache.lucene.store.Lock$With.run(Lock.java:99)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:259)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:204)
at
org.apache.oodt.cas.filemgr.catalog.LuceneCatalog.addCompleteProductToIndex(LuceneCatalog.java:993)
at
org.apache.oodt.cas.filemgr.catalog.LuceneCatalog.addProductReferences(LuceneCatalog.java:363)
at
org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.addProductReferences(XmlRpcFileManager.java:1122)
at
org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:740)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
at java.lang.Thread.run(Thread.java:662)
Best Regards,
Sanjaya
On Mon, Mar 25, 2013 at 8:38 AM, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Hi Sanjaya,
>
> Great work!
>
> I would also have a look at:
>
> https://cwiki.apache.org/OODT/cas-pge-learn-by-example.html
>
> https://cwiki.apache.org/confluence/display/OODT/OODT+Filemgr+User+Guide
>
>
> That should help learn how to use CAS-PGE (and file manager).
>
> Cheers,
> Chris
>
> On 3/24/13 6:43 PM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>
> >Hi Dev,
> > I have implemented a new module named as oodt-integration in Airavata
> >which currently contains implementation which extends Apache OODT PGE task
> >instance to stage files into a local directory. Still I am figuring out
> >best way to integrate configurations. My main concern is to how to set up
> >a
> >test environment to test the implementation. My plan is to follow the
> >following approach.
> > 1. Set up CAS File Manager using Airavata with the help of
> >https://cwiki.apache.org/OODT/oodt-filemgr-user-guide.html
> > 2. On Airavata is there any sample workflow that uses file
> >as an input to GFac. Please let me know what is the best sample workflow
> >that can be used to test my implementation.
> >
> >Best Regards,
> >Sanjaya
>
>
Re: Integrate Apache OODT file manager as an extension to GFac
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Thanks Chris!
I have followed the OODT File Manager user guide to start file manager
server to ingest file. I couldn't able to ingest a sample file due to
following exception.
I couldn't able to figure out the issue by googling on the issue. I have
also noticed that file named 'segments' should be available in catalog
directory. Any idea on possible root cause for the issue?
INFO: Running Met Extractor:
[org.apache.oodt.cas.filemgr.metadata.extractors.examples.FinalFileLocationExtractor]
for product type: [GenericFile]
Mar 26, 2013 12:40:19 AM org.apache.oodt.cas.filemgr.catalog.LuceneCatalog
addCompleteProductToIndex
WARNING: Unable to index product: [blah.txt]: Message: read past EOF
java.io.IOException: read past EOF
at
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:60)
at
org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:33)
at org.apache.lucene.store.IndexInput.readInt(IndexInput.java:46)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:47)
at org.apache.lucene.index.IndexWriter$1.doBody(IndexWriter.java:264)
at org.apache.lucene.store.Lock$With.run(Lock.java:99)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:259)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:204)
at
org.apache.oodt.cas.filemgr.catalog.LuceneCatalog.addCompleteProductToIndex(LuceneCatalog.java:993)
at
org.apache.oodt.cas.filemgr.catalog.LuceneCatalog.addProductReferences(LuceneCatalog.java:363)
at
org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.addProductReferences(XmlRpcFileManager.java:1122)
at
org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:740)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
at java.lang.Thread.run(Thread.java:662)
Best Regards,
Sanjaya
On Mon, Mar 25, 2013 at 8:38 AM, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Hi Sanjaya,
>
> Great work!
>
> I would also have a look at:
>
> https://cwiki.apache.org/OODT/cas-pge-learn-by-example.html
>
> https://cwiki.apache.org/confluence/display/OODT/OODT+Filemgr+User+Guide
>
>
> That should help learn how to use CAS-PGE (and file manager).
>
> Cheers,
> Chris
>
> On 3/24/13 6:43 PM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>
> >Hi Dev,
> > I have implemented a new module named as oodt-integration in Airavata
> >which currently contains implementation which extends Apache OODT PGE task
> >instance to stage files into a local directory. Still I am figuring out
> >best way to integrate configurations. My main concern is to how to set up
> >a
> >test environment to test the implementation. My plan is to follow the
> >following approach.
> > 1. Set up CAS File Manager using Airavata with the help of
> >https://cwiki.apache.org/OODT/oodt-filemgr-user-guide.html
> > 2. On Airavata is there any sample workflow that uses file
> >as an input to GFac. Please let me know what is the best sample workflow
> >that can be used to test my implementation.
> >
> >Best Regards,
> >Sanjaya
>
>
Re: Integrate Apache OODT file manager as an extension to GFac
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Sanjaya,
Great work!
I would also have a look at:
https://cwiki.apache.org/OODT/cas-pge-learn-by-example.html
https://cwiki.apache.org/confluence/display/OODT/OODT+Filemgr+User+Guide
That should help learn how to use CAS-PGE (and file manager).
Cheers,
Chris
On 3/24/13 6:43 PM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>Hi Dev,
> I have implemented a new module named as oodt-integration in Airavata
>which currently contains implementation which extends Apache OODT PGE task
>instance to stage files into a local directory. Still I am figuring out
>best way to integrate configurations. My main concern is to how to set up
>a
>test environment to test the implementation. My plan is to follow the
>following approach.
> 1. Set up CAS File Manager using Airavata with the help of
>https://cwiki.apache.org/OODT/oodt-filemgr-user-guide.html
> 2. On Airavata is there any sample workflow that uses file
>as an input to GFac. Please let me know what is the best sample workflow
>that can be used to test my implementation.
>
>Best Regards,
>Sanjaya
Re: Integrate Apache OODT file manager as an extension to GFac
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Sanjaya,
Great work!
I would also have a look at:
https://cwiki.apache.org/OODT/cas-pge-learn-by-example.html
https://cwiki.apache.org/confluence/display/OODT/OODT+Filemgr+User+Guide
That should help learn how to use CAS-PGE (and file manager).
Cheers,
Chris
On 3/24/13 6:43 PM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>Hi Dev,
> I have implemented a new module named as oodt-integration in Airavata
>which currently contains implementation which extends Apache OODT PGE task
>instance to stage files into a local directory. Still I am figuring out
>best way to integrate configurations. My main concern is to how to set up
>a
>test environment to test the implementation. My plan is to follow the
>following approach.
> 1. Set up CAS File Manager using Airavata with the help of
>https://cwiki.apache.org/OODT/oodt-filemgr-user-guide.html
> 2. On Airavata is there any sample workflow that uses file
>as an input to GFac. Please let me know what is the best sample workflow
>that can be used to test my implementation.
>
>Best Regards,
>Sanjaya