You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by Tom Barber <to...@meteorite.bi> on 2016/03/16 22:57:50 UTC
Multiple File Managers
Here's something I assume is possible but can't find on the wiki.
I know for workflow stuff you add resources to the pool, configure a
workflow manager and scale up.
If I want to actually do the "distributed" stuff though, say multiple file
managers, is this where the Grid components come in?
What I'm looking for is a setup of multiple file managers in different
locations but I'd like Joe User to query the whole lot like it was one. But
coming from single node Radix land there isn't much consumer info on the
scaling out.
Cheers
Tom
RE: Setting up filemanager with SOLR 5.5
Posted by Konstantinos Mavrommatis <km...@celgene.com>.
I created an issue and copied the relevant lines from my managed-schema file in the comments section of the issue
Let me know if this works.
Thanks
K
-----Original Message-----
From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
Sent: Thursday, March 24, 2016 6:07 PM
To: dev@oodt.apache.org
Subject: Re: Setting up filemanager with SOLR 5.5
Hi Kos,
The schema field definition in the file manager schema sound like they need a bit of an overhaul.
Are you able to file an issue against master and submit a PR?
Thanks
Lewis
On Thursday, March 24, 2016, Konstantinos Mavrommatis < kmavrommatis@celgene.com> wrote:
> Hi,
>
> I seem to have solved the issue, although I have not tested the setup
> extensively.
>
> 1. the actual url to solr needs to contain the core name. So the
> correct url that needs to be in the filemgr.properties file is "
> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8983_s
> olr_oodt_&d=CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-T2gHY95y7ZA&m=K-d7kCTHiATrIXnEOzkoLsBJQVSARmtX3WF1LeLp3S0&s=g3WzUToRlZMgmdcDjfEZbTDgwaUyAGW8el9v-bTNeGE&e= " where the last part 'oodt' corresponds to the name of the core.
> 2. The schema.xml provided with oodt is not compatible to the classes
> of this version of SOLR. Instead I setup the solr core using the
> data_driven_schema_configs method, and copied the definitions of the
> CAS.* fields in the file managed-schema.
>
> As a result I could now ingest files in SOLR but only partially. The
> first action of adding information about a document (i.e. CAS.*
> fields) was successful.
> But the second action of updating the record with additional
> information failed with errors of the type: HTTP method failed:
> HTTP/1.1 400 Bad Request
>
> In the solr.log file the error message indicated that it cannot index
> the field 'FileLocation'. Note that even simple curl commands that
> were trying to update documents by adding a single field at a time
> were failing with the same type of error. Somehow SOLR seemed
> incapable of adding new fields that were not explicitly defined in the managed-schema file.
>
> =================================================================================
> 2016-03-24 23:57:53.990 ERROR (qtp364656927-14) [ x:oodt]
> o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException:
> undefined
> field: "FileLocation"
> at
> org.apache.solr.schema.IndexSchema.getField(IndexSchema.java:1209)
> at
> org.apache.solr.update.processor.AtomicUpdateDocumentMerger.doAdd(AtomicUpdateDocumentMerger.java:133)
> at
> org.apache.solr.update.processor.AtomicUpdateDocumentMerger.merge(AtomicUpdateDocumentMerger.java:89)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.getUpdatedDocument(DistributedUpdateProcessor.java:1121)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1018)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:709)
> at
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
> at
> org.apache.solr.update.processor.AbstractDefaultValueUpdateProcessorFactory$DefaultValueUpdateProcessor.processAdd(AbstractDefaultValueUpdateProcessorFactory.java:93)
> at
> org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:250)
> at
> org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:177)
> at
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:94)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:69)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
>
> ======================================================================
> =========
>
> To resolve it I modified the managed-schema file replaced the line
> <copyField source="*" dest="_text_"/> line with the following one
> <dynamicField name="*" type="text_general" indexed="true"
> stored="true"/> Now the ingestion process works without problem.
> Since I am very new to SOLR, this may not be the correct approach but
> for the time works - if somebody has a better idea please chime in.
> Thanks
> K
>
> -----Original Message-----
> From: Tom Barber [mailto:tom.barber@meteorite.bi <javascript:;>]
> Sent: Thursday, March 24, 2016 12:37 AM
> To: dev@oodt.apache.org <javascript:;>
> Subject: Re: Setting up filemanager with SOLR 5.5
>
> Ooh, I see, 5.5! Sorry. not had my morning coffee yet. Yeah I tried
> 5.5 a while back out of curiosity rather than a use case and it
> certainly didn't work OOTB.
>
> Tom
>
> On Thu, Mar 24, 2016 at 7:32 AM, Tom Barber <tom.barber@meteorite.bi
> <javascript:;>> wrote:
>
> > I appreciate its cheating, but when you build OODT with Radix did
> > you use the Solr maven profile? We fixed it up in 0.12 and it seemed
> > to be working fine, although I wasn't the one who tested it fully.
> >
> > Tom
> >
> > On Thu, Mar 24, 2016 at 7:15 AM, Konstantinos Mavrommatis <
> > kmavrommatis@celgene.com <javascript:;>> wrote:
> >
> >> Hi,
> >> I am using oodt v 0.12
> >> Interestingly the file etc/logging.properties was not in its place
> >> although this was a clean installation on a clean, newly started
> >> Ubuntu
> >> 14.04 server on AWS (!!) I copied the file from the
> >> oodt-src/filemgr/target/classes/etc/logging.properties and set the
> >> logging level.
> >> Running the same command:
> >> At the end of the file there are following error lines
> >>
> >> INFO: Posting message:<add><doc><field
> >> name="id">93addb3e-4d02-41ca-bee6-fbb321d6c890</field><field
> >> name="CAS.ProductId">93addb3e-4d02-41ca-bee6-fbb321d6c890</field><f
> >> ie ld name="CAS.ProductName">test.txt</field><field
> >> name="CAS.ProductTypeName">GenericFile</field><field
> >> name="CAS.ProductReceivedTime">2016-03-24T07:12:01Z</field><field
> >> name="CAS.ProductTypeId">urn:oodt:GenericFile</field><field
> >> name="CAS.ProductStructure">Flat</field><field
> >> name="CAS.ProductTransferStatus">TRANSFERING</field></doc></add> to URL:
> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A898
> >> 3_
> >> solr_update&d=CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv
> >> 2h
> >> q2O6yvZ1Cs-T2gHY95y7ZA&m=RHpDT3nSOn4uWf0ifOCDn1AIwRXSw8CIY51DbEp5Lc
> >> 8& s=7cz1N0iI0WRMpROKkDmYSpLacfEzT9fKjg4N8Oh36w0&e=
> >> Mar 24, 2016 7:12:01 AM
> >> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient index
> >> SEVERE: HTTP method failed: HTTP/1.1 404 Not Found Mar 24, 2016
> >> 7:12:01 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager
> >> catalogProduct
> >> SEVERE: ingestProduct: CatalogException when adding Product:
> >> test.txt to
> >> Catalog: Message: HTTP method failed: HTTP/1.1 404 Not Found Mar
> >> 24,
> >> 2016 7:12:01 AM
> >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager
> >> ingestProductCore
> >> SEVERE: HTTP method failed: HTTP/1.1 404 Not Found Mar 24, 2016
> >> 7:12:01 AM
> >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient
> >> ingestProduct
> >> SEVERE: Failed to ingest product [ name:test.txt] :java.lang.Exception:
> >> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
> >> Error ingesting product
> >> [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb]
> >> : HTTP method failed: HTTP/1.1 404 Not Found -- rolling back ingest
> >> Mar 24, 2016 7:12:01 AM
> >> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient delete
> >> INFO: Posting message:<delete><query>id:null</query></delete> to URL:
> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A898
> >> 3_
> >> solr_update-3Fcommit-3Dtrue&d=CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wnd
> >> YZ
> >> 4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-T2gHY95y7ZA&m=RHpDT3nSOn4uWf0ifOCDn1AIwR
> >> XS w8CIY51DbEp5Lc8&s=jITvZPImZDY7_yKA18cyMffYS67ahR9Wz9IN9P1UNz0&e=
> >> Mar 24, 2016 7:12:01 AM
> >> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient delete
> >> SEVERE: HTTP method failed: HTTP/1.1 404 Not Found Mar 24, 2016
> >> 7:12:01 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager
> >> removeProduct
> >> WARNING: Exception modifying product: [null]: Message: HTTP method
> >> failed: HTTP/1.1 404 Not Found
> >> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
> >> HTTP method failed: HTTP/1.1 404 Not Found
> >> at
> >>
> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient.delete(SolrClient.
> java:130)
> >> at
> >>
> org.apache.oodt.cas.filemgr.catalog.solr.SolrCatalog.removeProduct(Sol
> rCatalog.java:165)
> >> at
> >>
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.removeProduct(Xml
> RpcFileManager.java:1113)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava:57)
> >> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> orImpl.java:43)
> >> at java.lang.reflect.Method.invoke(Method.java:606)
> >> at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
> >> at
> >> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
> >> at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
> >> at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
> >> at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
> >> at
> org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
> >> at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
> >> at java.lang.Thread.run(Thread.java:745)
> >>
> >> Mar 24, 2016 7:12:01 AM
> >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient
> >> ingestProduct
> >> SEVERE: Failed to rollback ingest of product
> >> [org.apache.oodt.cas.filemgr.structs.Product@2782f72b] :
> >> java.lang.Exception:
> >> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
> >> Error ingesting product
> >> [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb]
> >> : HTTP method failed: HTTP/1.1 404 Not Found Mar 24, 2016 7:12:01
> >> AM org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction
> >> execute
> >> SEVERE: java.lang.Exception:
> >> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
> >> Error ingesting product
> >> [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb]
> >> : HTTP method failed: HTTP/1.1 404 Not Found
> >> ERROR: Failed to ingest product 'test.txt' : java.lang.Exception:
> >> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
> >> Error ingesting product
> >> [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb]
> >> : HTTP method failed: HTTP/1.1 404 Not Found-----Original
> >> Message-----
> >>
> >>
> >>
> >> From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com
> <javascript:;>]
> >> Sent: Wednesday, March 23, 2016 7:25 PM
> >> To: dev@oodt.apache.org <javascript:;>
> >> Subject: Re: Setting up filemanager with SOLR 5.5
> >>
> >> Hi K,
> >> Which version of OODT are you using?
> >> Can you set logging to debug, restart your filemanager instance and
> >> tail the log for any more clues. If you get any more clues then
> >> post
> them here.
> >> Also, if you are running off of master branch then this is an
> >> excellent opportunity for us to improve the error message, printing
> >> the product ID as oppose the opaque object... the later is
> >> relatively
> useless.
> >>
> >>
> >>
> >> On Wed, Mar 23, 2016 at 6:48 PM, Konstantinos Mavrommatis <
> >> kmavrommatis@celgene.com <javascript:;>> wrote:
> >>
> >> > Hi,
> >> > I have setup oodt using RADiX.
> >> >
> >> > When I use the default Lucene catalog factory I manage to ingest
> >> > a file with no problem:
> >> > # ./filemgr-client --url
> >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A9
> >> > 00
> >> > 0&d
> >> > =CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1C
> >> > s-
> >> > T2g
> >> > HY95y7ZA&m=yypXYFp0GDrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=we1KbELs
> >> > o6 G5c QKnuCw6lUZ-uHRh9J4QgzUSy_xWj1A&e= --operation
> >> > --ingestProduct --productName test.txt --productStructure Flat
> >> > --productTypeName GenericFile --metadataFile
> >> > file:///tmp/test.txt.met --refs file:///tmp/test.txt
> >> > ingestProduct: Result: afda62a1-f15f-11e5-a7d4-7d8cde2ab6dd
> >> >
> >> >
> >> > Then I install solr 5.5/jetty and created a new core named oodt
> >> > #solr create_core -c oodt I run the example command in solr
> >> > documentation and verified that this instance is able to index
> >> > documents. I can also access solr dashboard at the following URL:
> >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8
> >> > 98
> >> > 3_s
> >> > olr&d=CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6
> >> > yv
> >> > Z1C
> >> > s-T2gHY95y7ZA&m=yypXYFp0GDrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=cuA
> >> > RM uJC 3xzy3KYRi1OQeZ7vboagVHfSOb7cYjedhfo&e=
> >> > I also modified the filemgr.properties file with the following lines:
> >> >
> >> > filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.solr.
> >> > So
> >> > lrC
> >> > atalogFactory
> >> > org.apache.oodt.cas.filemgr.catalog.solr.url=https://urldefense.p
> >> > roofpoint.com/v2/url?u=https-3A__urldefense.pro&d=CwIBaQ&c=CZZujK
> >> > 3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-T2gHY95y7ZA&m=
> >> > K-d7kCTHiATrIXnEOzkoLsBJQVSARmtX3WF1LeLp3S0&s=ArMo6JsTWboDKziCKS2
> >> > VG1Xgqm2_hk2AwfUrQXWYBAg&e=
> >> > ofpoint.com/v2/url?u=https-3A__urldefense.proofp&d=CwIBaQ&c=CZZuj
> >> > K3
> >> > G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-T2gHY95y7ZA&m=R
> >> > Hp
> >> > DT3nSOn4uWf0ifOCDn1AIwRXSw8CIY51DbEp5Lc8&s=d5QpgHP8PkmNWprxGQlAET
> >> > nn
> >> > GvhxDGfMjMPTXoc7oPI&e=
> >> > oint.com/v2/url?u=http-3A__localhost-3A8983_solr&d=CwIBaQ&c=CZZuj
> >> > K3
> >> > G2K
> >> > uXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-T2gHY95y7ZA&m=yypX
> >> > YF
> >> > p0G
> >> > DrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=cuARMuJC3xzy3KYRi1OQeZ7vboag
> >> > VH
> >> > fSO
> >> > b7cYjedhfo&e=
> >> >
> >> > I have tried to run this command with and without the schema.xml
> >> > in the core directory. In both cases I get the error:
> >> >
> >> > But now when I try to ingest a file I get:
> >> > # ./filemgr-client --url
> >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A9
> >> > 00
> >> > 0&d
> >> > =CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1C
> >> > s-
> >> > T2g
> >> > HY95y7ZA&m=yypXYFp0GDrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=we1KbELs
> >> > o6 G5c QKnuCw6lUZ-uHRh9J4QgzUSy_xWj1A&e= --operation
> >> > --ingestProduct --productName test.txt --productStructure Flat
> >> > --productTypeName GenericFile --metadataFile
> >> > file:///tmp/test.txt.met --refs file:///tmp/test.txt
> >> > ERROR: Failed to ingest product 'test.txt' : java.lang.Exception:
> >> > org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
> >> > Error ingesting product
> >> > [org.apache.oodt.cas.filemgr.structs.Product@5eebbe]
> >> :
> >> > HTTP method failed: HTTP/1.1 404 Not Found
> >> >
> >> > Any ideas what is going on?
> >> >
> >> > Thanks
> >> > K
> >> >
> >> > *********************************************************
> >> > THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL
> >> > AND MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR
> >> > THE USE OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
> >> > If the reader is not the intended recipient, or the employee or
> >> > agent responsible to deliver it to the intended recipient, you
> >> > are hereby notified that any dissemination, distribution or
> >> > copying of this communication is strictly prohibited. If you have
> >> > received this communication in error, please reply to the sender
> >> > to notify us of the error and delete the original message. Thank You.
> >> >
> >>
> >>
> >>
> >> --
> >> *Lewis*
> >>
> >> *********************************************************
> >> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND
> >> MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE
> >> USE OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
> >> If the reader is not the intended recipient, or the employee or
> >> agent responsible to deliver it to the intended recipient, you are
> >> hereby notified that any dissemination, distribution or copying of
> >> this communication is strictly prohibited. If you have received
> >> this communication in error, please reply to the sender to notify
> >> us of the error and delete the original message. Thank You.
> >>
> >
> >
>
> *********************************************************
> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND
> MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE USE
> OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
> If the reader is not the intended recipient, or the employee or agent
> responsible to deliver it to the intended recipient, you are hereby
> notified that any dissemination, distribution or copying of this
> communication is strictly prohibited. If you have received this
> communication in error, please reply to the sender to notify us of the
> error and delete the original message. Thank You.
>
--
*Lewis*
*********************************************************
THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS
CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED
INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL
OR INDIVIDUALS NAMED ABOVE.
If the reader is not the intended recipient, or the
employee or agent responsible to deliver it to the
intended recipient, you are hereby notified that any
dissemination, distribution or copying of this
communication is strictly prohibited. If you have
received this communication in error, please reply to the
sender to notify us of the error and delete the original
message. Thank You.
Re: Setting up filemanager with SOLR 5.5
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Kos,
The schema field definition in the file manager schema sound like they need
a bit of an overhaul.
Are you able to file an issue against master and submit a PR?
Thanks
Lewis
On Thursday, March 24, 2016, Konstantinos Mavrommatis <
kmavrommatis@celgene.com> wrote:
> Hi,
>
> I seem to have solved the issue, although I have not tested the setup
> extensively.
>
> 1. the actual url to solr needs to contain the core name. So the correct
> url that needs to be in the filemgr.properties file is "
> http://localhost:8983/solr/oodt/" where the last part 'oodt' corresponds
> to the name of the core.
> 2. The schema.xml provided with oodt is not compatible to the classes of
> this version of SOLR. Instead I setup the solr core using the
> data_driven_schema_configs method, and copied the definitions of the CAS.*
> fields in the file managed-schema.
>
> As a result I could now ingest files in SOLR but only partially. The first
> action of adding information about a document (i.e. CAS.* fields) was
> successful.
> But the second action of updating the record with additional information
> failed with errors of the type: HTTP method failed: HTTP/1.1 400 Bad Request
>
> In the solr.log file the error message indicated that it cannot index the
> field 'FileLocation'. Note that even simple curl commands that were trying
> to update documents by adding a single field at a time were failing with
> the same type of error. Somehow SOLR seemed incapable of adding new fields
> that were not explicitly defined in the managed-schema file.
>
> =================================================================================
> 2016-03-24 23:57:53.990 ERROR (qtp364656927-14) [ x:oodt]
> o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: undefined
> field: "FileLocation"
> at
> org.apache.solr.schema.IndexSchema.getField(IndexSchema.java:1209)
> at
> org.apache.solr.update.processor.AtomicUpdateDocumentMerger.doAdd(AtomicUpdateDocumentMerger.java:133)
> at
> org.apache.solr.update.processor.AtomicUpdateDocumentMerger.merge(AtomicUpdateDocumentMerger.java:89)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.getUpdatedDocument(DistributedUpdateProcessor.java:1121)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1018)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:709)
> at
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
> at
> org.apache.solr.update.processor.AbstractDefaultValueUpdateProcessorFactory$DefaultValueUpdateProcessor.processAdd(AbstractDefaultValueUpdateProcessorFactory.java:93)
> at
> org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:250)
> at
> org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:177)
> at
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:94)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:69)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
>
> ===============================================================================
>
> To resolve it I modified the managed-schema file
> replaced the line
> <copyField source="*" dest="_text_"/> line with the following one
> <dynamicField name="*" type="text_general" indexed="true" stored="true"/>
> Now the ingestion process works without problem.
> Since I am very new to SOLR, this may not be the correct approach but for
> the time works - if somebody has a better idea please chime in.
> Thanks
> K
>
> -----Original Message-----
> From: Tom Barber [mailto:tom.barber@meteorite.bi <javascript:;>]
> Sent: Thursday, March 24, 2016 12:37 AM
> To: dev@oodt.apache.org <javascript:;>
> Subject: Re: Setting up filemanager with SOLR 5.5
>
> Ooh, I see, 5.5! Sorry. not had my morning coffee yet. Yeah I tried 5.5 a
> while back out of curiosity rather than a use case and it certainly didn't
> work OOTB.
>
> Tom
>
> On Thu, Mar 24, 2016 at 7:32 AM, Tom Barber <tom.barber@meteorite.bi
> <javascript:;>> wrote:
>
> > I appreciate its cheating, but when you build OODT with Radix did you
> > use the Solr maven profile? We fixed it up in 0.12 and it seemed to be
> > working fine, although I wasn't the one who tested it fully.
> >
> > Tom
> >
> > On Thu, Mar 24, 2016 at 7:15 AM, Konstantinos Mavrommatis <
> > kmavrommatis@celgene.com <javascript:;>> wrote:
> >
> >> Hi,
> >> I am using oodt v 0.12
> >> Interestingly the file etc/logging.properties was not in its place
> >> although this was a clean installation on a clean, newly started
> >> Ubuntu
> >> 14.04 server on AWS (!!) I copied the file from the
> >> oodt-src/filemgr/target/classes/etc/logging.properties and set the
> >> logging level.
> >> Running the same command:
> >> At the end of the file there are following error lines
> >>
> >> INFO: Posting message:<add><doc><field
> >> name="id">93addb3e-4d02-41ca-bee6-fbb321d6c890</field><field
> >> name="CAS.ProductId">93addb3e-4d02-41ca-bee6-fbb321d6c890</field><fie
> >> ld name="CAS.ProductName">test.txt</field><field
> >> name="CAS.ProductTypeName">GenericFile</field><field
> >> name="CAS.ProductReceivedTime">2016-03-24T07:12:01Z</field><field
> >> name="CAS.ProductTypeId">urn:oodt:GenericFile</field><field
> >> name="CAS.ProductStructure">Flat</field><field
> >> name="CAS.ProductTransferStatus">TRANSFERING</field></doc></add> to URL:
> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8983_
> >> solr_update&d=CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2h
> >> q2O6yvZ1Cs-T2gHY95y7ZA&m=RHpDT3nSOn4uWf0ifOCDn1AIwRXSw8CIY51DbEp5Lc8&
> >> s=7cz1N0iI0WRMpROKkDmYSpLacfEzT9fKjg4N8Oh36w0&e=
> >> Mar 24, 2016 7:12:01 AM
> >> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient index
> >> SEVERE: HTTP method failed: HTTP/1.1 404 Not Found Mar 24, 2016
> >> 7:12:01 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager
> >> catalogProduct
> >> SEVERE: ingestProduct: CatalogException when adding Product: test.txt
> >> to
> >> Catalog: Message: HTTP method failed: HTTP/1.1 404 Not Found Mar 24,
> >> 2016 7:12:01 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager
> >> ingestProductCore
> >> SEVERE: HTTP method failed: HTTP/1.1 404 Not Found Mar 24, 2016
> >> 7:12:01 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient
> >> ingestProduct
> >> SEVERE: Failed to ingest product [ name:test.txt] :java.lang.Exception:
> >> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
> >> Error ingesting product
> >> [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb]
> >> : HTTP method failed: HTTP/1.1 404 Not Found -- rolling back ingest
> >> Mar 24, 2016 7:12:01 AM
> >> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient delete
> >> INFO: Posting message:<delete><query>id:null</query></delete> to URL:
> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8983_
> >> solr_update-3Fcommit-3Dtrue&d=CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ
> >> 4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-T2gHY95y7ZA&m=RHpDT3nSOn4uWf0ifOCDn1AIwRXS
> >> w8CIY51DbEp5Lc8&s=jITvZPImZDY7_yKA18cyMffYS67ahR9Wz9IN9P1UNz0&e=
> >> Mar 24, 2016 7:12:01 AM
> >> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient delete
> >> SEVERE: HTTP method failed: HTTP/1.1 404 Not Found Mar 24, 2016
> >> 7:12:01 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager
> >> removeProduct
> >> WARNING: Exception modifying product: [null]: Message: HTTP method
> >> failed: HTTP/1.1 404 Not Found
> >> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: HTTP
> >> method failed: HTTP/1.1 404 Not Found
> >> at
> >>
> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient.delete(SolrClient.java:130)
> >> at
> >>
> org.apache.oodt.cas.filemgr.catalog.solr.SolrCatalog.removeProduct(SolrCatalog.java:165)
> >> at
> >>
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.removeProduct(XmlRpcFileManager.java:1113)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> at java.lang.reflect.Method.invoke(Method.java:606)
> >> at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
> >> at
> >> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
> >> at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
> >> at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
> >> at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
> >> at
> org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
> >> at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
> >> at java.lang.Thread.run(Thread.java:745)
> >>
> >> Mar 24, 2016 7:12:01 AM
> >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient
> >> ingestProduct
> >> SEVERE: Failed to rollback ingest of product
> >> [org.apache.oodt.cas.filemgr.structs.Product@2782f72b] :
> >> java.lang.Exception:
> >> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
> >> Error ingesting product
> >> [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb]
> >> : HTTP method failed: HTTP/1.1 404 Not Found Mar 24, 2016 7:12:01 AM
> >> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction execute
> >> SEVERE: java.lang.Exception:
> >> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
> >> Error ingesting product
> >> [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb]
> >> : HTTP method failed: HTTP/1.1 404 Not Found
> >> ERROR: Failed to ingest product 'test.txt' : java.lang.Exception:
> >> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
> >> Error ingesting product
> >> [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb]
> >> : HTTP method failed: HTTP/1.1 404 Not Found-----Original
> >> Message-----
> >>
> >>
> >>
> >> From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com
> <javascript:;>]
> >> Sent: Wednesday, March 23, 2016 7:25 PM
> >> To: dev@oodt.apache.org <javascript:;>
> >> Subject: Re: Setting up filemanager with SOLR 5.5
> >>
> >> Hi K,
> >> Which version of OODT are you using?
> >> Can you set logging to debug, restart your filemanager instance and
> >> tail the log for any more clues. If you get any more clues then post
> them here.
> >> Also, if you are running off of master branch then this is an
> >> excellent opportunity for us to improve the error message, printing
> >> the product ID as oppose the opaque object... the later is relatively
> useless.
> >>
> >>
> >>
> >> On Wed, Mar 23, 2016 at 6:48 PM, Konstantinos Mavrommatis <
> >> kmavrommatis@celgene.com <javascript:;>> wrote:
> >>
> >> > Hi,
> >> > I have setup oodt using RADiX.
> >> >
> >> > When I use the default Lucene catalog factory I manage to ingest a
> >> > file with no problem:
> >> > # ./filemgr-client --url
> >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A900
> >> > 0&d
> >> > =CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-
> >> > T2g
> >> > HY95y7ZA&m=yypXYFp0GDrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=we1KbELso6
> >> > G5c QKnuCw6lUZ-uHRh9J4QgzUSy_xWj1A&e= --operation --ingestProduct
> >> > --productName test.txt --productStructure Flat --productTypeName
> >> > GenericFile --metadataFile file:///tmp/test.txt.met --refs
> >> > file:///tmp/test.txt
> >> > ingestProduct: Result: afda62a1-f15f-11e5-a7d4-7d8cde2ab6dd
> >> >
> >> >
> >> > Then I install solr 5.5/jetty and created a new core named oodt
> >> > #solr create_core -c oodt I run the example command in solr
> >> > documentation and verified that this instance is able to index
> >> > documents. I can also access solr dashboard at the following URL:
> >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A898
> >> > 3_s
> >> > olr&d=CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yv
> >> > Z1C
> >> > s-T2gHY95y7ZA&m=yypXYFp0GDrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=cuARM
> >> > uJC 3xzy3KYRi1OQeZ7vboagVHfSOb7cYjedhfo&e=
> >> > I also modified the filemgr.properties file with the following lines:
> >> >
> >> > filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.solr.So
> >> > lrC
> >> > atalogFactory
> >> > org.apache.oodt.cas.filemgr.catalog.solr.url=https://urldefense.pro
> >> > ofpoint.com/v2/url?u=https-3A__urldefense.proofp&d=CwIBaQ&c=CZZujK3
> >> > G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-T2gHY95y7ZA&m=RHp
> >> > DT3nSOn4uWf0ifOCDn1AIwRXSw8CIY51DbEp5Lc8&s=d5QpgHP8PkmNWprxGQlAETnn
> >> > GvhxDGfMjMPTXoc7oPI&e=
> >> > oint.com/v2/url?u=http-3A__localhost-3A8983_solr&d=CwIBaQ&c=CZZujK3
> >> > G2K
> >> > uXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-T2gHY95y7ZA&m=yypXYF
> >> > p0G
> >> > DrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=cuARMuJC3xzy3KYRi1OQeZ7vboagVH
> >> > fSO
> >> > b7cYjedhfo&e=
> >> >
> >> > I have tried to run this command with and without the schema.xml in
> >> > the core directory. In both cases I get the error:
> >> >
> >> > But now when I try to ingest a file I get:
> >> > # ./filemgr-client --url
> >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A900
> >> > 0&d
> >> > =CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-
> >> > T2g
> >> > HY95y7ZA&m=yypXYFp0GDrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=we1KbELso6
> >> > G5c QKnuCw6lUZ-uHRh9J4QgzUSy_xWj1A&e= --operation --ingestProduct
> >> > --productName test.txt --productStructure Flat --productTypeName
> >> > GenericFile --metadataFile file:///tmp/test.txt.met --refs
> >> > file:///tmp/test.txt
> >> > ERROR: Failed to ingest product 'test.txt' : java.lang.Exception:
> >> > org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
> >> > Error ingesting product
> >> > [org.apache.oodt.cas.filemgr.structs.Product@5eebbe]
> >> :
> >> > HTTP method failed: HTTP/1.1 404 Not Found
> >> >
> >> > Any ideas what is going on?
> >> >
> >> > Thanks
> >> > K
> >> >
> >> > *********************************************************
> >> > THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND
> >> > MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE
> >> > USE OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
> >> > If the reader is not the intended recipient, or the employee or
> >> > agent responsible to deliver it to the intended recipient, you are
> >> > hereby notified that any dissemination, distribution or copying of
> >> > this communication is strictly prohibited. If you have received
> >> > this communication in error, please reply to the sender to notify
> >> > us of the error and delete the original message. Thank You.
> >> >
> >>
> >>
> >>
> >> --
> >> *Lewis*
> >>
> >> *********************************************************
> >> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND
> >> MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE USE
> >> OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
> >> If the reader is not the intended recipient, or the employee or agent
> >> responsible to deliver it to the intended recipient, you are hereby
> >> notified that any dissemination, distribution or copying of this
> >> communication is strictly prohibited. If you have received this
> >> communication in error, please reply to the sender to notify us of
> >> the error and delete the original message. Thank You.
> >>
> >
> >
>
> *********************************************************
> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS
> CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED
> INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL
> OR INDIVIDUALS NAMED ABOVE.
> If the reader is not the intended recipient, or the
> employee or agent responsible to deliver it to the
> intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this
> communication is strictly prohibited. If you have
> received this communication in error, please reply to the
> sender to notify us of the error and delete the original
> message. Thank You.
>
--
*Lewis*
RE: Setting up filemanager with SOLR 5.5
Posted by Konstantinos Mavrommatis <km...@celgene.com>.
Hi,
I seem to have solved the issue, although I have not tested the setup extensively.
1. the actual url to solr needs to contain the core name. So the correct url that needs to be in the filemgr.properties file is "http://localhost:8983/solr/oodt/" where the last part 'oodt' corresponds to the name of the core.
2. The schema.xml provided with oodt is not compatible to the classes of this version of SOLR. Instead I setup the solr core using the data_driven_schema_configs method, and copied the definitions of the CAS.* fields in the file managed-schema.
As a result I could now ingest files in SOLR but only partially. The first action of adding information about a document (i.e. CAS.* fields) was successful.
But the second action of updating the record with additional information failed with errors of the type: HTTP method failed: HTTP/1.1 400 Bad Request
In the solr.log file the error message indicated that it cannot index the field 'FileLocation'. Note that even simple curl commands that were trying to update documents by adding a single field at a time were failing with the same type of error. Somehow SOLR seemed incapable of adding new fields that were not explicitly defined in the managed-schema file.
=================================================================================
2016-03-24 23:57:53.990 ERROR (qtp364656927-14) [ x:oodt] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: undefined field: "FileLocation"
at org.apache.solr.schema.IndexSchema.getField(IndexSchema.java:1209)
at org.apache.solr.update.processor.AtomicUpdateDocumentMerger.doAdd(AtomicUpdateDocumentMerger.java:133)
at org.apache.solr.update.processor.AtomicUpdateDocumentMerger.merge(AtomicUpdateDocumentMerger.java:89)
at org.apache.solr.update.processor.DistributedUpdateProcessor.getUpdatedDocument(DistributedUpdateProcessor.java:1121)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1018)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:709)
at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.AbstractDefaultValueUpdateProcessorFactory$DefaultValueUpdateProcessor.processAdd(AbstractDefaultValueUpdateProcessorFactory.java:93)
at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:250)
at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:177)
at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:94)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:69)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
===============================================================================
To resolve it I modified the managed-schema file
replaced the line
<copyField source="*" dest="_text_"/> line with the following one
<dynamicField name="*" type="text_general" indexed="true" stored="true"/>
Now the ingestion process works without problem.
Since I am very new to SOLR, this may not be the correct approach but for the time works - if somebody has a better idea please chime in.
Thanks
K
-----Original Message-----
From: Tom Barber [mailto:tom.barber@meteorite.bi]
Sent: Thursday, March 24, 2016 12:37 AM
To: dev@oodt.apache.org
Subject: Re: Setting up filemanager with SOLR 5.5
Ooh, I see, 5.5! Sorry. not had my morning coffee yet. Yeah I tried 5.5 a while back out of curiosity rather than a use case and it certainly didn't work OOTB.
Tom
On Thu, Mar 24, 2016 at 7:32 AM, Tom Barber <to...@meteorite.bi> wrote:
> I appreciate its cheating, but when you build OODT with Radix did you
> use the Solr maven profile? We fixed it up in 0.12 and it seemed to be
> working fine, although I wasn't the one who tested it fully.
>
> Tom
>
> On Thu, Mar 24, 2016 at 7:15 AM, Konstantinos Mavrommatis <
> kmavrommatis@celgene.com> wrote:
>
>> Hi,
>> I am using oodt v 0.12
>> Interestingly the file etc/logging.properties was not in its place
>> although this was a clean installation on a clean, newly started
>> Ubuntu
>> 14.04 server on AWS (!!) I copied the file from the
>> oodt-src/filemgr/target/classes/etc/logging.properties and set the
>> logging level.
>> Running the same command:
>> At the end of the file there are following error lines
>>
>> INFO: Posting message:<add><doc><field
>> name="id">93addb3e-4d02-41ca-bee6-fbb321d6c890</field><field
>> name="CAS.ProductId">93addb3e-4d02-41ca-bee6-fbb321d6c890</field><fie
>> ld name="CAS.ProductName">test.txt</field><field
>> name="CAS.ProductTypeName">GenericFile</field><field
>> name="CAS.ProductReceivedTime">2016-03-24T07:12:01Z</field><field
>> name="CAS.ProductTypeId">urn:oodt:GenericFile</field><field
>> name="CAS.ProductStructure">Flat</field><field
>> name="CAS.ProductTransferStatus">TRANSFERING</field></doc></add> to URL:
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8983_
>> solr_update&d=CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2h
>> q2O6yvZ1Cs-T2gHY95y7ZA&m=RHpDT3nSOn4uWf0ifOCDn1AIwRXSw8CIY51DbEp5Lc8&
>> s=7cz1N0iI0WRMpROKkDmYSpLacfEzT9fKjg4N8Oh36w0&e=
>> Mar 24, 2016 7:12:01 AM
>> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient index
>> SEVERE: HTTP method failed: HTTP/1.1 404 Not Found Mar 24, 2016
>> 7:12:01 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager
>> catalogProduct
>> SEVERE: ingestProduct: CatalogException when adding Product: test.txt
>> to
>> Catalog: Message: HTTP method failed: HTTP/1.1 404 Not Found Mar 24,
>> 2016 7:12:01 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager
>> ingestProductCore
>> SEVERE: HTTP method failed: HTTP/1.1 404 Not Found Mar 24, 2016
>> 7:12:01 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient
>> ingestProduct
>> SEVERE: Failed to ingest product [ name:test.txt] :java.lang.Exception:
>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>> Error ingesting product
>> [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb]
>> : HTTP method failed: HTTP/1.1 404 Not Found -- rolling back ingest
>> Mar 24, 2016 7:12:01 AM
>> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient delete
>> INFO: Posting message:<delete><query>id:null</query></delete> to URL:
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8983_
>> solr_update-3Fcommit-3Dtrue&d=CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ
>> 4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-T2gHY95y7ZA&m=RHpDT3nSOn4uWf0ifOCDn1AIwRXS
>> w8CIY51DbEp5Lc8&s=jITvZPImZDY7_yKA18cyMffYS67ahR9Wz9IN9P1UNz0&e=
>> Mar 24, 2016 7:12:01 AM
>> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient delete
>> SEVERE: HTTP method failed: HTTP/1.1 404 Not Found Mar 24, 2016
>> 7:12:01 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager
>> removeProduct
>> WARNING: Exception modifying product: [null]: Message: HTTP method
>> failed: HTTP/1.1 404 Not Found
>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: HTTP
>> method failed: HTTP/1.1 404 Not Found
>> at
>> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient.delete(SolrClient.java:130)
>> at
>> org.apache.oodt.cas.filemgr.catalog.solr.SolrCatalog.removeProduct(SolrCatalog.java:165)
>> at
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.removeProduct(XmlRpcFileManager.java:1113)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
>> at
>> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
>> at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
>> at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
>> at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
>> at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
>> at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Mar 24, 2016 7:12:01 AM
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient
>> ingestProduct
>> SEVERE: Failed to rollback ingest of product
>> [org.apache.oodt.cas.filemgr.structs.Product@2782f72b] :
>> java.lang.Exception:
>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>> Error ingesting product
>> [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb]
>> : HTTP method failed: HTTP/1.1 404 Not Found Mar 24, 2016 7:12:01 AM
>> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction execute
>> SEVERE: java.lang.Exception:
>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>> Error ingesting product
>> [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb]
>> : HTTP method failed: HTTP/1.1 404 Not Found
>> ERROR: Failed to ingest product 'test.txt' : java.lang.Exception:
>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>> Error ingesting product
>> [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb]
>> : HTTP method failed: HTTP/1.1 404 Not Found-----Original
>> Message-----
>>
>>
>>
>> From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
>> Sent: Wednesday, March 23, 2016 7:25 PM
>> To: dev@oodt.apache.org
>> Subject: Re: Setting up filemanager with SOLR 5.5
>>
>> Hi K,
>> Which version of OODT are you using?
>> Can you set logging to debug, restart your filemanager instance and
>> tail the log for any more clues. If you get any more clues then post them here.
>> Also, if you are running off of master branch then this is an
>> excellent opportunity for us to improve the error message, printing
>> the product ID as oppose the opaque object... the later is relatively useless.
>>
>>
>>
>> On Wed, Mar 23, 2016 at 6:48 PM, Konstantinos Mavrommatis <
>> kmavrommatis@celgene.com> wrote:
>>
>> > Hi,
>> > I have setup oodt using RADiX.
>> >
>> > When I use the default Lucene catalog factory I manage to ingest a
>> > file with no problem:
>> > # ./filemgr-client --url
>> > https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A900
>> > 0&d
>> > =CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-
>> > T2g
>> > HY95y7ZA&m=yypXYFp0GDrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=we1KbELso6
>> > G5c QKnuCw6lUZ-uHRh9J4QgzUSy_xWj1A&e= --operation --ingestProduct
>> > --productName test.txt --productStructure Flat --productTypeName
>> > GenericFile --metadataFile file:///tmp/test.txt.met --refs
>> > file:///tmp/test.txt
>> > ingestProduct: Result: afda62a1-f15f-11e5-a7d4-7d8cde2ab6dd
>> >
>> >
>> > Then I install solr 5.5/jetty and created a new core named oodt
>> > #solr create_core -c oodt I run the example command in solr
>> > documentation and verified that this instance is able to index
>> > documents. I can also access solr dashboard at the following URL:
>> > https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A898
>> > 3_s
>> > olr&d=CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yv
>> > Z1C
>> > s-T2gHY95y7ZA&m=yypXYFp0GDrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=cuARM
>> > uJC 3xzy3KYRi1OQeZ7vboagVHfSOb7cYjedhfo&e=
>> > I also modified the filemgr.properties file with the following lines:
>> >
>> > filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.solr.So
>> > lrC
>> > atalogFactory
>> > org.apache.oodt.cas.filemgr.catalog.solr.url=https://urldefense.pro
>> > ofpoint.com/v2/url?u=https-3A__urldefense.proofp&d=CwIBaQ&c=CZZujK3
>> > G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-T2gHY95y7ZA&m=RHp
>> > DT3nSOn4uWf0ifOCDn1AIwRXSw8CIY51DbEp5Lc8&s=d5QpgHP8PkmNWprxGQlAETnn
>> > GvhxDGfMjMPTXoc7oPI&e=
>> > oint.com/v2/url?u=http-3A__localhost-3A8983_solr&d=CwIBaQ&c=CZZujK3
>> > G2K
>> > uXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-T2gHY95y7ZA&m=yypXYF
>> > p0G
>> > DrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=cuARMuJC3xzy3KYRi1OQeZ7vboagVH
>> > fSO
>> > b7cYjedhfo&e=
>> >
>> > I have tried to run this command with and without the schema.xml in
>> > the core directory. In both cases I get the error:
>> >
>> > But now when I try to ingest a file I get:
>> > # ./filemgr-client --url
>> > https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A900
>> > 0&d
>> > =CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-
>> > T2g
>> > HY95y7ZA&m=yypXYFp0GDrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=we1KbELso6
>> > G5c QKnuCw6lUZ-uHRh9J4QgzUSy_xWj1A&e= --operation --ingestProduct
>> > --productName test.txt --productStructure Flat --productTypeName
>> > GenericFile --metadataFile file:///tmp/test.txt.met --refs
>> > file:///tmp/test.txt
>> > ERROR: Failed to ingest product 'test.txt' : java.lang.Exception:
>> > org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>> > Error ingesting product
>> > [org.apache.oodt.cas.filemgr.structs.Product@5eebbe]
>> :
>> > HTTP method failed: HTTP/1.1 404 Not Found
>> >
>> > Any ideas what is going on?
>> >
>> > Thanks
>> > K
>> >
>> > *********************************************************
>> > THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND
>> > MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE
>> > USE OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
>> > If the reader is not the intended recipient, or the employee or
>> > agent responsible to deliver it to the intended recipient, you are
>> > hereby notified that any dissemination, distribution or copying of
>> > this communication is strictly prohibited. If you have received
>> > this communication in error, please reply to the sender to notify
>> > us of the error and delete the original message. Thank You.
>> >
>>
>>
>>
>> --
>> *Lewis*
>>
>> *********************************************************
>> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND
>> MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE USE
>> OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
>> If the reader is not the intended recipient, or the employee or agent
>> responsible to deliver it to the intended recipient, you are hereby
>> notified that any dissemination, distribution or copying of this
>> communication is strictly prohibited. If you have received this
>> communication in error, please reply to the sender to notify us of
>> the error and delete the original message. Thank You.
>>
>
>
*********************************************************
THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS
CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED
INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL
OR INDIVIDUALS NAMED ABOVE.
If the reader is not the intended recipient, or the
employee or agent responsible to deliver it to the
intended recipient, you are hereby notified that any
dissemination, distribution or copying of this
communication is strictly prohibited. If you have
received this communication in error, please reply to the
sender to notify us of the error and delete the original
message. Thank You.
Re: Setting up filemanager with SOLR 5.5
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Kos,
Can you log an issue in JIRA?
We can try to reproduce and fix for Solr 5.x API. This may even be a
problem only with one version of Solr... Who knows.
Thanks
On Thursday, March 24, 2016, Tom Barber <to...@meteorite.bi> wrote:
> Ooh, I see, 5.5! Sorry. not had my morning coffee yet. Yeah I tried 5.5 a
> while back out of curiosity rather than a use case and it certainly didn't
> work OOTB.
>
> Tom
>
> On Thu, Mar 24, 2016 at 7:32 AM, Tom Barber <tom.barber@meteorite.bi
> <javascript:;>> wrote:
>
> > I appreciate its cheating, but when you build OODT with Radix did you use
> > the Solr maven profile? We fixed it up in 0.12 and it seemed to be
> working
> > fine, although I wasn't the one who tested it fully.
> >
> > Tom
> >
> > On Thu, Mar 24, 2016 at 7:15 AM, Konstantinos Mavrommatis <
> > kmavrommatis@celgene.com <javascript:;>> wrote:
> >
> >> Hi,
> >> I am using oodt v 0.12
> >> Interestingly the file etc/logging.properties was not in its place
> >> although this was a clean installation on a clean, newly started Ubuntu
> >> 14.04 server on AWS (!!) I copied the file from the
> >> oodt-src/filemgr/target/classes/etc/logging.properties and set the
> logging
> >> level.
> >> Running the same command:
> >> At the end of the file there are following error lines
> >>
> >> INFO: Posting message:<add><doc><field
> >> name="id">93addb3e-4d02-41ca-bee6-fbb321d6c890</field><field
> >> name="CAS.ProductId">93addb3e-4d02-41ca-bee6-fbb321d6c890</field><field
> >> name="CAS.ProductName">test.txt</field><field
> >> name="CAS.ProductTypeName">GenericFile</field><field
> >> name="CAS.ProductReceivedTime">2016-03-24T07:12:01Z</field><field
> >> name="CAS.ProductTypeId">urn:oodt:GenericFile</field><field
> >> name="CAS.ProductStructure">Flat</field><field
> >> name="CAS.ProductTransferStatus">TRANSFERING</field></doc></add> to URL:
> >> http://localhost:8983/solr/update
> >> Mar 24, 2016 7:12:01 AM
> >> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient index
> >> SEVERE: HTTP method failed: HTTP/1.1 404 Not Found
> >> Mar 24, 2016 7:12:01 AM
> >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager catalogProduct
> >> SEVERE: ingestProduct: CatalogException when adding Product: test.txt to
> >> Catalog: Message: HTTP method failed: HTTP/1.1 404 Not Found
> >> Mar 24, 2016 7:12:01 AM
> >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager ingestProductCore
> >> SEVERE: HTTP method failed: HTTP/1.1 404 Not Found
> >> Mar 24, 2016 7:12:01 AM
> >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient ingestProduct
> >> SEVERE: Failed to ingest product [ name:test.txt] :java.lang.Exception:
> >> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
> >> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb
> ]
> >> : HTTP method failed: HTTP/1.1 404 Not Found -- rolling back ingest
> >> Mar 24, 2016 7:12:01 AM
> >> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient delete
> >> INFO: Posting message:<delete><query>id:null</query></delete> to URL:
> >> http://localhost:8983/solr/update?commit=true
> >> Mar 24, 2016 7:12:01 AM
> >> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient delete
> >> SEVERE: HTTP method failed: HTTP/1.1 404 Not Found
> >> Mar 24, 2016 7:12:01 AM
> >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager removeProduct
> >> WARNING: Exception modifying product: [null]: Message: HTTP method
> >> failed: HTTP/1.1 404 Not Found
> >> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: HTTP
> >> method failed: HTTP/1.1 404 Not Found
> >> at
> >>
> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient.delete(SolrClient.java:130)
> >> at
> >>
> org.apache.oodt.cas.filemgr.catalog.solr.SolrCatalog.removeProduct(SolrCatalog.java:165)
> >> at
> >>
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.removeProduct(XmlRpcFileManager.java:1113)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> at java.lang.reflect.Method.invoke(Method.java:606)
> >> at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
> >> at
> >> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
> >> at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
> >> at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
> >> at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
> >> at
> org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
> >> at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
> >> at java.lang.Thread.run(Thread.java:745)
> >>
> >> Mar 24, 2016 7:12:01 AM
> >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient ingestProduct
> >> SEVERE: Failed to rollback ingest of product
> >> [org.apache.oodt.cas.filemgr.structs.Product@2782f72b] :
> >> java.lang.Exception:
> >> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
> >> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb
> ]
> >> : HTTP method failed: HTTP/1.1 404 Not Found
> >> Mar 24, 2016 7:12:01 AM
> >> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction execute
> >> SEVERE: java.lang.Exception:
> >> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
> >> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb
> ]
> >> : HTTP method failed: HTTP/1.1 404 Not Found
> >> ERROR: Failed to ingest product 'test.txt' : java.lang.Exception:
> >> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
> >> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb
> ]
> >> : HTTP method failed: HTTP/1.1 404 Not Found-----Original Message-----
> >>
> >>
> >>
> >> From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com
> <javascript:;>]
> >> Sent: Wednesday, March 23, 2016 7:25 PM
> >> To: dev@oodt.apache.org <javascript:;>
> >> Subject: Re: Setting up filemanager with SOLR 5.5
> >>
> >> Hi K,
> >> Which version of OODT are you using?
> >> Can you set logging to debug, restart your filemanager instance and tail
> >> the log for any more clues. If you get any more clues then post them
> here.
> >> Also, if you are running off of master branch then this is an excellent
> >> opportunity for us to improve the error message, printing the product
> ID as
> >> oppose the opaque object... the later is relatively useless.
> >>
> >>
> >>
> >> On Wed, Mar 23, 2016 at 6:48 PM, Konstantinos Mavrommatis <
> >> kmavrommatis@celgene.com <javascript:;>> wrote:
> >>
> >> > Hi,
> >> > I have setup oodt using RADiX.
> >> >
> >> > When I use the default Lucene catalog factory I manage to ingest a
> >> > file with no problem:
> >> > # ./filemgr-client --url
> >> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A9000&d
> >> > =CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-T2g
> >> > HY95y7ZA&m=yypXYFp0GDrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=we1KbELso6G5c
> >> > QKnuCw6lUZ-uHRh9J4QgzUSy_xWj1A&e= --operation --ingestProduct
> >> > --productName test.txt --productStructure Flat --productTypeName
> >> > GenericFile --metadataFile file:///tmp/test.txt.met --refs
> >> > file:///tmp/test.txt
> >> > ingestProduct: Result: afda62a1-f15f-11e5-a7d4-7d8cde2ab6dd
> >> >
> >> >
> >> > Then I install solr 5.5/jetty and created a new core named oodt #solr
> >> > create_core -c oodt I run the example command in solr documentation
> >> > and verified that this instance is able to index documents. I can also
> >> > access solr dashboard at the following URL:
> >> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8983_s
> >> > olr&d=CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1C
> >> > s-T2gHY95y7ZA&m=yypXYFp0GDrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=cuARMuJC
> >> > 3xzy3KYRi1OQeZ7vboagVHfSOb7cYjedhfo&e=
> >> > I also modified the filemgr.properties file with the following lines:
> >> >
> >> > filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.solr.SolrC
> >> > atalogFactory
> >> > org.apache.oodt.cas.filemgr.catalog.solr.url=
> https://urldefense.proofp
> >> >
> oint.com/v2/url?u=http-3A__localhost-3A8983_solr&d=CwIBaQ&c=CZZujK3G2K
> >> > uXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-T2gHY95y7ZA&m=yypXYFp0G
> >> > DrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=cuARMuJC3xzy3KYRi1OQeZ7vboagVHfSO
> >> > b7cYjedhfo&e=
> >> >
> >> > I have tried to run this command with and without the schema.xml in
> >> > the core directory. In both cases I get the error:
> >> >
> >> > But now when I try to ingest a file I get:
> >> > # ./filemgr-client --url
> >> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A9000&d
> >> > =CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-T2g
> >> > HY95y7ZA&m=yypXYFp0GDrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=we1KbELso6G5c
> >> > QKnuCw6lUZ-uHRh9J4QgzUSy_xWj1A&e= --operation --ingestProduct
> >> > --productName test.txt --productStructure Flat --productTypeName
> >> > GenericFile --metadataFile file:///tmp/test.txt.met --refs
> >> > file:///tmp/test.txt
> >> > ERROR: Failed to ingest product 'test.txt' : java.lang.Exception:
> >> > org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
> >> > ingesting product [org.apache.oodt.cas.filemgr.structs.Product@5eebbe
> ]
> >> :
> >> > HTTP method failed: HTTP/1.1 404 Not Found
> >> >
> >> > Any ideas what is going on?
> >> >
> >> > Thanks
> >> > K
> >> >
> >> > *********************************************************
> >> > THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND
> >> > MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE USE
> >> > OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
> >> > If the reader is not the intended recipient, or the employee or agent
> >> > responsible to deliver it to the intended recipient, you are hereby
> >> > notified that any dissemination, distribution or copying of this
> >> > communication is strictly prohibited. If you have received this
> >> > communication in error, please reply to the sender to notify us of the
> >> > error and delete the original message. Thank You.
> >> >
> >>
> >>
> >>
> >> --
> >> *Lewis*
> >>
> >> *********************************************************
> >> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS
> >> CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED
> >> INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL
> >> OR INDIVIDUALS NAMED ABOVE.
> >> If the reader is not the intended recipient, or the
> >> employee or agent responsible to deliver it to the
> >> intended recipient, you are hereby notified that any
> >> dissemination, distribution or copying of this
> >> communication is strictly prohibited. If you have
> >> received this communication in error, please reply to the
> >> sender to notify us of the error and delete the original
> >> message. Thank You.
> >>
> >
> >
>
--
*Lewis*
Re: Setting up filemanager with SOLR 5.5
Posted by Tom Barber <to...@meteorite.bi>.
Ooh, I see, 5.5! Sorry. not had my morning coffee yet. Yeah I tried 5.5 a
while back out of curiosity rather than a use case and it certainly didn't
work OOTB.
Tom
On Thu, Mar 24, 2016 at 7:32 AM, Tom Barber <to...@meteorite.bi> wrote:
> I appreciate its cheating, but when you build OODT with Radix did you use
> the Solr maven profile? We fixed it up in 0.12 and it seemed to be working
> fine, although I wasn't the one who tested it fully.
>
> Tom
>
> On Thu, Mar 24, 2016 at 7:15 AM, Konstantinos Mavrommatis <
> kmavrommatis@celgene.com> wrote:
>
>> Hi,
>> I am using oodt v 0.12
>> Interestingly the file etc/logging.properties was not in its place
>> although this was a clean installation on a clean, newly started Ubuntu
>> 14.04 server on AWS (!!) I copied the file from the
>> oodt-src/filemgr/target/classes/etc/logging.properties and set the logging
>> level.
>> Running the same command:
>> At the end of the file there are following error lines
>>
>> INFO: Posting message:<add><doc><field
>> name="id">93addb3e-4d02-41ca-bee6-fbb321d6c890</field><field
>> name="CAS.ProductId">93addb3e-4d02-41ca-bee6-fbb321d6c890</field><field
>> name="CAS.ProductName">test.txt</field><field
>> name="CAS.ProductTypeName">GenericFile</field><field
>> name="CAS.ProductReceivedTime">2016-03-24T07:12:01Z</field><field
>> name="CAS.ProductTypeId">urn:oodt:GenericFile</field><field
>> name="CAS.ProductStructure">Flat</field><field
>> name="CAS.ProductTransferStatus">TRANSFERING</field></doc></add> to URL:
>> http://localhost:8983/solr/update
>> Mar 24, 2016 7:12:01 AM
>> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient index
>> SEVERE: HTTP method failed: HTTP/1.1 404 Not Found
>> Mar 24, 2016 7:12:01 AM
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager catalogProduct
>> SEVERE: ingestProduct: CatalogException when adding Product: test.txt to
>> Catalog: Message: HTTP method failed: HTTP/1.1 404 Not Found
>> Mar 24, 2016 7:12:01 AM
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager ingestProductCore
>> SEVERE: HTTP method failed: HTTP/1.1 404 Not Found
>> Mar 24, 2016 7:12:01 AM
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient ingestProduct
>> SEVERE: Failed to ingest product [ name:test.txt] :java.lang.Exception:
>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
>> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb]
>> : HTTP method failed: HTTP/1.1 404 Not Found -- rolling back ingest
>> Mar 24, 2016 7:12:01 AM
>> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient delete
>> INFO: Posting message:<delete><query>id:null</query></delete> to URL:
>> http://localhost:8983/solr/update?commit=true
>> Mar 24, 2016 7:12:01 AM
>> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient delete
>> SEVERE: HTTP method failed: HTTP/1.1 404 Not Found
>> Mar 24, 2016 7:12:01 AM
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager removeProduct
>> WARNING: Exception modifying product: [null]: Message: HTTP method
>> failed: HTTP/1.1 404 Not Found
>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: HTTP
>> method failed: HTTP/1.1 404 Not Found
>> at
>> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient.delete(SolrClient.java:130)
>> at
>> org.apache.oodt.cas.filemgr.catalog.solr.SolrCatalog.removeProduct(SolrCatalog.java:165)
>> at
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.removeProduct(XmlRpcFileManager.java:1113)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
>> at
>> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
>> at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
>> at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
>> at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
>> at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
>> at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Mar 24, 2016 7:12:01 AM
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient ingestProduct
>> SEVERE: Failed to rollback ingest of product
>> [org.apache.oodt.cas.filemgr.structs.Product@2782f72b] :
>> java.lang.Exception:
>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
>> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb]
>> : HTTP method failed: HTTP/1.1 404 Not Found
>> Mar 24, 2016 7:12:01 AM
>> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction execute
>> SEVERE: java.lang.Exception:
>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
>> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb]
>> : HTTP method failed: HTTP/1.1 404 Not Found
>> ERROR: Failed to ingest product 'test.txt' : java.lang.Exception:
>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
>> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb]
>> : HTTP method failed: HTTP/1.1 404 Not Found-----Original Message-----
>>
>>
>>
>> From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
>> Sent: Wednesday, March 23, 2016 7:25 PM
>> To: dev@oodt.apache.org
>> Subject: Re: Setting up filemanager with SOLR 5.5
>>
>> Hi K,
>> Which version of OODT are you using?
>> Can you set logging to debug, restart your filemanager instance and tail
>> the log for any more clues. If you get any more clues then post them here.
>> Also, if you are running off of master branch then this is an excellent
>> opportunity for us to improve the error message, printing the product ID as
>> oppose the opaque object... the later is relatively useless.
>>
>>
>>
>> On Wed, Mar 23, 2016 at 6:48 PM, Konstantinos Mavrommatis <
>> kmavrommatis@celgene.com> wrote:
>>
>> > Hi,
>> > I have setup oodt using RADiX.
>> >
>> > When I use the default Lucene catalog factory I manage to ingest a
>> > file with no problem:
>> > # ./filemgr-client --url
>> > https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A9000&d
>> > =CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-T2g
>> > HY95y7ZA&m=yypXYFp0GDrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=we1KbELso6G5c
>> > QKnuCw6lUZ-uHRh9J4QgzUSy_xWj1A&e= --operation --ingestProduct
>> > --productName test.txt --productStructure Flat --productTypeName
>> > GenericFile --metadataFile file:///tmp/test.txt.met --refs
>> > file:///tmp/test.txt
>> > ingestProduct: Result: afda62a1-f15f-11e5-a7d4-7d8cde2ab6dd
>> >
>> >
>> > Then I install solr 5.5/jetty and created a new core named oodt #solr
>> > create_core -c oodt I run the example command in solr documentation
>> > and verified that this instance is able to index documents. I can also
>> > access solr dashboard at the following URL:
>> > https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8983_s
>> > olr&d=CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1C
>> > s-T2gHY95y7ZA&m=yypXYFp0GDrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=cuARMuJC
>> > 3xzy3KYRi1OQeZ7vboagVHfSOb7cYjedhfo&e=
>> > I also modified the filemgr.properties file with the following lines:
>> >
>> > filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.solr.SolrC
>> > atalogFactory
>> > org.apache.oodt.cas.filemgr.catalog.solr.url=https://urldefense.proofp
>> > oint.com/v2/url?u=http-3A__localhost-3A8983_solr&d=CwIBaQ&c=CZZujK3G2K
>> > uXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-T2gHY95y7ZA&m=yypXYFp0G
>> > DrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=cuARMuJC3xzy3KYRi1OQeZ7vboagVHfSO
>> > b7cYjedhfo&e=
>> >
>> > I have tried to run this command with and without the schema.xml in
>> > the core directory. In both cases I get the error:
>> >
>> > But now when I try to ingest a file I get:
>> > # ./filemgr-client --url
>> > https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A9000&d
>> > =CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-T2g
>> > HY95y7ZA&m=yypXYFp0GDrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=we1KbELso6G5c
>> > QKnuCw6lUZ-uHRh9J4QgzUSy_xWj1A&e= --operation --ingestProduct
>> > --productName test.txt --productStructure Flat --productTypeName
>> > GenericFile --metadataFile file:///tmp/test.txt.met --refs
>> > file:///tmp/test.txt
>> > ERROR: Failed to ingest product 'test.txt' : java.lang.Exception:
>> > org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
>> > ingesting product [org.apache.oodt.cas.filemgr.structs.Product@5eebbe]
>> :
>> > HTTP method failed: HTTP/1.1 404 Not Found
>> >
>> > Any ideas what is going on?
>> >
>> > Thanks
>> > K
>> >
>> > *********************************************************
>> > THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND
>> > MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE USE
>> > OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
>> > If the reader is not the intended recipient, or the employee or agent
>> > responsible to deliver it to the intended recipient, you are hereby
>> > notified that any dissemination, distribution or copying of this
>> > communication is strictly prohibited. If you have received this
>> > communication in error, please reply to the sender to notify us of the
>> > error and delete the original message. Thank You.
>> >
>>
>>
>>
>> --
>> *Lewis*
>>
>> *********************************************************
>> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS
>> CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED
>> INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL
>> OR INDIVIDUALS NAMED ABOVE.
>> If the reader is not the intended recipient, or the
>> employee or agent responsible to deliver it to the
>> intended recipient, you are hereby notified that any
>> dissemination, distribution or copying of this
>> communication is strictly prohibited. If you have
>> received this communication in error, please reply to the
>> sender to notify us of the error and delete the original
>> message. Thank You.
>>
>
>
Re: Setting up filemanager with SOLR 5.5
Posted by Tom Barber <to...@meteorite.bi>.
I appreciate its cheating, but when you build OODT with Radix did you use
the Solr maven profile? We fixed it up in 0.12 and it seemed to be working
fine, although I wasn't the one who tested it fully.
Tom
On Thu, Mar 24, 2016 at 7:15 AM, Konstantinos Mavrommatis <
kmavrommatis@celgene.com> wrote:
> Hi,
> I am using oodt v 0.12
> Interestingly the file etc/logging.properties was not in its place
> although this was a clean installation on a clean, newly started Ubuntu
> 14.04 server on AWS (!!) I copied the file from the
> oodt-src/filemgr/target/classes/etc/logging.properties and set the logging
> level.
> Running the same command:
> At the end of the file there are following error lines
>
> INFO: Posting message:<add><doc><field
> name="id">93addb3e-4d02-41ca-bee6-fbb321d6c890</field><field
> name="CAS.ProductId">93addb3e-4d02-41ca-bee6-fbb321d6c890</field><field
> name="CAS.ProductName">test.txt</field><field
> name="CAS.ProductTypeName">GenericFile</field><field
> name="CAS.ProductReceivedTime">2016-03-24T07:12:01Z</field><field
> name="CAS.ProductTypeId">urn:oodt:GenericFile</field><field
> name="CAS.ProductStructure">Flat</field><field
> name="CAS.ProductTransferStatus">TRANSFERING</field></doc></add> to URL:
> http://localhost:8983/solr/update
> Mar 24, 2016 7:12:01 AM
> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient index
> SEVERE: HTTP method failed: HTTP/1.1 404 Not Found
> Mar 24, 2016 7:12:01 AM
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager catalogProduct
> SEVERE: ingestProduct: CatalogException when adding Product: test.txt to
> Catalog: Message: HTTP method failed: HTTP/1.1 404 Not Found
> Mar 24, 2016 7:12:01 AM
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager ingestProductCore
> SEVERE: HTTP method failed: HTTP/1.1 404 Not Found
> Mar 24, 2016 7:12:01 AM
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient ingestProduct
> SEVERE: Failed to ingest product [ name:test.txt] :java.lang.Exception:
> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb]
> : HTTP method failed: HTTP/1.1 404 Not Found -- rolling back ingest
> Mar 24, 2016 7:12:01 AM
> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient delete
> INFO: Posting message:<delete><query>id:null</query></delete> to URL:
> http://localhost:8983/solr/update?commit=true
> Mar 24, 2016 7:12:01 AM
> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient delete
> SEVERE: HTTP method failed: HTTP/1.1 404 Not Found
> Mar 24, 2016 7:12:01 AM
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager removeProduct
> WARNING: Exception modifying product: [null]: Message: HTTP method failed:
> HTTP/1.1 404 Not Found
> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: HTTP
> method failed: HTTP/1.1 404 Not Found
> at
> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient.delete(SolrClient.java:130)
> at
> org.apache.oodt.cas.filemgr.catalog.solr.SolrCatalog.removeProduct(SolrCatalog.java:165)
> at
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.removeProduct(XmlRpcFileManager.java:1113)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
> at
> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
> at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
> at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
> at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
> at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
> at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
> at java.lang.Thread.run(Thread.java:745)
>
> Mar 24, 2016 7:12:01 AM
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient ingestProduct
> SEVERE: Failed to rollback ingest of product
> [org.apache.oodt.cas.filemgr.structs.Product@2782f72b] :
> java.lang.Exception:
> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb]
> : HTTP method failed: HTTP/1.1 404 Not Found
> Mar 24, 2016 7:12:01 AM
> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction execute
> SEVERE: java.lang.Exception:
> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb]
> : HTTP method failed: HTTP/1.1 404 Not Found
> ERROR: Failed to ingest product 'test.txt' : java.lang.Exception:
> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb]
> : HTTP method failed: HTTP/1.1 404 Not Found-----Original Message-----
>
>
>
> From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
> Sent: Wednesday, March 23, 2016 7:25 PM
> To: dev@oodt.apache.org
> Subject: Re: Setting up filemanager with SOLR 5.5
>
> Hi K,
> Which version of OODT are you using?
> Can you set logging to debug, restart your filemanager instance and tail
> the log for any more clues. If you get any more clues then post them here.
> Also, if you are running off of master branch then this is an excellent
> opportunity for us to improve the error message, printing the product ID as
> oppose the opaque object... the later is relatively useless.
>
>
>
> On Wed, Mar 23, 2016 at 6:48 PM, Konstantinos Mavrommatis <
> kmavrommatis@celgene.com> wrote:
>
> > Hi,
> > I have setup oodt using RADiX.
> >
> > When I use the default Lucene catalog factory I manage to ingest a
> > file with no problem:
> > # ./filemgr-client --url
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A9000&d
> > =CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-T2g
> > HY95y7ZA&m=yypXYFp0GDrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=we1KbELso6G5c
> > QKnuCw6lUZ-uHRh9J4QgzUSy_xWj1A&e= --operation --ingestProduct
> > --productName test.txt --productStructure Flat --productTypeName
> > GenericFile --metadataFile file:///tmp/test.txt.met --refs
> > file:///tmp/test.txt
> > ingestProduct: Result: afda62a1-f15f-11e5-a7d4-7d8cde2ab6dd
> >
> >
> > Then I install solr 5.5/jetty and created a new core named oodt #solr
> > create_core -c oodt I run the example command in solr documentation
> > and verified that this instance is able to index documents. I can also
> > access solr dashboard at the following URL:
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8983_s
> > olr&d=CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1C
> > s-T2gHY95y7ZA&m=yypXYFp0GDrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=cuARMuJC
> > 3xzy3KYRi1OQeZ7vboagVHfSOb7cYjedhfo&e=
> > I also modified the filemgr.properties file with the following lines:
> >
> > filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.solr.SolrC
> > atalogFactory
> > org.apache.oodt.cas.filemgr.catalog.solr.url=https://urldefense.proofp
> > oint.com/v2/url?u=http-3A__localhost-3A8983_solr&d=CwIBaQ&c=CZZujK3G2K
> > uXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-T2gHY95y7ZA&m=yypXYFp0G
> > DrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=cuARMuJC3xzy3KYRi1OQeZ7vboagVHfSO
> > b7cYjedhfo&e=
> >
> > I have tried to run this command with and without the schema.xml in
> > the core directory. In both cases I get the error:
> >
> > But now when I try to ingest a file I get:
> > # ./filemgr-client --url
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A9000&d
> > =CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-T2g
> > HY95y7ZA&m=yypXYFp0GDrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=we1KbELso6G5c
> > QKnuCw6lUZ-uHRh9J4QgzUSy_xWj1A&e= --operation --ingestProduct
> > --productName test.txt --productStructure Flat --productTypeName
> > GenericFile --metadataFile file:///tmp/test.txt.met --refs
> > file:///tmp/test.txt
> > ERROR: Failed to ingest product 'test.txt' : java.lang.Exception:
> > org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
> > ingesting product [org.apache.oodt.cas.filemgr.structs.Product@5eebbe] :
> > HTTP method failed: HTTP/1.1 404 Not Found
> >
> > Any ideas what is going on?
> >
> > Thanks
> > K
> >
> > *********************************************************
> > THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND
> > MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE USE
> > OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
> > If the reader is not the intended recipient, or the employee or agent
> > responsible to deliver it to the intended recipient, you are hereby
> > notified that any dissemination, distribution or copying of this
> > communication is strictly prohibited. If you have received this
> > communication in error, please reply to the sender to notify us of the
> > error and delete the original message. Thank You.
> >
>
>
>
> --
> *Lewis*
>
> *********************************************************
> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS
> CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED
> INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL
> OR INDIVIDUALS NAMED ABOVE.
> If the reader is not the intended recipient, or the
> employee or agent responsible to deliver it to the
> intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this
> communication is strictly prohibited. If you have
> received this communication in error, please reply to the
> sender to notify us of the error and delete the original
> message. Thank You.
>
RE: Setting up filemanager with SOLR 5.5
Posted by Konstantinos Mavrommatis <km...@celgene.com>.
Hi,
I am using oodt v 0.12
Interestingly the file etc/logging.properties was not in its place although this was a clean installation on a clean, newly started Ubuntu 14.04 server on AWS (!!) I copied the file from the oodt-src/filemgr/target/classes/etc/logging.properties and set the logging level.
Running the same command:
At the end of the file there are following error lines
INFO: Posting message:<add><doc><field name="id">93addb3e-4d02-41ca-bee6-fbb321d6c890</field><field name="CAS.ProductId">93addb3e-4d02-41ca-bee6-fbb321d6c890</field><field name="CAS.ProductName">test.txt</field><field name="CAS.ProductTypeName">GenericFile</field><field name="CAS.ProductReceivedTime">2016-03-24T07:12:01Z</field><field name="CAS.ProductTypeId">urn:oodt:GenericFile</field><field name="CAS.ProductStructure">Flat</field><field name="CAS.ProductTransferStatus">TRANSFERING</field></doc></add> to URL:http://localhost:8983/solr/update
Mar 24, 2016 7:12:01 AM org.apache.oodt.cas.filemgr.catalog.solr.SolrClient index
SEVERE: HTTP method failed: HTTP/1.1 404 Not Found
Mar 24, 2016 7:12:01 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager catalogProduct
SEVERE: ingestProduct: CatalogException when adding Product: test.txt to Catalog: Message: HTTP method failed: HTTP/1.1 404 Not Found
Mar 24, 2016 7:12:01 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager ingestProductCore
SEVERE: HTTP method failed: HTTP/1.1 404 Not Found
Mar 24, 2016 7:12:01 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient ingestProduct
SEVERE: Failed to ingest product [ name:test.txt] :java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error ingesting product [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb] : HTTP method failed: HTTP/1.1 404 Not Found -- rolling back ingest
Mar 24, 2016 7:12:01 AM org.apache.oodt.cas.filemgr.catalog.solr.SolrClient delete
INFO: Posting message:<delete><query>id:null</query></delete> to URL:http://localhost:8983/solr/update?commit=true
Mar 24, 2016 7:12:01 AM org.apache.oodt.cas.filemgr.catalog.solr.SolrClient delete
SEVERE: HTTP method failed: HTTP/1.1 404 Not Found
Mar 24, 2016 7:12:01 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager removeProduct
WARNING: Exception modifying product: [null]: Message: HTTP method failed: HTTP/1.1 404 Not Found
org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: HTTP method failed: HTTP/1.1 404 Not Found
at org.apache.oodt.cas.filemgr.catalog.solr.SolrClient.delete(SolrClient.java:130)
at org.apache.oodt.cas.filemgr.catalog.solr.SolrCatalog.removeProduct(SolrCatalog.java:165)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.removeProduct(XmlRpcFileManager.java:1113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
at java.lang.Thread.run(Thread.java:745)
Mar 24, 2016 7:12:01 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient ingestProduct
SEVERE: Failed to rollback ingest of product [org.apache.oodt.cas.filemgr.structs.Product@2782f72b] : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error ingesting product [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb] : HTTP method failed: HTTP/1.1 404 Not Found
Mar 24, 2016 7:12:01 AM org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction execute
SEVERE: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error ingesting product [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb] : HTTP method failed: HTTP/1.1 404 Not Found
ERROR: Failed to ingest product 'test.txt' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error ingesting product [org.apache.oodt.cas.filemgr.structs.Product@60a79fbb] : HTTP method failed: HTTP/1.1 404 Not Found-----Original Message-----
From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
Sent: Wednesday, March 23, 2016 7:25 PM
To: dev@oodt.apache.org
Subject: Re: Setting up filemanager with SOLR 5.5
Hi K,
Which version of OODT are you using?
Can you set logging to debug, restart your filemanager instance and tail the log for any more clues. If you get any more clues then post them here.
Also, if you are running off of master branch then this is an excellent opportunity for us to improve the error message, printing the product ID as oppose the opaque object... the later is relatively useless.
On Wed, Mar 23, 2016 at 6:48 PM, Konstantinos Mavrommatis < kmavrommatis@celgene.com> wrote:
> Hi,
> I have setup oodt using RADiX.
>
> When I use the default Lucene catalog factory I manage to ingest a
> file with no problem:
> # ./filemgr-client --url
> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A9000&d
> =CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-T2g
> HY95y7ZA&m=yypXYFp0GDrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=we1KbELso6G5c
> QKnuCw6lUZ-uHRh9J4QgzUSy_xWj1A&e= --operation --ingestProduct
> --productName test.txt --productStructure Flat --productTypeName
> GenericFile --metadataFile file:///tmp/test.txt.met --refs
> file:///tmp/test.txt
> ingestProduct: Result: afda62a1-f15f-11e5-a7d4-7d8cde2ab6dd
>
>
> Then I install solr 5.5/jetty and created a new core named oodt #solr
> create_core -c oodt I run the example command in solr documentation
> and verified that this instance is able to index documents. I can also
> access solr dashboard at the following URL:
> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8983_s
> olr&d=CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1C
> s-T2gHY95y7ZA&m=yypXYFp0GDrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=cuARMuJC
> 3xzy3KYRi1OQeZ7vboagVHfSOb7cYjedhfo&e=
> I also modified the filemgr.properties file with the following lines:
>
> filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.solr.SolrC
> atalogFactory
> org.apache.oodt.cas.filemgr.catalog.solr.url=https://urldefense.proofp
> oint.com/v2/url?u=http-3A__localhost-3A8983_solr&d=CwIBaQ&c=CZZujK3G2K
> uXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-T2gHY95y7ZA&m=yypXYFp0G
> DrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=cuARMuJC3xzy3KYRi1OQeZ7vboagVHfSO
> b7cYjedhfo&e=
>
> I have tried to run this command with and without the schema.xml in
> the core directory. In both cases I get the error:
>
> But now when I try to ingest a file I get:
> # ./filemgr-client --url
> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A9000&d
> =CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs-T2g
> HY95y7ZA&m=yypXYFp0GDrsimQXj-42kCPwJnfRFvCIjjJKzPQVqSw&s=we1KbELso6G5c
> QKnuCw6lUZ-uHRh9J4QgzUSy_xWj1A&e= --operation --ingestProduct
> --productName test.txt --productStructure Flat --productTypeName
> GenericFile --metadataFile file:///tmp/test.txt.met --refs
> file:///tmp/test.txt
> ERROR: Failed to ingest product 'test.txt' : java.lang.Exception:
> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@5eebbe] :
> HTTP method failed: HTTP/1.1 404 Not Found
>
> Any ideas what is going on?
>
> Thanks
> K
>
> *********************************************************
> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND
> MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE USE
> OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
> If the reader is not the intended recipient, or the employee or agent
> responsible to deliver it to the intended recipient, you are hereby
> notified that any dissemination, distribution or copying of this
> communication is strictly prohibited. If you have received this
> communication in error, please reply to the sender to notify us of the
> error and delete the original message. Thank You.
>
--
*Lewis*
*********************************************************
THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS
CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED
INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL
OR INDIVIDUALS NAMED ABOVE.
If the reader is not the intended recipient, or the
employee or agent responsible to deliver it to the
intended recipient, you are hereby notified that any
dissemination, distribution or copying of this
communication is strictly prohibited. If you have
received this communication in error, please reply to the
sender to notify us of the error and delete the original
message. Thank You.
Re: Setting up filemanager with SOLR 5.5
Posted by "Ramirez, Paul M (398M)" <pa...@jpl.nasa.gov>.
Isn’t there an oodt schema that should be used in this core? You may also have point to the core endpoint that you’ve created as I believe the default solr endpoint will not ingest documents.
--Paul Ramirez
On 3/23/16, 7:25 PM, "Lewis John Mcgibbney" <le...@gmail.com> wrote:
>Hi K,
>Which version of OODT are you using?
>Can you set logging to debug, restart your filemanager instance and tail
>the log for any more clues. If you get any more clues then post them here.
>Also, if you are running off of master branch then this is an excellent
>opportunity for us to improve the error message, printing the product ID as
>oppose the opaque object... the later is relatively useless.
>
>
>
>On Wed, Mar 23, 2016 at 6:48 PM, Konstantinos Mavrommatis <
>kmavrommatis@celgene.com> wrote:
>
>> Hi,
>> I have setup oodt using RADiX.
>>
>> When I use the default Lucene catalog factory I manage to ingest a file
>> with no problem:
>> # ./filemgr-client --url http://localhost:9000 --operation
>> --ingestProduct --productName test.txt --productStructure Flat
>> --productTypeName GenericFile --metadataFile file:///tmp/test.txt.met
>> --refs file:///tmp/test.txt
>> ingestProduct: Result: afda62a1-f15f-11e5-a7d4-7d8cde2ab6dd
>>
>>
>> Then I install solr 5.5/jetty and created a new core named oodt
>> #solr create_core -c oodt
>> I run the example command in solr documentation and verified that this
>> instance is able to index documents. I can also access solr dashboard at
>> the following URL: http://localhost:8983/solr
>> I also modified the filemgr.properties file with the following lines:
>>
>> filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.solr.SolrCatalogFactory
>> org.apache.oodt.cas.filemgr.catalog.solr.url=http://localhost:8983/solr
>>
>> I have tried to run this command with and without the schema.xml in the
>> core directory. In both cases I get the error:
>>
>> But now when I try to ingest a file I get:
>> # ./filemgr-client --url http://localhost:9000 --operation
>> --ingestProduct --productName test.txt --productStructure Flat
>> --productTypeName GenericFile --metadataFile file:///tmp/test.txt.met
>> --refs file:///tmp/test.txt
>> ERROR: Failed to ingest product 'test.txt' : java.lang.Exception:
>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
>> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@5eebbe] :
>> HTTP method failed: HTTP/1.1 404 Not Found
>>
>> Any ideas what is going on?
>>
>> Thanks
>> K
>>
>> *********************************************************
>> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS
>> CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED
>> INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL
>> OR INDIVIDUALS NAMED ABOVE.
>> If the reader is not the intended recipient, or the
>> employee or agent responsible to deliver it to the
>> intended recipient, you are hereby notified that any
>> dissemination, distribution or copying of this
>> communication is strictly prohibited. If you have
>> received this communication in error, please reply to the
>> sender to notify us of the error and delete the original
>> message. Thank You.
>>
>
>
>
>--
>*Lewis*
Re: Setting up filemanager with SOLR 5.5
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi K,
Which version of OODT are you using?
Can you set logging to debug, restart your filemanager instance and tail
the log for any more clues. If you get any more clues then post them here.
Also, if you are running off of master branch then this is an excellent
opportunity for us to improve the error message, printing the product ID as
oppose the opaque object... the later is relatively useless.
On Wed, Mar 23, 2016 at 6:48 PM, Konstantinos Mavrommatis <
kmavrommatis@celgene.com> wrote:
> Hi,
> I have setup oodt using RADiX.
>
> When I use the default Lucene catalog factory I manage to ingest a file
> with no problem:
> # ./filemgr-client --url http://localhost:9000 --operation
> --ingestProduct --productName test.txt --productStructure Flat
> --productTypeName GenericFile --metadataFile file:///tmp/test.txt.met
> --refs file:///tmp/test.txt
> ingestProduct: Result: afda62a1-f15f-11e5-a7d4-7d8cde2ab6dd
>
>
> Then I install solr 5.5/jetty and created a new core named oodt
> #solr create_core -c oodt
> I run the example command in solr documentation and verified that this
> instance is able to index documents. I can also access solr dashboard at
> the following URL: http://localhost:8983/solr
> I also modified the filemgr.properties file with the following lines:
>
> filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.solr.SolrCatalogFactory
> org.apache.oodt.cas.filemgr.catalog.solr.url=http://localhost:8983/solr
>
> I have tried to run this command with and without the schema.xml in the
> core directory. In both cases I get the error:
>
> But now when I try to ingest a file I get:
> # ./filemgr-client --url http://localhost:9000 --operation
> --ingestProduct --productName test.txt --productStructure Flat
> --productTypeName GenericFile --metadataFile file:///tmp/test.txt.met
> --refs file:///tmp/test.txt
> ERROR: Failed to ingest product 'test.txt' : java.lang.Exception:
> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@5eebbe] :
> HTTP method failed: HTTP/1.1 404 Not Found
>
> Any ideas what is going on?
>
> Thanks
> K
>
> *********************************************************
> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS
> CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED
> INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL
> OR INDIVIDUALS NAMED ABOVE.
> If the reader is not the intended recipient, or the
> employee or agent responsible to deliver it to the
> intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this
> communication is strictly prohibited. If you have
> received this communication in error, please reply to the
> sender to notify us of the error and delete the original
> message. Thank You.
>
--
*Lewis*
Setting up filemanager with SOLR 5.5
Posted by Konstantinos Mavrommatis <km...@celgene.com>.
Hi,
I have setup oodt using RADiX.
When I use the default Lucene catalog factory I manage to ingest a file with no problem:
# ./filemgr-client --url http://localhost:9000 --operation --ingestProduct --productName test.txt --productStructure Flat --productTypeName GenericFile --metadataFile file:///tmp/test.txt.met --refs file:///tmp/test.txt
ingestProduct: Result: afda62a1-f15f-11e5-a7d4-7d8cde2ab6dd
Then I install solr 5.5/jetty and created a new core named oodt
#solr create_core -c oodt
I run the example command in solr documentation and verified that this instance is able to index documents. I can also access solr dashboard at the following URL: http://localhost:8983/solr
I also modified the filemgr.properties file with the following lines:
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.solr.SolrCatalogFactory
org.apache.oodt.cas.filemgr.catalog.solr.url=http://localhost:8983/solr
I have tried to run this command with and without the schema.xml in the core directory. In both cases I get the error:
But now when I try to ingest a file I get:
# ./filemgr-client --url http://localhost:9000 --operation --ingestProduct --productName test.txt --productStructure Flat --productTypeName GenericFile --metadataFile file:///tmp/test.txt.met --refs file:///tmp/test.txt
ERROR: Failed to ingest product 'test.txt' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error ingesting product [org.apache.oodt.cas.filemgr.structs.Product@5eebbe] : HTTP method failed: HTTP/1.1 404 Not Found
Any ideas what is going on?
Thanks
K
*********************************************************
THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS
CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED
INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL
OR INDIVIDUALS NAMED ABOVE.
If the reader is not the intended recipient, or the
employee or agent responsible to deliver it to the
intended recipient, you are hereby notified that any
dissemination, distribution or copying of this
communication is strictly prohibited. If you have
received this communication in error, please reply to the
sender to notify us of the error and delete the original
message. Thank You.
Re: Multiple File Managers
Posted by Tom Barber <to...@meteorite.bi>.
Can spell, try again:
https://cwiki.apache.org/confluence/display/OODT/File+Manager+Scale+Out+Planning
On Thu, Mar 17, 2016 at 10:56 PM, Tom Barber <to...@meteorite.bi>
wrote:
> Hi guys
>
> I've started doing some planning about support for multiple filemanagers
> and scale out stuff, here:
> https://cwiki.apache.org/confluence/display/OODT/File+Manage+Scale+Out+Planning
>
> Work in progress and Chris you understand the underlying infrastructure
> far better than me, so let me know if I'm thinking about it the wrong way
> or going off piste.
>
> Val, add in any specific requirements you have to the doc and we'll look
> at addressing them.
>
> Cheers
>
> Tom
>
> On Thu, Mar 17, 2016 at 4:01 PM, Tom Barber <to...@meteorite.bi>
> wrote:
>
>> Okay clearly this is a requirement not just something I made up because I
>> clearly don't have enough work and could do with spending more time on
>> unpaid stuff! ;)
>>
>> I'll draft a plan in the wiki to try and formulate the best way for cross
>> FM comms and let people critique it before diving in head first.
>>
>> Tom
>>
>> On Thu, Mar 17, 2016 at 1:03 AM, Mallder, Valerie <
>> Valerie.Mallder@jhuapl.edu> wrote:
>>
>>> Hi Tom,
>>>
>>> When you figure out how how to do it could you document it, or post a
>>> complete example to the mailing list so that the info can be found with a
>>> search of the email archive?
>>>
>>> Val
>>>
>>>
>>>
>>> Sent with Good (www.good.com)
>>> ________________________________
>>> From: Tom Barber <to...@meteorite.bi>
>>> Sent: Wednesday, March 16, 2016 7:54:51 PM
>>> To: dev@oodt.apache.org
>>> Subject: Re: Multiple File Managers
>>>
>>> Yeah I was wondering if there was some catalog sharing going on. Okay,
>>> i'll
>>> have a play over the next few days, see what ideas I can come up with.
>>>
>>> Ta!
>>>
>>> On Wed, Mar 16, 2016 at 11:50 PM, Chris Mattmann <
>>> chris.mattmann@gmail.com>
>>> wrote:
>>>
>>> > yeah not as simple as a config file, but doable via PCS
>>> > as a base. I always wanted to use PCS to do multi-horizontal
>>> > scale out. Right now people just put up multiple FMs and WMs
>>> > and then write a script to coordinate them. Or rely on the fact
>>> > that they write to e.g., the same index, or same file system or
>>> > etc to deal with that.
>>> >
>>> > —
>>> > Chris Mattmann
>>> > chris.mattmann@gmail.com
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > -----Original Message-----
>>> > From: Tom Barber <to...@meteorite.bi>
>>> > Reply-To: <de...@oodt.apache.org>
>>> > Date: Wednesday, March 16, 2016 at 4:42 PM
>>> > To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>>> > Subject: Re: Multiple File Managers
>>> >
>>> > >Okay, I'm missing something here, if you have FM on Site A, FM on
>>> Site B,
>>> > >and PCS to unify them.....
>>> > >
>>> > >I look at ops ui and fm client configs and only see entries for single
>>> > >services, if I want to run filemgr-client on my local file manager,
>>> but in
>>> > >reality also search filemanager on Site B at the same time as if it
>>> were a
>>> > >single unit, how would that occur? If they are on separate networks
>>> they
>>> > >can't broadcast, but I also can't find a way to set that type of
>>> stuff up
>>> > >in a config file.
>>> > >
>>> > >Ta
>>> > >
>>> > >Tom
>>> > >
>>> > >On Wed, Mar 16, 2016 at 10:43 PM, Tom Barber <tom.barber@meteorite.bi
>>> >
>>> > >wrote:
>>> > >
>>> > >> I knew I was over googling it!
>>> > >>
>>> > >> Thanks
>>> > >>
>>> > >> On Wed, Mar 16, 2016 at 10:33 PM, Mattmann, Chris A (3980) <
>>> > >> chris.a.mattmann@jpl.nasa.gov> wrote:
>>> > >>
>>> > >>> multiple file managers is easy ;)
>>> > >>>
>>> > >>> just create them - the unifying framework is PCS when you do things
>>> > >>> like that.
>>> > >>>
>>> > >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > >>> Chris Mattmann, Ph.D.
>>> > >>> Chief Architect
>>> > >>> Instrument Software and Science Data Systems Section (398)
>>> > >>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> > >>> Office: 168-519, Mailstop: 168-527
>>> > >>> Email: chris.a.mattmann@nasa.gov
>>> > >>> WWW: http://sunset.usc.edu/~mattmann/
>>> > >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > >>> Director, Information Retrieval and Data Science Group (IRDS)
>>> > >>> Adjunct Associate Professor, Computer Science Department
>>> > >>> University of Southern California, Los Angeles, CA 90089 USA
>>> > >>> WWW: http://irds.usc.edu/
>>> > >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > >>>
>>> > >>>
>>> > >>>
>>> > >>>
>>> > >>>
>>> > >>> -----Original Message-----
>>> > >>> From: Tom Barber <to...@meteorite.bi>
>>> > >>> Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>>> > >>> Date: Wednesday, March 16, 2016 at 2:57 PM
>>> > >>> To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>>> > >>> Subject: Multiple File Managers
>>> > >>>
>>> > >>> >Here's something I assume is possible but can't find on the wiki.
>>> > >>> >
>>> > >>> >I know for workflow stuff you add resources to the pool,
>>> configure a
>>> > >>> >workflow manager and scale up.
>>> > >>> >
>>> > >>> >If I want to actually do the "distributed" stuff though, say
>>> multiple
>>> > >>> file
>>> > >>> >managers, is this where the Grid components come in?
>>> > >>> >
>>> > >>> >What I'm looking for is a setup of multiple file managers in
>>> different
>>> > >>> >locations but I'd like Joe User to query the whole lot like it was
>>> > >>>one.
>>> > >>> >But
>>> > >>> >coming from single node Radix land there isn't much consumer info
>>> on
>>> > >>>the
>>> > >>> >scaling out.
>>> > >>> >
>>> > >>> >Cheers
>>> > >>> >
>>> > >>> >Tom
>>> > >>>
>>> > >>>
>>> > >>
>>> >
>>> >
>>> >
>>>
>>
>>
>
Re: Multiple File Managers
Posted by Tom Barber <to...@meteorite.bi>.
Hi guys
I've started doing some planning about support for multiple filemanagers
and scale out stuff, here:
https://cwiki.apache.org/confluence/display/OODT/File+Manage+Scale+Out+Planning
Work in progress and Chris you understand the underlying infrastructure far
better than me, so let me know if I'm thinking about it the wrong way or
going off piste.
Val, add in any specific requirements you have to the doc and we'll look at
addressing them.
Cheers
Tom
On Thu, Mar 17, 2016 at 4:01 PM, Tom Barber <to...@meteorite.bi> wrote:
> Okay clearly this is a requirement not just something I made up because I
> clearly don't have enough work and could do with spending more time on
> unpaid stuff! ;)
>
> I'll draft a plan in the wiki to try and formulate the best way for cross
> FM comms and let people critique it before diving in head first.
>
> Tom
>
> On Thu, Mar 17, 2016 at 1:03 AM, Mallder, Valerie <
> Valerie.Mallder@jhuapl.edu> wrote:
>
>> Hi Tom,
>>
>> When you figure out how how to do it could you document it, or post a
>> complete example to the mailing list so that the info can be found with a
>> search of the email archive?
>>
>> Val
>>
>>
>>
>> Sent with Good (www.good.com)
>> ________________________________
>> From: Tom Barber <to...@meteorite.bi>
>> Sent: Wednesday, March 16, 2016 7:54:51 PM
>> To: dev@oodt.apache.org
>> Subject: Re: Multiple File Managers
>>
>> Yeah I was wondering if there was some catalog sharing going on. Okay,
>> i'll
>> have a play over the next few days, see what ideas I can come up with.
>>
>> Ta!
>>
>> On Wed, Mar 16, 2016 at 11:50 PM, Chris Mattmann <
>> chris.mattmann@gmail.com>
>> wrote:
>>
>> > yeah not as simple as a config file, but doable via PCS
>> > as a base. I always wanted to use PCS to do multi-horizontal
>> > scale out. Right now people just put up multiple FMs and WMs
>> > and then write a script to coordinate them. Or rely on the fact
>> > that they write to e.g., the same index, or same file system or
>> > etc to deal with that.
>> >
>> > —
>> > Chris Mattmann
>> > chris.mattmann@gmail.com
>> >
>> >
>> >
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: Tom Barber <to...@meteorite.bi>
>> > Reply-To: <de...@oodt.apache.org>
>> > Date: Wednesday, March 16, 2016 at 4:42 PM
>> > To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>> > Subject: Re: Multiple File Managers
>> >
>> > >Okay, I'm missing something here, if you have FM on Site A, FM on Site
>> B,
>> > >and PCS to unify them.....
>> > >
>> > >I look at ops ui and fm client configs and only see entries for single
>> > >services, if I want to run filemgr-client on my local file manager,
>> but in
>> > >reality also search filemanager on Site B at the same time as if it
>> were a
>> > >single unit, how would that occur? If they are on separate networks
>> they
>> > >can't broadcast, but I also can't find a way to set that type of stuff
>> up
>> > >in a config file.
>> > >
>> > >Ta
>> > >
>> > >Tom
>> > >
>> > >On Wed, Mar 16, 2016 at 10:43 PM, Tom Barber <to...@meteorite.bi>
>> > >wrote:
>> > >
>> > >> I knew I was over googling it!
>> > >>
>> > >> Thanks
>> > >>
>> > >> On Wed, Mar 16, 2016 at 10:33 PM, Mattmann, Chris A (3980) <
>> > >> chris.a.mattmann@jpl.nasa.gov> wrote:
>> > >>
>> > >>> multiple file managers is easy ;)
>> > >>>
>> > >>> just create them - the unifying framework is PCS when you do things
>> > >>> like that.
>> > >>>
>> > >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > >>> Chris Mattmann, Ph.D.
>> > >>> Chief Architect
>> > >>> Instrument Software and Science Data Systems Section (398)
>> > >>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> > >>> Office: 168-519, Mailstop: 168-527
>> > >>> Email: chris.a.mattmann@nasa.gov
>> > >>> WWW: http://sunset.usc.edu/~mattmann/
>> > >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > >>> Director, Information Retrieval and Data Science Group (IRDS)
>> > >>> Adjunct Associate Professor, Computer Science Department
>> > >>> University of Southern California, Los Angeles, CA 90089 USA
>> > >>> WWW: http://irds.usc.edu/
>> > >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>> -----Original Message-----
>> > >>> From: Tom Barber <to...@meteorite.bi>
>> > >>> Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>> > >>> Date: Wednesday, March 16, 2016 at 2:57 PM
>> > >>> To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>> > >>> Subject: Multiple File Managers
>> > >>>
>> > >>> >Here's something I assume is possible but can't find on the wiki.
>> > >>> >
>> > >>> >I know for workflow stuff you add resources to the pool, configure
>> a
>> > >>> >workflow manager and scale up.
>> > >>> >
>> > >>> >If I want to actually do the "distributed" stuff though, say
>> multiple
>> > >>> file
>> > >>> >managers, is this where the Grid components come in?
>> > >>> >
>> > >>> >What I'm looking for is a setup of multiple file managers in
>> different
>> > >>> >locations but I'd like Joe User to query the whole lot like it was
>> > >>>one.
>> > >>> >But
>> > >>> >coming from single node Radix land there isn't much consumer info
>> on
>> > >>>the
>> > >>> >scaling out.
>> > >>> >
>> > >>> >Cheers
>> > >>> >
>> > >>> >Tom
>> > >>>
>> > >>>
>> > >>
>> >
>> >
>> >
>>
>
>
Re: Multiple File Managers
Posted by Tom Barber <to...@meteorite.bi>.
Okay clearly this is a requirement not just something I made up because I
clearly don't have enough work and could do with spending more time on
unpaid stuff! ;)
I'll draft a plan in the wiki to try and formulate the best way for cross
FM comms and let people critique it before diving in head first.
Tom
On Thu, Mar 17, 2016 at 1:03 AM, Mallder, Valerie <
Valerie.Mallder@jhuapl.edu> wrote:
> Hi Tom,
>
> When you figure out how how to do it could you document it, or post a
> complete example to the mailing list so that the info can be found with a
> search of the email archive?
>
> Val
>
>
>
> Sent with Good (www.good.com)
> ________________________________
> From: Tom Barber <to...@meteorite.bi>
> Sent: Wednesday, March 16, 2016 7:54:51 PM
> To: dev@oodt.apache.org
> Subject: Re: Multiple File Managers
>
> Yeah I was wondering if there was some catalog sharing going on. Okay, i'll
> have a play over the next few days, see what ideas I can come up with.
>
> Ta!
>
> On Wed, Mar 16, 2016 at 11:50 PM, Chris Mattmann <chris.mattmann@gmail.com
> >
> wrote:
>
> > yeah not as simple as a config file, but doable via PCS
> > as a base. I always wanted to use PCS to do multi-horizontal
> > scale out. Right now people just put up multiple FMs and WMs
> > and then write a script to coordinate them. Or rely on the fact
> > that they write to e.g., the same index, or same file system or
> > etc to deal with that.
> >
> > —
> > Chris Mattmann
> > chris.mattmann@gmail.com
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Tom Barber <to...@meteorite.bi>
> > Reply-To: <de...@oodt.apache.org>
> > Date: Wednesday, March 16, 2016 at 4:42 PM
> > To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> > Subject: Re: Multiple File Managers
> >
> > >Okay, I'm missing something here, if you have FM on Site A, FM on Site
> B,
> > >and PCS to unify them.....
> > >
> > >I look at ops ui and fm client configs and only see entries for single
> > >services, if I want to run filemgr-client on my local file manager, but
> in
> > >reality also search filemanager on Site B at the same time as if it
> were a
> > >single unit, how would that occur? If they are on separate networks they
> > >can't broadcast, but I also can't find a way to set that type of stuff
> up
> > >in a config file.
> > >
> > >Ta
> > >
> > >Tom
> > >
> > >On Wed, Mar 16, 2016 at 10:43 PM, Tom Barber <to...@meteorite.bi>
> > >wrote:
> > >
> > >> I knew I was over googling it!
> > >>
> > >> Thanks
> > >>
> > >> On Wed, Mar 16, 2016 at 10:33 PM, Mattmann, Chris A (3980) <
> > >> chris.a.mattmann@jpl.nasa.gov> wrote:
> > >>
> > >>> multiple file managers is easy ;)
> > >>>
> > >>> just create them - the unifying framework is PCS when you do things
> > >>> like that.
> > >>>
> > >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>> Chris Mattmann, Ph.D.
> > >>> Chief Architect
> > >>> Instrument Software and Science Data Systems Section (398)
> > >>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > >>> Office: 168-519, Mailstop: 168-527
> > >>> Email: chris.a.mattmann@nasa.gov
> > >>> WWW: http://sunset.usc.edu/~mattmann/
> > >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>> Director, Information Retrieval and Data Science Group (IRDS)
> > >>> Adjunct Associate Professor, Computer Science Department
> > >>> University of Southern California, Los Angeles, CA 90089 USA
> > >>> WWW: http://irds.usc.edu/
> > >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> -----Original Message-----
> > >>> From: Tom Barber <to...@meteorite.bi>
> > >>> Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> > >>> Date: Wednesday, March 16, 2016 at 2:57 PM
> > >>> To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> > >>> Subject: Multiple File Managers
> > >>>
> > >>> >Here's something I assume is possible but can't find on the wiki.
> > >>> >
> > >>> >I know for workflow stuff you add resources to the pool, configure a
> > >>> >workflow manager and scale up.
> > >>> >
> > >>> >If I want to actually do the "distributed" stuff though, say
> multiple
> > >>> file
> > >>> >managers, is this where the Grid components come in?
> > >>> >
> > >>> >What I'm looking for is a setup of multiple file managers in
> different
> > >>> >locations but I'd like Joe User to query the whole lot like it was
> > >>>one.
> > >>> >But
> > >>> >coming from single node Radix land there isn't much consumer info on
> > >>>the
> > >>> >scaling out.
> > >>> >
> > >>> >Cheers
> > >>> >
> > >>> >Tom
> > >>>
> > >>>
> > >>
> >
> >
> >
>
RE: Multiple File Managers
Posted by "Mallder, Valerie" <Va...@jhuapl.edu>.
Hi Tom,
When you figure out how how to do it could you document it, or post a complete example to the mailing list so that the info can be found with a search of the email archive?
Val
Sent with Good (www.good.com)
________________________________
From: Tom Barber <to...@meteorite.bi>
Sent: Wednesday, March 16, 2016 7:54:51 PM
To: dev@oodt.apache.org
Subject: Re: Multiple File Managers
Yeah I was wondering if there was some catalog sharing going on. Okay, i'll
have a play over the next few days, see what ideas I can come up with.
Ta!
On Wed, Mar 16, 2016 at 11:50 PM, Chris Mattmann <ch...@gmail.com>
wrote:
> yeah not as simple as a config file, but doable via PCS
> as a base. I always wanted to use PCS to do multi-horizontal
> scale out. Right now people just put up multiple FMs and WMs
> and then write a script to coordinate them. Or rely on the fact
> that they write to e.g., the same index, or same file system or
> etc to deal with that.
>
> —
> Chris Mattmann
> chris.mattmann@gmail.com
>
>
>
>
>
>
> -----Original Message-----
> From: Tom Barber <to...@meteorite.bi>
> Reply-To: <de...@oodt.apache.org>
> Date: Wednesday, March 16, 2016 at 4:42 PM
> To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> Subject: Re: Multiple File Managers
>
> >Okay, I'm missing something here, if you have FM on Site A, FM on Site B,
> >and PCS to unify them.....
> >
> >I look at ops ui and fm client configs and only see entries for single
> >services, if I want to run filemgr-client on my local file manager, but in
> >reality also search filemanager on Site B at the same time as if it were a
> >single unit, how would that occur? If they are on separate networks they
> >can't broadcast, but I also can't find a way to set that type of stuff up
> >in a config file.
> >
> >Ta
> >
> >Tom
> >
> >On Wed, Mar 16, 2016 at 10:43 PM, Tom Barber <to...@meteorite.bi>
> >wrote:
> >
> >> I knew I was over googling it!
> >>
> >> Thanks
> >>
> >> On Wed, Mar 16, 2016 at 10:33 PM, Mattmann, Chris A (3980) <
> >> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>
> >>> multiple file managers is easy ;)
> >>>
> >>> just create them - the unifying framework is PCS when you do things
> >>> like that.
> >>>
> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> Chris Mattmann, Ph.D.
> >>> Chief Architect
> >>> Instrument Software and Science Data Systems Section (398)
> >>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>> Office: 168-519, Mailstop: 168-527
> >>> Email: chris.a.mattmann@nasa.gov
> >>> WWW: http://sunset.usc.edu/~mattmann/
> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> Director, Information Retrieval and Data Science Group (IRDS)
> >>> Adjunct Associate Professor, Computer Science Department
> >>> University of Southern California, Los Angeles, CA 90089 USA
> >>> WWW: http://irds.usc.edu/
> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Tom Barber <to...@meteorite.bi>
> >>> Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> >>> Date: Wednesday, March 16, 2016 at 2:57 PM
> >>> To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> >>> Subject: Multiple File Managers
> >>>
> >>> >Here's something I assume is possible but can't find on the wiki.
> >>> >
> >>> >I know for workflow stuff you add resources to the pool, configure a
> >>> >workflow manager and scale up.
> >>> >
> >>> >If I want to actually do the "distributed" stuff though, say multiple
> >>> file
> >>> >managers, is this where the Grid components come in?
> >>> >
> >>> >What I'm looking for is a setup of multiple file managers in different
> >>> >locations but I'd like Joe User to query the whole lot like it was
> >>>one.
> >>> >But
> >>> >coming from single node Radix land there isn't much consumer info on
> >>>the
> >>> >scaling out.
> >>> >
> >>> >Cheers
> >>> >
> >>> >Tom
> >>>
> >>>
> >>
>
>
>
Re: Multiple File Managers
Posted by Tom Barber <to...@meteorite.bi>.
Yeah I was wondering if there was some catalog sharing going on. Okay, i'll
have a play over the next few days, see what ideas I can come up with.
Ta!
On Wed, Mar 16, 2016 at 11:50 PM, Chris Mattmann <ch...@gmail.com>
wrote:
> yeah not as simple as a config file, but doable via PCS
> as a base. I always wanted to use PCS to do multi-horizontal
> scale out. Right now people just put up multiple FMs and WMs
> and then write a script to coordinate them. Or rely on the fact
> that they write to e.g., the same index, or same file system or
> etc to deal with that.
>
> —
> Chris Mattmann
> chris.mattmann@gmail.com
>
>
>
>
>
>
> -----Original Message-----
> From: Tom Barber <to...@meteorite.bi>
> Reply-To: <de...@oodt.apache.org>
> Date: Wednesday, March 16, 2016 at 4:42 PM
> To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> Subject: Re: Multiple File Managers
>
> >Okay, I'm missing something here, if you have FM on Site A, FM on Site B,
> >and PCS to unify them.....
> >
> >I look at ops ui and fm client configs and only see entries for single
> >services, if I want to run filemgr-client on my local file manager, but in
> >reality also search filemanager on Site B at the same time as if it were a
> >single unit, how would that occur? If they are on separate networks they
> >can't broadcast, but I also can't find a way to set that type of stuff up
> >in a config file.
> >
> >Ta
> >
> >Tom
> >
> >On Wed, Mar 16, 2016 at 10:43 PM, Tom Barber <to...@meteorite.bi>
> >wrote:
> >
> >> I knew I was over googling it!
> >>
> >> Thanks
> >>
> >> On Wed, Mar 16, 2016 at 10:33 PM, Mattmann, Chris A (3980) <
> >> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>
> >>> multiple file managers is easy ;)
> >>>
> >>> just create them - the unifying framework is PCS when you do things
> >>> like that.
> >>>
> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> Chris Mattmann, Ph.D.
> >>> Chief Architect
> >>> Instrument Software and Science Data Systems Section (398)
> >>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>> Office: 168-519, Mailstop: 168-527
> >>> Email: chris.a.mattmann@nasa.gov
> >>> WWW: http://sunset.usc.edu/~mattmann/
> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> Director, Information Retrieval and Data Science Group (IRDS)
> >>> Adjunct Associate Professor, Computer Science Department
> >>> University of Southern California, Los Angeles, CA 90089 USA
> >>> WWW: http://irds.usc.edu/
> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Tom Barber <to...@meteorite.bi>
> >>> Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> >>> Date: Wednesday, March 16, 2016 at 2:57 PM
> >>> To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> >>> Subject: Multiple File Managers
> >>>
> >>> >Here's something I assume is possible but can't find on the wiki.
> >>> >
> >>> >I know for workflow stuff you add resources to the pool, configure a
> >>> >workflow manager and scale up.
> >>> >
> >>> >If I want to actually do the "distributed" stuff though, say multiple
> >>> file
> >>> >managers, is this where the Grid components come in?
> >>> >
> >>> >What I'm looking for is a setup of multiple file managers in different
> >>> >locations but I'd like Joe User to query the whole lot like it was
> >>>one.
> >>> >But
> >>> >coming from single node Radix land there isn't much consumer info on
> >>>the
> >>> >scaling out.
> >>> >
> >>> >Cheers
> >>> >
> >>> >Tom
> >>>
> >>>
> >>
>
>
>
Re: Multiple File Managers
Posted by Chris Mattmann <ch...@gmail.com>.
yeah not as simple as a config file, but doable via PCS
as a base. I always wanted to use PCS to do multi-horizontal
scale out. Right now people just put up multiple FMs and WMs
and then write a script to coordinate them. Or rely on the fact
that they write to e.g., the same index, or same file system or
etc to deal with that.
—
Chris Mattmann
chris.mattmann@gmail.com
-----Original Message-----
From: Tom Barber <to...@meteorite.bi>
Reply-To: <de...@oodt.apache.org>
Date: Wednesday, March 16, 2016 at 4:42 PM
To: "dev@oodt.apache.org" <de...@oodt.apache.org>
Subject: Re: Multiple File Managers
>Okay, I'm missing something here, if you have FM on Site A, FM on Site B,
>and PCS to unify them.....
>
>I look at ops ui and fm client configs and only see entries for single
>services, if I want to run filemgr-client on my local file manager, but in
>reality also search filemanager on Site B at the same time as if it were a
>single unit, how would that occur? If they are on separate networks they
>can't broadcast, but I also can't find a way to set that type of stuff up
>in a config file.
>
>Ta
>
>Tom
>
>On Wed, Mar 16, 2016 at 10:43 PM, Tom Barber <to...@meteorite.bi>
>wrote:
>
>> I knew I was over googling it!
>>
>> Thanks
>>
>> On Wed, Mar 16, 2016 at 10:33 PM, Mattmann, Chris A (3980) <
>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>
>>> multiple file managers is easy ;)
>>>
>>> just create them - the unifying framework is PCS when you do things
>>> like that.
>>>
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Chris Mattmann, Ph.D.
>>> Chief Architect
>>> Instrument Software and Science Data Systems Section (398)
>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> Office: 168-519, Mailstop: 168-527
>>> Email: chris.a.mattmann@nasa.gov
>>> WWW: http://sunset.usc.edu/~mattmann/
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Director, Information Retrieval and Data Science Group (IRDS)
>>> Adjunct Associate Professor, Computer Science Department
>>> University of Southern California, Los Angeles, CA 90089 USA
>>> WWW: http://irds.usc.edu/
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Tom Barber <to...@meteorite.bi>
>>> Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>>> Date: Wednesday, March 16, 2016 at 2:57 PM
>>> To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>>> Subject: Multiple File Managers
>>>
>>> >Here's something I assume is possible but can't find on the wiki.
>>> >
>>> >I know for workflow stuff you add resources to the pool, configure a
>>> >workflow manager and scale up.
>>> >
>>> >If I want to actually do the "distributed" stuff though, say multiple
>>> file
>>> >managers, is this where the Grid components come in?
>>> >
>>> >What I'm looking for is a setup of multiple file managers in different
>>> >locations but I'd like Joe User to query the whole lot like it was
>>>one.
>>> >But
>>> >coming from single node Radix land there isn't much consumer info on
>>>the
>>> >scaling out.
>>> >
>>> >Cheers
>>> >
>>> >Tom
>>>
>>>
>>
Re: Multiple File Managers
Posted by Tom Barber <to...@meteorite.bi>.
Okay, I'm missing something here, if you have FM on Site A, FM on Site B,
and PCS to unify them.....
I look at ops ui and fm client configs and only see entries for single
services, if I want to run filemgr-client on my local file manager, but in
reality also search filemanager on Site B at the same time as if it were a
single unit, how would that occur? If they are on separate networks they
can't broadcast, but I also can't find a way to set that type of stuff up
in a config file.
Ta
Tom
On Wed, Mar 16, 2016 at 10:43 PM, Tom Barber <to...@meteorite.bi>
wrote:
> I knew I was over googling it!
>
> Thanks
>
> On Wed, Mar 16, 2016 at 10:33 PM, Mattmann, Chris A (3980) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> multiple file managers is easy ;)
>>
>> just create them - the unifying framework is PCS when you do things
>> like that.
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Chief Architect
>> Instrument Software and Science Data Systems Section (398)
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 168-519, Mailstop: 168-527
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Director, Information Retrieval and Data Science Group (IRDS)
>> Adjunct Associate Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> WWW: http://irds.usc.edu/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Tom Barber <to...@meteorite.bi>
>> Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>> Date: Wednesday, March 16, 2016 at 2:57 PM
>> To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>> Subject: Multiple File Managers
>>
>> >Here's something I assume is possible but can't find on the wiki.
>> >
>> >I know for workflow stuff you add resources to the pool, configure a
>> >workflow manager and scale up.
>> >
>> >If I want to actually do the "distributed" stuff though, say multiple
>> file
>> >managers, is this where the Grid components come in?
>> >
>> >What I'm looking for is a setup of multiple file managers in different
>> >locations but I'd like Joe User to query the whole lot like it was one.
>> >But
>> >coming from single node Radix land there isn't much consumer info on the
>> >scaling out.
>> >
>> >Cheers
>> >
>> >Tom
>>
>>
>
Re: Multiple File Managers
Posted by Tom Barber <to...@meteorite.bi>.
I knew I was over googling it!
Thanks
On Wed, Mar 16, 2016 at 10:33 PM, Mattmann, Chris A (3980) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> multiple file managers is easy ;)
>
> just create them - the unifying framework is PCS when you do things
> like that.
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Director, Information Retrieval and Data Science Group (IRDS)
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> WWW: http://irds.usc.edu/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
> -----Original Message-----
> From: Tom Barber <to...@meteorite.bi>
> Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> Date: Wednesday, March 16, 2016 at 2:57 PM
> To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> Subject: Multiple File Managers
>
> >Here's something I assume is possible but can't find on the wiki.
> >
> >I know for workflow stuff you add resources to the pool, configure a
> >workflow manager and scale up.
> >
> >If I want to actually do the "distributed" stuff though, say multiple file
> >managers, is this where the Grid components come in?
> >
> >What I'm looking for is a setup of multiple file managers in different
> >locations but I'd like Joe User to query the whole lot like it was one.
> >But
> >coming from single node Radix land there isn't much consumer info on the
> >scaling out.
> >
> >Cheers
> >
> >Tom
>
>
Re: Multiple File Managers
Posted by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>.
multiple file managers is easy ;)
just create them - the unifying framework is PCS when you do things
like that.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Director, Information Retrieval and Data Science Group (IRDS)
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
WWW: http://irds.usc.edu/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Tom Barber <to...@meteorite.bi>
Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
Date: Wednesday, March 16, 2016 at 2:57 PM
To: "dev@oodt.apache.org" <de...@oodt.apache.org>
Subject: Multiple File Managers
>Here's something I assume is possible but can't find on the wiki.
>
>I know for workflow stuff you add resources to the pool, configure a
>workflow manager and scale up.
>
>If I want to actually do the "distributed" stuff though, say multiple file
>managers, is this where the Grid components come in?
>
>What I'm looking for is a setup of multiple file managers in different
>locations but I'd like Joe User to query the whole lot like it was one.
>But
>coming from single node Radix land there isn't much consumer info on the
>scaling out.
>
>Cheers
>
>Tom