You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@manifoldcf.apache.org by Marisol Redondo <ma...@gmail.com> on 2017/02/21 13:42:40 UTC
Metadata adjuster
Hi.
I'm trying to use metadata adjuster to add one field to the solr index but
doesn't inject the field into a solr's field.
Maybe I'm misundertaning the use of the metada adjuster, but I have read in
the documentation (https://manifoldcf.apache.org/release/release-1.10/en_
US/end-user-documentation.html) that I can add metadata to the document
that is going to be indexed into solr, but the solr instance gave me the
error "missing required field: facetContentType".
ManifoldCF Job pipeline:
1. Repository (type web repository)
2. Transformation (Tikka Metadata Extractor)
3. Transformation (type Metada Adjuster)
4. Output (Solr 6)
ManifoldCF Job Metadata Expressions tab:
Parameter name: "facetContentType"
Remove this parameter: false
Expresion: xxxx (the literal text value I want in facetContentType)
Solr schema:
.....
<field name="facetContentType" type="string" indexed="true" stored="true"
required="true"/>
....
The error logged in ManifoldCF is:
Error from server at http://solrServer:port/solr/c
<http://revnetsolrdev:8983/solr/sites>ore: [doc=https://....../index.aspx]
missing required field: facetContentType.
Thanks for your help
Re: Metadata adjuster
Posted by Karl Wright <da...@gmail.com>.
Hi Marisol,
I proved here that the Metadata Adjuster works as described in the
documentation, and that it can add metadata fields as promised. I posted
the proof. So I really don't know where the disconnect lies.
Karl
On Tue, Feb 28, 2017 at 5:35 AM, Marisol Redondo <
marisol.redondo.garcia@gmail.com> wrote:
> Hi, sorry for not answering early. I have been trying to create my own
> tranformation or changing the NullOutput one, but I'm still trying to do it
> (I'm having problems installing MCF to modifiy it)
>
> But, we have found a way of fixing the problem.
>
> The Metadata ajuster doesn't add new metadata into the document before
> injecting into solr, it only can change the value of a existed metadata (at
> least, this is the behaviour we are seeing).
> So, to make it works we have:
> - In the Tika transformer we are using, we add the next field mapping:
> Metadata field Name: *existedMetadata*
> Final field name: facetContentType
> - In the Metadata Adjuster:
> Parameter Name: facetContentType
> Expresion: *myText*
>
> Thanks for your help.
>
>
>
Re: Metadata adjuster
Posted by Marisol Redondo <ma...@gmail.com>.
Hi, sorry for not answering early. I have been trying to create my own
tranformation or changing the NullOutput one, but I'm still trying to do it
(I'm having problems installing MCF to modifiy it)
But, we have found a way of fixing the problem.
The Metadata ajuster doesn't add new metadata into the document before
injecting into solr, it only can change the value of a existed metadata (at
least, this is the behaviour we are seeing).
So, to make it works we have:
- In the Tika transformer we are using, we add the next field mapping:
Metadata field Name: *existedMetadata*
Final field name: facetContentType
- In the Metadata Adjuster:
Parameter Name: facetContentType
Expresion: *myText*
Thanks for your help.
Re: Metadata adjuster
Posted by Karl Wright <da...@gmail.com>.
Ok, I added a bit of extra info to the null output connector document
ingestion simple history logging on trunk. This extra info summarizes
attributes and their counts. I then created a job, with an attribute and
fixed value, and left all other defaults in place. The output looks like
this:
>>>>>>
02-22-2017 17:49:18.851 document ingest (null)
file:/C:/wip/mcf/trunk/README.txt
OK 4970 1 "myAttribute":1,"uri":1
02-22-2017 17:49:18.850 document ingest (null)
file:/C:/wip/mcf/trunk/Livelink.patch
OK 2201 1 "myAttribute":1,"uri":1
02-22-2017 17:49:18.840 document ingest (null)
file:/C:/wip/mcf/trunk/lib/c3p0-0.9.1.1.jar
OK 608376 1 "myAttribute":1,"uri":1
02-22-2017 17:49:18.830 document ingest (null)
file:/C:/wip/mcf/trunk/build.xml.svnpatch.rej
OK 515 1 "myAttribute":1,"uri":1
<<<<<<
It clearly picks up the attribute I injected, and indeed another one that
comes from the file system repository connector too. Then I unchecked both
checkboxes in the metadata adjuster stage, and ran some more documents.
This is the result:
>>>>>>
02-22-2017 17:54:31.020 document ingest (null)
file:/C:/wip/mcf/trunk/dist/connector-common-lib/jna-4.1.0.jar
OK 914597 1 "myAttribute":1
02-22-2017 17:54:30.985 document ingest (null)
file:/C:/wip/mcf/trunk/lib/google-http-client-jackson2-1.19.0...
.jar
OK 6720 1 "myAttribute":1
<<<<<<
As you can see, it continued to inject the attribute, but now it no longer
passes through the upstream attribute. This is working as designed.
So it seems clear that the issue must be related to either the Solr output
connector, or to the Solr configuration. If you are using MCF 2.4, that
does *not* have the SolrJ 6.x version you will need to work with Solr 6.x.
That may well be where the trouble lies. Please upgrade to MCF 2.6 to rule
out that possibility. If that does not fix the issue, then I will bring
one of our resident Solr experts into the conversation.
Thanks,
Karl
On Wed, Feb 22, 2017 at 11:51 AM, Karl Wright <da...@gmail.com> wrote:
> Ah, sorry once again. It is definitely the update/extract handler in the
> log entry you sent.
>
> I am quite busy at the moment and will review this evening further.
>
> Thanks,
> Karl
>
>
> On Wed, Feb 22, 2017 at 11:21 AM, Karl Wright <da...@gmail.com> wrote:
>
>> Hi Marisol,
>>
>> The [INFO] log statement you sent earlier was not an /update/extract
>> request, and your Solr connection is set up to send to the Solr Cell
>> /update/extract endpoint. Can you look again in your logs and find the
>> *right* [INFO] statement? Thanks!!
>>
>> Karl
>>
>>
>> On Wed, Feb 22, 2017 at 10:52 AM, Marisol Redondo <
>> marisol.redondo.garcia@gmail.com> wrote:
>>
>>> I have formatted it so you have all the information
>>>
>>> Name: Sites solr dev Description: sites core in solr dev
>>> ________________________________________
>>> Connection type: Solr Max connections: 10
>>> ________________________________________
>>> Parameters:
>>> User ID=
>>> ZooKeeper znode path=
>>> Socket timeout=900
>>> Server remove handler=/update
>>> Included mime types=
>>> Use extract update handler=true
>>> Solr created date field name=
>>> ZooKeeper client timeout=60
>>> Solr modified date field name=
>>> Solr core name=sites
>>> Server protocol=http
>>> Realm=
>>> Server name=solrdev
>>> Server status handler=/admin/ping
>>> Password=********
>>> Excluded mime types=
>>> Commits=true
>>> Maximum document length=
>>> Server port=8983
>>> Connection timeout=60
>>> Solr type=standard
>>> Solr filename field name=
>>> Commit within=
>>> Solr id field name=id
>>> Solr mime type field name=
>>> ZooKeeper connect timeout=60
>>> Collection=collection1
>>> Server update handler=/update/extract
>>> Server web application=solr
>>> Solr original size field name=
>>> Solr indexed date field name=
>>> Solr content field name=
>>> ZooKeeper hosts: Host Port:
>>> localhost 2181
>>>
>>> Arguments: Name Value
>>> No arguments
>>>
>>> ________________________________________
>>> Connection status: Connection working
>>>
>>>
>>>
>>>
>>
>
Re: Metadata adjuster
Posted by Karl Wright <da...@gmail.com>.
Ah, sorry once again. It is definitely the update/extract handler in the
log entry you sent.
I am quite busy at the moment and will review this evening further.
Thanks,
Karl
On Wed, Feb 22, 2017 at 11:21 AM, Karl Wright <da...@gmail.com> wrote:
> Hi Marisol,
>
> The [INFO] log statement you sent earlier was not an /update/extract
> request, and your Solr connection is set up to send to the Solr Cell
> /update/extract endpoint. Can you look again in your logs and find the
> *right* [INFO] statement? Thanks!!
>
> Karl
>
>
> On Wed, Feb 22, 2017 at 10:52 AM, Marisol Redondo <
> marisol.redondo.garcia@gmail.com> wrote:
>
>> I have formatted it so you have all the information
>>
>> Name: Sites solr dev Description: sites core in solr dev
>> ________________________________________
>> Connection type: Solr Max connections: 10
>> ________________________________________
>> Parameters:
>> User ID=
>> ZooKeeper znode path=
>> Socket timeout=900
>> Server remove handler=/update
>> Included mime types=
>> Use extract update handler=true
>> Solr created date field name=
>> ZooKeeper client timeout=60
>> Solr modified date field name=
>> Solr core name=sites
>> Server protocol=http
>> Realm=
>> Server name=solrdev
>> Server status handler=/admin/ping
>> Password=********
>> Excluded mime types=
>> Commits=true
>> Maximum document length=
>> Server port=8983
>> Connection timeout=60
>> Solr type=standard
>> Solr filename field name=
>> Commit within=
>> Solr id field name=id
>> Solr mime type field name=
>> ZooKeeper connect timeout=60
>> Collection=collection1
>> Server update handler=/update/extract
>> Server web application=solr
>> Solr original size field name=
>> Solr indexed date field name=
>> Solr content field name=
>> ZooKeeper hosts: Host Port:
>> localhost 2181
>>
>> Arguments: Name Value
>> No arguments
>>
>> ________________________________________
>> Connection status: Connection working
>>
>>
>>
>>
>
Re: Metadata adjuster
Posted by Karl Wright <da...@gmail.com>.
Hi Marisol,
The [INFO] log statement you sent earlier was not an /update/extract
request, and your Solr connection is set up to send to the Solr Cell
/update/extract endpoint. Can you look again in your logs and find the
*right* [INFO] statement? Thanks!!
Karl
On Wed, Feb 22, 2017 at 10:52 AM, Marisol Redondo <
marisol.redondo.garcia@gmail.com> wrote:
> I have formatted it so you have all the information
>
> Name: Sites solr dev Description: sites core in solr dev
> ________________________________________
> Connection type: Solr Max connections: 10
> ________________________________________
> Parameters:
> User ID=
> ZooKeeper znode path=
> Socket timeout=900
> Server remove handler=/update
> Included mime types=
> Use extract update handler=true
> Solr created date field name=
> ZooKeeper client timeout=60
> Solr modified date field name=
> Solr core name=sites
> Server protocol=http
> Realm=
> Server name=solrdev
> Server status handler=/admin/ping
> Password=********
> Excluded mime types=
> Commits=true
> Maximum document length=
> Server port=8983
> Connection timeout=60
> Solr type=standard
> Solr filename field name=
> Commit within=
> Solr id field name=id
> Solr mime type field name=
> ZooKeeper connect timeout=60
> Collection=collection1
> Server update handler=/update/extract
> Server web application=solr
> Solr original size field name=
> Solr indexed date field name=
> Solr content field name=
> ZooKeeper hosts: Host Port:
> localhost 2181
>
> Arguments: Name Value
> No arguments
>
> ________________________________________
> Connection status: Connection working
>
>
>
>
Re: Metadata adjuster
Posted by Marisol Redondo <ma...@gmail.com>.
I have formatted it so you have all the information
Name: Sites solr dev Description: sites core in solr dev
________________________________________
Connection type: Solr Max connections: 10
________________________________________
Parameters:
User ID=
ZooKeeper znode path=
Socket timeout=900
Server remove handler=/update
Included mime types=
Use extract update handler=true
Solr created date field name=
ZooKeeper client timeout=60
Solr modified date field name=
Solr core name=sites
Server protocol=http
Realm=
Server name=solrdev
Server status handler=/admin/ping
Password=********
Excluded mime types=
Commits=true
Maximum document length=
Server port=8983
Connection timeout=60
Solr type=standard
Solr filename field name=
Commit within=
Solr id field name=id
Solr mime type field name=
ZooKeeper connect timeout=60
Collection=collection1
Server update handler=/update/extract
Server web application=solr
Solr original size field name=
Solr indexed date field name=
Solr content field name=
ZooKeeper hosts: Host Port:
localhost 2181
Arguments: Name Value
No arguments
________________________________________
Connection status: Connection working
Re: Metadata adjuster
Posted by Karl Wright <da...@gmail.com>.
Ah, never mind -- I need you instead to view the Solr connection, and paste
that in an email. Basically, I want to be sure you are not inadvertantly
disabling metadata to Solr.
Thanks,
Karl
On Wed, Feb 22, 2017 at 10:39 AM, Karl Wright <da...@gmail.com> wrote:
> This is how the email appears here:
>
> >>>>>>
>
> 4.
>
> Bottom of Form
>
>
>
> Marisol Redondo
>
> Email: mredondo@revenue.ie
>
> Phone: 35428
>
>
>
> Please note that Revenue cannot guarantee that any personal and sensitive data, sent in plain text via standard email, is fully secure. Customers who choose to use this channel are deemed to have accepted any risk involved. The alternative communication methods offered by Revenue include standard post and the option to use our (encrypted) MyEnquiries service which is available within myAccount and ROS. You can register for either myAccount or ROS on the Revenue website.
>
>
>
> Tabhair faoi deara nach féidir leis na Coimisinéirí Ioncaim ráthaíocht a thabhairt go bhfuil aon sonraí pearsanta agus íogair a gcuirtear isteach i ngnáth-théacs trí r-phost caighdeánach go huile is go hiomlán slán. Meastar go nglacann custaiméirí a úsáideann an cainéal seo le haon riosca bainteach. I measc na modhanna cumarsáide eile atá ag na Coimisinéirí ná post caighdeánach agus an rogha ár seirbhís (criptithe) M'Fhiosruithe a úsáid, tá sí ar fáil laistigh de MoChúrsaí agus ROS. Is féidir leat clárú le haghaidh ceachtar MoChúrsaí nó ROS ar shuíomh gréasáin na gCoimisinéirí.
>
> <<<<<<
>
> In other words I cannot see anything from the 4. stage.
>
>
> Thanks,
>
> Karl
>
>
> On Wed, Feb 22, 2017 at 10:37 AM, Marisol Redondo <
> marisol.redondo.garcia@gmail.com> wrote:
>
>> I was trying with "Keep all incoming metadata" set to false and too true,
>> but I'll take your advice and set to true.
>>
>> I don't know why you can't see it, but it's the 4 stage
>>
>> On 22 February 2017 at 15:26, Karl Wright <da...@gmail.com> wrote:
>>
>>> Hi Marisol,
>>>
>>> Some observations.
>>> (1) It makes no sense to have "Keep all incoming metadata" set to false,
>>> since that will filter out everything that your tika extractor extracts. I
>>> doubt that is what you have intended.
>>> (2) I can't see the Solr output configuration -- looks like it got
>>> truncated?
>>>
>>> Thanks,
>>> Karl
>>>
>>>
>>> On Wed, Feb 22, 2017 at 10:12 AM, Marisol Redondo <
>>> marisol.redondo.garcia@gmail.com> wrote:
>>>
>>>> Here you are:
>>>>
>>>> View a Job
>>>>
>>>> Top of Form
>>>>
>>>>
>>>> ------------------------------
>>>>
>>>> Name:
>>>>
>>>> revenueToSites
>>>> ------------------------------
>>>>
>>>> Pipeline:
>>>>
>>>> Stage
>>>>
>>>> Type
>>>>
>>>> Precedent
>>>>
>>>> Description
>>>>
>>>> Connection name
>>>>
>>>> 1.
>>>>
>>>> Repository
>>>>
>>>> Revenue Website
>>>>
>>>> 2.
>>>>
>>>> Transformation
>>>>
>>>> 1.
>>>>
>>>> Tikka Metadata Extractor
>>>>
>>>> 3.
>>>>
>>>> Transformation
>>>>
>>>> 2.
>>>>
>>>> Set mimeType and facetContentType
>>>>
>>>> customField
>>>>
>>>> 4.
>>>>
>>>> Output
>>>>
>>>> 3.
>>>>
>>>> sites solr dev
>>>>
>>>> Notifications:
>>>>
>>>> Stage
>>>>
>>>> Description
>>>>
>>>> Connection name
>>>>
>>>> No notification connections
>>>> ------------------------------
>>>>
>>>> Priority:
>>>>
>>>> 5
>>>>
>>>> Start method:
>>>>
>>>> Don't automatically start
>>>> ------------------------------
>>>>
>>>> Schedule type:
>>>>
>>>> Scan every document once
>>>>
>>>> Minimum recrawl interval:
>>>>
>>>> Not applicable
>>>>
>>>> Maximum recrawl interval:
>>>>
>>>> Not applicable
>>>>
>>>> Expiration interval:
>>>>
>>>> Not applicable
>>>>
>>>> Reseed interval:
>>>>
>>>> Not applicable
>>>> ------------------------------
>>>>
>>>> No scheduled run times
>>>> ------------------------------
>>>>
>>>> Maximum hop count for link type 'link':
>>>>
>>>> Unlimited
>>>>
>>>> Maximum hop count for link type 'redirect':
>>>>
>>>> Unlimited
>>>> ------------------------------
>>>>
>>>> Hop count mode:
>>>>
>>>> Delete unreachable documents
>>>> ------------------------------
>>>>
>>>> 1.
>>>>
>>>> Seeds:
>>>>
>>>> https://xxxxxx/index.aspx
>>>> <https://preview.revenuedomain.ie/en/press-office/index.aspx>
>>>> ------------------------------
>>>>
>>>> No canonicalization specified - all URLs will be reordered and have all
>>>> sessions removed
>>>> ------------------------------
>>>>
>>>> No mappings specified; will accept all URLs
>>>> ------------------------------
>>>>
>>>> Include only hosts matching seeds?
>>>>
>>>> yes
>>>> ------------------------------
>>>>
>>>> Include in crawl:
>>>>
>>>> .*
>>>> ------------------------------
>>>>
>>>> Include in index:
>>>>
>>>> .*
>>>> ------------------------------
>>>>
>>>> Exclude from crawl:
>>>>
>>>> \.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|w
>>>> mf|WMF|zip|ZIP|mpg|MPG|gz|GZ|rpm|RPM|tgz|TGZ|mov|MOV|exe|EXE
>>>> |jpeg|JPEG|bmp|BMP|js|JS|<script>|</script>|<script
>>>> type="text/javascript">)
>>>> [?*!@=].*
>>>> ------------------------------
>>>>
>>>> Exclude from index:
>>>> ------------------------------
>>>>
>>>> Exclude content from index:
>>>> ------------------------------
>>>>
>>>> No access tokens specified
>>>> ------------------------------
>>>>
>>>> Excluded headers:
>>>>
>>>> last-modified
>>>> ------------------------------
>>>>
>>>> 2.
>>>>
>>>> Field mappings:
>>>>
>>>> Metadata field name
>>>>
>>>> Final field name
>>>>
>>>> No field mapping specified
>>>> ------------------------------
>>>>
>>>> Keep all metadata:
>>>>
>>>> true
>>>> ------------------------------
>>>>
>>>> Lower names:
>>>>
>>>> false
>>>> ------------------------------
>>>>
>>>> Write limit:
>>>> ------------------------------
>>>>
>>>> Ignore Tika exceptions:
>>>>
>>>> true
>>>> ------------------------------
>>>>
>>>> Boilerplate extractor:
>>>>
>>>> -- No extraction selected --
>>>> ------------------------------
>>>>
>>>> 3.
>>>>
>>>> Metadata expressions:
>>>>
>>>> Parameter name
>>>>
>>>> Remove this parameter?
>>>>
>>>> Expression ("${fieldname}" references a field)
>>>>
>>>> facetContentType
>>>>
>>>> false
>>>>
>>>> site.ie
>>>> ------------------------------
>>>>
>>>> Keep all incoming metadata
>>>>
>>>> false
>>>>
>>>> Remove empty metadata values
>>>>
>>>> false
>>>> ------------------------------
>>>>
>>>> 4.
>>>>
>>>> Bottom of Form
>>>>
>>>>
>>>>
>>>> Marisol Redondo
>>>>
>>>> Email: mredondo@revenue.ie
>>>>
>>>> Phone: 35428
>>>>
>>>>
>>>>
>>>> Please note that Revenue cannot guarantee that any personal and sensitive data, sent in plain text via standard email, is fully secure. Customers who choose to use this channel are deemed to have accepted any risk involved. The alternative communication methods offered by Revenue include standard post and the option to use our (encrypted) MyEnquiries service which is available within myAccount and ROS. You can register for either myAccount or ROS on the Revenue website.
>>>>
>>>>
>>>>
>>>> Tabhair faoi deara nach féidir leis na Coimisinéirí Ioncaim ráthaíocht a thabhairt go bhfuil aon sonraí pearsanta agus íogair a gcuirtear isteach i ngnáth-théacs trí r-phost caighdeánach go huile is go hiomlán slán. Meastar go nglacann custaiméirí a úsáideann an cainéal seo le haon riosca bainteach. I measc na modhanna cumarsáide eile atá ag na Coimisinéirí ná post caighdeánach agus an rogha ár seirbhís (criptithe) M'Fhiosruithe a úsáid, tá sí ar fáil laistigh de MoChúrsaí agus ROS. Is féidir leat clárú le haghaidh ceachtar MoChúrsaí nó ROS ar shuíomh gréasáin na gCoimisinéirí.
>>>>
>>>>
>>>>
>>>> On 22 February 2017 at 14:53, Karl Wright <da...@gmail.com> wrote:
>>>>
>>>>> Hi Marisol,
>>>>>
>>>>> The [INFO] log entries indicate that your document has almost no
>>>>> metadata at all. But the Metadata Adjuster transformation connector is
>>>>> designed to do exactly what you want.
>>>>>
>>>>> Can you view your job, and cut and paste the View Job page into an
>>>>> email, so I can see how your metadata adjuster transformation connection
>>>>> and your solr output connections are configured? Thanks!
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Feb 22, 2017 at 8:57 AM, Marisol Redondo <
>>>>> marisol.redondo.garcia@gmail.com> wrote:
>>>>>
>>>>>> Hi Karl and thank you for this quick answer.
>>>>>>
>>>>>> I was reading the documentation of MCF 1.10 but I'm using MCF 2.5,
>>>>>> sorry for the confusion, and I think this version is compatible with solr6.
>>>>>> The pdf doesn't have any metadata or field called facetContentType,
>>>>>> this is because I'd been trying to use the Metadata Adjuster, to add a new
>>>>>> metadata/property to the doc so solr can index by this field when I'm
>>>>>> injecting the doc.
>>>>>> Should I use other transformation or is there any other way of duing
>>>>>> it?
>>>>>> I am migrating from nutch to ManifoldCF and in nutch we can do it
>>>>>> with plugins, and I was thinking that the plugins in nutch are the same as
>>>>>> the transformation connectors in MCF
>>>>>>
>>>>>> The completed error in solr is :
>>>>>>
>>>>>> 017-02-21 13:19:32.108 INFO (qtp1854778591-18) [ x:sites]
>>>>>>> o.a.s.c.PluginBag Going to create a new requestHandler with {type =
>>>>>>> requestHandler,name = /update/extract,class = solr.extraction.ExtractingRequestHandler,args
>>>>>>> = {defaults={lowernames=true,fmap.meta=ignored_,fmap.content=_
>>>>>>> text_,update.chain=add-unknown-fields-to-the-schema,df=_text_}}}
>>>>>>
>>>>>> 2017-02-21 13:19:32.454 INFO (qtp1854778591-18) [ x:sites]
>>>>>>> o.a.s.u.p.LogUpdateProcessorFactory [sites] webapp=/solr path=/up
>>>>>>
>>>>>> date/extract params={resource.name=introduction.pdf&literal.id
>>>>>>> =https://...../introduction.pdf&wt=xml&version=2.2}{} 0 347
>>>>>>
>>>>>> 2017-02-21 13:19:32.455 ERROR (qtp1854778591-18) [ x:sites]
>>>>>>> o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: [
>>>>>>
>>>>>> doc=https://....../introduction.pdf] missing required field:
>>>>>>> facetContentType
>>>>>>
>>>>>> at org.apache.solr.update.Documen
>>>>>>> tBuilder.toDocument(DocumentBuilder.java:197)
>>>>>>
>>>>>> at org.apache.solr.update.AddUpda
>>>>>>> teCommand.getLuceneDocument(AddUpdateCommand.java:82)
>>>>>>
>>>>>> at org.apache.solr.update.DirectU
>>>>>>> pdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:277)
>>>>>>
>>>>>> at org.apache.solr.update.DirectU
>>>>>>> pdateHandler2.addDoc0(DirectUpdateHandler2.java:211)
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>> On 21 February 2017 at 14:52, Karl Wright <da...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Marisol,
>>>>>>>
>>>>>>> Can you find the [INFO] entry in the Solr log for this document?
>>>>>>> That should help clear up any confusion.
>>>>>>>
>>>>>>> Also, for what it is worth, MCF 1.10 is not using a SolrJ that is up
>>>>>>> to date with Solr 6.x. That could be the source of the problem Is there
>>>>>>> any reason you are using a 1.x version of MCF?
>>>>>>>
>>>>>>> Karl
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Feb 21, 2017 at 8:42 AM, Marisol Redondo <
>>>>>>> marisol.redondo.garcia@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi.
>>>>>>>>
>>>>>>>> I'm trying to use metadata adjuster to add one field to the solr
>>>>>>>> index but doesn't inject the field into a solr's field.
>>>>>>>> Maybe I'm misundertaning the use of the metada adjuster, but I have
>>>>>>>> read in the documentation (https://manifoldcf.apache.org
>>>>>>>> /release/release-1.10/en_US/end-user-documentation.html) that I
>>>>>>>> can add metadata to the document that is going to be indexed into solr, but
>>>>>>>> the solr instance gave me the error "missing required field:
>>>>>>>> facetContentType".
>>>>>>>>
>>>>>>>> ManifoldCF Job pipeline:
>>>>>>>> 1. Repository (type web repository)
>>>>>>>> 2. Transformation (Tikka Metadata Extractor)
>>>>>>>> 3. Transformation (type Metada Adjuster)
>>>>>>>> 4. Output (Solr 6)
>>>>>>>>
>>>>>>>> ManifoldCF Job Metadata Expressions tab:
>>>>>>>> Parameter name: "facetContentType"
>>>>>>>> Remove this parameter: false
>>>>>>>> Expresion: xxxx (the literal text value I want in
>>>>>>>> facetContentType)
>>>>>>>>
>>>>>>>> Solr schema:
>>>>>>>> .....
>>>>>>>> <field name="facetContentType" type="string" indexed="true"
>>>>>>>> stored="true" required="true"/>
>>>>>>>> ....
>>>>>>>>
>>>>>>>> The error logged in ManifoldCF is:
>>>>>>>> Error from server at http://solrServer:port/solr/c
>>>>>>>> <http://revnetsolrdev:8983/solr/sites>ore: [doc=https://
>>>>>>>> ....../index.aspx] missing required field: facetContentType.
>>>>>>>>
>>>>>>>> Thanks for your help
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Metadata adjuster
Posted by Karl Wright <da...@gmail.com>.
This is how the email appears here:
>>>>>>
4.
Bottom of Form
Marisol Redondo
Email: mredondo@revenue.ie
Phone: 35428
Please note that Revenue cannot guarantee that any personal and
sensitive data, sent in plain text via standard email, is fully
secure. Customers who choose to use this channel are deemed to have
accepted any risk involved. The alternative communication methods
offered by Revenue include standard post and the option to use our
(encrypted) MyEnquiries service which is available within myAccount
and ROS. You can register for either myAccount or ROS on the Revenue
website.
Tabhair faoi deara nach féidir leis na Coimisinéirí Ioncaim ráthaíocht
a thabhairt go bhfuil aon sonraí pearsanta agus íogair a gcuirtear
isteach i ngnáth-théacs trí r-phost caighdeánach go huile is go
hiomlán slán. Meastar go nglacann custaiméirí a úsáideann an cainéal
seo le haon riosca bainteach. I measc na modhanna cumarsáide eile atá
ag na Coimisinéirí ná post caighdeánach agus an rogha ár seirbhís
(criptithe) M'Fhiosruithe a úsáid, tá sí ar fáil laistigh de MoChúrsaí
agus ROS. Is féidir leat clárú le haghaidh ceachtar MoChúrsaí nó ROS
ar shuíomh gréasáin na gCoimisinéirí.
<<<<<<
In other words I cannot see anything from the 4. stage.
Thanks,
Karl
On Wed, Feb 22, 2017 at 10:37 AM, Marisol Redondo <
marisol.redondo.garcia@gmail.com> wrote:
> I was trying with "Keep all incoming metadata" set to false and too true,
> but I'll take your advice and set to true.
>
> I don't know why you can't see it, but it's the 4 stage
>
> On 22 February 2017 at 15:26, Karl Wright <da...@gmail.com> wrote:
>
>> Hi Marisol,
>>
>> Some observations.
>> (1) It makes no sense to have "Keep all incoming metadata" set to false,
>> since that will filter out everything that your tika extractor extracts. I
>> doubt that is what you have intended.
>> (2) I can't see the Solr output configuration -- looks like it got
>> truncated?
>>
>> Thanks,
>> Karl
>>
>>
>> On Wed, Feb 22, 2017 at 10:12 AM, Marisol Redondo <
>> marisol.redondo.garcia@gmail.com> wrote:
>>
>>> Here you are:
>>>
>>> View a Job
>>>
>>> Top of Form
>>>
>>>
>>> ------------------------------
>>>
>>> Name:
>>>
>>> revenueToSites
>>> ------------------------------
>>>
>>> Pipeline:
>>>
>>> Stage
>>>
>>> Type
>>>
>>> Precedent
>>>
>>> Description
>>>
>>> Connection name
>>>
>>> 1.
>>>
>>> Repository
>>>
>>> Revenue Website
>>>
>>> 2.
>>>
>>> Transformation
>>>
>>> 1.
>>>
>>> Tikka Metadata Extractor
>>>
>>> 3.
>>>
>>> Transformation
>>>
>>> 2.
>>>
>>> Set mimeType and facetContentType
>>>
>>> customField
>>>
>>> 4.
>>>
>>> Output
>>>
>>> 3.
>>>
>>> sites solr dev
>>>
>>> Notifications:
>>>
>>> Stage
>>>
>>> Description
>>>
>>> Connection name
>>>
>>> No notification connections
>>> ------------------------------
>>>
>>> Priority:
>>>
>>> 5
>>>
>>> Start method:
>>>
>>> Don't automatically start
>>> ------------------------------
>>>
>>> Schedule type:
>>>
>>> Scan every document once
>>>
>>> Minimum recrawl interval:
>>>
>>> Not applicable
>>>
>>> Maximum recrawl interval:
>>>
>>> Not applicable
>>>
>>> Expiration interval:
>>>
>>> Not applicable
>>>
>>> Reseed interval:
>>>
>>> Not applicable
>>> ------------------------------
>>>
>>> No scheduled run times
>>> ------------------------------
>>>
>>> Maximum hop count for link type 'link':
>>>
>>> Unlimited
>>>
>>> Maximum hop count for link type 'redirect':
>>>
>>> Unlimited
>>> ------------------------------
>>>
>>> Hop count mode:
>>>
>>> Delete unreachable documents
>>> ------------------------------
>>>
>>> 1.
>>>
>>> Seeds:
>>>
>>> https://xxxxxx/index.aspx
>>> <https://preview.revenuedomain.ie/en/press-office/index.aspx>
>>> ------------------------------
>>>
>>> No canonicalization specified - all URLs will be reordered and have all
>>> sessions removed
>>> ------------------------------
>>>
>>> No mappings specified; will accept all URLs
>>> ------------------------------
>>>
>>> Include only hosts matching seeds?
>>>
>>> yes
>>> ------------------------------
>>>
>>> Include in crawl:
>>>
>>> .*
>>> ------------------------------
>>>
>>> Include in index:
>>>
>>> .*
>>> ------------------------------
>>>
>>> Exclude from crawl:
>>>
>>> \.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|w
>>> mf|WMF|zip|ZIP|mpg|MPG|gz|GZ|rpm|RPM|tgz|TGZ|mov|MOV|exe|EXE
>>> |jpeg|JPEG|bmp|BMP|js|JS|<script>|</script>|<script
>>> type="text/javascript">)
>>> [?*!@=].*
>>> ------------------------------
>>>
>>> Exclude from index:
>>> ------------------------------
>>>
>>> Exclude content from index:
>>> ------------------------------
>>>
>>> No access tokens specified
>>> ------------------------------
>>>
>>> Excluded headers:
>>>
>>> last-modified
>>> ------------------------------
>>>
>>> 2.
>>>
>>> Field mappings:
>>>
>>> Metadata field name
>>>
>>> Final field name
>>>
>>> No field mapping specified
>>> ------------------------------
>>>
>>> Keep all metadata:
>>>
>>> true
>>> ------------------------------
>>>
>>> Lower names:
>>>
>>> false
>>> ------------------------------
>>>
>>> Write limit:
>>> ------------------------------
>>>
>>> Ignore Tika exceptions:
>>>
>>> true
>>> ------------------------------
>>>
>>> Boilerplate extractor:
>>>
>>> -- No extraction selected --
>>> ------------------------------
>>>
>>> 3.
>>>
>>> Metadata expressions:
>>>
>>> Parameter name
>>>
>>> Remove this parameter?
>>>
>>> Expression ("${fieldname}" references a field)
>>>
>>> facetContentType
>>>
>>> false
>>>
>>> site.ie
>>> ------------------------------
>>>
>>> Keep all incoming metadata
>>>
>>> false
>>>
>>> Remove empty metadata values
>>>
>>> false
>>> ------------------------------
>>>
>>> 4.
>>>
>>> Bottom of Form
>>>
>>>
>>>
>>> Marisol Redondo
>>>
>>> Email: mredondo@revenue.ie
>>>
>>> Phone: 35428
>>>
>>>
>>>
>>> Please note that Revenue cannot guarantee that any personal and sensitive data, sent in plain text via standard email, is fully secure. Customers who choose to use this channel are deemed to have accepted any risk involved. The alternative communication methods offered by Revenue include standard post and the option to use our (encrypted) MyEnquiries service which is available within myAccount and ROS. You can register for either myAccount or ROS on the Revenue website.
>>>
>>>
>>>
>>> Tabhair faoi deara nach féidir leis na Coimisinéirí Ioncaim ráthaíocht a thabhairt go bhfuil aon sonraí pearsanta agus íogair a gcuirtear isteach i ngnáth-théacs trí r-phost caighdeánach go huile is go hiomlán slán. Meastar go nglacann custaiméirí a úsáideann an cainéal seo le haon riosca bainteach. I measc na modhanna cumarsáide eile atá ag na Coimisinéirí ná post caighdeánach agus an rogha ár seirbhís (criptithe) M'Fhiosruithe a úsáid, tá sí ar fáil laistigh de MoChúrsaí agus ROS. Is féidir leat clárú le haghaidh ceachtar MoChúrsaí nó ROS ar shuíomh gréasáin na gCoimisinéirí.
>>>
>>>
>>>
>>> On 22 February 2017 at 14:53, Karl Wright <da...@gmail.com> wrote:
>>>
>>>> Hi Marisol,
>>>>
>>>> The [INFO] log entries indicate that your document has almost no
>>>> metadata at all. But the Metadata Adjuster transformation connector is
>>>> designed to do exactly what you want.
>>>>
>>>> Can you view your job, and cut and paste the View Job page into an
>>>> email, so I can see how your metadata adjuster transformation connection
>>>> and your solr output connections are configured? Thanks!
>>>>
>>>> Karl
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Feb 22, 2017 at 8:57 AM, Marisol Redondo <
>>>> marisol.redondo.garcia@gmail.com> wrote:
>>>>
>>>>> Hi Karl and thank you for this quick answer.
>>>>>
>>>>> I was reading the documentation of MCF 1.10 but I'm using MCF 2.5,
>>>>> sorry for the confusion, and I think this version is compatible with solr6.
>>>>> The pdf doesn't have any metadata or field called facetContentType,
>>>>> this is because I'd been trying to use the Metadata Adjuster, to add a new
>>>>> metadata/property to the doc so solr can index by this field when I'm
>>>>> injecting the doc.
>>>>> Should I use other transformation or is there any other way of duing
>>>>> it?
>>>>> I am migrating from nutch to ManifoldCF and in nutch we can do it with
>>>>> plugins, and I was thinking that the plugins in nutch are the same as the
>>>>> transformation connectors in MCF
>>>>>
>>>>> The completed error in solr is :
>>>>>
>>>>> 017-02-21 13:19:32.108 INFO (qtp1854778591-18) [ x:sites]
>>>>>> o.a.s.c.PluginBag Going to create a new requestHandler with {type =
>>>>>> requestHandler,name = /update/extract,class = solr.extraction.ExtractingRequestHandler,args
>>>>>> = {defaults={lowernames=true,fmap.meta=ignored_,fmap.content=_
>>>>>> text_,update.chain=add-unknown-fields-to-the-schema,df=_text_}}}
>>>>>
>>>>> 2017-02-21 13:19:32.454 INFO (qtp1854778591-18) [ x:sites]
>>>>>> o.a.s.u.p.LogUpdateProcessorFactory [sites] webapp=/solr path=/up
>>>>>
>>>>> date/extract params={resource.name=introduction.pdf&literal.id
>>>>>> =https://...../introduction.pdf&wt=xml&version=2.2}{} 0 347
>>>>>
>>>>> 2017-02-21 13:19:32.455 ERROR (qtp1854778591-18) [ x:sites]
>>>>>> o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: [
>>>>>
>>>>> doc=https://....../introduction.pdf] missing required field:
>>>>>> facetContentType
>>>>>
>>>>> at org.apache.solr.update.Documen
>>>>>> tBuilder.toDocument(DocumentBuilder.java:197)
>>>>>
>>>>> at org.apache.solr.update.AddUpda
>>>>>> teCommand.getLuceneDocument(AddUpdateCommand.java:82)
>>>>>
>>>>> at org.apache.solr.update.DirectU
>>>>>> pdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:277)
>>>>>
>>>>> at org.apache.solr.update.DirectU
>>>>>> pdateHandler2.addDoc0(DirectUpdateHandler2.java:211)
>>>>>
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>> On 21 February 2017 at 14:52, Karl Wright <da...@gmail.com> wrote:
>>>>>
>>>>>> Hi Marisol,
>>>>>>
>>>>>> Can you find the [INFO] entry in the Solr log for this document?
>>>>>> That should help clear up any confusion.
>>>>>>
>>>>>> Also, for what it is worth, MCF 1.10 is not using a SolrJ that is up
>>>>>> to date with Solr 6.x. That could be the source of the problem Is there
>>>>>> any reason you are using a 1.x version of MCF?
>>>>>>
>>>>>> Karl
>>>>>>
>>>>>>
>>>>>> On Tue, Feb 21, 2017 at 8:42 AM, Marisol Redondo <
>>>>>> marisol.redondo.garcia@gmail.com> wrote:
>>>>>>
>>>>>>> Hi.
>>>>>>>
>>>>>>> I'm trying to use metadata adjuster to add one field to the solr
>>>>>>> index but doesn't inject the field into a solr's field.
>>>>>>> Maybe I'm misundertaning the use of the metada adjuster, but I have
>>>>>>> read in the documentation (https://manifoldcf.apache.org
>>>>>>> /release/release-1.10/en_US/end-user-documentation.html) that I can
>>>>>>> add metadata to the document that is going to be indexed into solr, but the
>>>>>>> solr instance gave me the error "missing required field:
>>>>>>> facetContentType".
>>>>>>>
>>>>>>> ManifoldCF Job pipeline:
>>>>>>> 1. Repository (type web repository)
>>>>>>> 2. Transformation (Tikka Metadata Extractor)
>>>>>>> 3. Transformation (type Metada Adjuster)
>>>>>>> 4. Output (Solr 6)
>>>>>>>
>>>>>>> ManifoldCF Job Metadata Expressions tab:
>>>>>>> Parameter name: "facetContentType"
>>>>>>> Remove this parameter: false
>>>>>>> Expresion: xxxx (the literal text value I want in
>>>>>>> facetContentType)
>>>>>>>
>>>>>>> Solr schema:
>>>>>>> .....
>>>>>>> <field name="facetContentType" type="string" indexed="true"
>>>>>>> stored="true" required="true"/>
>>>>>>> ....
>>>>>>>
>>>>>>> The error logged in ManifoldCF is:
>>>>>>> Error from server at http://solrServer:port/solr/c
>>>>>>> <http://revnetsolrdev:8983/solr/sites>ore: [doc=https://
>>>>>>> ....../index.aspx] missing required field: facetContentType.
>>>>>>>
>>>>>>> Thanks for your help
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Metadata adjuster
Posted by Marisol Redondo <ma...@gmail.com>.
I was trying with "Keep all incoming metadata" set to false and too true,
but I'll take your advice and set to true.
I don't know why you can't see it, but it's the 4 stage
On 22 February 2017 at 15:26, Karl Wright <da...@gmail.com> wrote:
> Hi Marisol,
>
> Some observations.
> (1) It makes no sense to have "Keep all incoming metadata" set to false,
> since that will filter out everything that your tika extractor extracts. I
> doubt that is what you have intended.
> (2) I can't see the Solr output configuration -- looks like it got
> truncated?
>
> Thanks,
> Karl
>
>
> On Wed, Feb 22, 2017 at 10:12 AM, Marisol Redondo <
> marisol.redondo.garcia@gmail.com> wrote:
>
>> Here you are:
>>
>> View a Job
>>
>> Top of Form
>>
>>
>> ------------------------------
>>
>> Name:
>>
>> revenueToSites
>> ------------------------------
>>
>> Pipeline:
>>
>> Stage
>>
>> Type
>>
>> Precedent
>>
>> Description
>>
>> Connection name
>>
>> 1.
>>
>> Repository
>>
>> Revenue Website
>>
>> 2.
>>
>> Transformation
>>
>> 1.
>>
>> Tikka Metadata Extractor
>>
>> 3.
>>
>> Transformation
>>
>> 2.
>>
>> Set mimeType and facetContentType
>>
>> customField
>>
>> 4.
>>
>> Output
>>
>> 3.
>>
>> sites solr dev
>>
>> Notifications:
>>
>> Stage
>>
>> Description
>>
>> Connection name
>>
>> No notification connections
>> ------------------------------
>>
>> Priority:
>>
>> 5
>>
>> Start method:
>>
>> Don't automatically start
>> ------------------------------
>>
>> Schedule type:
>>
>> Scan every document once
>>
>> Minimum recrawl interval:
>>
>> Not applicable
>>
>> Maximum recrawl interval:
>>
>> Not applicable
>>
>> Expiration interval:
>>
>> Not applicable
>>
>> Reseed interval:
>>
>> Not applicable
>> ------------------------------
>>
>> No scheduled run times
>> ------------------------------
>>
>> Maximum hop count for link type 'link':
>>
>> Unlimited
>>
>> Maximum hop count for link type 'redirect':
>>
>> Unlimited
>> ------------------------------
>>
>> Hop count mode:
>>
>> Delete unreachable documents
>> ------------------------------
>>
>> 1.
>>
>> Seeds:
>>
>> https://xxxxxx/index.aspx
>> <https://preview.revenuedomain.ie/en/press-office/index.aspx>
>> ------------------------------
>>
>> No canonicalization specified - all URLs will be reordered and have all
>> sessions removed
>> ------------------------------
>>
>> No mappings specified; will accept all URLs
>> ------------------------------
>>
>> Include only hosts matching seeds?
>>
>> yes
>> ------------------------------
>>
>> Include in crawl:
>>
>> .*
>> ------------------------------
>>
>> Include in index:
>>
>> .*
>> ------------------------------
>>
>> Exclude from crawl:
>>
>> \.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|
>> wmf|WMF|zip|ZIP|mpg|MPG|gz|GZ|rpm|RPM|tgz|TGZ|mov|MOV|exe|
>> EXE|jpeg|JPEG|bmp|BMP|js|JS|<script>|</script>|<script
>> type="text/javascript">)
>> [?*!@=].*
>> ------------------------------
>>
>> Exclude from index:
>> ------------------------------
>>
>> Exclude content from index:
>> ------------------------------
>>
>> No access tokens specified
>> ------------------------------
>>
>> Excluded headers:
>>
>> last-modified
>> ------------------------------
>>
>> 2.
>>
>> Field mappings:
>>
>> Metadata field name
>>
>> Final field name
>>
>> No field mapping specified
>> ------------------------------
>>
>> Keep all metadata:
>>
>> true
>> ------------------------------
>>
>> Lower names:
>>
>> false
>> ------------------------------
>>
>> Write limit:
>> ------------------------------
>>
>> Ignore Tika exceptions:
>>
>> true
>> ------------------------------
>>
>> Boilerplate extractor:
>>
>> -- No extraction selected --
>> ------------------------------
>>
>> 3.
>>
>> Metadata expressions:
>>
>> Parameter name
>>
>> Remove this parameter?
>>
>> Expression ("${fieldname}" references a field)
>>
>> facetContentType
>>
>> false
>>
>> site.ie
>> ------------------------------
>>
>> Keep all incoming metadata
>>
>> false
>>
>> Remove empty metadata values
>>
>> false
>> ------------------------------
>>
>> 4.
>>
>> Bottom of Form
>>
>>
>>
>> Marisol Redondo
>>
>> Email: mredondo@revenue.ie
>>
>> Phone: 35428
>>
>>
>>
>> Please note that Revenue cannot guarantee that any personal and sensitive data, sent in plain text via standard email, is fully secure. Customers who choose to use this channel are deemed to have accepted any risk involved. The alternative communication methods offered by Revenue include standard post and the option to use our (encrypted) MyEnquiries service which is available within myAccount and ROS. You can register for either myAccount or ROS on the Revenue website.
>>
>>
>>
>> Tabhair faoi deara nach féidir leis na Coimisinéirí Ioncaim ráthaíocht a thabhairt go bhfuil aon sonraí pearsanta agus íogair a gcuirtear isteach i ngnáth-théacs trí r-phost caighdeánach go huile is go hiomlán slán. Meastar go nglacann custaiméirí a úsáideann an cainéal seo le haon riosca bainteach. I measc na modhanna cumarsáide eile atá ag na Coimisinéirí ná post caighdeánach agus an rogha ár seirbhís (criptithe) M'Fhiosruithe a úsáid, tá sí ar fáil laistigh de MoChúrsaí agus ROS. Is féidir leat clárú le haghaidh ceachtar MoChúrsaí nó ROS ar shuíomh gréasáin na gCoimisinéirí.
>>
>>
>>
>> On 22 February 2017 at 14:53, Karl Wright <da...@gmail.com> wrote:
>>
>>> Hi Marisol,
>>>
>>> The [INFO] log entries indicate that your document has almost no
>>> metadata at all. But the Metadata Adjuster transformation connector is
>>> designed to do exactly what you want.
>>>
>>> Can you view your job, and cut and paste the View Job page into an
>>> email, so I can see how your metadata adjuster transformation connection
>>> and your solr output connections are configured? Thanks!
>>>
>>> Karl
>>>
>>>
>>>
>>>
>>> On Wed, Feb 22, 2017 at 8:57 AM, Marisol Redondo <
>>> marisol.redondo.garcia@gmail.com> wrote:
>>>
>>>> Hi Karl and thank you for this quick answer.
>>>>
>>>> I was reading the documentation of MCF 1.10 but I'm using MCF 2.5,
>>>> sorry for the confusion, and I think this version is compatible with solr6.
>>>> The pdf doesn't have any metadata or field called facetContentType,
>>>> this is because I'd been trying to use the Metadata Adjuster, to add a new
>>>> metadata/property to the doc so solr can index by this field when I'm
>>>> injecting the doc.
>>>> Should I use other transformation or is there any other way of duing it?
>>>> I am migrating from nutch to ManifoldCF and in nutch we can do it with
>>>> plugins, and I was thinking that the plugins in nutch are the same as the
>>>> transformation connectors in MCF
>>>>
>>>> The completed error in solr is :
>>>>
>>>> 017-02-21 13:19:32.108 INFO (qtp1854778591-18) [ x:sites]
>>>>> o.a.s.c.PluginBag Going to create a new requestHandler with {type =
>>>>> requestHandler,name = /update/extract,class = solr.extraction.ExtractingRequestHandler,args
>>>>> = {defaults={lowernames=true,fmap.meta=ignored_,fmap.content=_
>>>>> text_,update.chain=add-unknown-fields-to-the-schema,df=_text_}}}
>>>>
>>>> 2017-02-21 13:19:32.454 INFO (qtp1854778591-18) [ x:sites]
>>>>> o.a.s.u.p.LogUpdateProcessorFactory [sites] webapp=/solr path=/up
>>>>
>>>> date/extract params={resource.name=introduction.pdf&literal.id=https://
>>>>> ...../introduction.pdf&wt=xml&version=2.2}{} 0 347
>>>>
>>>> 2017-02-21 13:19:32.455 ERROR (qtp1854778591-18) [ x:sites]
>>>>> o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: [
>>>>
>>>> doc=https://....../introduction.pdf] missing required field:
>>>>> facetContentType
>>>>
>>>> at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBu
>>>>> ilder.java:197)
>>>>
>>>> at org.apache.solr.update.AddUpdateCommand.getLuceneDocument(Ad
>>>>> dUpdateCommand.java:82)
>>>>
>>>> at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(D
>>>>> irectUpdateHandler2.java:277)
>>>>
>>>> at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUp
>>>>> dateHandler2.java:211)
>>>>
>>>>
>>>>
>>>> Thanks
>>>>
>>>>
>>>> On 21 February 2017 at 14:52, Karl Wright <da...@gmail.com> wrote:
>>>>
>>>>> Hi Marisol,
>>>>>
>>>>> Can you find the [INFO] entry in the Solr log for this document? That
>>>>> should help clear up any confusion.
>>>>>
>>>>> Also, for what it is worth, MCF 1.10 is not using a SolrJ that is up
>>>>> to date with Solr 6.x. That could be the source of the problem Is there
>>>>> any reason you are using a 1.x version of MCF?
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>> On Tue, Feb 21, 2017 at 8:42 AM, Marisol Redondo <
>>>>> marisol.redondo.garcia@gmail.com> wrote:
>>>>>
>>>>>> Hi.
>>>>>>
>>>>>> I'm trying to use metadata adjuster to add one field to the solr
>>>>>> index but doesn't inject the field into a solr's field.
>>>>>> Maybe I'm misundertaning the use of the metada adjuster, but I have
>>>>>> read in the documentation (https://manifoldcf.apache.org
>>>>>> /release/release-1.10/en_US/end-user-documentation.html) that I can
>>>>>> add metadata to the document that is going to be indexed into solr, but the
>>>>>> solr instance gave me the error "missing required field:
>>>>>> facetContentType".
>>>>>>
>>>>>> ManifoldCF Job pipeline:
>>>>>> 1. Repository (type web repository)
>>>>>> 2. Transformation (Tikka Metadata Extractor)
>>>>>> 3. Transformation (type Metada Adjuster)
>>>>>> 4. Output (Solr 6)
>>>>>>
>>>>>> ManifoldCF Job Metadata Expressions tab:
>>>>>> Parameter name: "facetContentType"
>>>>>> Remove this parameter: false
>>>>>> Expresion: xxxx (the literal text value I want in facetContentType)
>>>>>>
>>>>>> Solr schema:
>>>>>> .....
>>>>>> <field name="facetContentType" type="string" indexed="true"
>>>>>> stored="true" required="true"/>
>>>>>> ....
>>>>>>
>>>>>> The error logged in ManifoldCF is:
>>>>>> Error from server at http://solrServer:port/solr/c
>>>>>> <http://revnetsolrdev:8983/solr/sites>ore: [doc=https://
>>>>>> ....../index.aspx] missing required field: facetContentType.
>>>>>>
>>>>>> Thanks for your help
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Metadata adjuster
Posted by Karl Wright <da...@gmail.com>.
Hi Marisol,
Some observations.
(1) It makes no sense to have "Keep all incoming metadata" set to false,
since that will filter out everything that your tika extractor extracts. I
doubt that is what you have intended.
(2) I can't see the Solr output configuration -- looks like it got
truncated?
Thanks,
Karl
On Wed, Feb 22, 2017 at 10:12 AM, Marisol Redondo <
marisol.redondo.garcia@gmail.com> wrote:
> Here you are:
>
> View a Job
>
> Top of Form
>
>
> ------------------------------
>
> Name:
>
> revenueToSites
> ------------------------------
>
> Pipeline:
>
> Stage
>
> Type
>
> Precedent
>
> Description
>
> Connection name
>
> 1.
>
> Repository
>
> Revenue Website
>
> 2.
>
> Transformation
>
> 1.
>
> Tikka Metadata Extractor
>
> 3.
>
> Transformation
>
> 2.
>
> Set mimeType and facetContentType
>
> customField
>
> 4.
>
> Output
>
> 3.
>
> sites solr dev
>
> Notifications:
>
> Stage
>
> Description
>
> Connection name
>
> No notification connections
> ------------------------------
>
> Priority:
>
> 5
>
> Start method:
>
> Don't automatically start
> ------------------------------
>
> Schedule type:
>
> Scan every document once
>
> Minimum recrawl interval:
>
> Not applicable
>
> Maximum recrawl interval:
>
> Not applicable
>
> Expiration interval:
>
> Not applicable
>
> Reseed interval:
>
> Not applicable
> ------------------------------
>
> No scheduled run times
> ------------------------------
>
> Maximum hop count for link type 'link':
>
> Unlimited
>
> Maximum hop count for link type 'redirect':
>
> Unlimited
> ------------------------------
>
> Hop count mode:
>
> Delete unreachable documents
> ------------------------------
>
> 1.
>
> Seeds:
>
> https://xxxxxx/index.aspx
> <https://preview.revenuedomain.ie/en/press-office/index.aspx>
> ------------------------------
>
> No canonicalization specified - all URLs will be reordered and have all
> sessions removed
> ------------------------------
>
> No mappings specified; will accept all URLs
> ------------------------------
>
> Include only hosts matching seeds?
>
> yes
> ------------------------------
>
> Include in crawl:
>
> .*
> ------------------------------
>
> Include in index:
>
> .*
> ------------------------------
>
> Exclude from crawl:
>
> \.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|
> EPS|wmf|WMF|zip|ZIP|mpg|MPG|gz|GZ|rpm|RPM|tgz|TGZ|mov|MOV|
> exe|EXE|jpeg|JPEG|bmp|BMP|js|JS|<script>|</script>|<script
> type="text/javascript">)
> [?*!@=].*
> ------------------------------
>
> Exclude from index:
> ------------------------------
>
> Exclude content from index:
> ------------------------------
>
> No access tokens specified
> ------------------------------
>
> Excluded headers:
>
> last-modified
> ------------------------------
>
> 2.
>
> Field mappings:
>
> Metadata field name
>
> Final field name
>
> No field mapping specified
> ------------------------------
>
> Keep all metadata:
>
> true
> ------------------------------
>
> Lower names:
>
> false
> ------------------------------
>
> Write limit:
> ------------------------------
>
> Ignore Tika exceptions:
>
> true
> ------------------------------
>
> Boilerplate extractor:
>
> -- No extraction selected --
> ------------------------------
>
> 3.
>
> Metadata expressions:
>
> Parameter name
>
> Remove this parameter?
>
> Expression ("${fieldname}" references a field)
>
> facetContentType
>
> false
>
> site.ie
> ------------------------------
>
> Keep all incoming metadata
>
> false
>
> Remove empty metadata values
>
> false
> ------------------------------
>
> 4.
>
> Bottom of Form
>
>
>
> Marisol Redondo
>
> Email: mredondo@revenue.ie
>
> Phone: 35428
>
>
>
> Please note that Revenue cannot guarantee that any personal and sensitive data, sent in plain text via standard email, is fully secure. Customers who choose to use this channel are deemed to have accepted any risk involved. The alternative communication methods offered by Revenue include standard post and the option to use our (encrypted) MyEnquiries service which is available within myAccount and ROS. You can register for either myAccount or ROS on the Revenue website.
>
>
>
> Tabhair faoi deara nach féidir leis na Coimisinéirí Ioncaim ráthaíocht a thabhairt go bhfuil aon sonraí pearsanta agus íogair a gcuirtear isteach i ngnáth-théacs trí r-phost caighdeánach go huile is go hiomlán slán. Meastar go nglacann custaiméirí a úsáideann an cainéal seo le haon riosca bainteach. I measc na modhanna cumarsáide eile atá ag na Coimisinéirí ná post caighdeánach agus an rogha ár seirbhís (criptithe) M'Fhiosruithe a úsáid, tá sí ar fáil laistigh de MoChúrsaí agus ROS. Is féidir leat clárú le haghaidh ceachtar MoChúrsaí nó ROS ar shuíomh gréasáin na gCoimisinéirí.
>
>
>
> On 22 February 2017 at 14:53, Karl Wright <da...@gmail.com> wrote:
>
>> Hi Marisol,
>>
>> The [INFO] log entries indicate that your document has almost no metadata
>> at all. But the Metadata Adjuster transformation connector is designed to
>> do exactly what you want.
>>
>> Can you view your job, and cut and paste the View Job page into an email,
>> so I can see how your metadata adjuster transformation connection and your
>> solr output connections are configured? Thanks!
>>
>> Karl
>>
>>
>>
>>
>> On Wed, Feb 22, 2017 at 8:57 AM, Marisol Redondo <
>> marisol.redondo.garcia@gmail.com> wrote:
>>
>>> Hi Karl and thank you for this quick answer.
>>>
>>> I was reading the documentation of MCF 1.10 but I'm using MCF 2.5, sorry
>>> for the confusion, and I think this version is compatible with solr6.
>>> The pdf doesn't have any metadata or field called facetContentType, this
>>> is because I'd been trying to use the Metadata Adjuster, to add a new
>>> metadata/property to the doc so solr can index by this field when I'm
>>> injecting the doc.
>>> Should I use other transformation or is there any other way of duing it?
>>> I am migrating from nutch to ManifoldCF and in nutch we can do it with
>>> plugins, and I was thinking that the plugins in nutch are the same as the
>>> transformation connectors in MCF
>>>
>>> The completed error in solr is :
>>>
>>> 017-02-21 13:19:32.108 INFO (qtp1854778591-18) [ x:sites]
>>>> o.a.s.c.PluginBag Going to create a new requestHandler with {type =
>>>> requestHandler,name = /update/extract,class = solr.extraction.ExtractingRequestHandler,args
>>>> = {defaults={lowernames=true,fmap.meta=ignored_,fmap.content=_
>>>> text_,update.chain=add-unknown-fields-to-the-schema,df=_text_}}}
>>>
>>> 2017-02-21 13:19:32.454 INFO (qtp1854778591-18) [ x:sites]
>>>> o.a.s.u.p.LogUpdateProcessorFactory [sites] webapp=/solr path=/up
>>>
>>> date/extract params={resource.name=introduction.pdf&literal.id=https://
>>>> ...../introduction.pdf&wt=xml&version=2.2}{} 0 347
>>>
>>> 2017-02-21 13:19:32.455 ERROR (qtp1854778591-18) [ x:sites]
>>>> o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: [
>>>
>>> doc=https://....../introduction.pdf] missing required field:
>>>> facetContentType
>>>
>>> at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBu
>>>> ilder.java:197)
>>>
>>> at org.apache.solr.update.AddUpdateCommand.getLuceneDocument(Ad
>>>> dUpdateCommand.java:82)
>>>
>>> at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(D
>>>> irectUpdateHandler2.java:277)
>>>
>>> at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUp
>>>> dateHandler2.java:211)
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>> On 21 February 2017 at 14:52, Karl Wright <da...@gmail.com> wrote:
>>>
>>>> Hi Marisol,
>>>>
>>>> Can you find the [INFO] entry in the Solr log for this document? That
>>>> should help clear up any confusion.
>>>>
>>>> Also, for what it is worth, MCF 1.10 is not using a SolrJ that is up to
>>>> date with Solr 6.x. That could be the source of the problem Is there any
>>>> reason you are using a 1.x version of MCF?
>>>>
>>>> Karl
>>>>
>>>>
>>>> On Tue, Feb 21, 2017 at 8:42 AM, Marisol Redondo <
>>>> marisol.redondo.garcia@gmail.com> wrote:
>>>>
>>>>> Hi.
>>>>>
>>>>> I'm trying to use metadata adjuster to add one field to the solr index
>>>>> but doesn't inject the field into a solr's field.
>>>>> Maybe I'm misundertaning the use of the metada adjuster, but I have
>>>>> read in the documentation (https://manifoldcf.apache.org
>>>>> /release/release-1.10/en_US/end-user-documentation.html) that I can
>>>>> add metadata to the document that is going to be indexed into solr, but the
>>>>> solr instance gave me the error "missing required field:
>>>>> facetContentType".
>>>>>
>>>>> ManifoldCF Job pipeline:
>>>>> 1. Repository (type web repository)
>>>>> 2. Transformation (Tikka Metadata Extractor)
>>>>> 3. Transformation (type Metada Adjuster)
>>>>> 4. Output (Solr 6)
>>>>>
>>>>> ManifoldCF Job Metadata Expressions tab:
>>>>> Parameter name: "facetContentType"
>>>>> Remove this parameter: false
>>>>> Expresion: xxxx (the literal text value I want in facetContentType)
>>>>>
>>>>> Solr schema:
>>>>> .....
>>>>> <field name="facetContentType" type="string" indexed="true"
>>>>> stored="true" required="true"/>
>>>>> ....
>>>>>
>>>>> The error logged in ManifoldCF is:
>>>>> Error from server at http://solrServer:port/solr/c
>>>>> <http://revnetsolrdev:8983/solr/sites>ore: [doc=https://
>>>>> ....../index.aspx] missing required field: facetContentType.
>>>>>
>>>>> Thanks for your help
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Metadata adjuster
Posted by Marisol Redondo <ma...@gmail.com>.
Here you are:
View a Job
Top of Form
------------------------------
Name:
revenueToSites
------------------------------
Pipeline:
Stage
Type
Precedent
Description
Connection name
1.
Repository
Revenue Website
2.
Transformation
1.
Tikka Metadata Extractor
3.
Transformation
2.
Set mimeType and facetContentType
customField
4.
Output
3.
sites solr dev
Notifications:
Stage
Description
Connection name
No notification connections
------------------------------
Priority:
5
Start method:
Don't automatically start
------------------------------
Schedule type:
Scan every document once
Minimum recrawl interval:
Not applicable
Maximum recrawl interval:
Not applicable
Expiration interval:
Not applicable
Reseed interval:
Not applicable
------------------------------
No scheduled run times
------------------------------
Maximum hop count for link type 'link':
Unlimited
Maximum hop count for link type 'redirect':
Unlimited
------------------------------
Hop count mode:
Delete unreachable documents
------------------------------
1.
Seeds:
https://xxxxxx/index.aspx
<https://preview.revenuedomain.ie/en/press-office/index.aspx>
------------------------------
No canonicalization specified - all URLs will be reordered and have all
sessions removed
------------------------------
No mappings specified; will accept all URLs
------------------------------
Include only hosts matching seeds?
yes
------------------------------
Include in crawl:
.*
------------------------------
Include in index:
.*
------------------------------
Exclude from crawl:
\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|wmf|WMF|zip|ZIP|mpg|MPG|gz|GZ|rpm|RPM|tgz|TGZ|mov|MOV|exe|EXE|jpeg|JPEG|bmp|BMP|js|JS|<script>|</script>|<script
type="text/javascript">)
[?*!@=].*
------------------------------
Exclude from index:
------------------------------
Exclude content from index:
------------------------------
No access tokens specified
------------------------------
Excluded headers:
last-modified
------------------------------
2.
Field mappings:
Metadata field name
Final field name
No field mapping specified
------------------------------
Keep all metadata:
true
------------------------------
Lower names:
false
------------------------------
Write limit:
------------------------------
Ignore Tika exceptions:
true
------------------------------
Boilerplate extractor:
-- No extraction selected --
------------------------------
3.
Metadata expressions:
Parameter name
Remove this parameter?
Expression ("${fieldname}" references a field)
facetContentType
false
site.ie
------------------------------
Keep all incoming metadata
false
Remove empty metadata values
false
------------------------------
4.
Bottom of Form
Marisol Redondo
Email: mredondo@revenue.ie
Phone: 35428
Please note that Revenue cannot guarantee that any personal and
sensitive data, sent in plain text via standard email, is fully
secure. Customers who choose to use this channel are deemed to have
accepted any risk involved. The alternative communication methods
offered by Revenue include standard post and the option to use our
(encrypted) MyEnquiries service which is available within myAccount
and ROS. You can register for either myAccount or ROS on the Revenue
website.
Tabhair faoi deara nach féidir leis na Coimisinéirí Ioncaim ráthaíocht
a thabhairt go bhfuil aon sonraí pearsanta agus íogair a gcuirtear
isteach i ngnáth-théacs trí r-phost caighdeánach go huile is go
hiomlán slán. Meastar go nglacann custaiméirí a úsáideann an cainéal
seo le haon riosca bainteach. I measc na modhanna cumarsáide eile atá
ag na Coimisinéirí ná post caighdeánach agus an rogha ár seirbhís
(criptithe) M'Fhiosruithe a úsáid, tá sí ar fáil laistigh de MoChúrsaí
agus ROS. Is féidir leat clárú le haghaidh ceachtar MoChúrsaí nó ROS
ar shuíomh gréasáin na gCoimisinéirí.
On 22 February 2017 at 14:53, Karl Wright <da...@gmail.com> wrote:
> Hi Marisol,
>
> The [INFO] log entries indicate that your document has almost no metadata
> at all. But the Metadata Adjuster transformation connector is designed to
> do exactly what you want.
>
> Can you view your job, and cut and paste the View Job page into an email,
> so I can see how your metadata adjuster transformation connection and your
> solr output connections are configured? Thanks!
>
> Karl
>
>
>
>
> On Wed, Feb 22, 2017 at 8:57 AM, Marisol Redondo <
> marisol.redondo.garcia@gmail.com> wrote:
>
>> Hi Karl and thank you for this quick answer.
>>
>> I was reading the documentation of MCF 1.10 but I'm using MCF 2.5, sorry
>> for the confusion, and I think this version is compatible with solr6.
>> The pdf doesn't have any metadata or field called facetContentType, this
>> is because I'd been trying to use the Metadata Adjuster, to add a new
>> metadata/property to the doc so solr can index by this field when I'm
>> injecting the doc.
>> Should I use other transformation or is there any other way of duing it?
>> I am migrating from nutch to ManifoldCF and in nutch we can do it with
>> plugins, and I was thinking that the plugins in nutch are the same as the
>> transformation connectors in MCF
>>
>> The completed error in solr is :
>>
>> 017-02-21 13:19:32.108 INFO (qtp1854778591-18) [ x:sites]
>>> o.a.s.c.PluginBag Going to create a new requestHandler with {type =
>>> requestHandler,name = /update/extract,class = solr.extraction.ExtractingRequestHandler,args
>>> = {defaults={lowernames=true,fmap.meta=ignored_,fmap.content=_
>>> text_,update.chain=add-unknown-fields-to-the-schema,df=_text_}}}
>>
>> 2017-02-21 13:19:32.454 INFO (qtp1854778591-18) [ x:sites]
>>> o.a.s.u.p.LogUpdateProcessorFactory [sites] webapp=/solr path=/up
>>
>> date/extract params={resource.name=introduction.pdf&literal.id=https://..
>>> .../introduction.pdf&wt=xml&version=2.2}{} 0 347
>>
>> 2017-02-21 13:19:32.455 ERROR (qtp1854778591-18) [ x:sites]
>>> o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: [
>>
>> doc=https://....../introduction.pdf] missing required field:
>>> facetContentType
>>
>> at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBu
>>> ilder.java:197)
>>
>> at org.apache.solr.update.AddUpdateCommand.getLuceneDocument(Ad
>>> dUpdateCommand.java:82)
>>
>> at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(D
>>> irectUpdateHandler2.java:277)
>>
>> at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUp
>>> dateHandler2.java:211)
>>
>>
>>
>> Thanks
>>
>>
>> On 21 February 2017 at 14:52, Karl Wright <da...@gmail.com> wrote:
>>
>>> Hi Marisol,
>>>
>>> Can you find the [INFO] entry in the Solr log for this document? That
>>> should help clear up any confusion.
>>>
>>> Also, for what it is worth, MCF 1.10 is not using a SolrJ that is up to
>>> date with Solr 6.x. That could be the source of the problem Is there any
>>> reason you are using a 1.x version of MCF?
>>>
>>> Karl
>>>
>>>
>>> On Tue, Feb 21, 2017 at 8:42 AM, Marisol Redondo <
>>> marisol.redondo.garcia@gmail.com> wrote:
>>>
>>>> Hi.
>>>>
>>>> I'm trying to use metadata adjuster to add one field to the solr index
>>>> but doesn't inject the field into a solr's field.
>>>> Maybe I'm misundertaning the use of the metada adjuster, but I have
>>>> read in the documentation (https://manifoldcf.apache.org
>>>> /release/release-1.10/en_US/end-user-documentation.html) that I can
>>>> add metadata to the document that is going to be indexed into solr, but the
>>>> solr instance gave me the error "missing required field:
>>>> facetContentType".
>>>>
>>>> ManifoldCF Job pipeline:
>>>> 1. Repository (type web repository)
>>>> 2. Transformation (Tikka Metadata Extractor)
>>>> 3. Transformation (type Metada Adjuster)
>>>> 4. Output (Solr 6)
>>>>
>>>> ManifoldCF Job Metadata Expressions tab:
>>>> Parameter name: "facetContentType"
>>>> Remove this parameter: false
>>>> Expresion: xxxx (the literal text value I want in facetContentType)
>>>>
>>>> Solr schema:
>>>> .....
>>>> <field name="facetContentType" type="string" indexed="true"
>>>> stored="true" required="true"/>
>>>> ....
>>>>
>>>> The error logged in ManifoldCF is:
>>>> Error from server at http://solrServer:port/solr/c
>>>> <http://revnetsolrdev:8983/solr/sites>ore: [doc=https://
>>>> ....../index.aspx] missing required field: facetContentType.
>>>>
>>>> Thanks for your help
>>>>
>>>
>>>
>>
>
Re: Metadata adjuster
Posted by Karl Wright <da...@gmail.com>.
Hi Marisol,
The [INFO] log entries indicate that your document has almost no metadata
at all. But the Metadata Adjuster transformation connector is designed to
do exactly what you want.
Can you view your job, and cut and paste the View Job page into an email,
so I can see how your metadata adjuster transformation connection and your
solr output connections are configured? Thanks!
Karl
On Wed, Feb 22, 2017 at 8:57 AM, Marisol Redondo <
marisol.redondo.garcia@gmail.com> wrote:
> Hi Karl and thank you for this quick answer.
>
> I was reading the documentation of MCF 1.10 but I'm using MCF 2.5, sorry
> for the confusion, and I think this version is compatible with solr6.
> The pdf doesn't have any metadata or field called facetContentType, this
> is because I'd been trying to use the Metadata Adjuster, to add a new
> metadata/property to the doc so solr can index by this field when I'm
> injecting the doc.
> Should I use other transformation or is there any other way of duing it?
> I am migrating from nutch to ManifoldCF and in nutch we can do it with
> plugins, and I was thinking that the plugins in nutch are the same as the
> transformation connectors in MCF
>
> The completed error in solr is :
>
> 017-02-21 13:19:32.108 INFO (qtp1854778591-18) [ x:sites]
>> o.a.s.c.PluginBag Going to create a new requestHandler with {type =
>> requestHandler,name = /update/extract,class = solr.extraction.ExtractingRequestHandler,args
>> = {defaults={lowernames=true,fmap.meta=ignored_,fmap.
>> content=_text_,update.chain=add-unknown-fields-to-the-schema,df=_text_}}}
>
> 2017-02-21 13:19:32.454 INFO (qtp1854778591-18) [ x:sites] o.a.s.u.p.LogUpdateProcessorFactory
>> [sites] webapp=/solr path=/up
>
> date/extract params={resource.name=introduction.pdf&literal.id=https://
>> ...../introduction.pdf&wt=xml&version=2.2}{} 0 347
>
> 2017-02-21 13:19:32.455 ERROR (qtp1854778591-18) [ x:sites]
>> o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: [
>
> doc=https://....../introduction.pdf] missing required field:
>> facetContentType
>
> at org.apache.solr.update.DocumentBuilder.toDocument(
>> DocumentBuilder.java:197)
>
> at org.apache.solr.update.AddUpdateCommand.getLuceneDocument(
>> AddUpdateCommand.java:82)
>
> at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(
>> DirectUpdateHandler2.java:277)
>
> at org.apache.solr.update.DirectUpdateHandler2.addDoc0(
>> DirectUpdateHandler2.java:211)
>
>
>
> Thanks
>
>
> On 21 February 2017 at 14:52, Karl Wright <da...@gmail.com> wrote:
>
>> Hi Marisol,
>>
>> Can you find the [INFO] entry in the Solr log for this document? That
>> should help clear up any confusion.
>>
>> Also, for what it is worth, MCF 1.10 is not using a SolrJ that is up to
>> date with Solr 6.x. That could be the source of the problem Is there any
>> reason you are using a 1.x version of MCF?
>>
>> Karl
>>
>>
>> On Tue, Feb 21, 2017 at 8:42 AM, Marisol Redondo <
>> marisol.redondo.garcia@gmail.com> wrote:
>>
>>> Hi.
>>>
>>> I'm trying to use metadata adjuster to add one field to the solr index
>>> but doesn't inject the field into a solr's field.
>>> Maybe I'm misundertaning the use of the metada adjuster, but I have read
>>> in the documentation (https://manifoldcf.apache.org
>>> /release/release-1.10/en_US/end-user-documentation.html) that I can add
>>> metadata to the document that is going to be indexed into solr, but the
>>> solr instance gave me the error "missing required field:
>>> facetContentType".
>>>
>>> ManifoldCF Job pipeline:
>>> 1. Repository (type web repository)
>>> 2. Transformation (Tikka Metadata Extractor)
>>> 3. Transformation (type Metada Adjuster)
>>> 4. Output (Solr 6)
>>>
>>> ManifoldCF Job Metadata Expressions tab:
>>> Parameter name: "facetContentType"
>>> Remove this parameter: false
>>> Expresion: xxxx (the literal text value I want in facetContentType)
>>>
>>> Solr schema:
>>> .....
>>> <field name="facetContentType" type="string" indexed="true"
>>> stored="true" required="true"/>
>>> ....
>>>
>>> The error logged in ManifoldCF is:
>>> Error from server at http://solrServer:port/solr/c
>>> <http://revnetsolrdev:8983/solr/sites>ore: [doc=https://
>>> ....../index.aspx] missing required field: facetContentType.
>>>
>>> Thanks for your help
>>>
>>
>>
>
Re: Metadata adjuster
Posted by Marisol Redondo <ma...@gmail.com>.
Hi Karl and thank you for this quick answer.
I was reading the documentation of MCF 1.10 but I'm using MCF 2.5, sorry
for the confusion, and I think this version is compatible with solr6.
The pdf doesn't have any metadata or field called facetContentType, this is
because I'd been trying to use the Metadata Adjuster, to add a new
metadata/property to the doc so solr can index by this field when I'm
injecting the doc.
Should I use other transformation or is there any other way of duing it?
I am migrating from nutch to ManifoldCF and in nutch we can do it with
plugins, and I was thinking that the plugins in nutch are the same as the
transformation connectors in MCF
The completed error in solr is :
017-02-21 13:19:32.108 INFO (qtp1854778591-18) [ x:sites]
> o.a.s.c.PluginBag Going to create a new requestHandler with {type =
> requestHandler,name = /update/extract,class =
> solr.extraction.ExtractingRequestHandler,args =
> {defaults={lowernames=true,fmap.
> meta=ignored_,fmap.content=_text_,update.chain=add-unknown-fields-to-the-schema,df=_text_}}}
2017-02-21 13:19:32.454 INFO (qtp1854778591-18) [ x:sites]
> o.a.s.u.p.LogUpdateProcessorFactory [sites] webapp=/solr path=/up
date/extract params={resource.name=introduction.pdf&literal.id=https://...../introduction.pdf&wt=xml&version=2.2}{}
> 0 347
2017-02-21 13:19:32.455 ERROR (qtp1854778591-18) [ x:sites]
> o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: [
doc=https://....../introduction.pdf] missing required field:
> facetContentType
at
> org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:197)
at
> org.apache.solr.update.AddUpdateCommand.getLuceneDocument(AddUpdateCommand.java:82)
at
> org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:277)
at
> org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:211)
Thanks
On 21 February 2017 at 14:52, Karl Wright <da...@gmail.com> wrote:
> Hi Marisol,
>
> Can you find the [INFO] entry in the Solr log for this document? That
> should help clear up any confusion.
>
> Also, for what it is worth, MCF 1.10 is not using a SolrJ that is up to
> date with Solr 6.x. That could be the source of the problem Is there any
> reason you are using a 1.x version of MCF?
>
> Karl
>
>
> On Tue, Feb 21, 2017 at 8:42 AM, Marisol Redondo <
> marisol.redondo.garcia@gmail.com> wrote:
>
>> Hi.
>>
>> I'm trying to use metadata adjuster to add one field to the solr index
>> but doesn't inject the field into a solr's field.
>> Maybe I'm misundertaning the use of the metada adjuster, but I have read
>> in the documentation (https://manifoldcf.apache.org
>> /release/release-1.10/en_US/end-user-documentation.html) that I can add
>> metadata to the document that is going to be indexed into solr, but the
>> solr instance gave me the error "missing required field:
>> facetContentType".
>>
>> ManifoldCF Job pipeline:
>> 1. Repository (type web repository)
>> 2. Transformation (Tikka Metadata Extractor)
>> 3. Transformation (type Metada Adjuster)
>> 4. Output (Solr 6)
>>
>> ManifoldCF Job Metadata Expressions tab:
>> Parameter name: "facetContentType"
>> Remove this parameter: false
>> Expresion: xxxx (the literal text value I want in facetContentType)
>>
>> Solr schema:
>> .....
>> <field name="facetContentType" type="string" indexed="true"
>> stored="true" required="true"/>
>> ....
>>
>> The error logged in ManifoldCF is:
>> Error from server at http://solrServer:port/solr/c
>> <http://revnetsolrdev:8983/solr/sites>ore: [doc=https://....../index.aspx]
>> missing required field: facetContentType.
>>
>> Thanks for your help
>>
>
>
Re: Metadata adjuster
Posted by Karl Wright <da...@gmail.com>.
Hi Marisol,
Can you find the [INFO] entry in the Solr log for this document? That
should help clear up any confusion.
Also, for what it is worth, MCF 1.10 is not using a SolrJ that is up to
date with Solr 6.x. That could be the source of the problem Is there any
reason you are using a 1.x version of MCF?
Karl
On Tue, Feb 21, 2017 at 8:42 AM, Marisol Redondo <
marisol.redondo.garcia@gmail.com> wrote:
> Hi.
>
> I'm trying to use metadata adjuster to add one field to the solr index but
> doesn't inject the field into a solr's field.
> Maybe I'm misundertaning the use of the metada adjuster, but I have read
> in the documentation (https://manifoldcf.apache.org
> /release/release-1.10/en_US/end-user-documentation.html) that I can add
> metadata to the document that is going to be indexed into solr, but the
> solr instance gave me the error "missing required field:
> facetContentType".
>
> ManifoldCF Job pipeline:
> 1. Repository (type web repository)
> 2. Transformation (Tikka Metadata Extractor)
> 3. Transformation (type Metada Adjuster)
> 4. Output (Solr 6)
>
> ManifoldCF Job Metadata Expressions tab:
> Parameter name: "facetContentType"
> Remove this parameter: false
> Expresion: xxxx (the literal text value I want in facetContentType)
>
> Solr schema:
> .....
> <field name="facetContentType" type="string" indexed="true"
> stored="true" required="true"/>
> ....
>
> The error logged in ManifoldCF is:
> Error from server at http://solrServer:port/solr/c
> <http://revnetsolrdev:8983/solr/sites>ore: [doc=https://....../index.aspx]
> missing required field: facetContentType.
>
> Thanks for your help
>