You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "John Call (JIRA)" <ji...@apache.org> on 2016/10/05 01:37:20 UTC
[jira] [Updated] (SOLR-9579) Reuse lucene FieldType in createField
flow during ingestion
[ https://issues.apache.org/jira/browse/SOLR-9579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Call updated SOLR-9579:
----------------------------
Attachment: SOLR-9579.patch
Made SchemaField implement IndexableFieldType and updated lucene.document.Field to be dependent on IndexableFieldType instead of FieldType. Not sure if I should renaming some of the existing methods in SchemaField to conform to IndexableFieldType instead of having duplicate methods as this will touch a large number of files. (e.g. tokenized/IsTokenized, storeTermVector/storeTermVectors etc)
> Reuse lucene FieldType in createField flow during ingestion
> -----------------------------------------------------------
>
> Key: SOLR-9579
> URL: https://issues.apache.org/jira/browse/SOLR-9579
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 6.x, master (7.0)
> Environment: This has been primarily tested on Windows 8 and Windows Server 2012 R2
> Reporter: John Call
> Priority: Minor
> Labels: gc, memory, reuse
> Fix For: 6.x, master (7.0)
>
> Attachments: SOLR-9579.patch, SOLR-9579.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> During ingestion createField in FieldType is being called for each field on each document. For the subclasses of FieldType without their own implementation of createField the lucene version of FieldType is created to be stored along with the value. However the lucene FieldType object is identical when created from the same SchemaField. In testing ingestion of one million rows with 22 field each we were creating 22 million lucene FieldType objects when only 22 are needed. Solr should lazily initialize a lucene FieldType for each SchemaField and reuse them for future ingestion. Not only does this relieve memory usage but also relieves significant pressure on the gc.
> There are also subclasses of Solr FieldType which create separate Lucene FieldType for stored fields instead of reusing the static in StoredField.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org