You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Zheng Lin Edwin Yeo <ed...@gmail.com> on 2015/09/02 11:53:13 UTC

String bytes can be at most 32766 characters in length?

Hi,

I would like to check, is the string bytes must be at most 32766 characters
in length?

I'm trying to do a copyField of my rich-text documents content to a field
with fieldType=string to try out my getting distinct result for content, as
there are several documents with the exact same content, and we only want
to list one of them during searching.

However, I get the following errors in some of the documents when I tried
to index them with the copyField. Some of my documents are quite large in
size, and there is a possibility that it exceed 32766 characters. Is there
any other ways to overcome this problem?


org.apache.solr.common.SolrException: Exception writing document id
collection1_polymer100 to the index; possible analysis error.
at
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:167)
at
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:955)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1110)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:706)
at
org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:104)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
at
org.apache.solr.update.processor.LanguageIdentifierUpdateProcessor.processAdd(LanguageIdentifierUpdateProcessor.java:207)
at
org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(ExtractingDocumentLoader.java:122)
at
org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(ExtractingDocumentLoader.java:127)
at
org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:235)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Document contains at least
one immense term in field="signature" (whose UTF8 encoding is longer than
the max length 32766), all of which were skipped.  Please correct the
analyzer to not produce such terms.  The prefix of the first immense term
is: '[32, 60, 112, 62, 60, 98, 114, 62, 32, 32, 32, 60, 98, 114, 62, 56,
48, 56, 32, 72, 97, 110, 100, 98, 111, 111, 107, 32, 111, 102]...',
original message: bytes can be at most 32766 in length; got 49960
at
org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:670)
at
org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:344)
at
org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:300)
at
org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:232)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:458)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1363)
at
org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:239)
at
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:163)
... 38 more
Caused by:
org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes
can be at most 32766 in length; got 49960
at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:284)
at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:154)
at
org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:660)
... 45 more


Regards,
Edwin

Re: String bytes can be at most 32766 characters in length?

Posted by Zheng Lin Edwin Yeo <ed...@gmail.com>.
Hi Erick,

I couldn't really find anything special in the logs. The indexing process
just went on normally, but after that when I check the index, there is
nothing indexed.

This is what I see from the logs. Looks the same as when the indexing works
fine.

INFO  - 2015-09-03 01:24:35.316; [collection1 shard1 core_node2
collection1] org.apache.solr.handler.extraction.SolrContentHandler; Content
1 = content
INFO  - 2015-09-03 01:24:35.319; [collection1 shard1 core_node2
collection1] org.apache.solr.handler.extraction.SolrContentHandler; Content
2 = content
INFO  - 2015-09-03 01:24:35.482; [collection1 shard1 core_node1
collection1_shard1_replica2] org.apache.solr.core.SolrCore;
[collection1_shard1_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=
http://localhost:8983/solr/collection1/&wt=javabin&version=2
<http://192.168.23.52:8983/edm/collection1/&wt=javabin&version=2>} status=0
QTime=4
INFO  - 2015-09-03 01:24:35.483; [collection1 shard1 core_node2
collection1] org.apache.solr.update.processor.LogUpdateProcessor;
[collection1] webapp=/solr path=/update/extract
params={literal.geolocation=1,103&literal.popularity=0&literal.title=cat&literal.entity=Growhill&
resource.name
=C:\Users\edwin_000\Desktop\edwin\solr-5.2.1\IndexingDocuments\collection1\cat.pdf&
literal.id=collection1_cat&literal.location=Singapore&literal.accessgroup=VIP&literal.content_cat=test3&literal.crossreference=science&literal.accesslevel=8&literal.importance=5&literal.userid=edwin&literal.reference=science&literal.url=C:\Users\edwin_000\Desktop\edwin\solr-5.2.1\IndexingDocuments\collection1\cat.pdf&literal.content_subcat=test3&literal.visibility=Public}
{add=[collection1_cat (1511253318382387200)]} 0 437
INFO  - 2015-09-03 01:24:36.218; [collection1 shard1 core_node2
collection1] org.apache.solr.handler.extraction.SolrContentHandler; Content
1 = content
INFO  - 2015-09-03 01:24:36.225; [collection1 shard1 core_node2
collection1] org.apache.solr.handler.extraction.SolrContentHandler; Content
2 = content
INFO  - 2015-09-03 01:24:36.487; [collection1 shard1 core_node1
collection1_shard1_replica2] org.apache.solr.core.SolrCore;
[collection1_shard1_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=
http://localhost:8983/solr/collection1/&wt=javabin&version=2
<http://192.168.23.52:8983/edm/collection1/&wt=javabin&version=2>} status=0
QTime=6


Regards,
Edwin


On 2 September 2015 at 23:34, Erick Erickson <er...@gmail.com>
wrote:

> _How_ does it fail? You must be seeing something in the logs....
>
>
>
> On Wed, Sep 2, 2015 at 8:29 AM, Zheng Lin Edwin Yeo
> <ed...@gmail.com> wrote:
> > Hi Erick,
> >
> > Yes, i'm trying out the De-Duplication too. But I'm facing a problem with
> > that, which is the indexing stops working once I put in the following
> > De-Duplication code in solrconfig.xml. The problem seems to be with this
> <str
> > name="update.chain">dedupe</str> line.
> >
> >   <requestHandler name="/update" class="solr.UpdateRequestHandler">
> >   <lst name="defaults">
> > <str name="update.chain">dedupe</str>
> >   </lst>
> >   </requestHandler>
> >
> >
> >     <updateRequestProcessorChain name="dedupe">
> >   <processor class="solr.processor.SignatureUpdateProcessorFactory">
> > <bool name="enabled">true</bool>
> > <str name="signatureField">signature</str>
> > <bool name="overwriteDupes">false</bool>
> > <str name="fields">content</str>
> > <str name="signatureClass">solr.processor.Lookup3Signature</str>
> >   </processor>
> > </updateRequestProcessorChain>
> >
> >
> > Regards,
> > Edwin
> >
> > On 2 September 2015 at 23:10, Erick Erickson <er...@gmail.com>
> > wrote:
> >
> >> Yes, that is an intentional limit for the size of a single token,
> >> which strings are.
> >>
> >> Why not use deduplication? See:
> >> https://cwiki.apache.org/confluence/display/solr/De-Duplication
> >>
> >> You don't have to replace the existing documents, and Solr will
> >> compute a hash that can be used to identify identical documents
> >> and you can use_that_.
> >>
> >> Best
> >> Erick
> >>
> >> On Wed, Sep 2, 2015 at 2:53 AM, Zheng Lin Edwin Yeo
> >> <ed...@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > I would like to check, is the string bytes must be at most 32766
> >> characters
> >> > in length?
> >> >
> >> > I'm trying to do a copyField of my rich-text documents content to a
> field
> >> > with fieldType=string to try out my getting distinct result for
> content,
> >> as
> >> > there are several documents with the exact same content, and we only
> want
> >> > to list one of them during searching.
> >> >
> >> > However, I get the following errors in some of the documents when I
> tried
> >> > to index them with the copyField. Some of my documents are quite
> large in
> >> > size, and there is a possibility that it exceed 32766 characters. Is
> >> there
> >> > any other ways to overcome this problem?
> >> >
> >> >
> >> > org.apache.solr.common.SolrException: Exception writing document id
> >> > collection1_polymer100 to the index; possible analysis error.
> >> > at
> >> >
> >>
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:167)
> >> > at
> >> >
> >>
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
> >> > at
> >> >
> >>
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
> >> > at
> >> >
> >>
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:955)
> >> > at
> >> >
> >>
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1110)
> >> > at
> >> >
> >>
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:706)
> >> > at
> >> >
> >>
> org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:104)
> >> > at
> >> >
> >>
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
> >> > at
> >> >
> >>
> org.apache.solr.update.processor.LanguageIdentifierUpdateProcessor.processAdd(LanguageIdentifierUpdateProcessor.java:207)
> >> > at
> >> >
> >>
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(ExtractingDocumentLoader.java:122)
> >> > at
> >> >
> >>
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(ExtractingDocumentLoader.java:127)
> >> > at
> >> >
> >>
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:235)
> >> > at
> >> >
> >>
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> >> > at
> >> >
> >>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> >> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
> >> > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
> >> > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
> >> > at
> >> >
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
> >> > at
> >> >
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
> >> > at
> >> >
> >>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> >> > at
> >> >
> >>
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> >> > at
> >> >
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> >> > at
> >> >
> >>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> >> > at
> >> >
> >>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> >> > at
> >> >
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> >> > at
> >>
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> >> > at
> >> >
> >>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> >> > at
> >> >
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> >> > at
> >> >
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> >> > at
> >> >
> >>
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> >> > at
> >> >
> >>
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> >> > at
> >> >
> >>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> >> > at org.eclipse.jetty.server.Server.handle(Server.java:497)
> >> > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> >> > at
> >> >
> >>
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> >> > at
> >> >
> >>
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> >> > at
> >> >
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> >> > at
> >> >
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> >> > at java.lang.Thread.run(Thread.java:745)
> >> > Caused by: java.lang.IllegalArgumentException: Document contains at
> least
> >> > one immense term in field="signature" (whose UTF8 encoding is longer
> than
> >> > the max length 32766), all of which were skipped.  Please correct the
> >> > analyzer to not produce such terms.  The prefix of the first immense
> term
> >> > is: '[32, 60, 112, 62, 60, 98, 114, 62, 32, 32, 32, 60, 98, 114, 62,
> 56,
> >> > 48, 56, 32, 72, 97, 110, 100, 98, 111, 111, 107, 32, 111, 102]...',
> >> > original message: bytes can be at most 32766 in length; got 49960
> >> > at
> >> >
> >>
> org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:670)
> >> > at
> >> >
> >>
> org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:344)
> >> > at
> >> >
> >>
> org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:300)
> >> > at
> >> >
> >>
> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:232)
> >> > at
> >> >
> >>
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:458)
> >> > at
> >>
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1363)
> >> > at
> >> >
> >>
> org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:239)
> >> > at
> >> >
> >>
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:163)
> >> > ... 38 more
> >> > Caused by:
> >> > org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException:
> >> bytes
> >> > can be at most 32766 in length; got 49960
> >> > at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:284)
> >> > at
> >>
> org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:154)
> >> > at
> >> >
> >>
> org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:660)
> >> > ... 45 more
> >> >
> >> >
> >> > Regards,
> >> > Edwin
> >>
>

Re: String bytes can be at most 32766 characters in length?

Posted by Erick Erickson <er...@gmail.com>.
_How_ does it fail? You must be seeing something in the logs....



On Wed, Sep 2, 2015 at 8:29 AM, Zheng Lin Edwin Yeo
<ed...@gmail.com> wrote:
> Hi Erick,
>
> Yes, i'm trying out the De-Duplication too. But I'm facing a problem with
> that, which is the indexing stops working once I put in the following
> De-Duplication code in solrconfig.xml. The problem seems to be with this <str
> name="update.chain">dedupe</str> line.
>
>   <requestHandler name="/update" class="solr.UpdateRequestHandler">
>   <lst name="defaults">
> <str name="update.chain">dedupe</str>
>   </lst>
>   </requestHandler>
>
>
>     <updateRequestProcessorChain name="dedupe">
>   <processor class="solr.processor.SignatureUpdateProcessorFactory">
> <bool name="enabled">true</bool>
> <str name="signatureField">signature</str>
> <bool name="overwriteDupes">false</bool>
> <str name="fields">content</str>
> <str name="signatureClass">solr.processor.Lookup3Signature</str>
>   </processor>
> </updateRequestProcessorChain>
>
>
> Regards,
> Edwin
>
> On 2 September 2015 at 23:10, Erick Erickson <er...@gmail.com>
> wrote:
>
>> Yes, that is an intentional limit for the size of a single token,
>> which strings are.
>>
>> Why not use deduplication? See:
>> https://cwiki.apache.org/confluence/display/solr/De-Duplication
>>
>> You don't have to replace the existing documents, and Solr will
>> compute a hash that can be used to identify identical documents
>> and you can use_that_.
>>
>> Best
>> Erick
>>
>> On Wed, Sep 2, 2015 at 2:53 AM, Zheng Lin Edwin Yeo
>> <ed...@gmail.com> wrote:
>> > Hi,
>> >
>> > I would like to check, is the string bytes must be at most 32766
>> characters
>> > in length?
>> >
>> > I'm trying to do a copyField of my rich-text documents content to a field
>> > with fieldType=string to try out my getting distinct result for content,
>> as
>> > there are several documents with the exact same content, and we only want
>> > to list one of them during searching.
>> >
>> > However, I get the following errors in some of the documents when I tried
>> > to index them with the copyField. Some of my documents are quite large in
>> > size, and there is a possibility that it exceed 32766 characters. Is
>> there
>> > any other ways to overcome this problem?
>> >
>> >
>> > org.apache.solr.common.SolrException: Exception writing document id
>> > collection1_polymer100 to the index; possible analysis error.
>> > at
>> >
>> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:167)
>> > at
>> >
>> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
>> > at
>> >
>> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
>> > at
>> >
>> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:955)
>> > at
>> >
>> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1110)
>> > at
>> >
>> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:706)
>> > at
>> >
>> org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:104)
>> > at
>> >
>> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
>> > at
>> >
>> org.apache.solr.update.processor.LanguageIdentifierUpdateProcessor.processAdd(LanguageIdentifierUpdateProcessor.java:207)
>> > at
>> >
>> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(ExtractingDocumentLoader.java:122)
>> > at
>> >
>> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(ExtractingDocumentLoader.java:127)
>> > at
>> >
>> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:235)
>> > at
>> >
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
>> > at
>> >
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
>> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
>> > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
>> > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
>> > at
>> >
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
>> > at
>> >
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
>> > at
>> >
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>> > at
>> >
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>> > at
>> >
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>> > at
>> >
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
>> > at
>> >
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
>> > at
>> >
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>> > at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>> > at
>> >
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>> > at
>> >
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>> > at
>> >
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>> > at
>> >
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
>> > at
>> >
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
>> > at
>> >
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>> > at org.eclipse.jetty.server.Server.handle(Server.java:497)
>> > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>> > at
>> >
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>> > at
>> >
>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>> > at
>> >
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>> > at
>> >
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>> > at java.lang.Thread.run(Thread.java:745)
>> > Caused by: java.lang.IllegalArgumentException: Document contains at least
>> > one immense term in field="signature" (whose UTF8 encoding is longer than
>> > the max length 32766), all of which were skipped.  Please correct the
>> > analyzer to not produce such terms.  The prefix of the first immense term
>> > is: '[32, 60, 112, 62, 60, 98, 114, 62, 32, 32, 32, 60, 98, 114, 62, 56,
>> > 48, 56, 32, 72, 97, 110, 100, 98, 111, 111, 107, 32, 111, 102]...',
>> > original message: bytes can be at most 32766 in length; got 49960
>> > at
>> >
>> org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:670)
>> > at
>> >
>> org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:344)
>> > at
>> >
>> org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:300)
>> > at
>> >
>> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:232)
>> > at
>> >
>> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:458)
>> > at
>> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1363)
>> > at
>> >
>> org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:239)
>> > at
>> >
>> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:163)
>> > ... 38 more
>> > Caused by:
>> > org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException:
>> bytes
>> > can be at most 32766 in length; got 49960
>> > at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:284)
>> > at
>> org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:154)
>> > at
>> >
>> org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:660)
>> > ... 45 more
>> >
>> >
>> > Regards,
>> > Edwin
>>

Re: String bytes can be at most 32766 characters in length?

Posted by Zheng Lin Edwin Yeo <ed...@gmail.com>.
Thanks for your advice Alexandre.

On 3 September 2015 at 20:29, Alexandre Rafalovitch <ar...@gmail.com>
wrote:

> Probably because your signatureField and your fields are the same! You
> need to point signatureField at a new (not-ID) field.
>
> You will still get duplicates, as you requested that in your other
> emails, but now you would be able to group on that new signature
> field.
>
> If you have any further problems, you also need to start a new thread
> with a new subject, as the current question is no longer related.
>
> Regards,
>    Alex.
> ----
> Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
> http://www.solr-start.com/
>
>
> On 2 September 2015 at 22:21, Zheng Lin Edwin Yeo <ed...@gmail.com>
> wrote:
> > Hi Alexandre,
> >
> > Thanks for pointing out the error. I'm able to get the documents to be
> > indexed after adding in the two processors.
> >
> > However, I'm still seeing all the similar documents being search in the
> > content without being de-duplicated. My content is currently indexed as
> > fieldType=text_general.
> >
> >     <updateRequestProcessorChain name="dedupe">
> >  <processor class="solr.processor.SignatureUpdateProcessorFactory">
> > <bool name="enabled">true</bool>
> > <str name="signatureField">content</str>
> > <bool name="overwriteDupes">false</bool>
> > <str name="fields">content</str>
> > <str name="signatureClass">solr.processor.Lookup3Signature</str>
> >  </processor>
> > <processor class="solr.LogUpdateProcessorFactory" />
> > <processor class="solr.RunUpdateProcessorFactory" />
> > </updateRequestProcessorChain>
> >
> > Regards,
> > Edwin
> >
> >
> > On 3 September 2015 at 09:46, Alexandre Rafalovitch <ar...@gmail.com>
> > wrote:
> >
> >> And that's because you have an incomplete chain. If you look at the
> >> full example in solrconfig.xml, it shows:
> >>      <updateRequestProcessorChain name="dedupe">
> >>        <processor
> class="solr.processor.SignatureUpdateProcessorFactory">
> >>          <bool name="enabled">true</bool>
> >>          <str name="signatureField">id</str>
> >>          <bool name="overwriteDupes">false</bool>
> >>          <str name="fields">name,features,cat</str>
> >>          <str
> name="signatureClass">solr.processor.Lookup3Signature</str>
> >>        </processor>
> >>        <processor class="solr.LogUpdateProcessorFactory" />
> >>        <processor class="solr.RunUpdateProcessorFactory" />
> >>      </updateRequestProcessorChain>
> >>
> >>
> >> Notice, the last two processors. If you don't have those, nothing gets
> >> indexed. You chain is missing them, for whatever reasons. Try adding
> >> them back in, reloading the core and reindexing.
> >>
> >> Regards,
> >>    Alex.
> >> ----
> >> Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
> >> http://www.solr-start.com/
> >>
> >>
> >> On 2 September 2015 at 11:29, Zheng Lin Edwin Yeo <edwinyeozl@gmail.com
> >
> >> wrote:
> >> > Hi Erick,
> >> >
> >> > Yes, i'm trying out the De-Duplication too. But I'm facing a problem
> with
> >> > that, which is the indexing stops working once I put in the following
> >> > De-Duplication code in solrconfig.xml. The problem seems to be with
> this
> >> <str
> >> > name="update.chain">dedupe</str> line.
> >> >
> >> >   <requestHandler name="/update" class="solr.UpdateRequestHandler">
> >> >   <lst name="defaults">
> >> > <str name="update.chain">dedupe</str>
> >> >   </lst>
> >> >   </requestHandler>
> >> >
> >> >
> >> >     <updateRequestProcessorChain name="dedupe">
> >> >   <processor class="solr.processor.SignatureUpdateProcessorFactory">
> >> > <bool name="enabled">true</bool>
> >> > <str name="signatureField">signature</str>
> >> > <bool name="overwriteDupes">false</bool>
> >> > <str name="fields">content</str>
> >> > <str name="signatureClass">solr.processor.Lookup3Signature</str>
> >> >   </processor>
> >> > </updateRequestProcessorChain>
> >> >
> >> >
> >> > Regards,
> >> > Edwin
> >> >
> >> > On 2 September 2015 at 23:10, Erick Erickson <erickerickson@gmail.com
> >
> >> > wrote:
> >> >
> >> >> Yes, that is an intentional limit for the size of a single token,
> >> >> which strings are.
> >> >>
> >> >> Why not use deduplication? See:
> >> >> https://cwiki.apache.org/confluence/display/solr/De-Duplication
> >> >>
> >> >> You don't have to replace the existing documents, and Solr will
> >> >> compute a hash that can be used to identify identical documents
> >> >> and you can use_that_.
> >> >>
> >> >> Best
> >> >> Erick
> >> >>
> >> >> On Wed, Sep 2, 2015 at 2:53 AM, Zheng Lin Edwin Yeo
> >> >> <ed...@gmail.com> wrote:
> >> >> > Hi,
> >> >> >
> >> >> > I would like to check, is the string bytes must be at most 32766
> >> >> characters
> >> >> > in length?
> >> >> >
> >> >> > I'm trying to do a copyField of my rich-text documents content to a
> >> field
> >> >> > with fieldType=string to try out my getting distinct result for
> >> content,
> >> >> as
> >> >> > there are several documents with the exact same content, and we
> only
> >> want
> >> >> > to list one of them during searching.
> >> >> >
> >> >> > However, I get the following errors in some of the documents when I
> >> tried
> >> >> > to index them with the copyField. Some of my documents are quite
> >> large in
> >> >> > size, and there is a possibility that it exceed 32766 characters.
> Is
> >> >> there
> >> >> > any other ways to overcome this problem?
> >> >> >
> >> >> >
> >> >> > org.apache.solr.common.SolrException: Exception writing document id
> >> >> > collection1_polymer100 to the index; possible analysis error.
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:167)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:955)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1110)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:706)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:104)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.solr.update.processor.LanguageIdentifierUpdateProcessor.processAdd(LanguageIdentifierUpdateProcessor.java:207)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(ExtractingDocumentLoader.java:122)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(ExtractingDocumentLoader.java:127)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:235)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> >> >> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
> >> >> > at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
> >> >> > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> >> >> > at
> >> >>
> >>
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> >> >> > at org.eclipse.jetty.server.Server.handle(Server.java:497)
> >> >> > at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> >> >> > at java.lang.Thread.run(Thread.java:745)
> >> >> > Caused by: java.lang.IllegalArgumentException: Document contains at
> >> least
> >> >> > one immense term in field="signature" (whose UTF8 encoding is
> longer
> >> than
> >> >> > the max length 32766), all of which were skipped.  Please correct
> the
> >> >> > analyzer to not produce such terms.  The prefix of the first
> immense
> >> term
> >> >> > is: '[32, 60, 112, 62, 60, 98, 114, 62, 32, 32, 32, 60, 98, 114,
> 62,
> >> 56,
> >> >> > 48, 56, 32, 72, 97, 110, 100, 98, 111, 111, 107, 32, 111, 102]...',
> >> >> > original message: bytes can be at most 32766 in length; got 49960
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:670)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:344)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:300)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:232)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:458)
> >> >> > at
> >> >>
> >>
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1363)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:239)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:163)
> >> >> > ... 38 more
> >> >> > Caused by:
> >> >> >
> org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException:
> >> >> bytes
> >> >> > can be at most 32766 in length; got 49960
> >> >> > at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:284)
> >> >> > at
> >> >>
> >>
> org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:154)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:660)
> >> >> > ... 45 more
> >> >> >
> >> >> >
> >> >> > Regards,
> >> >> > Edwin
> >> >>
> >>
>

Re: String bytes can be at most 32766 characters in length?

Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Probably because your signatureField and your fields are the same! You
need to point signatureField at a new (not-ID) field.

You will still get duplicates, as you requested that in your other
emails, but now you would be able to group on that new signature
field.

If you have any further problems, you also need to start a new thread
with a new subject, as the current question is no longer related.

Regards,
   Alex.
----
Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
http://www.solr-start.com/


On 2 September 2015 at 22:21, Zheng Lin Edwin Yeo <ed...@gmail.com> wrote:
> Hi Alexandre,
>
> Thanks for pointing out the error. I'm able to get the documents to be
> indexed after adding in the two processors.
>
> However, I'm still seeing all the similar documents being search in the
> content without being de-duplicated. My content is currently indexed as
> fieldType=text_general.
>
>     <updateRequestProcessorChain name="dedupe">
>  <processor class="solr.processor.SignatureUpdateProcessorFactory">
> <bool name="enabled">true</bool>
> <str name="signatureField">content</str>
> <bool name="overwriteDupes">false</bool>
> <str name="fields">content</str>
> <str name="signatureClass">solr.processor.Lookup3Signature</str>
>  </processor>
> <processor class="solr.LogUpdateProcessorFactory" />
> <processor class="solr.RunUpdateProcessorFactory" />
> </updateRequestProcessorChain>
>
> Regards,
> Edwin
>
>
> On 3 September 2015 at 09:46, Alexandre Rafalovitch <ar...@gmail.com>
> wrote:
>
>> And that's because you have an incomplete chain. If you look at the
>> full example in solrconfig.xml, it shows:
>>      <updateRequestProcessorChain name="dedupe">
>>        <processor class="solr.processor.SignatureUpdateProcessorFactory">
>>          <bool name="enabled">true</bool>
>>          <str name="signatureField">id</str>
>>          <bool name="overwriteDupes">false</bool>
>>          <str name="fields">name,features,cat</str>
>>          <str name="signatureClass">solr.processor.Lookup3Signature</str>
>>        </processor>
>>        <processor class="solr.LogUpdateProcessorFactory" />
>>        <processor class="solr.RunUpdateProcessorFactory" />
>>      </updateRequestProcessorChain>
>>
>>
>> Notice, the last two processors. If you don't have those, nothing gets
>> indexed. You chain is missing them, for whatever reasons. Try adding
>> them back in, reloading the core and reindexing.
>>
>> Regards,
>>    Alex.
>> ----
>> Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
>> http://www.solr-start.com/
>>
>>
>> On 2 September 2015 at 11:29, Zheng Lin Edwin Yeo <ed...@gmail.com>
>> wrote:
>> > Hi Erick,
>> >
>> > Yes, i'm trying out the De-Duplication too. But I'm facing a problem with
>> > that, which is the indexing stops working once I put in the following
>> > De-Duplication code in solrconfig.xml. The problem seems to be with this
>> <str
>> > name="update.chain">dedupe</str> line.
>> >
>> >   <requestHandler name="/update" class="solr.UpdateRequestHandler">
>> >   <lst name="defaults">
>> > <str name="update.chain">dedupe</str>
>> >   </lst>
>> >   </requestHandler>
>> >
>> >
>> >     <updateRequestProcessorChain name="dedupe">
>> >   <processor class="solr.processor.SignatureUpdateProcessorFactory">
>> > <bool name="enabled">true</bool>
>> > <str name="signatureField">signature</str>
>> > <bool name="overwriteDupes">false</bool>
>> > <str name="fields">content</str>
>> > <str name="signatureClass">solr.processor.Lookup3Signature</str>
>> >   </processor>
>> > </updateRequestProcessorChain>
>> >
>> >
>> > Regards,
>> > Edwin
>> >
>> > On 2 September 2015 at 23:10, Erick Erickson <er...@gmail.com>
>> > wrote:
>> >
>> >> Yes, that is an intentional limit for the size of a single token,
>> >> which strings are.
>> >>
>> >> Why not use deduplication? See:
>> >> https://cwiki.apache.org/confluence/display/solr/De-Duplication
>> >>
>> >> You don't have to replace the existing documents, and Solr will
>> >> compute a hash that can be used to identify identical documents
>> >> and you can use_that_.
>> >>
>> >> Best
>> >> Erick
>> >>
>> >> On Wed, Sep 2, 2015 at 2:53 AM, Zheng Lin Edwin Yeo
>> >> <ed...@gmail.com> wrote:
>> >> > Hi,
>> >> >
>> >> > I would like to check, is the string bytes must be at most 32766
>> >> characters
>> >> > in length?
>> >> >
>> >> > I'm trying to do a copyField of my rich-text documents content to a
>> field
>> >> > with fieldType=string to try out my getting distinct result for
>> content,
>> >> as
>> >> > there are several documents with the exact same content, and we only
>> want
>> >> > to list one of them during searching.
>> >> >
>> >> > However, I get the following errors in some of the documents when I
>> tried
>> >> > to index them with the copyField. Some of my documents are quite
>> large in
>> >> > size, and there is a possibility that it exceed 32766 characters. Is
>> >> there
>> >> > any other ways to overcome this problem?
>> >> >
>> >> >
>> >> > org.apache.solr.common.SolrException: Exception writing document id
>> >> > collection1_polymer100 to the index; possible analysis error.
>> >> > at
>> >> >
>> >>
>> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:167)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:955)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1110)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:706)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:104)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.update.processor.LanguageIdentifierUpdateProcessor.processAdd(LanguageIdentifierUpdateProcessor.java:207)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(ExtractingDocumentLoader.java:122)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(ExtractingDocumentLoader.java:127)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:235)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
>> >> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
>> >> > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
>> >> > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
>> >> > at
>> >> >
>> >>
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>> >> > at
>> >> >
>> >>
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>> >> > at
>> >> >
>> >>
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>> >> > at
>> >> >
>> >>
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
>> >> > at
>> >> >
>> >>
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
>> >> > at
>> >> >
>> >>
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>> >> > at
>> >>
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>> >> > at
>> >> >
>> >>
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>> >> > at
>> >> >
>> >>
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>> >> > at
>> >> >
>> >>
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>> >> > at
>> >> >
>> >>
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
>> >> > at
>> >> >
>> >>
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
>> >> > at
>> >> >
>> >>
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>> >> > at org.eclipse.jetty.server.Server.handle(Server.java:497)
>> >> > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>> >> > at
>> >> >
>> >>
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>> >> > at
>> >> >
>> >>
>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>> >> > at
>> >> >
>> >>
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>> >> > at
>> >> >
>> >>
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>> >> > at java.lang.Thread.run(Thread.java:745)
>> >> > Caused by: java.lang.IllegalArgumentException: Document contains at
>> least
>> >> > one immense term in field="signature" (whose UTF8 encoding is longer
>> than
>> >> > the max length 32766), all of which were skipped.  Please correct the
>> >> > analyzer to not produce such terms.  The prefix of the first immense
>> term
>> >> > is: '[32, 60, 112, 62, 60, 98, 114, 62, 32, 32, 32, 60, 98, 114, 62,
>> 56,
>> >> > 48, 56, 32, 72, 97, 110, 100, 98, 111, 111, 107, 32, 111, 102]...',
>> >> > original message: bytes can be at most 32766 in length; got 49960
>> >> > at
>> >> >
>> >>
>> org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:670)
>> >> > at
>> >> >
>> >>
>> org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:344)
>> >> > at
>> >> >
>> >>
>> org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:300)
>> >> > at
>> >> >
>> >>
>> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:232)
>> >> > at
>> >> >
>> >>
>> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:458)
>> >> > at
>> >>
>> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1363)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:239)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:163)
>> >> > ... 38 more
>> >> > Caused by:
>> >> > org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException:
>> >> bytes
>> >> > can be at most 32766 in length; got 49960
>> >> > at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:284)
>> >> > at
>> >>
>> org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:154)
>> >> > at
>> >> >
>> >>
>> org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:660)
>> >> > ... 45 more
>> >> >
>> >> >
>> >> > Regards,
>> >> > Edwin
>> >>
>>

Re: String bytes can be at most 32766 characters in length?

Posted by Zheng Lin Edwin Yeo <ed...@gmail.com>.
Hi Alexandre,

Thanks for pointing out the error. I'm able to get the documents to be
indexed after adding in the two processors.

However, I'm still seeing all the similar documents being search in the
content without being de-duplicated. My content is currently indexed as
fieldType=text_general.

    <updateRequestProcessorChain name="dedupe">
 <processor class="solr.processor.SignatureUpdateProcessorFactory">
<bool name="enabled">true</bool>
<str name="signatureField">content</str>
<bool name="overwriteDupes">false</bool>
<str name="fields">content</str>
<str name="signatureClass">solr.processor.Lookup3Signature</str>
 </processor>
<processor class="solr.LogUpdateProcessorFactory" />
<processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>

Regards,
Edwin


On 3 September 2015 at 09:46, Alexandre Rafalovitch <ar...@gmail.com>
wrote:

> And that's because you have an incomplete chain. If you look at the
> full example in solrconfig.xml, it shows:
>      <updateRequestProcessorChain name="dedupe">
>        <processor class="solr.processor.SignatureUpdateProcessorFactory">
>          <bool name="enabled">true</bool>
>          <str name="signatureField">id</str>
>          <bool name="overwriteDupes">false</bool>
>          <str name="fields">name,features,cat</str>
>          <str name="signatureClass">solr.processor.Lookup3Signature</str>
>        </processor>
>        <processor class="solr.LogUpdateProcessorFactory" />
>        <processor class="solr.RunUpdateProcessorFactory" />
>      </updateRequestProcessorChain>
>
>
> Notice, the last two processors. If you don't have those, nothing gets
> indexed. You chain is missing them, for whatever reasons. Try adding
> them back in, reloading the core and reindexing.
>
> Regards,
>    Alex.
> ----
> Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
> http://www.solr-start.com/
>
>
> On 2 September 2015 at 11:29, Zheng Lin Edwin Yeo <ed...@gmail.com>
> wrote:
> > Hi Erick,
> >
> > Yes, i'm trying out the De-Duplication too. But I'm facing a problem with
> > that, which is the indexing stops working once I put in the following
> > De-Duplication code in solrconfig.xml. The problem seems to be with this
> <str
> > name="update.chain">dedupe</str> line.
> >
> >   <requestHandler name="/update" class="solr.UpdateRequestHandler">
> >   <lst name="defaults">
> > <str name="update.chain">dedupe</str>
> >   </lst>
> >   </requestHandler>
> >
> >
> >     <updateRequestProcessorChain name="dedupe">
> >   <processor class="solr.processor.SignatureUpdateProcessorFactory">
> > <bool name="enabled">true</bool>
> > <str name="signatureField">signature</str>
> > <bool name="overwriteDupes">false</bool>
> > <str name="fields">content</str>
> > <str name="signatureClass">solr.processor.Lookup3Signature</str>
> >   </processor>
> > </updateRequestProcessorChain>
> >
> >
> > Regards,
> > Edwin
> >
> > On 2 September 2015 at 23:10, Erick Erickson <er...@gmail.com>
> > wrote:
> >
> >> Yes, that is an intentional limit for the size of a single token,
> >> which strings are.
> >>
> >> Why not use deduplication? See:
> >> https://cwiki.apache.org/confluence/display/solr/De-Duplication
> >>
> >> You don't have to replace the existing documents, and Solr will
> >> compute a hash that can be used to identify identical documents
> >> and you can use_that_.
> >>
> >> Best
> >> Erick
> >>
> >> On Wed, Sep 2, 2015 at 2:53 AM, Zheng Lin Edwin Yeo
> >> <ed...@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > I would like to check, is the string bytes must be at most 32766
> >> characters
> >> > in length?
> >> >
> >> > I'm trying to do a copyField of my rich-text documents content to a
> field
> >> > with fieldType=string to try out my getting distinct result for
> content,
> >> as
> >> > there are several documents with the exact same content, and we only
> want
> >> > to list one of them during searching.
> >> >
> >> > However, I get the following errors in some of the documents when I
> tried
> >> > to index them with the copyField. Some of my documents are quite
> large in
> >> > size, and there is a possibility that it exceed 32766 characters. Is
> >> there
> >> > any other ways to overcome this problem?
> >> >
> >> >
> >> > org.apache.solr.common.SolrException: Exception writing document id
> >> > collection1_polymer100 to the index; possible analysis error.
> >> > at
> >> >
> >>
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:167)
> >> > at
> >> >
> >>
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
> >> > at
> >> >
> >>
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
> >> > at
> >> >
> >>
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:955)
> >> > at
> >> >
> >>
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1110)
> >> > at
> >> >
> >>
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:706)
> >> > at
> >> >
> >>
> org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:104)
> >> > at
> >> >
> >>
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
> >> > at
> >> >
> >>
> org.apache.solr.update.processor.LanguageIdentifierUpdateProcessor.processAdd(LanguageIdentifierUpdateProcessor.java:207)
> >> > at
> >> >
> >>
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(ExtractingDocumentLoader.java:122)
> >> > at
> >> >
> >>
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(ExtractingDocumentLoader.java:127)
> >> > at
> >> >
> >>
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:235)
> >> > at
> >> >
> >>
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> >> > at
> >> >
> >>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> >> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
> >> > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
> >> > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
> >> > at
> >> >
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
> >> > at
> >> >
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
> >> > at
> >> >
> >>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> >> > at
> >> >
> >>
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> >> > at
> >> >
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> >> > at
> >> >
> >>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> >> > at
> >> >
> >>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> >> > at
> >> >
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> >> > at
> >>
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> >> > at
> >> >
> >>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> >> > at
> >> >
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> >> > at
> >> >
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> >> > at
> >> >
> >>
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> >> > at
> >> >
> >>
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> >> > at
> >> >
> >>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> >> > at org.eclipse.jetty.server.Server.handle(Server.java:497)
> >> > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> >> > at
> >> >
> >>
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> >> > at
> >> >
> >>
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> >> > at
> >> >
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> >> > at
> >> >
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> >> > at java.lang.Thread.run(Thread.java:745)
> >> > Caused by: java.lang.IllegalArgumentException: Document contains at
> least
> >> > one immense term in field="signature" (whose UTF8 encoding is longer
> than
> >> > the max length 32766), all of which were skipped.  Please correct the
> >> > analyzer to not produce such terms.  The prefix of the first immense
> term
> >> > is: '[32, 60, 112, 62, 60, 98, 114, 62, 32, 32, 32, 60, 98, 114, 62,
> 56,
> >> > 48, 56, 32, 72, 97, 110, 100, 98, 111, 111, 107, 32, 111, 102]...',
> >> > original message: bytes can be at most 32766 in length; got 49960
> >> > at
> >> >
> >>
> org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:670)
> >> > at
> >> >
> >>
> org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:344)
> >> > at
> >> >
> >>
> org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:300)
> >> > at
> >> >
> >>
> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:232)
> >> > at
> >> >
> >>
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:458)
> >> > at
> >>
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1363)
> >> > at
> >> >
> >>
> org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:239)
> >> > at
> >> >
> >>
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:163)
> >> > ... 38 more
> >> > Caused by:
> >> > org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException:
> >> bytes
> >> > can be at most 32766 in length; got 49960
> >> > at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:284)
> >> > at
> >>
> org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:154)
> >> > at
> >> >
> >>
> org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:660)
> >> > ... 45 more
> >> >
> >> >
> >> > Regards,
> >> > Edwin
> >>
>

Re: String bytes can be at most 32766 characters in length?

Posted by Alexandre Rafalovitch <ar...@gmail.com>.
And that's because you have an incomplete chain. If you look at the
full example in solrconfig.xml, it shows:
     <updateRequestProcessorChain name="dedupe">
       <processor class="solr.processor.SignatureUpdateProcessorFactory">
         <bool name="enabled">true</bool>
         <str name="signatureField">id</str>
         <bool name="overwriteDupes">false</bool>
         <str name="fields">name,features,cat</str>
         <str name="signatureClass">solr.processor.Lookup3Signature</str>
       </processor>
       <processor class="solr.LogUpdateProcessorFactory" />
       <processor class="solr.RunUpdateProcessorFactory" />
     </updateRequestProcessorChain>


Notice, the last two processors. If you don't have those, nothing gets
indexed. You chain is missing them, for whatever reasons. Try adding
them back in, reloading the core and reindexing.

Regards,
   Alex.
----
Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
http://www.solr-start.com/


On 2 September 2015 at 11:29, Zheng Lin Edwin Yeo <ed...@gmail.com> wrote:
> Hi Erick,
>
> Yes, i'm trying out the De-Duplication too. But I'm facing a problem with
> that, which is the indexing stops working once I put in the following
> De-Duplication code in solrconfig.xml. The problem seems to be with this <str
> name="update.chain">dedupe</str> line.
>
>   <requestHandler name="/update" class="solr.UpdateRequestHandler">
>   <lst name="defaults">
> <str name="update.chain">dedupe</str>
>   </lst>
>   </requestHandler>
>
>
>     <updateRequestProcessorChain name="dedupe">
>   <processor class="solr.processor.SignatureUpdateProcessorFactory">
> <bool name="enabled">true</bool>
> <str name="signatureField">signature</str>
> <bool name="overwriteDupes">false</bool>
> <str name="fields">content</str>
> <str name="signatureClass">solr.processor.Lookup3Signature</str>
>   </processor>
> </updateRequestProcessorChain>
>
>
> Regards,
> Edwin
>
> On 2 September 2015 at 23:10, Erick Erickson <er...@gmail.com>
> wrote:
>
>> Yes, that is an intentional limit for the size of a single token,
>> which strings are.
>>
>> Why not use deduplication? See:
>> https://cwiki.apache.org/confluence/display/solr/De-Duplication
>>
>> You don't have to replace the existing documents, and Solr will
>> compute a hash that can be used to identify identical documents
>> and you can use_that_.
>>
>> Best
>> Erick
>>
>> On Wed, Sep 2, 2015 at 2:53 AM, Zheng Lin Edwin Yeo
>> <ed...@gmail.com> wrote:
>> > Hi,
>> >
>> > I would like to check, is the string bytes must be at most 32766
>> characters
>> > in length?
>> >
>> > I'm trying to do a copyField of my rich-text documents content to a field
>> > with fieldType=string to try out my getting distinct result for content,
>> as
>> > there are several documents with the exact same content, and we only want
>> > to list one of them during searching.
>> >
>> > However, I get the following errors in some of the documents when I tried
>> > to index them with the copyField. Some of my documents are quite large in
>> > size, and there is a possibility that it exceed 32766 characters. Is
>> there
>> > any other ways to overcome this problem?
>> >
>> >
>> > org.apache.solr.common.SolrException: Exception writing document id
>> > collection1_polymer100 to the index; possible analysis error.
>> > at
>> >
>> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:167)
>> > at
>> >
>> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
>> > at
>> >
>> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
>> > at
>> >
>> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:955)
>> > at
>> >
>> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1110)
>> > at
>> >
>> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:706)
>> > at
>> >
>> org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:104)
>> > at
>> >
>> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
>> > at
>> >
>> org.apache.solr.update.processor.LanguageIdentifierUpdateProcessor.processAdd(LanguageIdentifierUpdateProcessor.java:207)
>> > at
>> >
>> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(ExtractingDocumentLoader.java:122)
>> > at
>> >
>> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(ExtractingDocumentLoader.java:127)
>> > at
>> >
>> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:235)
>> > at
>> >
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
>> > at
>> >
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
>> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
>> > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
>> > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
>> > at
>> >
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
>> > at
>> >
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
>> > at
>> >
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>> > at
>> >
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>> > at
>> >
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>> > at
>> >
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
>> > at
>> >
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
>> > at
>> >
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>> > at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>> > at
>> >
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>> > at
>> >
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>> > at
>> >
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>> > at
>> >
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
>> > at
>> >
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
>> > at
>> >
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>> > at org.eclipse.jetty.server.Server.handle(Server.java:497)
>> > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>> > at
>> >
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>> > at
>> >
>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>> > at
>> >
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>> > at
>> >
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>> > at java.lang.Thread.run(Thread.java:745)
>> > Caused by: java.lang.IllegalArgumentException: Document contains at least
>> > one immense term in field="signature" (whose UTF8 encoding is longer than
>> > the max length 32766), all of which were skipped.  Please correct the
>> > analyzer to not produce such terms.  The prefix of the first immense term
>> > is: '[32, 60, 112, 62, 60, 98, 114, 62, 32, 32, 32, 60, 98, 114, 62, 56,
>> > 48, 56, 32, 72, 97, 110, 100, 98, 111, 111, 107, 32, 111, 102]...',
>> > original message: bytes can be at most 32766 in length; got 49960
>> > at
>> >
>> org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:670)
>> > at
>> >
>> org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:344)
>> > at
>> >
>> org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:300)
>> > at
>> >
>> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:232)
>> > at
>> >
>> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:458)
>> > at
>> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1363)
>> > at
>> >
>> org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:239)
>> > at
>> >
>> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:163)
>> > ... 38 more
>> > Caused by:
>> > org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException:
>> bytes
>> > can be at most 32766 in length; got 49960
>> > at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:284)
>> > at
>> org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:154)
>> > at
>> >
>> org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:660)
>> > ... 45 more
>> >
>> >
>> > Regards,
>> > Edwin
>>

Re: String bytes can be at most 32766 characters in length?

Posted by Zheng Lin Edwin Yeo <ed...@gmail.com>.
Hi Erick,

Yes, i'm trying out the De-Duplication too. But I'm facing a problem with
that, which is the indexing stops working once I put in the following
De-Duplication code in solrconfig.xml. The problem seems to be with this <str
name="update.chain">dedupe</str> line.

  <requestHandler name="/update" class="solr.UpdateRequestHandler">
  <lst name="defaults">
<str name="update.chain">dedupe</str>
  </lst>
  </requestHandler>


    <updateRequestProcessorChain name="dedupe">
  <processor class="solr.processor.SignatureUpdateProcessorFactory">
<bool name="enabled">true</bool>
<str name="signatureField">signature</str>
<bool name="overwriteDupes">false</bool>
<str name="fields">content</str>
<str name="signatureClass">solr.processor.Lookup3Signature</str>
  </processor>
</updateRequestProcessorChain>


Regards,
Edwin

On 2 September 2015 at 23:10, Erick Erickson <er...@gmail.com>
wrote:

> Yes, that is an intentional limit for the size of a single token,
> which strings are.
>
> Why not use deduplication? See:
> https://cwiki.apache.org/confluence/display/solr/De-Duplication
>
> You don't have to replace the existing documents, and Solr will
> compute a hash that can be used to identify identical documents
> and you can use_that_.
>
> Best
> Erick
>
> On Wed, Sep 2, 2015 at 2:53 AM, Zheng Lin Edwin Yeo
> <ed...@gmail.com> wrote:
> > Hi,
> >
> > I would like to check, is the string bytes must be at most 32766
> characters
> > in length?
> >
> > I'm trying to do a copyField of my rich-text documents content to a field
> > with fieldType=string to try out my getting distinct result for content,
> as
> > there are several documents with the exact same content, and we only want
> > to list one of them during searching.
> >
> > However, I get the following errors in some of the documents when I tried
> > to index them with the copyField. Some of my documents are quite large in
> > size, and there is a possibility that it exceed 32766 characters. Is
> there
> > any other ways to overcome this problem?
> >
> >
> > org.apache.solr.common.SolrException: Exception writing document id
> > collection1_polymer100 to the index; possible analysis error.
> > at
> >
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:167)
> > at
> >
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
> > at
> >
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
> > at
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:955)
> > at
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1110)
> > at
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:706)
> > at
> >
> org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:104)
> > at
> >
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
> > at
> >
> org.apache.solr.update.processor.LanguageIdentifierUpdateProcessor.processAdd(LanguageIdentifierUpdateProcessor.java:207)
> > at
> >
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(ExtractingDocumentLoader.java:122)
> > at
> >
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(ExtractingDocumentLoader.java:127)
> > at
> >
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:235)
> > at
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> > at
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
> > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
> > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> > at
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> > at
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> > at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> > at
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> > at org.eclipse.jetty.server.Server.handle(Server.java:497)
> > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> > at
> >
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> > at
> >
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> > at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> > at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.lang.IllegalArgumentException: Document contains at least
> > one immense term in field="signature" (whose UTF8 encoding is longer than
> > the max length 32766), all of which were skipped.  Please correct the
> > analyzer to not produce such terms.  The prefix of the first immense term
> > is: '[32, 60, 112, 62, 60, 98, 114, 62, 32, 32, 32, 60, 98, 114, 62, 56,
> > 48, 56, 32, 72, 97, 110, 100, 98, 111, 111, 107, 32, 111, 102]...',
> > original message: bytes can be at most 32766 in length; got 49960
> > at
> >
> org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:670)
> > at
> >
> org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:344)
> > at
> >
> org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:300)
> > at
> >
> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:232)
> > at
> >
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:458)
> > at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1363)
> > at
> >
> org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:239)
> > at
> >
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:163)
> > ... 38 more
> > Caused by:
> > org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException:
> bytes
> > can be at most 32766 in length; got 49960
> > at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:284)
> > at
> org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:154)
> > at
> >
> org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:660)
> > ... 45 more
> >
> >
> > Regards,
> > Edwin
>

Re: String bytes can be at most 32766 characters in length?

Posted by Erick Erickson <er...@gmail.com>.
Yes, that is an intentional limit for the size of a single token,
which strings are.

Why not use deduplication? See:
https://cwiki.apache.org/confluence/display/solr/De-Duplication

You don't have to replace the existing documents, and Solr will
compute a hash that can be used to identify identical documents
and you can use_that_.

Best
Erick

On Wed, Sep 2, 2015 at 2:53 AM, Zheng Lin Edwin Yeo
<ed...@gmail.com> wrote:
> Hi,
>
> I would like to check, is the string bytes must be at most 32766 characters
> in length?
>
> I'm trying to do a copyField of my rich-text documents content to a field
> with fieldType=string to try out my getting distinct result for content, as
> there are several documents with the exact same content, and we only want
> to list one of them during searching.
>
> However, I get the following errors in some of the documents when I tried
> to index them with the copyField. Some of my documents are quite large in
> size, and there is a possibility that it exceed 32766 characters. Is there
> any other ways to overcome this problem?
>
>
> org.apache.solr.common.SolrException: Exception writing document id
> collection1_polymer100 to the index; possible analysis error.
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:167)
> at
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:955)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1110)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:706)
> at
> org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:104)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
> at
> org.apache.solr.update.processor.LanguageIdentifierUpdateProcessor.processAdd(LanguageIdentifierUpdateProcessor.java:207)
> at
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(ExtractingDocumentLoader.java:122)
> at
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(ExtractingDocumentLoader.java:127)
> at
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:235)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:497)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException: Document contains at least
> one immense term in field="signature" (whose UTF8 encoding is longer than
> the max length 32766), all of which were skipped.  Please correct the
> analyzer to not produce such terms.  The prefix of the first immense term
> is: '[32, 60, 112, 62, 60, 98, 114, 62, 32, 32, 32, 60, 98, 114, 62, 56,
> 48, 56, 32, 72, 97, 110, 100, 98, 111, 111, 107, 32, 111, 102]...',
> original message: bytes can be at most 32766 in length; got 49960
> at
> org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:670)
> at
> org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:344)
> at
> org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:300)
> at
> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:232)
> at
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:458)
> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1363)
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:239)
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:163)
> ... 38 more
> Caused by:
> org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes
> can be at most 32766 in length; got 49960
> at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:284)
> at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:154)
> at
> org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:660)
> ... 45 more
>
>
> Regards,
> Edwin