You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Simon Willnauer (JIRA)" <ji...@apache.org> on 2011/05/19 16:15:47 UTC
[jira] [Created] (SOLR-2530) Remove Noggit CharArr from FieldType
Remove Noggit CharArr from FieldType
------------------------------------
Key: SOLR-2530
URL: https://issues.apache.org/jira/browse/SOLR-2530
Project: Solr
Issue Type: Improvement
Components: Schema and Analysis
Affects Versions: 4.0
Reporter: Simon Willnauer
Assignee: Simon Willnauer
Priority: Minor
Fix For: 4.0
FieldType#indexedToReadable(BytesRef, CharArr) uses a noggit dependency that also spreads into ByteUtils. The uses of this method area all convert to String which makes this extra reference and the dependency unnecessary. I refactored it to simply return string and removed ByteUtils entirely. The only leftover from BytesUtils is a constant, i moved that one to Lucenes UnicodeUtils. I will upload a patch in a second
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2530) Remove Noggit CharArr from FieldType
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037631#comment-13037631 ]
Simon Willnauer commented on SOLR-2530:
---------------------------------------
bq. I think its probably best to avoid this and go back to using a spare.
unintentionally I will fix later.
bq. 1. I dont think the BIGTERM should be in UnicodeUtil, its completely unrelated to unicode.
Any good place for it?
bq. I dont like the way UTF8toUTF16 sets the output, i think it should be consistent with UTF16toUTF8 (always make the result output start at 0). The problem can be seen by the size allocation:
Agreed. I will work on a new patch tomorrow.
> Remove Noggit CharArr from FieldType
> ------------------------------------
>
> Key: SOLR-2530
> URL: https://issues.apache.org/jira/browse/SOLR-2530
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Priority: Minor
> Labels: api-change
> Fix For: 4.0
>
> Attachments: SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch
>
>
> FieldType#indexedToReadable(BytesRef, CharArr) uses a noggit dependency that also spreads into ByteUtils. The uses of this method area all convert to String which makes this extra reference and the dependency unnecessary. I refactored it to simply return string and removed ByteUtils entirely. The only leftover from BytesUtils is a constant, i moved that one to Lucenes UnicodeUtils. I will upload a patch in a second
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2530) Remove Noggit CharArr from FieldType
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036219#comment-13036219 ]
Yonik Seeley commented on SOLR-2530:
------------------------------------
There are some efficiency losses here:
- A reusable CharArr allows one to avoid extra object creation. See TermsComponent which can update a CharArr and then compare it against a pattern w/o having to create a String object.
- We should not replace the previous toString with BytesRef.utf8String... it's much slower, esp for small strings like will be common here.
So rather than just removing ByteUtils.UTF8toUTF16, how about moving it to BytesRef and use it in BytesRTef.utf8String?
> Remove Noggit CharArr from FieldType
> ------------------------------------
>
> Key: SOLR-2530
> URL: https://issues.apache.org/jira/browse/SOLR-2530
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Priority: Minor
> Labels: api-change
> Fix For: 4.0
>
> Attachments: SOLR-2530.patch
>
>
> FieldType#indexedToReadable(BytesRef, CharArr) uses a noggit dependency that also spreads into ByteUtils. The uses of this method area all convert to String which makes this extra reference and the dependency unnecessary. I refactored it to simply return string and removed ByteUtils entirely. The only leftover from BytesUtils is a constant, i moved that one to Lucenes UnicodeUtils. I will upload a patch in a second
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2530) Remove Noggit CharArr from FieldType
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated SOLR-2530:
----------------------------------
Attachment: SOLR-2530.patch
updated patch including CharsRef.java...
It seems like we can get rid of UTF16Result entirely. I will wait for comments first.
> Remove Noggit CharArr from FieldType
> ------------------------------------
>
> Key: SOLR-2530
> URL: https://issues.apache.org/jira/browse/SOLR-2530
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Priority: Minor
> Labels: api-change
> Fix For: 4.0
>
> Attachments: SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch
>
>
> FieldType#indexedToReadable(BytesRef, CharArr) uses a noggit dependency that also spreads into ByteUtils. The uses of this method area all convert to String which makes this extra reference and the dependency unnecessary. I refactored it to simply return string and removed ByteUtils entirely. The only leftover from BytesUtils is a constant, i moved that one to Lucenes UnicodeUtils. I will upload a patch in a second
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2530) Remove Noggit CharArr from FieldType
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038993#comment-13038993 ]
Simon Willnauer commented on SOLR-2530:
---------------------------------------
bq. I am backporting this to 3.x now
not needed since the method doesn't use CharArr in 3.x
> Remove Noggit CharArr from FieldType
> ------------------------------------
>
> Key: SOLR-2530
> URL: https://issues.apache.org/jira/browse/SOLR-2530
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Priority: Minor
> Labels: api-change
> Fix For: 4.0
>
> Attachments: SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch
>
>
> FieldType#indexedToReadable(BytesRef, CharArr) uses a noggit dependency that also spreads into ByteUtils. The uses of this method area all convert to String which makes this extra reference and the dependency unnecessary. I refactored it to simply return string and removed ByteUtils entirely. The only leftover from BytesUtils is a constant, i moved that one to Lucenes UnicodeUtils. I will upload a patch in a second
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2530) Remove Noggit CharArr from FieldType
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038784#comment-13038784 ]
Simon Willnauer commented on SOLR-2530:
---------------------------------------
yeah I wonder if yonik could look at this nocommit?
> Remove Noggit CharArr from FieldType
> ------------------------------------
>
> Key: SOLR-2530
> URL: https://issues.apache.org/jira/browse/SOLR-2530
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Priority: Minor
> Labels: api-change
> Fix For: 4.0
>
> Attachments: SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch
>
>
> FieldType#indexedToReadable(BytesRef, CharArr) uses a noggit dependency that also spreads into ByteUtils. The uses of this method area all convert to String which makes this extra reference and the dependency unnecessary. I refactored it to simply return string and removed ByteUtils entirely. The only leftover from BytesUtils is a constant, i moved that one to Lucenes UnicodeUtils. I will upload a patch in a second
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2530) Remove Noggit CharArr from FieldType
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038875#comment-13038875 ]
Simon Willnauer commented on SOLR-2530:
---------------------------------------
I am backporting this to 3.x now
> Remove Noggit CharArr from FieldType
> ------------------------------------
>
> Key: SOLR-2530
> URL: https://issues.apache.org/jira/browse/SOLR-2530
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Priority: Minor
> Labels: api-change
> Fix For: 4.0
>
> Attachments: SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch
>
>
> FieldType#indexedToReadable(BytesRef, CharArr) uses a noggit dependency that also spreads into ByteUtils. The uses of this method area all convert to String which makes this extra reference and the dependency unnecessary. I refactored it to simply return string and removed ByteUtils entirely. The only leftover from BytesUtils is a constant, i moved that one to Lucenes UnicodeUtils. I will upload a patch in a second
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2530) Remove Noggit CharArr from FieldType
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated SOLR-2530:
----------------------------------
Attachment: SOLR-2530.patch
updated patch to new suggest module and replaced the nocommit in SimpleFacets with a TODO. I will commit this soon if nobody objects.
> Remove Noggit CharArr from FieldType
> ------------------------------------
>
> Key: SOLR-2530
> URL: https://issues.apache.org/jira/browse/SOLR-2530
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Priority: Minor
> Labels: api-change
> Fix For: 4.0
>
> Attachments: SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch
>
>
> FieldType#indexedToReadable(BytesRef, CharArr) uses a noggit dependency that also spreads into ByteUtils. The uses of this method area all convert to String which makes this extra reference and the dependency unnecessary. I refactored it to simply return string and removed ByteUtils entirely. The only leftover from BytesUtils is a constant, i moved that one to Lucenes UnicodeUtils. I will upload a patch in a second
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2530) Remove Noggit CharArr from FieldType
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038777#comment-13038777 ]
Robert Muir commented on SOLR-2530:
-----------------------------------
bq. I think this is ready though, all tests pass.
Looks good to me, there is still the one nocommit about interned fields but I don't know the answer to that one either! The rest of the patch looks great, good to finally get rid of UTF16Result/incremental api and to have a CharsRef.
> Remove Noggit CharArr from FieldType
> ------------------------------------
>
> Key: SOLR-2530
> URL: https://issues.apache.org/jira/browse/SOLR-2530
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Priority: Minor
> Labels: api-change
> Fix For: 4.0
>
> Attachments: SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch
>
>
> FieldType#indexedToReadable(BytesRef, CharArr) uses a noggit dependency that also spreads into ByteUtils. The uses of this method area all convert to String which makes this extra reference and the dependency unnecessary. I refactored it to simply return string and removed ByteUtils entirely. The only leftover from BytesUtils is a constant, i moved that one to Lucenes UnicodeUtils. I will upload a patch in a second
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2530) Remove Noggit CharArr from FieldType
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated SOLR-2530:
------------------------------
Attachment: SOLR-2530.patch
{quote}
It seems like we can get rid of UTF16Result entirely. I will wait for comments first.
{quote}
done!
upon review there are two things i dont like:
1. I dont think the BIGTERM should be in UnicodeUtil, its completely unrelated to unicode.
2. I dont like the way UTF8toUTF16 sets the output, i think it should be consistent with UTF16toUTF8 (always make the result output start at 0). The problem can be seen by the size allocation:
{noformat}
int out_offset = chars.offset;
final char[] out = chars.chars = ArrayUtil.grow(chars.chars, out_offset + length);
{noformat}
> Remove Noggit CharArr from FieldType
> ------------------------------------
>
> Key: SOLR-2530
> URL: https://issues.apache.org/jira/browse/SOLR-2530
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Priority: Minor
> Labels: api-change
> Fix For: 4.0
>
> Attachments: SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch
>
>
> FieldType#indexedToReadable(BytesRef, CharArr) uses a noggit dependency that also spreads into ByteUtils. The uses of this method area all convert to String which makes this extra reference and the dependency unnecessary. I refactored it to simply return string and removed ByteUtils entirely. The only leftover from BytesUtils is a constant, i moved that one to Lucenes UnicodeUtils. I will upload a patch in a second
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2530) Remove Noggit CharArr from FieldType
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated SOLR-2530:
----------------------------------
Attachment: SOLR-2530.patch
here is a patch
> Remove Noggit CharArr from FieldType
> ------------------------------------
>
> Key: SOLR-2530
> URL: https://issues.apache.org/jira/browse/SOLR-2530
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Priority: Minor
> Labels: api-change
> Fix For: 4.0
>
> Attachments: SOLR-2530.patch
>
>
> FieldType#indexedToReadable(BytesRef, CharArr) uses a noggit dependency that also spreads into ByteUtils. The uses of this method area all convert to String which makes this extra reference and the dependency unnecessary. I refactored it to simply return string and removed ByteUtils entirely. The only leftover from BytesUtils is a constant, i moved that one to Lucenes UnicodeUtils. I will upload a patch in a second
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2530) Remove Noggit CharArr from FieldType
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036851#comment-13036851 ]
Simon Willnauer commented on SOLR-2530:
---------------------------------------
seem like CharsRef is not in the patch... I will upload a new one ASAP
> Remove Noggit CharArr from FieldType
> ------------------------------------
>
> Key: SOLR-2530
> URL: https://issues.apache.org/jira/browse/SOLR-2530
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Priority: Minor
> Labels: api-change
> Fix For: 4.0
>
> Attachments: SOLR-2530.patch, SOLR-2530.patch
>
>
> FieldType#indexedToReadable(BytesRef, CharArr) uses a noggit dependency that also spreads into ByteUtils. The uses of this method area all convert to String which makes this extra reference and the dependency unnecessary. I refactored it to simply return string and removed ByteUtils entirely. The only leftover from BytesUtils is a constant, i moved that one to Lucenes UnicodeUtils. I will upload a patch in a second
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (SOLR-2530) Remove Noggit CharArr
from FieldType
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036737#comment-13036737 ]
Simon Willnauer edited comment on SOLR-2530 at 5/20/11 2:34 PM:
----------------------------------------------------------------
Here is a new patch. Thanks yonik I didn't realize that the utf8String() method is slow / slower. I moved the UTF8ToUTF16 from ByteUtils to UnicodeUtils (plus test) and use it in BytesRef now. I also switched all FT#indexedToReadable to use a CharsRef (o.a.l.util).
I fixed some minor stuff along the way (generics) and added one nocommit to SimpleFacets since we are comparing Strings with == there. If thats ok there we should add a comment!
was (Author: simonw):
Here is a new patch. Thanks yonik I didn't realize that the utf8String() method is slow / slower. I moved the UTF8ToUTF16 from ByteUtils to UnicodeUtils (plus test) and use it in BytesRef now. I also switched all FT#indexedToReadable to use a CharsRef (o.a.l.util).
I fixed some minor stuff along the way (generics) and added one nocommit to SimpleFacets since we are comparing Strings with != there. If thats ok there we should add a comment!
> Remove Noggit CharArr from FieldType
> ------------------------------------
>
> Key: SOLR-2530
> URL: https://issues.apache.org/jira/browse/SOLR-2530
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Priority: Minor
> Labels: api-change
> Fix For: 4.0
>
> Attachments: SOLR-2530.patch, SOLR-2530.patch
>
>
> FieldType#indexedToReadable(BytesRef, CharArr) uses a noggit dependency that also spreads into ByteUtils. The uses of this method area all convert to String which makes this extra reference and the dependency unnecessary. I refactored it to simply return string and removed ByteUtils entirely. The only leftover from BytesUtils is a constant, i moved that one to Lucenes UnicodeUtils. I will upload a patch in a second
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2530) Remove Noggit CharArr from FieldType
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038817#comment-13038817 ]
Mark Miller commented on SOLR-2530:
-----------------------------------
bq. nocommit in SimpleFacets with a TODO.
I've looked into this in the past - its fine. Perhaps it's safer to change it (equals or enum or whatever), but all current usage uses the same constant.
> Remove Noggit CharArr from FieldType
> ------------------------------------
>
> Key: SOLR-2530
> URL: https://issues.apache.org/jira/browse/SOLR-2530
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Priority: Minor
> Labels: api-change
> Fix For: 4.0
>
> Attachments: SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch
>
>
> FieldType#indexedToReadable(BytesRef, CharArr) uses a noggit dependency that also spreads into ByteUtils. The uses of this method area all convert to String which makes this extra reference and the dependency unnecessary. I refactored it to simply return string and removed ByteUtils entirely. The only leftover from BytesUtils is a constant, i moved that one to Lucenes UnicodeUtils. I will upload a patch in a second
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2530) Remove Noggit CharArr from FieldType
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated SOLR-2530:
----------------------------------
Attachment: SOLR-2530.patch
Here is a new patch. Thanks yonik I didn't realize that the utf8String() method is slow / slower. I moved the UTF8ToUTF16 from ByteUtils to UnicodeUtils (plus test) and use it in BytesRef now. I also switched all FT#indexedToReadable to use a CharsRef (o.a.l.util).
I fixed some minor stuff along the way (generics) and added one nocommit to SimpleFacets since we are comparing Strings with != there. If thats ok there we should add a comment!
> Remove Noggit CharArr from FieldType
> ------------------------------------
>
> Key: SOLR-2530
> URL: https://issues.apache.org/jira/browse/SOLR-2530
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Priority: Minor
> Labels: api-change
> Fix For: 4.0
>
> Attachments: SOLR-2530.patch, SOLR-2530.patch
>
>
> FieldType#indexedToReadable(BytesRef, CharArr) uses a noggit dependency that also spreads into ByteUtils. The uses of this method area all convert to String which makes this extra reference and the dependency unnecessary. I refactored it to simply return string and removed ByteUtils entirely. The only leftover from BytesUtils is a constant, i moved that one to Lucenes UnicodeUtils. I will upload a patch in a second
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2530) Remove Noggit CharArr from FieldType
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036224#comment-13036224 ]
Robert Muir commented on SOLR-2530:
-----------------------------------
My recommendation: add CharsRef. We already have BytesRef and IntsRef...
> Remove Noggit CharArr from FieldType
> ------------------------------------
>
> Key: SOLR-2530
> URL: https://issues.apache.org/jira/browse/SOLR-2530
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Priority: Minor
> Labels: api-change
> Fix For: 4.0
>
> Attachments: SOLR-2530.patch
>
>
> FieldType#indexedToReadable(BytesRef, CharArr) uses a noggit dependency that also spreads into ByteUtils. The uses of this method area all convert to String which makes this extra reference and the dependency unnecessary. I refactored it to simply return string and removed ByteUtils entirely. The only leftover from BytesUtils is a constant, i moved that one to Lucenes UnicodeUtils. I will upload a patch in a second
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (SOLR-2530) Remove Noggit CharArr from FieldType
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer resolved SOLR-2530.
-----------------------------------
Resolution: Fixed
Committed in revision 1127326.
> Remove Noggit CharArr from FieldType
> ------------------------------------
>
> Key: SOLR-2530
> URL: https://issues.apache.org/jira/browse/SOLR-2530
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Priority: Minor
> Labels: api-change
> Fix For: 4.0
>
> Attachments: SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch
>
>
> FieldType#indexedToReadable(BytesRef, CharArr) uses a noggit dependency that also spreads into ByteUtils. The uses of this method area all convert to String which makes this extra reference and the dependency unnecessary. I refactored it to simply return string and removed ByteUtils entirely. The only leftover from BytesUtils is a constant, i moved that one to Lucenes UnicodeUtils. I will upload a patch in a second
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2530) Remove Noggit CharArr from FieldType
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated SOLR-2530:
----------------------------------
Attachment: SOLR-2530.patch
next iteration. The UTF8 to UTF16 conversion starts the output at 0 now. I fixed the places where I missed to reuse a spare instance and replaced some occurrences of BytesRef#utf8ToString with BytesRef#utf8ToChars where I can reuse a CharsRef instance.
I also added a comment / TODO to UnicodeUtils#BIG_TERM to move it elsewhere once we have a good place for it or don't need it anymore.
I think this is ready though, all tests pass.
> Remove Noggit CharArr from FieldType
> ------------------------------------
>
> Key: SOLR-2530
> URL: https://issues.apache.org/jira/browse/SOLR-2530
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Priority: Minor
> Labels: api-change
> Fix For: 4.0
>
> Attachments: SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch
>
>
> FieldType#indexedToReadable(BytesRef, CharArr) uses a noggit dependency that also spreads into ByteUtils. The uses of this method area all convert to String which makes this extra reference and the dependency unnecessary. I refactored it to simply return string and removed ByteUtils entirely. The only leftover from BytesUtils is a constant, i moved that one to Lucenes UnicodeUtils. I will upload a patch in a second
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2530) Remove Noggit CharArr from FieldType
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036240#comment-13036240 ]
Yonik Seeley commented on SOLR-2530:
------------------------------------
Minor nit: renaming bigTerm to UnicodeUtil.BIG_UTF8_TERM is a bit misleading since it's not UTF8 at all.
> Remove Noggit CharArr from FieldType
> ------------------------------------
>
> Key: SOLR-2530
> URL: https://issues.apache.org/jira/browse/SOLR-2530
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Priority: Minor
> Labels: api-change
> Fix For: 4.0
>
> Attachments: SOLR-2530.patch
>
>
> FieldType#indexedToReadable(BytesRef, CharArr) uses a noggit dependency that also spreads into ByteUtils. The uses of this method area all convert to String which makes this extra reference and the dependency unnecessary. I refactored it to simply return string and removed ByteUtils entirely. The only leftover from BytesUtils is a constant, i moved that one to Lucenes UnicodeUtils. I will upload a patch in a second
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2530) Remove Noggit CharArr from FieldType
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037607#comment-13037607 ]
Robert Muir commented on SOLR-2530:
-----------------------------------
Also, it seems in a few places in Solr, there was a spare CharArr being reused for an entire loop, but not anymore (there is now a new CharsRef being made each time).
I think its probably best to avoid this and go back to using a spare.
> Remove Noggit CharArr from FieldType
> ------------------------------------
>
> Key: SOLR-2530
> URL: https://issues.apache.org/jira/browse/SOLR-2530
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Priority: Minor
> Labels: api-change
> Fix For: 4.0
>
> Attachments: SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch
>
>
> FieldType#indexedToReadable(BytesRef, CharArr) uses a noggit dependency that also spreads into ByteUtils. The uses of this method area all convert to String which makes this extra reference and the dependency unnecessary. I refactored it to simply return string and removed ByteUtils entirely. The only leftover from BytesUtils is a constant, i moved that one to Lucenes UnicodeUtils. I will upload a patch in a second
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2530) Remove Noggit CharArr from FieldType
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038852#comment-13038852 ]
Simon Willnauer commented on SOLR-2530:
---------------------------------------
bq. I've looked into this in the past - its fine. Perhaps it's safer to change it (equals or enum or whatever), but all current usage uses the same constant.
alright, I changed the todo to point out that we should cut over to an enum or something similar. This is totally unrelated but wanted to make sure it doesn't get lost.
I will commit in a minute
> Remove Noggit CharArr from FieldType
> ------------------------------------
>
> Key: SOLR-2530
> URL: https://issues.apache.org/jira/browse/SOLR-2530
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Priority: Minor
> Labels: api-change
> Fix For: 4.0
>
> Attachments: SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch, SOLR-2530.patch
>
>
> FieldType#indexedToReadable(BytesRef, CharArr) uses a noggit dependency that also spreads into ByteUtils. The uses of this method area all convert to String which makes this extra reference and the dependency unnecessary. I refactored it to simply return string and removed ByteUtils entirely. The only leftover from BytesUtils is a constant, i moved that one to Lucenes UnicodeUtils. I will upload a patch in a second
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org