Tokenizer violates TokenStream.reset() contract?

[Dan M <mr...@gmail.com>]

Hello,

It seems that Tokenizer may violate the contract put forth by the
TokenStream.reset function. Specifically, TokenStream.reset states:

"*Resets this stream to a clean state. Stateful implementations must
implement this method so that they can be reused, just as if they had been
created fresh.*"

Tokenizer does not do this. Tokenizers can only be reset one time. On
subsequent resets IllegalStateReader is swapped in as the Reader, and
incrementToken throws an exception.

The complication arises because Tokenizer takes a Reader and LUCENE-2387
was filed to intentionally unset the input (Reader) to prevent memory leak.
However, unsetting it means we can never read from the Tokenizer a 2nd time
(unless you set the Reader again) and thus it violates the contract.

Should there be a way to reuse Tokenizers?

Thanks,
Dan

Re: 8.0.0-SNAPSHOT snapshot repo poms broken?

[Tim Allison <ta...@apache.org>]

User error...<face_palm/>...please ignore.

On Thu, Jan 17, 2019 at 4:36 PM Tim Allison <ta...@apache.org> wrote:
>
> All,
>   I recently tried to upgrade a project that relies on the snapshot
> repos[1], but maven wasn't able to pull lucene-highlighter,
> lucene-test-framework, lucene-memory, among a few others.  However,
> maven was able to pull lucene-core and most other artifacts for
> 8.0.0-SNAPSHOT.  I manually checked that the jars and poms for the
> artifacts that maven wasn't able to pull were in fact there.
>   Is this user error or something wrong with the poms or something else?
>
>    Thank you.
>
>                  Best,
>
>                        Tim
>
>
> [1]
> <repositories>
>     <repository>
>         <id>apache-snapshot</id>
>         <url>http://repository.apache.org/snapshots/</url>
>     </repository>
> </repositories>

8.0.0-SNAPSHOT snapshot repo poms broken?

[Tim Allison <ta...@apache.org>]

All,
  I recently tried to upgrade a project that relies on the snapshot
repos[1], but maven wasn't able to pull lucene-highlighter,
lucene-test-framework, lucene-memory, among a few others.  However,
maven was able to pull lucene-core and most other artifacts for
8.0.0-SNAPSHOT.  I manually checked that the jars and poms for the
artifacts that maven wasn't able to pull were in fact there.
  Is this user error or something wrong with the poms or something else?

   Thank you.

                 Best,

                       Tim


[1]
<repositories>
    <repository>
        <id>apache-snapshot</id>
        <url>http://repository.apache.org/snapshots/</url>
    </repository>
</repositories>