You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucenenet.apache.org by ni...@apache.org on 2019/10/26 18:51:55 UTC
[lucenenet] branch master updated (ecf6212 -> 5c43454)
This is an automated email from the ASF dual-hosted git repository.
nightowl888 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/lucenenet.git.
from ecf6212 Lucene.Net.TestFramework: Moved StackTraceHelper from Util to Support folder
new 2e2710e Lucene.Net.TestFramework.Analysis.ICheckClearAttributesAttribute: Converted property AndResetClearCalled back to a method GetAndResetClearCalled(), as it has a side-effect, which makes it a poor candidate for a property.
new 2d29c1e Lucene.Net.TestFramework.Codecs.Lucene3x.PreFlexRWNormsConsumer: Changed exceptions from InvalidOperationException to AssertionExeption (as they were in Lucene)
new fd664cd Lucene.Net.Store.NativeFSLockFactory, Lucene.Net.Support.IO.FileSupport: Changed implementations to provoke lock/sharing/alredy exists exceptions during initialization so the values of the current OS can be used for comparison of HResult values at runtime (closes LUCENENET-618)
new eecbcda Lucene.Net.TestFramework: Refactored Assert classes to utilize FileSupport.IsFileAlreadyExistsException(). Deleted Support/TestFramework/AssertCommon.cs.
new b2352ea build/TestReferences.Common.targets: Removed TeamCity references and System.Runtime.InteropServices.RuntimeInformation
new 55503dd Lucene.Net.Tests._J-S: Added System.Runtime.InteropServices.RuntimeInformation for net451 target only
new 9bd6520 SWEEP: Changed test target framework from netcoreapp1.0 to netcoreapp1.1
new d8395be run-tests-on-os.yml: Added name to jobs so the ones that are running can be displayed in the logs
new 4cc10c9 build.ps1: Fixed where expression (dotnet vstest uses --TestCaseFilter, not --filter, and must be separated with a colon)
new f93597b run-tests-on-os.yml, azure-pipelines.yml: Added test expression to filter tests, filtered netcoreapp1.1 tests so long running tests are excluded. Changed VM image to vs2017-win2016 because .NET Core support has been dropped from the windows-2019 image.
new 9a23529 azure-pipelines.yml: Increased maximumParallelJobs on net451 from 7 to 8
new aee8690 Lucene.Net.Util.AttributeSource: Added TODO
new 5c43454 Ported Lucene.Net.Analysis.OpenNLP + tests
The 13 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
Lucene.Net.sln | 12 +
README.md | 5 +-
TestTargetFramework.props | 4 +-
azure-pipelines.yml | 13 +-
build/Dependencies.props | 3 +-
build/TestReferences.Common.targets | 4 -
.../publish-test-results-for-target-frameworks.yml | 2 +-
.../publish-test-results-for-test-projects.yml | 10 +
build/azure-templates/run-tests-on-os.yml | 14 +-
build/build.ps1 | 20 +-
.../Lucene.Net.Analysis.OpenNLP.csproj} | 41 +--
.../OpenNLPChunkerFilter.cs | 115 +++++++
.../OpenNLPChunkerFilterFactory.cs | 90 +++++
.../OpenNLPLemmatizerFilter.cs | 129 +++++++
.../OpenNLPLemmatizerFilterFactory.cs | 95 ++++++
.../OpenNLPPOSFilter.cs | 102 ++++++
.../OpenNLPPOSFilterFactory.cs | 79 +++++
.../OpenNLPSentenceBreakIterator.cs | 277 +++++++++++++++
.../OpenNLPTokenizer.cs | 112 +++++++
.../OpenNLPTokenizerFactory.cs | 85 +++++
.../Tools/NLPChunkerOp.cs} | 33 +-
.../Tools/NLPLemmatizerOp.cs | 93 ++++++
.../Tools/NLPNERTaggerOp.cs | 61 ++++
.../Tools/NLPPOSTaggerOp.cs} | 29 +-
.../Tools/NLPSentenceDetectorOp.cs} | 46 ++-
.../Tools/NLPTokenizerOp.cs} | 39 ++-
.../Tools/OpenNLPOpsFactory.cs | 225 +++++++++++++
.../Support/TestFramework/Assert.cs | 3 +-
.../Support/TestFramework/Assert.cs | 3 +-
.../Support/TestFramework/Assert.cs | 3 +-
.../Analysis/BaseTokenStreamTestCase.cs | 99 +++++-
.../Codecs/Lucene3x/PreFlexRWNormsConsumer.cs | 7 +-
.../Support/TestFramework/AssertCommon.cs | 41 ---
.../Analysis/Core/TestStopFilter.cs | 2 +-
.../Lucene.Net.Tests.Analysis.Common.csproj | 4 +-
.../Lucene.Net.Tests.Analysis.Kuromoji.csproj | 6 +-
.../Lucene.Net.Tests.Analysis.OpenNLP.csproj} | 55 +--
.../TestOpenNLPChunkerFilterFactory.cs | 122 +++++++
.../TestOpenNLPLemmatizerFilterFactory.cs | 372 +++++++++++++++++++++
.../TestOpenNLPPOSFilterFactory.cs | 160 +++++++++
.../TestOpenNLPSentenceBreakIterator.cs | 252 ++++++++++++++
.../TestOpenNLPTokenizerFactory.cs | 140 ++++++++
.../en-test-chunker.bin | Bin 0 -> 89914 bytes
.../en-test-lemmas.dict | 12 +
.../en-test-lemmatizer.bin | Bin 0 -> 7370 bytes
.../en-test-ner.bin | Bin 0 -> 2049 bytes
.../en-test-pos-maxent.bin | Bin 0 -> 18494 bytes
.../en-test-sent.bin | Bin 0 -> 1051 bytes
.../en-test-tokenizer.bin | Bin 0 -> 15100 bytes
.../Lucene.Net.Tests.Analysis.Phonetic.csproj | 4 +-
.../TestOpenNLPChunkerFilterFactory.cs | 86 +++++
.../Lucene.Net.Tests.Analysis.SmartCn.csproj | 6 +-
.../Lucene.Net.Tests.Analysis.Stempel.csproj | 4 +-
.../Lucene.Net.Tests.Benchmark.csproj | 6 +-
.../Lucene.Net.Tests.Classification.csproj | 4 +-
.../Lucene.Net.Tests.Codecs.csproj | 4 +-
.../Lucene.Net.Tests.Demo.csproj | 4 +-
.../Lucene.Net.Tests.Expressions.csproj | 4 +-
.../Lucene.Net.Tests.Facet.csproj | 4 +-
.../Lucene.Net.Tests.Grouping.csproj | 4 +-
.../Lucene.Net.Tests.Highlighter.csproj | 6 +-
.../Lucene.Net.Tests.Join.csproj | 4 +-
.../Lucene.Net.Tests.Memory.csproj | 4 +-
.../Lucene.Net.Tests.Misc.csproj | 4 +-
.../Lucene.Net.Tests.Queries.csproj | 4 +-
.../Lucene.Net.Tests.QueryParser.csproj | 4 +-
.../Lucene.Net.Tests.Replicator.csproj | 4 +-
.../Lucene.Net.Tests.Sandbox.csproj | 4 +-
.../Lucene.Net.Tests.Spatial.csproj | 6 +-
.../Lucene.Net.Tests.Suggest.csproj | 4 +-
.../Lucene.Net.Tests.TestFramework.NUnit.csproj | 4 +-
.../Lucene.Net.Tests.TestFramework.xUnit.csproj | 4 +-
.../Lucene.Net.Tests._A-D.csproj | 4 +-
.../Lucene.Net.Tests._E-I.csproj | 4 +-
.../Lucene.Net.Tests._J-S.csproj | 14 +-
.../Lucene.Net.Tests._T-U.csproj | 4 +-
.../Lucene.Net.Tests._U-Z.csproj | 4 +-
src/Lucene.Net.Tests/Support/IO/TestFileSupport.cs | 3 +-
.../Support/TestLurchTableThreading.cs | 4 +-
src/Lucene.Net/Store/NativeFSLockFactory.cs | 345 ++++++++++++++-----
src/Lucene.Net/Support/IO/FileSupport.cs | 81 ++++-
src/Lucene.Net/Util/AttributeSource.cs | 2 +-
.../Lucene.Net.Tests.ICU.csproj | 6 +-
83 files changed, 3302 insertions(+), 370 deletions(-)
copy src/{Lucene.Net.Analysis.Phonetic/Lucene.Net.Analysis.Phonetic.csproj => Lucene.Net.Analysis.OpenNLP/Lucene.Net.Analysis.OpenNLP.csproj} (57%)
create mode 100644 src/Lucene.Net.Analysis.OpenNLP/OpenNLPChunkerFilter.cs
create mode 100644 src/Lucene.Net.Analysis.OpenNLP/OpenNLPChunkerFilterFactory.cs
create mode 100644 src/Lucene.Net.Analysis.OpenNLP/OpenNLPLemmatizerFilter.cs
create mode 100644 src/Lucene.Net.Analysis.OpenNLP/OpenNLPLemmatizerFilterFactory.cs
create mode 100644 src/Lucene.Net.Analysis.OpenNLP/OpenNLPPOSFilter.cs
create mode 100644 src/Lucene.Net.Analysis.OpenNLP/OpenNLPPOSFilterFactory.cs
create mode 100644 src/Lucene.Net.Analysis.OpenNLP/OpenNLPSentenceBreakIterator.cs
create mode 100644 src/Lucene.Net.Analysis.OpenNLP/OpenNLPTokenizer.cs
create mode 100644 src/Lucene.Net.Analysis.OpenNLP/OpenNLPTokenizerFactory.cs
copy src/{Lucene.Net.Benchmark/ByTask/Tasks/CommitTaxonomyIndexTask.cs => Lucene.Net.Analysis.OpenNLP/Tools/NLPChunkerOp.cs} (60%)
create mode 100644 src/Lucene.Net.Analysis.OpenNLP/Tools/NLPLemmatizerOp.cs
create mode 100644 src/Lucene.Net.Analysis.OpenNLP/Tools/NLPNERTaggerOp.cs
copy src/{Lucene.Net.Memory/MemoryIndexNormDocValues.cs => Lucene.Net.Analysis.OpenNLP/Tools/NLPPOSTaggerOp.cs} (66%)
copy src/{Lucene.Net.Benchmark/ByTask/Tasks/CommitIndexTask.cs => Lucene.Net.Analysis.OpenNLP/Tools/NLPSentenceDetectorOp.cs} (53%)
copy src/{Lucene.Net.Benchmark/ByTask/Tasks/RollbackIndexTask.cs => Lucene.Net.Analysis.OpenNLP/Tools/NLPTokenizerOp.cs} (56%)
create mode 100644 src/Lucene.Net.Analysis.OpenNLP/Tools/OpenNLPOpsFactory.cs
delete mode 100644 src/Lucene.Net.TestFramework/Support/TestFramework/AssertCommon.cs
copy src/{Lucene.Net.Tests.Analysis.Kuromoji/Lucene.Net.Tests.Analysis.Kuromoji.csproj => Lucene.Net.Tests.Analysis.OpenNLP/Lucene.Net.Tests.Analysis.OpenNLP.csproj} (55%)
create mode 100644 src/Lucene.Net.Tests.Analysis.OpenNLP/TestOpenNLPChunkerFilterFactory.cs
create mode 100644 src/Lucene.Net.Tests.Analysis.OpenNLP/TestOpenNLPLemmatizerFilterFactory.cs
create mode 100644 src/Lucene.Net.Tests.Analysis.OpenNLP/TestOpenNLPPOSFilterFactory.cs
create mode 100644 src/Lucene.Net.Tests.Analysis.OpenNLP/TestOpenNLPSentenceBreakIterator.cs
create mode 100644 src/Lucene.Net.Tests.Analysis.OpenNLP/TestOpenNLPTokenizerFactory.cs
create mode 100644 src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-chunker.bin
create mode 100644 src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-lemmas.dict
create mode 100644 src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-lemmatizer.bin
create mode 100644 src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-ner.bin
create mode 100644 src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-pos-maxent.bin
create mode 100644 src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-sent.bin
create mode 100644 src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-tokenizer.bin
create mode 100644 src/Lucene.Net.Tests.Analysis.Phonetic/TestOpenNLPChunkerFilterFactory.cs
[lucenenet] 06/13: Lucene.Net.Tests._J-S: Added
System.Runtime.InteropServices.RuntimeInformation for net451 target only
Posted by ni...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
nightowl888 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/lucenenet.git
commit 55503dd20ff3bd03ac5b6227ecc11cc24bac8d92
Author: Shad Storhaug <sh...@shadstorhaug.com>
AuthorDate: Sat Oct 26 00:56:27 2019 +0700
Lucene.Net.Tests._J-S: Added System.Runtime.InteropServices.RuntimeInformation for net451 target only
---
build/Dependencies.props | 1 +
src/Lucene.Net.Tests._J-S/Lucene.Net.Tests._J-S.csproj | 8 +++++++-
src/Lucene.Net.Tests/Support/IO/TestFileSupport.cs | 3 ++-
3 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/build/Dependencies.props b/build/Dependencies.props
index 625e6a7..1cb92e6 100644
--- a/build/Dependencies.props
+++ b/build/Dependencies.props
@@ -69,6 +69,7 @@
<SystemReflectionEmitPackageVersion>4.3.0</SystemReflectionEmitPackageVersion>
<SystemReflectionEmitILGenerationPackageVersion>4.3.0</SystemReflectionEmitILGenerationPackageVersion>
<SystemReflectionTypeExtensionsPackageVersion>4.3.0</SystemReflectionTypeExtensionsPackageVersion>
+ <SystemRuntimeInteropServicesRuntimeInformationPackageVersion>4.3.0</SystemRuntimeInteropServicesRuntimeInformationPackageVersion>
<SystemTextEncodingCodePagesPackageVersion>4.3.0</SystemTextEncodingCodePagesPackageVersion>
<SystemThreadingThreadPackageVersion>4.3.0</SystemThreadingThreadPackageVersion>
<SystemThreadingThreadPoolPackageVersion>4.3.0</SystemThreadingThreadPoolPackageVersion>
diff --git a/src/Lucene.Net.Tests._J-S/Lucene.Net.Tests._J-S.csproj b/src/Lucene.Net.Tests._J-S/Lucene.Net.Tests._J-S.csproj
index 449e134..4c7236c 100644
--- a/src/Lucene.Net.Tests._J-S/Lucene.Net.Tests._J-S.csproj
+++ b/src/Lucene.Net.Tests._J-S/Lucene.Net.Tests._J-S.csproj
@@ -50,7 +50,7 @@
<ProjectReference Include="..\Lucene.Net.QueryParser\Lucene.Net.QueryParser.csproj" />
<ProjectReference Include="..\Lucene.Net.TestFramework.NUnit\Lucene.Net.TestFramework.NUnit.csproj" />
</ItemGroup>
-
+
<Import Project="$(SolutionDir)build/TestReferences.Common.targets" />
<PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp2.1' ">
@@ -59,6 +59,8 @@
</PropertyGroup>
<PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD;NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
@@ -71,6 +73,10 @@
<ItemGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
<Compile Remove="..\Lucene.Net.Tests\Support\TestCase.cs" />
</ItemGroup>
+
+ <ItemGroup Condition=" '$(TargetFramework)' == 'net451' ">
+ <PackageReference Include="System.Runtime.InteropServices.RuntimeInformation" Version="$(SystemRuntimeInteropServicesRuntimeInformationPackageVersion)" />
+ </ItemGroup>
<ItemGroup Condition=" '$(TargetFramework)' == 'net451' ">
<Reference Include="System" />
diff --git a/src/Lucene.Net.Tests/Support/IO/TestFileSupport.cs b/src/Lucene.Net.Tests/Support/IO/TestFileSupport.cs
index b5ce877..244f848 100644
--- a/src/Lucene.Net.Tests/Support/IO/TestFileSupport.cs
+++ b/src/Lucene.Net.Tests/Support/IO/TestFileSupport.cs
@@ -4,6 +4,7 @@ using NUnit.Framework;
using System;
using System.Globalization;
using System.IO;
+using System.Runtime.InteropServices;
namespace Lucene.Net.Support.IO
{
@@ -26,7 +27,7 @@ namespace Lucene.Net.Support.IO
public class TestFileSupport : LuceneTestCase
{
- private static String platformId = System.Runtime.InteropServices.RuntimeInformation.FrameworkDescription.Replace('.', '-');
+ private static String platformId = RuntimeInformation.FrameworkDescription.Replace('.', '-');
/** Location to store tests in */
private DirectoryInfo tempDirectory;
[lucenenet] 08/13: run-tests-on-os.yml: Added name to jobs so the
ones that are running can be displayed in the logs
Posted by ni...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
nightowl888 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/lucenenet.git
commit d8395be593c74313d39c9763f0e1508f6015d430
Author: Shad Storhaug <sh...@shadstorhaug.com>
AuthorDate: Sat Oct 26 05:40:04 2019 +0700
run-tests-on-os.yml: Added name to jobs so the ones that are running can be displayed in the logs
---
build/azure-templates/run-tests-on-os.yml | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/build/azure-templates/run-tests-on-os.yml b/build/azure-templates/run-tests-on-os.yml
index 0bd9f33..875cfdf 100644
--- a/build/azure-templates/run-tests-on-os.yml
+++ b/build/azure-templates/run-tests-on-os.yml
@@ -161,7 +161,7 @@ steps:
Start-Sleep -Milliseconds 500
# Execute the jobs in parallel
- Start-Job $scriptBlock -ArgumentList $testExpression
+ Start-Job -Name "$testName,$framework" -ScriptBlock $scriptBlock -ArgumentList $testExpression
}
}
}
@@ -177,6 +177,12 @@ steps:
if ($running.Count -gt 0) {
Write-Host ""
Write-Host " Almost finished, only $($running.Count) projects left..." -ForegroundColor Cyan
+ [int]$number = 0
+ foreach ($runningJob in $running) {
+ $number++
+ $jobName = $runningJob | Select -ExpandProperty Name
+ Write-Host "$number. $jobName"
+ }
$running | Wait-Job -Any
}
} until ($running.Count -eq 0)
[lucenenet] 03/13: Lucene.Net.Store.NativeFSLockFactory,
Lucene.Net.Support.IO.FileSupport: Changed implementations to
provoke lock/sharing/alredy exists exceptions during initialization so the
values of the current OS can be used for comparison of HResult values at
runtime (closes LUCENENET-618)
Posted by ni...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
nightowl888 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/lucenenet.git
commit fd664cd12afcdfbae9d7b3dab4a9bfcd89489221
Author: Shad Storhaug <sh...@shadstorhaug.com>
AuthorDate: Thu Oct 24 20:19:18 2019 +0700
Lucene.Net.Store.NativeFSLockFactory, Lucene.Net.Support.IO.FileSupport: Changed implementations to provoke lock/sharing/alredy exists exceptions during initialization so the values of the current OS can be used for comparison of HResult values at runtime (closes LUCENENET-618)
---
src/Lucene.Net/Store/NativeFSLockFactory.cs | 345 +++++++++++++++++++++-------
src/Lucene.Net/Support/IO/FileSupport.cs | 81 ++++++-
2 files changed, 338 insertions(+), 88 deletions(-)
diff --git a/src/Lucene.Net/Store/NativeFSLockFactory.cs b/src/Lucene.Net/Store/NativeFSLockFactory.cs
index b5b65fc..b07b1af 100644
--- a/src/Lucene.Net/Store/NativeFSLockFactory.cs
+++ b/src/Lucene.Net/Store/NativeFSLockFactory.cs
@@ -54,8 +54,99 @@ namespace Lucene.Net.Store
/// <see cref="LockVerifyServer"/> and <see cref="LockStressTest"/>.</para>
/// </summary>
/// <seealso cref="LockFactory"/>
+ // LUCENENET specific - this class has been refactored significantly from its Java counterpart
+ // to take advantage of .NET FileShare locking in the Windows and Linux environments.
public class NativeFSLockFactory : FSLockFactory
{
+ private enum FSLockingStrategy
+ {
+ FileStreamLockViolation,
+ FileSharingViolation,
+ Fallback
+ }
+
+ // LUCENENET: This controls the locking strategy used for the current operating system and framework
+ private static FSLockingStrategy LockingStrategy
+ {
+ get
+ {
+ if (IS_FILESTREAM_LOCKING_PLATFORM && HRESULT_FILE_LOCK_VIOLATION.HasValue)
+ return FSLockingStrategy.FileStreamLockViolation;
+ else if (HRESULT_FILE_SHARE_VIOLATION.HasValue)
+ return FSLockingStrategy.FileSharingViolation;
+ else
+ // Fallback implementation for unknown platforms that don't rely on HResult
+ return FSLockingStrategy.Fallback;
+ }
+ }
+
+
+ // LUCNENENET NOTE: Lookup the HResult value we are interested in for the current OS
+ // by provoking the exception during initialization and caching its HResult value for later.
+ // We optimize for Windows because those HResult values are known and documented, but for
+ // other platforms, this is the only way we can reliably determine the HResult values
+ // we are interested in.
+ //
+ // Reference: https://stackoverflow.com/q/46380483
+ private static bool IS_FILESTREAM_LOCKING_PLATFORM = LoadIsFileStreamLockingPlatform();
+
+ private const int WIN_HRESULT_FILE_LOCK_VIOLATION = unchecked((int)0x80070021);
+ private const int WIN_HRESULT_FILE_SHARE_VIOLATION = unchecked((int)0x80070020);
+
+ internal static readonly int? HRESULT_FILE_LOCK_VIOLATION = LoadFileLockViolationHResult();
+ internal static readonly int? HRESULT_FILE_SHARE_VIOLATION = LoadFileShareViolationHResult();
+
+ private static bool LoadIsFileStreamLockingPlatform()
+ {
+#if FEATURE_FILESTREAM_LOCK
+ return Constants.WINDOWS; // LUCENENET: See: https://github.com/dotnet/corefx/issues/5964
+#else
+ return false;
+#endif
+ }
+
+ private static int? LoadFileLockViolationHResult()
+ {
+ if (Constants.WINDOWS)
+ return WIN_HRESULT_FILE_LOCK_VIOLATION;
+
+ // Skip provoking the exception unless we know we will use the value
+ if (IS_FILESTREAM_LOCKING_PLATFORM)
+ {
+ return FileSupport.GetFileIOExceptionHResult(provokeException: (fileName) =>
+ {
+ using (var lockStream = new FileStream(fileName, FileMode.OpenOrCreate, FileAccess.Write, FileShare.ReadWrite))
+ {
+ lockStream.Lock(0, 1); // Create an exclusive lock
+ using (var stream = new FileStream(fileName, FileMode.Open, FileAccess.Write, FileShare.ReadWrite))
+ {
+ // try to find out if the file is locked by writing a byte. Note that we need to flush the stream to find out.
+ stream.WriteByte(0);
+ stream.Flush(); // this *may* throw an IOException if the file is locked, but...
+ // ... closing the stream is the real test
+ }
+ }
+ });
+ }
+
+ return null;
+ }
+
+ private static int? LoadFileShareViolationHResult()
+ {
+ if (Constants.WINDOWS)
+ return WIN_HRESULT_FILE_SHARE_VIOLATION;
+
+ return FileSupport.GetFileIOExceptionHResult(provokeException: (fileName) =>
+ {
+ using (var lockStream = new FileStream(fileName, FileMode.OpenOrCreate, FileAccess.Write, FileShare.None, 1, FileOptions.None))
+ // Try to get an exclusive lock on the file - this should throw an IOException with the current platform's HResult value for FileShare violation
+ using (var stream = new FileStream(fileName, FileMode.Open, FileAccess.Write, FileShare.None, 1, FileOptions.None))
+ {
+ }
+ });
+ }
+
/// <summary>
/// Create a <see cref="NativeFSLockFactory"/> instance, with <c>null</c> (unset)
/// lock directory. When you pass this factory to a <see cref="FSDirectory"/>
@@ -119,21 +210,25 @@ namespace Lucene.Net.Store
// Internal for testing
internal virtual Lock NewLock(string path)
{
- if (Constants.WINDOWS)
- return new WindowsNativeFSLock(this, m_lockDir, path);
-
- // Fallback implementation for unknown platforms that don't rely on HResult
- return new NativeFSLock(this, m_lockDir, path);
+ switch (LockingStrategy)
+ {
+ case FSLockingStrategy.FileStreamLockViolation:
+ return new NativeFSLock(m_lockDir, path);
+ case FSLockingStrategy.FileSharingViolation:
+ return new SharingNativeFSLock(m_lockDir, path);
+ default:
+ // Fallback implementation for unknown platforms that don't rely on HResult
+ return new FallbackNativeFSLock(m_lockDir, path);
+ }
}
public override void ClearLock(string lockName)
{
var path = GetCanonicalPathOfLockFile(lockName);
- Lock l;
// this is the reason why we can't use ConcurrentDictionary: we need the removal and disposal of the lock to be atomic
// otherwise it may clash with MakeLock making a lock and ClearLock disposing of it in another thread.
lock (_locks)
- if (_locks.TryGetValue(path, out l))
+ if (_locks.TryGetValue(path, out Lock l))
{
_locks.Remove(path);
l.Dispose();
@@ -145,29 +240,19 @@ namespace Lucene.Net.Store
// LUCENENET NOTE: We use this implementation as a fallback for platforms that we don't
// know the HResult numbers for lock and file sharing errors.
//
- // Note that using SharingAwareNativeFSLock would be ideal for all platforms. However,
- // at the time of this writing there is no cross-platform way to identify sharing errors
- // or lock errors because the values of HResult depend on the specific OS platform we
- // are running on. Unfortunately, researching what these numbers may be even for Linux and
- // OSx turned up nothing, and it is also unclear whether different flavors of Linux/Unix will have
- // different HResult numbers. The best we can hope for is for people to contribute subclasses
- // of SharingAwareNativeFSLock for the most popular platforms and fall back to this (substandard)
- // implementation for all of the other platforms. See WindowsNativeFSLock for an example of what
- // one of these subclasses should look like. The NativeFSLockFactory.NewLock() factory method
- // is the only place where adding the platform-specific logic which class to instantiate is required.
+ // Note that using NativeFSLock would be ideal for all platforms. However, there is a
+ // small chance that provoking lock/share exceptions will fail. In that rare case, we
+ // fallback to this substandard implementation.
//
// Reference: https://stackoverflow.com/q/46380483
- internal class NativeFSLock : Lock
+ internal class FallbackNativeFSLock : Lock
{
- private readonly NativeFSLockFactory outerInstance;
-
private FileStream channel;
private readonly string path;
private readonly DirectoryInfo lockDir;
- public NativeFSLock(NativeFSLockFactory outerInstance, DirectoryInfo lockDir, string path)
+ public FallbackNativeFSLock(DirectoryInfo lockDir, string path)
{
- this.outerInstance = outerInstance;
this.lockDir = lockDir;
this.path = path;
}
@@ -323,28 +408,30 @@ namespace Lucene.Net.Store
public override string ToString()
{
- return "NativeFSLock@" + path;
+ return "{nameof(FallbackNativeFSLock)}@{path}";
}
}
- // Abstract class that can be used to create additional native locks that
- // work with OS-specific HResult values.
- internal abstract class SharingAwareNativeFSLock : Lock
+ // Locks the entire file. macOS requires this approach.
+ internal class SharingNativeFSLock : Lock
{
- private readonly NativeFSLockFactory outerInstance;
-
private FileStream channel;
private readonly string path;
private readonly DirectoryInfo lockDir;
- public SharingAwareNativeFSLock(NativeFSLockFactory outerInstance, DirectoryInfo lockDir, string path)
+ public SharingNativeFSLock(DirectoryInfo lockDir, string path)
{
- this.outerInstance = outerInstance;
this.lockDir = lockDir;
this.path = path;
}
- protected abstract bool IsLockOrShareViolation(IOException e);
+ /// <summary>
+ /// Return true if the <see cref="IOException"/> is the result of a share violation
+ /// </summary>
+ private bool IsShareViolation(IOException e)
+ {
+ return e.HResult == NativeFSLockFactory.HRESULT_FILE_SHARE_VIOLATION;
+ }
private FileStream GetLockFileStream(FileMode mode)
{
@@ -367,11 +454,7 @@ namespace Lucene.Net.Store
throw new IOException("Found regular file where directory expected: " + lockDir.FullName);
}
-#if FEATURE_FILESTREAM_LOCK
- return new FileStream(path, mode, FileAccess.Write, FileShare.ReadWrite);
-#else
return new FileStream(path, mode, FileAccess.Write, FileShare.None, 1, mode == FileMode.Open ? FileOptions.None : FileOptions.DeleteOnClose);
-#endif
}
public override bool Obtain()
@@ -385,12 +468,13 @@ namespace Lucene.Net.Store
// Our instance is already locked:
return false;
}
-
-#if FEATURE_FILESTREAM_LOCK
- FileStream stream = null;
try
{
- stream = GetLockFileStream(FileMode.OpenOrCreate);
+ channel = GetLockFileStream(FileMode.OpenOrCreate);
+ }
+ catch (IOException e) when (IsShareViolation(e))
+ {
+ // no failure reason to be recorded, since this is the expected error if a lock exists
}
catch (IOException e)
{
@@ -410,29 +494,137 @@ namespace Lucene.Net.Store
// if it fails to get the lock.
FailureReason = e;
}
+ return channel != null;
+ }
+ }
- if (stream != null)
+ protected override void Dispose(bool disposing)
+ {
+ if (disposing)
+ {
+ lock (this)
{
+ // whether or not we have created a file, we need to remove
+ // the lock instance from the dictionary that tracks them.
try
{
- stream.Lock(0, 1);
- // only assign the channel if the lock succeeds
- channel = stream;
+ lock (NativeFSLockFactory._locks)
+ NativeFSLockFactory._locks.Remove(path);
}
- catch (Exception e)
+ finally
{
- FailureReason = e;
- IOUtils.DisposeWhileHandlingException(stream);
+ if (channel != null)
+ {
+ try
+ {
+ IOUtils.DisposeWhileHandlingException(channel);
+ }
+ finally
+ {
+ channel = null;
+ }
+ }
}
}
-#else
+ }
+ }
+
+ public override bool IsLocked()
+ {
+ lock (this)
+ {
+ // First a shortcut, if a lock reference in this instance is available
+ if (channel != null)
+ {
+ return true;
+ }
+
try
{
- channel = GetLockFileStream(FileMode.OpenOrCreate);
+ using (var stream = GetLockFileStream(FileMode.Open))
+ {
+ }
+ return false;
}
- catch (IOException e) when (IsLockOrShareViolation(e))
+ catch (IOException e) when (IsShareViolation(e))
{
- // no failure reason to be recorded, since this is the expected error if a lock exists
+ return true;
+ }
+ catch (FileNotFoundException)
+ {
+ // if the file doesn't exists, there can be no lock active
+ return false;
+ }
+ }
+ }
+
+ public override string ToString()
+ {
+ return $"{nameof(SharingNativeFSLock)}@{path}";
+ }
+ }
+
+ // Uses FileStream locking of file pages.
+ internal class NativeFSLock : Lock
+ {
+ private FileStream channel;
+ private readonly string path;
+ private readonly DirectoryInfo lockDir;
+
+ public NativeFSLock(DirectoryInfo lockDir, string path)
+ {
+ this.lockDir = lockDir;
+ this.path = path;
+ }
+
+ /// <summary>
+ /// Return true if the <see cref="IOException"/> is the result of a lock violation
+ /// </summary>
+ private bool IsLockViolation(IOException e)
+ {
+ return e.HResult == NativeFSLockFactory.HRESULT_FILE_LOCK_VIOLATION;
+ }
+
+ private FileStream GetLockFileStream(FileMode mode)
+ {
+ if (!System.IO.Directory.Exists(lockDir.FullName))
+ {
+ try
+ {
+ System.IO.Directory.CreateDirectory(lockDir.FullName);
+ }
+ catch (Exception e)
+ {
+ // note that several processes might have been trying to create the same directory at the same time.
+ // if one succeeded, the directory will exist and the exception can be ignored. In all other cases we should report it.
+ if (!System.IO.Directory.Exists(lockDir.FullName))
+ throw new IOException("Cannot create directory: " + lockDir.FullName, e);
+ }
+ }
+ else if (File.Exists(lockDir.FullName))
+ {
+ throw new IOException("Found regular file where directory expected: " + lockDir.FullName);
+ }
+
+ return new FileStream(path, mode, FileAccess.Write, FileShare.ReadWrite);
+ }
+
+ public override bool Obtain()
+ {
+ lock (this)
+ {
+ FailureReason = null;
+
+ if (channel != null)
+ {
+ // Our instance is already locked:
+ return false;
+ }
+
+ FileStream stream = null;
+ try
+ {
+ stream = GetLockFileStream(FileMode.OpenOrCreate);
}
catch (IOException e)
{
@@ -452,7 +644,21 @@ namespace Lucene.Net.Store
// if it fails to get the lock.
FailureReason = e;
}
-#endif
+
+ if (stream != null)
+ {
+ try
+ {
+ stream.Lock(0, 1);
+ // only assign the channel if the lock succeeds
+ channel = stream;
+ }
+ catch (Exception e)
+ {
+ FailureReason = e;
+ IOUtils.DisposeWhileHandlingException(stream);
+ }
+ }
return channel != null;
}
}
@@ -482,7 +688,6 @@ namespace Lucene.Net.Store
{
channel = null;
}
-#if FEATURE_FILESTREAM_LOCK
// try to delete the file if we created it, but it's not an error if we can't.
try
{
@@ -491,7 +696,6 @@ namespace Lucene.Net.Store
catch
{
}
-#endif
}
}
}
@@ -512,16 +716,14 @@ namespace Lucene.Net.Store
{
using (var stream = GetLockFileStream(FileMode.Open))
{
-#if FEATURE_FILESTREAM_LOCK
// try to find out if the file is locked by writing a byte. Note that we need to flush the stream to find out.
stream.WriteByte(0);
stream.Flush(); // this *may* throw an IOException if the file is locked, but...
// ... closing the stream is the real test
-#endif
}
return false;
}
- catch (IOException e) when (IsLockOrShareViolation(e))
+ catch (IOException e) when (IsLockViolation(e))
{
return true;
}
@@ -535,36 +737,17 @@ namespace Lucene.Net.Store
public override string ToString()
{
- return "NativeFSLock@" + path;
+ return $"{nameof(NativeFSLock)}@{path}";
}
}
-
- // LUCENENET: Lock that uses HResult native to Windows
- internal class WindowsNativeFSLock : SharingAwareNativeFSLock
+#if !FEATURE_FILESTREAM_LOCK
+ internal static class FileStreamExtensions
{
-#if FEATURE_FILESTREAM_LOCK
- private const int ERROR_LOCK_VIOLATION = 0x21;
-#else
- private const int ERROR_SHARE_VIOLATION = 0x20;
-#endif
-
- public WindowsNativeFSLock(NativeFSLockFactory outerInstance, DirectoryInfo lockDir, string path)
- : base(outerInstance, lockDir, path)
+ // Dummy lock method to ensure we can compile even if the feature is unavailable
+ public static void Lock(this FileStream stream, long position, long length)
{
}
-
- /// <summary>
- /// Return true if the <see cref="IOException"/> is the result of a lock violation
- /// </summary>
- protected override bool IsLockOrShareViolation(IOException e)
- {
- var result = e.HResult & 0xFFFF;
-#if FEATURE_FILESTREAM_LOCK
- return result == ERROR_LOCK_VIOLATION;
-#else
- return result == ERROR_SHARE_VIOLATION;
-#endif
- }
}
+#endif
}
\ No newline at end of file
diff --git a/src/Lucene.Net/Support/IO/FileSupport.cs b/src/Lucene.Net/Support/IO/FileSupport.cs
index 35734e6..c988016 100644
--- a/src/Lucene.Net/Support/IO/FileSupport.cs
+++ b/src/Lucene.Net/Support/IO/FileSupport.cs
@@ -29,7 +29,62 @@ namespace Lucene.Net.Support.IO
/// </summary>
public static class FileSupport
{
- private static int ERROR_FILE_EXISTS = 0x0050;
+ // LUCNENENET NOTE: Lookup the HResult value we are interested in for the current OS
+ // by provoking the exception during initialization and caching its HResult value for later.
+ // We optimize for Windows because those HResult values are known and documented, but for
+ // other platforms, this is the only way we can reliably determine the HResult values
+ // we are interested in.
+ //
+ // Reference: https://stackoverflow.com/q/46380483
+ private const int WIN_HRESULT_FILE_ALREADY_EXISTS = unchecked((int)0x80070050);
+ private static readonly int? HRESULT_FILE_ALREADY_EXISTS = LoadFileAlreadyExistsHResult();
+
+ private static int? LoadFileAlreadyExistsHResult()
+ {
+ if (Constants.WINDOWS)
+ return WIN_HRESULT_FILE_ALREADY_EXISTS;
+
+ return GetFileIOExceptionHResult(provokeException: (fileName) =>
+ {
+ //Try to create the file again -this should throw an IOException with the correct HResult for the current platform
+ using (var stream = new FileStream(fileName, FileMode.CreateNew, FileAccess.Write, FileShare.Read)) { }
+ });
+ }
+
+ internal static int? GetFileIOExceptionHResult(Action<string> provokeException)
+ {
+ string fileName;
+ try
+ {
+ // This could throw, but we don't care about this HResult value.
+ fileName = Path.GetTempFileName();
+ }
+ catch
+ {
+ return null; // We couldn't create a temp file
+ }
+ try
+ {
+ provokeException(fileName);
+ }
+ catch (IOException ex) when (ex.HResult != 0) // Assume 0 means the platform is not completely implemented, thus unknown
+ {
+ return ex.HResult;
+ }
+ catch
+ {
+ return null; // Unknown exception
+ }
+ finally
+ {
+ try
+ {
+ File.Delete(fileName);
+ }
+ catch { }
+ }
+ return null; // Should never get here
+ }
/// <summary>
/// Creates a new empty file in a random subdirectory of <see cref="Path.GetTempPath()"/>, using the given prefix and
@@ -144,14 +199,26 @@ namespace Lucene.Net.Support.IO
return new FileInfo(fileName);
}
- private static bool IsFileAlreadyExistsException(IOException e, string fileName)
+ /// <summary>
+ /// Tests whether the passed in <see cref="Exception"/> is an <see cref="IOException"/>
+ /// corresponding to the underlying operating system's "File Already Exists" violation.
+ /// This works by forcing the exception to occur during initialization and caching the
+ /// <see cref="Exception.HResult"/> value for the current OS.
+ /// </summary>
+ /// <param name="ex">An exception, for comparison.</param>
+ /// <param name="filePath">The path of the file to check. This is used as a fallback in case the
+ /// current OS doesn't have an HResult (an edge case).</param>
+ /// <returns><c>true</c> if the exception passed is an <see cref="IOException"/> with an
+ /// <see cref="Exception.HResult"/> corresponding to the operating system's "File Already Exists" violation, which
+ /// occurs when an attempt is made to create a file that already exists.</returns>
+ public static bool IsFileAlreadyExistsException(Exception ex, string filePath)
{
- // On Windows, we can rely on the constant, but we need to fallback
- // to doing a physical file check to be portable across platforms.
- if (Constants.WINDOWS)
- return (e.HResult & 0xFFFF) == ERROR_FILE_EXISTS;
+ if (!typeof(IOException).Equals(ex))
+ return false;
+ else if (HRESULT_FILE_ALREADY_EXISTS.HasValue)
+ return ex.HResult == HRESULT_FILE_ALREADY_EXISTS;
else
- return File.Exists(fileName);
+ return File.Exists(filePath);
}
/// <summary>
[lucenenet] 12/13: Lucene.Net.Util.AttributeSource: Added TODO
Posted by ni...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
nightowl888 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/lucenenet.git
commit aee8690c7c49c4f2a580fe2883a92656aa389641
Author: Shad Storhaug <sh...@shadstorhaug.com>
AuthorDate: Tue Oct 22 11:37:53 2019 +0700
Lucene.Net.Util.AttributeSource: Added TODO
---
src/Lucene.Net/Util/AttributeSource.cs | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/Lucene.Net/Util/AttributeSource.cs b/src/Lucene.Net/Util/AttributeSource.cs
index f3b8b66..7b46b88 100644
--- a/src/Lucene.Net/Util/AttributeSource.cs
+++ b/src/Lucene.Net/Util/AttributeSource.cs
@@ -40,7 +40,7 @@ namespace Lucene.Net.Util
/// <summary>
/// An <see cref="AttributeFactory"/> creates instances of <see cref="Attribute"/>s.
/// </summary>
- public abstract class AttributeFactory
+ public abstract class AttributeFactory // LUCENENET TODO: API - de-nest
{
/// <summary>
/// returns an <see cref="Attribute"/> for the supplied <see cref="IAttribute"/> interface.
[lucenenet] 01/13:
Lucene.Net.TestFramework.Analysis.ICheckClearAttributesAttribute: Converted
property AndResetClearCalled back to a method GetAndResetClearCalled(),
as it has a side-effect, which makes it a poor candidate for a property.
Posted by ni...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
nightowl888 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/lucenenet.git
commit 2e2710edf5777e8338bed16862c0cf862540a6f1
Author: Shad Storhaug <sh...@shadstorhaug.com>
AuthorDate: Sat Oct 26 23:20:12 2019 +0700
Lucene.Net.TestFramework.Analysis.ICheckClearAttributesAttribute: Converted property AndResetClearCalled back to a method GetAndResetClearCalled(), as it has a side-effect, which makes it a poor candidate for a property.
---
.../Analysis/BaseTokenStreamTestCase.cs | 19 ++++++++-----------
1 file changed, 8 insertions(+), 11 deletions(-)
diff --git a/src/Lucene.Net.TestFramework/Analysis/BaseTokenStreamTestCase.cs b/src/Lucene.Net.TestFramework/Analysis/BaseTokenStreamTestCase.cs
index 7624056..483036d 100644
--- a/src/Lucene.Net.TestFramework/Analysis/BaseTokenStreamTestCase.cs
+++ b/src/Lucene.Net.TestFramework/Analysis/BaseTokenStreamTestCase.cs
@@ -53,7 +53,7 @@ namespace Lucene.Net.Analysis
/// </summary>
public interface ICheckClearAttributesAttribute : IAttribute
{
- bool AndResetClearCalled { get; }
+ bool GetAndResetClearCalled();
}
/// <summary>
@@ -64,14 +64,11 @@ namespace Lucene.Net.Analysis
{
private bool clearCalled = false;
- public bool AndResetClearCalled
+ public bool GetAndResetClearCalled()
{
- get
- {
- bool old = clearCalled;
- clearCalled = false;
- return old;
- }
+ bool old = clearCalled;
+ clearCalled = false;
+ return old;
}
public override void Clear()
@@ -228,7 +225,7 @@ namespace Lucene.Net.Analysis
keywordAtt.IsKeyword = (i & 1) == 0;
}
- bool reset = checkClearAtt.AndResetClearCalled; // reset it, because we called clearAttribute() before
+ bool reset = checkClearAtt.GetAndResetClearCalled(); // reset it, because we called clearAttribute() before
Assert.IsTrue(ts.IncrementToken(), "token " + i + " does not exist");
Assert.IsTrue(reset, "ClearAttributes() was not called correctly in TokenStream chain");
@@ -362,10 +359,10 @@ namespace Lucene.Net.Analysis
posLengthAtt.PositionLength = 45987653;
}
- var reset_ = checkClearAtt.AndResetClearCalled; // reset it, because we called clearAttribute() before
+ var reset_ = checkClearAtt.GetAndResetClearCalled(); // reset it, because we called clearAttribute() before
ts.End();
- Assert.IsTrue(checkClearAtt.AndResetClearCalled, "super.End()/ClearAttributes() was not called correctly in End()");
+ Assert.IsTrue(checkClearAtt.GetAndResetClearCalled(), "base.End()/ClearAttributes() was not called correctly in End()");
if (finalOffset != null)
{
[lucenenet] 13/13: Ported Lucene.Net.Analysis.OpenNLP + tests
Posted by ni...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
nightowl888 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/lucenenet.git
commit 5c434545b31e8131485e00657b37685610358677
Author: Shad Storhaug <sh...@shadstorhaug.com>
AuthorDate: Tue Oct 22 11:36:01 2019 +0700
Ported Lucene.Net.Analysis.OpenNLP + tests
---
Lucene.Net.sln | 12 +
README.md | 5 +-
build/Dependencies.props | 1 +
.../publish-test-results-for-test-projects.yml | 10 +
build/build.ps1 | 14 +-
.../Lucene.Net.Analysis.OpenNLP.csproj | 55 +++
.../OpenNLPChunkerFilter.cs | 115 +++++++
.../OpenNLPChunkerFilterFactory.cs | 90 +++++
.../OpenNLPLemmatizerFilter.cs | 129 +++++++
.../OpenNLPLemmatizerFilterFactory.cs | 95 ++++++
.../OpenNLPPOSFilter.cs | 102 ++++++
.../OpenNLPPOSFilterFactory.cs | 79 +++++
.../OpenNLPSentenceBreakIterator.cs | 277 +++++++++++++++
.../OpenNLPTokenizer.cs | 112 +++++++
.../OpenNLPTokenizerFactory.cs | 85 +++++
.../Tools/NLPChunkerOp.cs | 47 +++
.../Tools/NLPLemmatizerOp.cs | 93 ++++++
.../Tools/NLPNERTaggerOp.cs | 61 ++++
.../Tools/NLPPOSTaggerOp.cs | 43 +++
.../Tools/NLPSentenceDetectorOp.cs | 58 ++++
.../Tools/NLPTokenizerOp.cs | 55 +++
.../Tools/OpenNLPOpsFactory.cs | 225 +++++++++++++
.../Analysis/BaseTokenStreamTestCase.cs | 80 ++++-
.../Analysis/Core/TestStopFilter.cs | 2 +-
.../Lucene.Net.Tests.Analysis.OpenNLP.csproj | 71 ++++
.../TestOpenNLPChunkerFilterFactory.cs | 122 +++++++
.../TestOpenNLPLemmatizerFilterFactory.cs | 372 +++++++++++++++++++++
.../TestOpenNLPPOSFilterFactory.cs | 160 +++++++++
.../TestOpenNLPSentenceBreakIterator.cs | 252 ++++++++++++++
.../TestOpenNLPTokenizerFactory.cs | 140 ++++++++
.../en-test-chunker.bin | Bin 0 -> 89914 bytes
.../en-test-lemmas.dict | 12 +
.../en-test-lemmatizer.bin | Bin 0 -> 7370 bytes
.../en-test-ner.bin | Bin 0 -> 2049 bytes
.../en-test-pos-maxent.bin | Bin 0 -> 18494 bytes
.../en-test-sent.bin | Bin 0 -> 1051 bytes
.../en-test-tokenizer.bin | Bin 0 -> 15100 bytes
.../TestOpenNLPChunkerFilterFactory.cs | 86 +++++
38 files changed, 3055 insertions(+), 5 deletions(-)
diff --git a/Lucene.Net.sln b/Lucene.Net.sln
index 70c88dc..192f434 100644
--- a/Lucene.Net.sln
+++ b/Lucene.Net.sln
@@ -193,6 +193,10 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Lucene.Net.Tests.TestFramew
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Lucene.Net.TestFramework.MSTest", "src\Lucene.Net.TestFramework.MSTest\Lucene.Net.TestFramework.MSTest.csproj", "{48520313-3B78-40D9-AE34-4864BFADF747}"
EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Lucene.Net.Analysis.OpenNLP", "src\Lucene.Net.Analysis.OpenNLP\Lucene.Net.Analysis.OpenNLP.csproj", "{CC2CE069-5BBB-429E-8510-7C3FBA8069D5}"
+EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Lucene.Net.Tests.Analysis.OpenNLP", "src\Lucene.Net.Tests.Analysis.OpenNLP\Lucene.Net.Tests.Analysis.OpenNLP.csproj", "{88D6D124-711D-4232-AD70-F22AB6AF9EA1}"
+EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
@@ -439,6 +443,14 @@ Global
{48520313-3B78-40D9-AE34-4864BFADF747}.Debug|Any CPU.Build.0 = Debug|Any CPU
{48520313-3B78-40D9-AE34-4864BFADF747}.Release|Any CPU.ActiveCfg = Release|Any CPU
{48520313-3B78-40D9-AE34-4864BFADF747}.Release|Any CPU.Build.0 = Release|Any CPU
+ {CC2CE069-5BBB-429E-8510-7C3FBA8069D5}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+ {CC2CE069-5BBB-429E-8510-7C3FBA8069D5}.Debug|Any CPU.Build.0 = Debug|Any CPU
+ {CC2CE069-5BBB-429E-8510-7C3FBA8069D5}.Release|Any CPU.ActiveCfg = Release|Any CPU
+ {CC2CE069-5BBB-429E-8510-7C3FBA8069D5}.Release|Any CPU.Build.0 = Release|Any CPU
+ {88D6D124-711D-4232-AD70-F22AB6AF9EA1}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+ {88D6D124-711D-4232-AD70-F22AB6AF9EA1}.Debug|Any CPU.Build.0 = Debug|Any CPU
+ {88D6D124-711D-4232-AD70-F22AB6AF9EA1}.Release|Any CPU.ActiveCfg = Release|Any CPU
+ {88D6D124-711D-4232-AD70-F22AB6AF9EA1}.Release|Any CPU.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
diff --git a/README.md b/README.md
index 7d0a5da..385ff7f 100644
--- a/README.md
+++ b/README.md
@@ -60,7 +60,10 @@ PM> Install-Package Lucene.Net -Pre
<!--- TO BE ADDED WHEN RELEASED
-- [Lucene.Net.Analysis.UIMA](https://www.nuget.org/packages/Lucene.Net.Analysis.UIMA/) - Analysis integration with Apache UIMA)
+- [Lucene.Net.Analysis.OpenNLP](https://www.nuget.org/packages/Lucene.Net.Analysis.OpenNLP/) - OpenNLP Library Integration
+- [Lucene.Net.Analysis.Nori](https://www.nuget.org/packages/Lucene.Net.Analysis.Nori/) - Korean Morphological Analyzer
+- [Lucene.Net.Analysis.Morfologik](https://www.nuget.org/packages/Lucene.Net.Analysis.Morfologik/) - Analyzer for dictionary stemming, built-in Polish dictionary
+- [Lucene.Net.TestFramework](https://www.nuget.org/packages/Lucene.Net.TestFramework/) - Framework for testing Lucene-based applications
-->
diff --git a/build/Dependencies.props b/build/Dependencies.props
index 1cb92e6..6852efd 100644
--- a/build/Dependencies.props
+++ b/build/Dependencies.props
@@ -50,6 +50,7 @@
<NewtonsoftJsonPackageVersion>10.0.3</NewtonsoftJsonPackageVersion>
<NUnit3TestAdapterPackageVersion>3.13.0</NUnit3TestAdapterPackageVersion>
<NUnitPackageVersion>3.9.0</NUnitPackageVersion>
+ <OpenNLPNETPackageVersion>1.9.1</OpenNLPNETPackageVersion>
<SharpZipLibPackageVersion>1.1.0</SharpZipLibPackageVersion>
<SharpZipLibNETStandardPackageVersion>0.86.0.1</SharpZipLibNETStandardPackageVersion>
<Spatial4nCorePackageVersion>0.4.1</Spatial4nCorePackageVersion>
diff --git a/build/azure-templates/publish-test-results-for-test-projects.yml b/build/azure-templates/publish-test-results-for-test-projects.yml
index af30e2c..1b61c3c 100644
--- a/build/azure-templates/publish-test-results-for-test-projects.yml
+++ b/build/azure-templates/publish-test-results-for-test-projects.yml
@@ -81,6 +81,16 @@ steps:
testResultsArtifactName: '${{ parameters.testResultsArtifactName }}'
testResultsFileName: '${{ parameters.testResultsFileName }}'
+# Special case: Only supports .net45
+- template: publish-test-results.yml
+ parameters:
+ framework: 'net451'
+ testProjectName: 'Lucene.Net.Tests.Analysis.OpenNLP'
+ osName: '${{ parameters.osName }}'
+ testResultsFormat: '${{ parameters.testResultsFormat }}'
+ testResultsArtifactName: '${{ parameters.testResultsArtifactName }}'
+ testResultsFileName: '${{ parameters.testResultsFileName }}'
+
- template: publish-test-results-for-target-frameworks.yml
parameters:
testProjectName: 'Lucene.Net.Tests._A-D'
diff --git a/build/build.ps1 b/build/build.ps1
index 14513f3..d9725b6 100644
--- a/build/build.ps1
+++ b/build/build.ps1
@@ -215,6 +215,11 @@ task Publish -depends Compile -description "This task uses dotnet publish to pac
continue
}
+ # Special case - OpenNLP.NET only supports .NET Framework
+ if ($projectName.Contains("Tests.Analysis.OpenNLP") -and (!$framework.StartsWith("net45"))) {
+ continue
+ }
+
$logPath = "$outDirectory/$framework"
$outputPath = "$logPath/$projectName"
@@ -272,7 +277,7 @@ task Test -depends InstallSDK, UpdateLocalSDKVersion, Restore -description "This
$testProjects = $testProjects | Sort-Object -Property FullName
$frameworksToTest = Get-FrameworksToTest
-
+
Write-Host "frameworksToTest: $frameworksToTest" -ForegroundColor Yellow
[int]$totalProjects = $testProjects.Length * $frameworksToTest.Length
@@ -291,6 +296,13 @@ task Test -depends InstallSDK, UpdateLocalSDKVersion, Restore -description "This
$remainingProjects--
continue
}
+
+ # Special case - OpenNLP.NET only supports .NET Framework
+ if ($testName.Contains("Tests.Analysis.OpenNLP") -and (!$framework.StartsWith("net45"))) {
+ $totalProjects--
+ $remainingProjects--
+ continue
+ }
Write-Host " Next Project in Queue: $testName, Framework: $framework" -ForegroundColor Yellow
diff --git a/src/Lucene.Net.Analysis.OpenNLP/Lucene.Net.Analysis.OpenNLP.csproj b/src/Lucene.Net.Analysis.OpenNLP/Lucene.Net.Analysis.OpenNLP.csproj
new file mode 100644
index 0000000..aabca38
--- /dev/null
+++ b/src/Lucene.Net.Analysis.OpenNLP/Lucene.Net.Analysis.OpenNLP.csproj
@@ -0,0 +1,55 @@
+<Project Sdk="Microsoft.NET.Sdk">
+
+ <Import Project="$(SolutionDir)build/NuGet.props" />
+
+ <PropertyGroup>
+ <!--<TargetFrameworks>netstandard2.0;netstandard1.6</TargetFrameworks>-->
+ <TargetFrameworks Condition="$([MSBuild]::IsOsPlatform('Windows'))">$(TargetFrameworks);net45</TargetFrameworks>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netstandard1.6' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+
+ <AssemblyTitle>Lucene.Net.Analysis.OpenNLP</AssemblyTitle>
+ <Description>Analyzer for indexing phonetic signatures (for sounds-alike search) for the Lucene.Net full-text search engine library from The Apache Software Foundation.</Description>
+ <PackageTags>$(PackageTags);analysis;soundex;double;metaphone;sounds;like;beider;morse;cologne;caverphone;nysiis;match;rating</PackageTags>
+ <DocumentationFile>bin\$(Configuration)\$(TargetFramework)\$(AssemblyName).xml</DocumentationFile>
+ <NoWarn>$(NoWarn);1591;1573</NoWarn>
+ <RootNamespace>Lucene.Net.Analysis.OpenNlp</RootNamespace>
+ </PropertyGroup>
+
+ <!-- OpenNLP is not strong-named -->
+ <PropertyGroup Label="Assembly Signing">
+ <SignAssembly>false</SignAssembly>
+ </PropertyGroup>
+
+ <ItemGroup>
+ <Compile Include="..\CommonAssemblyKeys.cs" Link="Properties\CommonAssemblyKeys.cs" />
+ </ItemGroup>
+
+ <ItemGroup>
+ <ProjectReference Include="..\dotnet\Lucene.Net.ICU\Lucene.Net.ICU.csproj" />
+ <ProjectReference Include="..\Lucene.Net\Lucene.Net.csproj" />
+ <ProjectReference Include="..\Lucene.Net.Analysis.Common\Lucene.Net.Analysis.Common.csproj" />
+ </ItemGroup>
+
+ <ItemGroup>
+ <PackageReference Include="ICU4N" Version="$(ICU4NPackageVersion)" />
+ <PackageReference Include="OpenNLP.NET" Version="$(OpenNLPNETPackageVersion)" />
+ </ItemGroup>
+
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netstandard2.0' ">
+ <DebugType>portable</DebugType>
+ </PropertyGroup>
+
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netstandard1.6' ">
+ <DebugType>portable</DebugType>
+ </PropertyGroup>
+
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'net45' ">
+ <DebugType>full</DebugType>
+ </PropertyGroup>
+
+ <ItemGroup Condition=" '$(TargetFramework)' == 'net45' ">
+ <Reference Include="System" />
+ <Reference Include="Microsoft.CSharp" />
+ </ItemGroup>
+
+</Project>
diff --git a/src/Lucene.Net.Analysis.OpenNLP/OpenNLPChunkerFilter.cs b/src/Lucene.Net.Analysis.OpenNLP/OpenNLPChunkerFilter.cs
new file mode 100644
index 0000000..9144475
--- /dev/null
+++ b/src/Lucene.Net.Analysis.OpenNLP/OpenNLPChunkerFilter.cs
@@ -0,0 +1,115 @@
+using Lucene.Net.Analysis.OpenNlp.Tools;
+using Lucene.Net.Analysis.TokenAttributes;
+using Lucene.Net.Util;
+using System.Collections.Generic;
+using System.Linq;
+
+namespace Lucene.Net.Analysis.OpenNlp
+{
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ /// <summary>
+ /// Run OpenNLP chunker. Prerequisite: the <see cref="OpenNLPTokenizer"/> and <see cref="OpenNLPPOSFilter"/> must precede this filter.
+ /// Tags terms in the TypeAttribute, replacing the POS tags previously put there by <see cref="OpenNLPPOSFilter"/>.
+ /// </summary>
+ public sealed class OpenNLPChunkerFilter : TokenFilter
+ {
+ private List<AttributeSource> sentenceTokenAttrs = new List<AttributeSource>();
+ private int tokenNum = 0;
+ private bool moreTokensAvailable = true;
+ private string[] sentenceTerms = null;
+ private string[] sentenceTermPOSTags = null;
+
+ private readonly NLPChunkerOp chunkerOp;
+ private readonly ITypeAttribute typeAtt;
+ private readonly IFlagsAttribute flagsAtt;
+ private readonly ICharTermAttribute termAtt;
+
+ public OpenNLPChunkerFilter(TokenStream input, NLPChunkerOp chunkerOp)
+ : base(input)
+ {
+ this.chunkerOp = chunkerOp;
+ this.typeAtt = AddAttribute<ITypeAttribute>();
+ this.flagsAtt = AddAttribute<IFlagsAttribute>();
+ this.termAtt = AddAttribute<ICharTermAttribute>();
+ }
+
+ public override sealed bool IncrementToken()
+ {
+ if (!moreTokensAvailable)
+ {
+ Clear();
+ return false;
+ }
+ if (tokenNum == sentenceTokenAttrs.Count)
+ {
+ NextSentence();
+ if (sentenceTerms == null)
+ {
+ Clear();
+ return false;
+ }
+ AssignTokenTypes(chunkerOp.GetChunks(sentenceTerms, sentenceTermPOSTags, null));
+ tokenNum = 0;
+ }
+ ClearAttributes();
+ sentenceTokenAttrs[tokenNum++].CopyTo(this);
+ return true;
+ }
+
+ private void NextSentence()
+ {
+ IList<string> termList = new List<string>();
+ IList<string> posTagList = new List<string>();
+ sentenceTokenAttrs.Clear();
+ bool endOfSentence = false;
+ while (!endOfSentence && (moreTokensAvailable = m_input.IncrementToken()))
+ {
+ termList.Add(termAtt.ToString());
+ posTagList.Add(typeAtt.Type);
+ endOfSentence = 0 != (flagsAtt.Flags & OpenNLPTokenizer.EOS_FLAG_BIT);
+ sentenceTokenAttrs.Add(m_input.CloneAttributes());
+ }
+ sentenceTerms = termList.Count > 0 ? termList.ToArray() : null;
+ sentenceTermPOSTags = posTagList.Count > 0 ? posTagList.ToArray() : null;
+ }
+
+ private void AssignTokenTypes(string[] tags)
+ {
+ for (int i = 0; i < tags.Length; ++i)
+ {
+ sentenceTokenAttrs[i].GetAttribute<ITypeAttribute>().Type = tags[i];
+ }
+ }
+
+ public override void Reset()
+ {
+ base.Reset();
+ moreTokensAvailable = true;
+ Clear();
+ }
+
+ private void Clear()
+ {
+ sentenceTokenAttrs.Clear();
+ sentenceTerms = null;
+ sentenceTermPOSTags = null;
+ tokenNum = 0;
+ }
+ }
+}
diff --git a/src/Lucene.Net.Analysis.OpenNLP/OpenNLPChunkerFilterFactory.cs b/src/Lucene.Net.Analysis.OpenNLP/OpenNLPChunkerFilterFactory.cs
new file mode 100644
index 0000000..5669603
--- /dev/null
+++ b/src/Lucene.Net.Analysis.OpenNLP/OpenNLPChunkerFilterFactory.cs
@@ -0,0 +1,90 @@
+using Lucene.Net.Analysis.OpenNlp.Tools;
+using Lucene.Net.Analysis.Util;
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+
+namespace Lucene.Net.Analysis.OpenNlp
+{
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ /// <summary>
+ /// Factory for <see cref="OpenNLPChunkerFilter"/>.
+ /// <code>
+ /// <fieldType name="text_opennlp_chunked" class="solr.TextField" positionIncrementGap="100">
+ /// <analyzer>
+ /// <tokenizer class="solr.OpenNLPTokenizerFactory" sentenceModel="filename" tokenizerModel="filename"/>
+ /// <filter class="solr.OpenNLPPOSFilterFactory" posTaggerModel="filename"/>
+ /// <filter class="solr.OpenNLPChunkerFilterFactory" chunkerModel="filename"/>
+ /// </analyzer>
+ /// </fieldType>
+ /// </code>
+ /// </summary>
+ /// <since>7.3.0</since>
+ public class OpenNLPChunkerFilterFactory : TokenFilterFactory, IResourceLoaderAware
+ {
+ public static readonly string CHUNKER_MODEL = "chunkerModel";
+
+ private readonly string chunkerModelFile;
+
+ public OpenNLPChunkerFilterFactory(IDictionary<string, string> args)
+ : base(args)
+ {
+ chunkerModelFile = Get(args, CHUNKER_MODEL);
+ if (args.Any())
+ {
+ throw new ArgumentException("Unknown parameters: " + args);
+ }
+ }
+
+ public override TokenStream Create(TokenStream input)
+ {
+ try
+ {
+ NLPChunkerOp chunkerOp = null;
+
+ if (chunkerModelFile != null)
+ {
+ chunkerOp = OpenNLPOpsFactory.GetChunker(chunkerModelFile);
+ }
+ return new OpenNLPChunkerFilter(input, chunkerOp);
+ }
+ catch (IOException e)
+ {
+ throw new ArgumentException(e.ToString(), e);
+ }
+ }
+
+ public virtual void Inform(IResourceLoader loader)
+ {
+ try
+ {
+ // load and register read-only models in cache with file/resource names
+ if (chunkerModelFile != null)
+ {
+ OpenNLPOpsFactory.GetChunkerModel(chunkerModelFile, loader);
+ }
+ }
+ catch (IOException e)
+ {
+ throw new ArgumentException(e.ToString(), e);
+ }
+ }
+ }
+}
diff --git a/src/Lucene.Net.Analysis.OpenNLP/OpenNLPLemmatizerFilter.cs b/src/Lucene.Net.Analysis.OpenNLP/OpenNLPLemmatizerFilter.cs
new file mode 100644
index 0000000..977d98a
--- /dev/null
+++ b/src/Lucene.Net.Analysis.OpenNLP/OpenNLPLemmatizerFilter.cs
@@ -0,0 +1,129 @@
+using Lucene.Net.Analysis.OpenNlp.Tools;
+using Lucene.Net.Analysis.TokenAttributes;
+using Lucene.Net.Util;
+using System.Collections.Generic;
+using System.Linq;
+
+namespace Lucene.Net.Analysis.OpenNlp
+{
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ /// <summary>
+ /// Runs OpenNLP dictionary-based and/or MaxEnt lemmatizers.
+ /// <para/>
+ /// Both a dictionary-based lemmatizer and a MaxEnt lemmatizer are supported,
+ /// via the "dictionary" and "lemmatizerModel" params, respectively.
+ /// If both are configured, the dictionary-based lemmatizer is tried first,
+ /// and then the MaxEnt lemmatizer is consulted for out-of-vocabulary tokens.
+ /// <para/>
+ /// The dictionary file must be encoded as UTF-8, with one entry per line,
+ /// in the form <c>word[tab]lemma[tab]part-of-speech</c>
+ /// </summary>
+ public class OpenNLPLemmatizerFilter : TokenFilter
+ {
+ private readonly NLPLemmatizerOp lemmatizerOp;
+ private readonly ICharTermAttribute termAtt;
+ private readonly ITypeAttribute typeAtt;
+ private readonly IKeywordAttribute keywordAtt;
+ private readonly IFlagsAttribute flagsAtt;
+ private IList<AttributeSource> sentenceTokenAttrs = new List<AttributeSource>();
+ private IEnumerator<AttributeSource> sentenceTokenAttrsIter = null;
+ private bool moreTokensAvailable = true;
+ private string[] sentenceTokens = null; // non-keyword tokens
+ private string[] sentenceTokenTypes = null; // types for non-keyword tokens
+ private string[] lemmas = null; // lemmas for non-keyword tokens
+ private int lemmaNum = 0; // lemma counter
+
+ public OpenNLPLemmatizerFilter(TokenStream input, NLPLemmatizerOp lemmatizerOp)
+ : base(input)
+ {
+ this.lemmatizerOp = lemmatizerOp;
+ this.termAtt = AddAttribute<ICharTermAttribute>();
+ this.typeAtt = AddAttribute<ITypeAttribute>();
+ this.keywordAtt = AddAttribute<IKeywordAttribute>();
+ this.flagsAtt = AddAttribute<IFlagsAttribute>();
+ }
+
+ public override sealed bool IncrementToken()
+ {
+ if (!moreTokensAvailable)
+ {
+ Clear();
+ return false;
+ }
+ if (sentenceTokenAttrsIter == null || !sentenceTokenAttrsIter.MoveNext())
+ {
+ NextSentence();
+ if (sentenceTokens == null)
+ { // zero non-keyword tokens
+ Clear();
+ return false;
+ }
+ lemmas = lemmatizerOp.Lemmatize(sentenceTokens, sentenceTokenTypes);
+ lemmaNum = 0;
+ sentenceTokenAttrsIter = sentenceTokenAttrs.GetEnumerator();
+ sentenceTokenAttrsIter.MoveNext();
+ }
+ ClearAttributes();
+ sentenceTokenAttrsIter.Current.CopyTo(this);
+ if (!keywordAtt.IsKeyword)
+ {
+ termAtt.SetEmpty().Append(lemmas[lemmaNum++]);
+ }
+ return true;
+
+ }
+
+ private void NextSentence()
+ {
+ IList<string> tokenList = new List<string>();
+ IList<string> typeList = new List<string>();
+ sentenceTokenAttrs.Clear();
+ bool endOfSentence = false;
+ while (!endOfSentence && (moreTokensAvailable = m_input.IncrementToken()))
+ {
+ if (!keywordAtt.IsKeyword)
+ {
+ tokenList.Add(termAtt.ToString());
+ typeList.Add(typeAtt.Type);
+ }
+ endOfSentence = 0 != (flagsAtt.Flags & OpenNLPTokenizer.EOS_FLAG_BIT);
+ sentenceTokenAttrs.Add(m_input.CloneAttributes());
+ }
+ sentenceTokens = tokenList.Count > 0 ? tokenList.ToArray() : null;
+ sentenceTokenTypes = typeList.Count > 0 ? typeList.ToArray() : null;
+ }
+
+ public override void Reset()
+ {
+ base.Reset();
+ moreTokensAvailable = true;
+ Clear();
+ }
+
+ private void Clear()
+ {
+ sentenceTokenAttrs.Clear();
+ sentenceTokenAttrsIter = null;
+ sentenceTokens = null;
+ sentenceTokenTypes = null;
+ lemmas = null;
+ lemmaNum = 0;
+ }
+ }
+}
diff --git a/src/Lucene.Net.Analysis.OpenNLP/OpenNLPLemmatizerFilterFactory.cs b/src/Lucene.Net.Analysis.OpenNLP/OpenNLPLemmatizerFilterFactory.cs
new file mode 100644
index 0000000..e8fa98c
--- /dev/null
+++ b/src/Lucene.Net.Analysis.OpenNLP/OpenNLPLemmatizerFilterFactory.cs
@@ -0,0 +1,95 @@
+using Lucene.Net.Analysis.OpenNlp.Tools;
+using Lucene.Net.Analysis.Util;
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+
+namespace Lucene.Net.Analysis.OpenNlp
+{
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ /// <summary>
+ /// Factory for <see cref="OpenNLPLemmatizerFilter"/>.
+ /// <code>
+ /// <fieldType name="text_opennlp_lemma" class="solr.TextField" positionIncrementGap="100"
+ /// <analyzer>
+ /// <tokenizer class="solr.OpenNLPTokenizerFactory"
+ /// sentenceModel="filename"
+ /// tokenizerModel="filename"/>
+ /// />
+ /// <filter class="solr.OpenNLPLemmatizerFilterFactory"
+ /// dictionary="filename"
+ /// lemmatizerModel="filename"/>
+ /// </analyzer>
+ /// </fieldType>
+ /// </code>
+ /// </summary>
+ public class OpenNLPLemmatizerFilterFactory : TokenFilterFactory, IResourceLoaderAware
+ {
+ public static readonly string DICTIONARY = "dictionary";
+ public static readonly string LEMMATIZER_MODEL = "lemmatizerModel";
+
+ private readonly string dictionaryFile;
+ private readonly string lemmatizerModelFile;
+
+ public OpenNLPLemmatizerFilterFactory(IDictionary<string, string> args)
+ : base(args)
+ {
+ dictionaryFile = Get(args, DICTIONARY);
+ lemmatizerModelFile = Get(args, LEMMATIZER_MODEL);
+
+ if (dictionaryFile == null && lemmatizerModelFile == null)
+ {
+ throw new ArgumentException("Configuration Error: missing parameter: at least one of '"
+ + DICTIONARY + "' and '" + LEMMATIZER_MODEL + "' must be provided.");
+ }
+
+ if (args.Any())
+ {
+ throw new ArgumentException("Unknown parameters: " + args);
+ }
+ }
+
+ public override TokenStream Create(TokenStream input)
+ {
+ try
+ {
+ NLPLemmatizerOp lemmatizerOp = OpenNLPOpsFactory.GetLemmatizer(dictionaryFile, lemmatizerModelFile);
+ return new OpenNLPLemmatizerFilter(input, lemmatizerOp);
+ }
+ catch (IOException e)
+ {
+ throw new Exception(e.ToString(), e);
+ }
+ }
+
+ public virtual void Inform(IResourceLoader loader)
+ {
+ // register models in cache with file/resource names
+ if (dictionaryFile != null)
+ {
+ OpenNLPOpsFactory.GetLemmatizerDictionary(dictionaryFile, loader);
+ }
+ if (lemmatizerModelFile != null)
+ {
+ OpenNLPOpsFactory.GetLemmatizerModel(lemmatizerModelFile, loader);
+ }
+ }
+ }
+}
diff --git a/src/Lucene.Net.Analysis.OpenNLP/OpenNLPPOSFilter.cs b/src/Lucene.Net.Analysis.OpenNLP/OpenNLPPOSFilter.cs
new file mode 100644
index 0000000..449f5ef
--- /dev/null
+++ b/src/Lucene.Net.Analysis.OpenNLP/OpenNLPPOSFilter.cs
@@ -0,0 +1,102 @@
+using Lucene.Net.Analysis.OpenNlp.Tools;
+using Lucene.Net.Analysis.TokenAttributes;
+using Lucene.Net.Util;
+using System.Collections.Generic;
+using System.Linq;
+
+namespace Lucene.Net.Analysis.OpenNlp
+{
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ /// <summary>
+ /// Run OpenNLP POS tagger. Tags all terms in the <see cref="ITypeAttribute"/>.
+ /// </summary>
+ public sealed class OpenNLPPOSFilter : TokenFilter
+ {
+ private IList<AttributeSource> sentenceTokenAttrs = new List<AttributeSource>();
+ string[] tags = null;
+ private int tokenNum = 0;
+ private bool moreTokensAvailable = true;
+
+ private readonly NLPPOSTaggerOp posTaggerOp;
+ private readonly ITypeAttribute typeAtt;
+ private readonly IFlagsAttribute flagsAtt;
+ private readonly ICharTermAttribute termAtt;
+
+ public OpenNLPPOSFilter(TokenStream input, NLPPOSTaggerOp posTaggerOp)
+ : base(input)
+ {
+ this.posTaggerOp = posTaggerOp;
+ this.typeAtt = AddAttribute<ITypeAttribute>();
+ this.flagsAtt = AddAttribute<IFlagsAttribute>();
+ this.termAtt = AddAttribute<ICharTermAttribute>();
+ }
+
+ public override sealed bool IncrementToken()
+ {
+ if (!moreTokensAvailable)
+ {
+ Clear();
+ return false;
+ }
+ if (tokenNum == sentenceTokenAttrs.Count)
+ { // beginning of stream, or previous sentence exhausted
+ string[] sentenceTokens = NextSentence();
+ if (sentenceTokens == null)
+ {
+ Clear();
+ return false;
+ }
+ tags = posTaggerOp.GetPOSTags(sentenceTokens);
+ tokenNum = 0;
+ }
+ ClearAttributes();
+ sentenceTokenAttrs[tokenNum].CopyTo(this);
+ typeAtt.Type = tags[tokenNum++];
+ return true;
+ }
+
+ private string[] NextSentence()
+ {
+ IList<string> termList = new List<string>();
+ sentenceTokenAttrs.Clear();
+ bool endOfSentence = false;
+ while (!endOfSentence && (moreTokensAvailable = m_input.IncrementToken()))
+ {
+ termList.Add(termAtt.ToString());
+ endOfSentence = 0 != (flagsAtt.Flags & OpenNLPTokenizer.EOS_FLAG_BIT);
+ sentenceTokenAttrs.Add(m_input.CloneAttributes());
+ }
+ return termList.Count > 0 ? termList.ToArray() : null;
+ }
+
+ public override void Reset()
+ {
+ base.Reset();
+ moreTokensAvailable = true;
+ Clear();
+ }
+
+ private void Clear()
+ {
+ sentenceTokenAttrs.Clear();
+ tags = null;
+ tokenNum = 0;
+ }
+ }
+}
diff --git a/src/Lucene.Net.Analysis.OpenNLP/OpenNLPPOSFilterFactory.cs b/src/Lucene.Net.Analysis.OpenNLP/OpenNLPPOSFilterFactory.cs
new file mode 100644
index 0000000..57c3262
--- /dev/null
+++ b/src/Lucene.Net.Analysis.OpenNLP/OpenNLPPOSFilterFactory.cs
@@ -0,0 +1,79 @@
+using Lucene.Net.Analysis.OpenNlp.Tools;
+using Lucene.Net.Analysis.Util;
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+
+namespace Lucene.Net.Analysis.OpenNlp
+{
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ /// <summary>
+ /// Factory for <see cref="OpenNLPPOSFilter"/>.
+ /// <code>
+ /// <fieldType name="text_opennlp_pos" class="solr.TextField" positionIncrementGap="100">
+ /// <analyzer>
+ /// <tokenizer class="solr.OpenNLPTokenizerFactory" sentenceModel="filename" tokenizerModel="filename"/>
+ /// <filter class="solr.OpenNLPPOSFilterFactory" posTaggerModel="filename"/>
+ /// </analyzer>
+ /// </fieldType>
+ /// </code>
+ /// </summary>
+ /// <since>7.3.0</since>
+ public class OpenNLPPOSFilterFactory : TokenFilterFactory, IResourceLoaderAware
+ {
+ public const string POS_TAGGER_MODEL = "posTaggerModel";
+
+ private readonly string posTaggerModelFile;
+
+ public OpenNLPPOSFilterFactory(IDictionary<string, string> args)
+ : base(args)
+ {
+ posTaggerModelFile = Require(args, POS_TAGGER_MODEL);
+ if (args.Any())
+ {
+ throw new ArgumentException("Unknown parameters: " + args);
+ }
+ }
+
+ public override TokenStream Create(TokenStream input)
+ {
+ try
+ {
+ return new OpenNLPPOSFilter(input, OpenNLPOpsFactory.GetPOSTagger(posTaggerModelFile));
+ }
+ catch (IOException e)
+ {
+ throw new ArgumentException(e.ToString(), e);
+ }
+ }
+
+ public virtual void Inform(IResourceLoader loader)
+ {
+ try
+ { // load and register the read-only model in cache with file/resource name
+ OpenNLPOpsFactory.GetPOSTaggerModel(posTaggerModelFile, loader);
+ }
+ catch (IOException e)
+ {
+ throw new ArgumentException(e.ToString(), e);
+ }
+ }
+ }
+}
diff --git a/src/Lucene.Net.Analysis.OpenNLP/OpenNLPSentenceBreakIterator.cs b/src/Lucene.Net.Analysis.OpenNLP/OpenNLPSentenceBreakIterator.cs
new file mode 100644
index 0000000..426f7b6
--- /dev/null
+++ b/src/Lucene.Net.Analysis.OpenNLP/OpenNLPSentenceBreakIterator.cs
@@ -0,0 +1,277 @@
+using ICU4N.Support.Text;
+using ICU4N.Text;
+using Lucene.Net.Analysis.OpenNlp.Tools;
+using Lucene.Net.Analysis.Util;
+using opennlp.tools.util;
+using System;
+using System.Diagnostics;
+using System.Text;
+
+namespace Lucene.Net.Analysis.OpenNlp
+{
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ /// <summary>
+ /// A <see cref="BreakIterator"/> that splits sentences using an OpenNLP sentence chunking model.
+ /// </summary>
+ public sealed class OpenNLPSentenceBreakIterator : BreakIterator
+ {
+ private CharacterIterator text;
+ private int currentSentence;
+ private int[] sentenceStarts;
+ private NLPSentenceDetectorOp sentenceOp;
+
+ public OpenNLPSentenceBreakIterator(NLPSentenceDetectorOp sentenceOp)
+ {
+ this.sentenceOp = sentenceOp;
+ }
+
+ public override int Current => text.Index;
+
+ public override int First()
+ {
+ currentSentence = 0;
+ text.SetIndex(text.BeginIndex);
+ return Current;
+ }
+
+ public override int Last()
+ {
+ if (sentenceStarts.Length > 0)
+ {
+ currentSentence = sentenceStarts.Length - 1;
+ text.SetIndex(text.EndIndex);
+ }
+ else
+ { // there are no sentences; both the first and last positions are the begin index
+ currentSentence = 0;
+ text.SetIndex(text.BeginIndex);
+ }
+ return Current;
+ }
+
+ public override int Next()
+ {
+ if (text.Index == text.EndIndex || 0 == sentenceStarts.Length)
+ {
+ return Done;
+ }
+ else if (currentSentence < sentenceStarts.Length - 1)
+ {
+ text.SetIndex(sentenceStarts[++currentSentence]);
+ return Current;
+ }
+ else
+ {
+ return Last();
+ }
+ }
+
+ public override int Following(int pos)
+ {
+ if (pos < text.BeginIndex || pos > text.EndIndex)
+ {
+ throw new ArgumentException("offset out of bounds");
+ }
+ else if (0 == sentenceStarts.Length)
+ {
+ text.SetIndex(text.BeginIndex);
+ return Done;
+ }
+ else if (pos >= sentenceStarts[sentenceStarts.Length - 1])
+ {
+ // this conflicts with the javadocs, but matches actual behavior (Oracle has a bug in something)
+ // https://bugs.openjdk.java.net/browse/JDK-8015110
+ text.SetIndex(text.EndIndex);
+ currentSentence = sentenceStarts.Length - 1;
+ return Done;
+ }
+ else
+ { // there are at least two sentences
+ currentSentence = (sentenceStarts.Length - 1) / 2; // start search from the middle
+ MoveToSentenceAt(pos, 0, sentenceStarts.Length - 2);
+ text.SetIndex(sentenceStarts[++currentSentence]);
+ return Current;
+ }
+ }
+
+ /// <summary>Binary search over sentences</summary>
+ private void MoveToSentenceAt(int pos, int minSentence, int maxSentence)
+ {
+ if (minSentence != maxSentence)
+ {
+ if (pos < sentenceStarts[currentSentence])
+ {
+ int newMaxSentence = currentSentence - 1;
+ currentSentence = minSentence + (currentSentence - minSentence) / 2;
+ MoveToSentenceAt(pos, minSentence, newMaxSentence);
+ }
+ else if (pos >= sentenceStarts[currentSentence + 1])
+ {
+ int newMinSentence = currentSentence + 1;
+ currentSentence = maxSentence - (maxSentence - currentSentence) / 2;
+ MoveToSentenceAt(pos, newMinSentence, maxSentence);
+ }
+ }
+ else
+ {
+ Debug.Assert(currentSentence == minSentence);
+ Debug.Assert(pos >= sentenceStarts[currentSentence]);
+ Debug.Assert((currentSentence == sentenceStarts.Length - 1 && pos <= text.EndIndex)
+ || pos < sentenceStarts[currentSentence + 1]);
+ }
+ // we have arrived - nothing to do
+ }
+
+ public override int Previous()
+ {
+ if (text.Index == text.BeginIndex)
+ {
+ return Done;
+ }
+ else
+ {
+ if (0 == sentenceStarts.Length)
+ {
+ text.SetIndex(text.BeginIndex);
+ return Done;
+ }
+ if (text.Index == text.EndIndex)
+ {
+ text.SetIndex(sentenceStarts[currentSentence]);
+ }
+ else
+ {
+ text.SetIndex(sentenceStarts[--currentSentence]);
+ }
+ return Current;
+ }
+ }
+
+ public override int Preceding(int pos)
+ {
+ if (pos < text.BeginIndex || pos > text.EndIndex)
+ {
+ throw new ArgumentException("offset out of bounds");
+ }
+ else if (0 == sentenceStarts.Length)
+ {
+ text.SetIndex(text.BeginIndex);
+ currentSentence = 0;
+ return Done;
+ }
+ else if (pos < sentenceStarts[0])
+ {
+ // this conflicts with the javadocs, but matches actual behavior (Oracle has a bug in something)
+ // https://bugs.openjdk.java.net/browse/JDK-8015110
+ text.SetIndex(text.BeginIndex);
+ currentSentence = 0;
+ return Done;
+ }
+ else
+ {
+ currentSentence = sentenceStarts.Length / 2; // start search from the middle
+ MoveToSentenceAt(pos, 0, sentenceStarts.Length - 1);
+ if (0 == currentSentence)
+ {
+ text.SetIndex(text.BeginIndex);
+ return Done;
+ }
+ else
+ {
+ text.SetIndex(sentenceStarts[--currentSentence]);
+ return Current;
+ }
+ }
+ }
+
+ public override int Next(int n)
+ {
+ currentSentence += n;
+ if (n < 0)
+ {
+ if (text.Index == text.EndIndex)
+ {
+ ++currentSentence;
+ }
+ if (currentSentence < 0)
+ {
+ currentSentence = 0;
+ text.SetIndex(text.BeginIndex);
+ return Done;
+ }
+ else
+ {
+ text.SetIndex(sentenceStarts[currentSentence]);
+ }
+ }
+ else if (n > 0)
+ {
+ if (currentSentence >= sentenceStarts.Length)
+ {
+ currentSentence = sentenceStarts.Length - 1;
+ text.SetIndex(text.EndIndex);
+ return Done;
+ }
+ else
+ {
+ text.SetIndex(sentenceStarts[currentSentence]);
+ }
+ }
+ return Current;
+ }
+
+ public override CharacterIterator Text => text;
+
+ public override void SetText(CharacterIterator newText)
+ {
+ text = newText;
+ text.SetIndex(text.BeginIndex);
+ currentSentence = 0;
+ Span[] spans = sentenceOp.SplitSentences(CharacterIteratorToString());
+ sentenceStarts = new int[spans.Length];
+ for (int i = 0; i < spans.Length; ++i)
+ {
+ // Adjust start positions to match those of the passed-in CharacterIterator
+ sentenceStarts[i] = spans[i].getStart() + text.BeginIndex;
+ }
+ }
+
+ private string CharacterIteratorToString()
+ {
+ string fullText;
+ if (text is CharArrayIterator)
+ {
+ CharArrayIterator charArrayIterator = (CharArrayIterator)text;
+ fullText = new string(charArrayIterator.Text, charArrayIterator.Start, charArrayIterator.Length);
+ }
+ else
+ {
+ // TODO: is there a better way to extract full text from arbitrary CharacterIterators?
+ StringBuilder builder = new StringBuilder();
+ for (char ch = text.First(); ch != CharacterIterator.Done; ch = text.Next())
+ {
+ builder.Append(ch);
+ }
+ fullText = builder.ToString();
+ text.SetIndex(text.BeginIndex);
+ }
+ return fullText;
+ }
+ }
+}
diff --git a/src/Lucene.Net.Analysis.OpenNLP/OpenNLPTokenizer.cs b/src/Lucene.Net.Analysis.OpenNLP/OpenNLPTokenizer.cs
new file mode 100644
index 0000000..0a45b1e
--- /dev/null
+++ b/src/Lucene.Net.Analysis.OpenNLP/OpenNLPTokenizer.cs
@@ -0,0 +1,112 @@
+using Lucene.Net.Analysis.OpenNlp.Tools;
+using Lucene.Net.Analysis.TokenAttributes;
+using Lucene.Net.Analysis.Util;
+using opennlp.tools.util;
+using System;
+using System.IO;
+
+namespace Lucene.Net.Analysis.OpenNlp
+{
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ /// <summary>
+ /// Run OpenNLP SentenceDetector and <see cref="Tokenizer"/>.
+ /// The last token in each sentence is marked by setting the <see cref="EOS_FLAG_BIT"/> in the <see cref="IFlagsAttribute"/>;
+ /// following filters can use this information to apply operations to tokens one sentence at a time.
+ /// </summary>
+ public sealed class OpenNLPTokenizer : SegmentingTokenizerBase
+ {
+ public static int EOS_FLAG_BIT = 1;
+
+ private readonly ICharTermAttribute termAtt;
+ private readonly IFlagsAttribute flagsAtt;
+ private readonly IOffsetAttribute offsetAtt;
+
+ private Span[] termSpans = null;
+ private int termNum = 0;
+ private int sentenceStart = 0;
+
+ private NLPSentenceDetectorOp sentenceOp = null;
+ private NLPTokenizerOp tokenizerOp = null;
+
+ /// <summary>
+ /// Creates a new <see cref="OpenNLPTokenizer"/> </summary>
+ public OpenNLPTokenizer(TextReader reader, NLPSentenceDetectorOp sentenceOp, NLPTokenizerOp tokenizerOp) // LUCENENET 4.8.0 specific overload to default AttributeFactory
+ : this(AttributeFactory.DEFAULT_ATTRIBUTE_FACTORY, reader, sentenceOp, tokenizerOp)
+ {
+ }
+
+ public OpenNLPTokenizer(AttributeFactory factory, TextReader reader, NLPSentenceDetectorOp sentenceOp, NLPTokenizerOp tokenizerOp) // LUCENENET: Added reader param for compatibility with 4.8 - remove when upgrading
+ : base(factory, reader, new OpenNLPSentenceBreakIterator(sentenceOp))
+ {
+ if (sentenceOp == null || tokenizerOp == null)
+ {
+ throw new ArgumentException("OpenNLPTokenizer: both a Sentence Detector and a Tokenizer are required");
+ }
+ this.sentenceOp = sentenceOp;
+ this.tokenizerOp = tokenizerOp;
+ this.termAtt = AddAttribute<ICharTermAttribute>();
+ this.flagsAtt = AddAttribute<IFlagsAttribute>();
+ this.offsetAtt = AddAttribute<IOffsetAttribute>();
+ }
+
+ protected override void Dispose(bool disposing)
+ {
+ base.Dispose(disposing);
+ if (disposing)
+ {
+ termSpans = null;
+ termNum = sentenceStart = 0;
+ }
+ }
+
+ protected override void SetNextSentence(int sentenceStart, int sentenceEnd)
+ {
+ this.sentenceStart = sentenceStart;
+ string sentenceText = new string(m_buffer, sentenceStart, sentenceEnd - sentenceStart);
+ termSpans = tokenizerOp.GetTerms(sentenceText);
+ termNum = 0;
+ }
+
+ protected override bool IncrementWord()
+ {
+ if (termSpans == null || termNum == termSpans.Length)
+ {
+ return false;
+ }
+ ClearAttributes();
+ Span term = termSpans[termNum];
+ termAtt.CopyBuffer(m_buffer, sentenceStart + term.getStart(), term.length());
+ offsetAtt.SetOffset(CorrectOffset(m_offset + sentenceStart + term.getStart()),
+ CorrectOffset(m_offset + sentenceStart + term.getEnd()));
+ if (termNum == termSpans.Length - 1)
+ {
+ flagsAtt.Flags = flagsAtt.Flags | EOS_FLAG_BIT; // mark the last token in the sentence with EOS_FLAG_BIT
+ }
+ ++termNum;
+ return true;
+ }
+
+ public override void Reset()
+ {
+ base.Reset();
+ termSpans = null;
+ termNum = sentenceStart = 0;
+ }
+ }
+}
diff --git a/src/Lucene.Net.Analysis.OpenNLP/OpenNLPTokenizerFactory.cs b/src/Lucene.Net.Analysis.OpenNLP/OpenNLPTokenizerFactory.cs
new file mode 100644
index 0000000..d015d10
--- /dev/null
+++ b/src/Lucene.Net.Analysis.OpenNLP/OpenNLPTokenizerFactory.cs
@@ -0,0 +1,85 @@
+using Lucene.Net.Analysis.OpenNlp.Tools;
+using Lucene.Net.Analysis.Util;
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using AttributeFactory = Lucene.Net.Util.AttributeSource.AttributeFactory;
+
+namespace Lucene.Net.Analysis.OpenNlp
+{
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ /// <summary>
+ /// Factory for <see cref="OpenNLPTokenizer"/>.
+ /// <code>
+ /// <fieldType name="text_opennlp" class="solr.TextField" positionIncrementGap="100"
+ /// <analyzer>
+ /// <tokenizer class="solr.OpenNLPTokenizerFactory" sentenceModel="filename" tokenizerModel="filename"/>
+ /// </analyzer>
+ /// </fieldType>
+ /// </code>
+ /// </summary>
+ /// <since>7.3.0</since>
+ public class OpenNLPTokenizerFactory : TokenizerFactory, IResourceLoaderAware
+ {
+ public const string SENTENCE_MODEL = "sentenceModel";
+ public const string TOKENIZER_MODEL = "tokenizerModel";
+
+ private readonly string sentenceModelFile;
+ private readonly string tokenizerModelFile;
+
+ public OpenNLPTokenizerFactory(IDictionary<string, string> args)
+ : base(args)
+ {
+ sentenceModelFile = Require(args, SENTENCE_MODEL);
+ tokenizerModelFile = Require(args, TOKENIZER_MODEL);
+ if (args.Any())
+ {
+ throw new ArgumentException("Unknown parameters: " + args);
+ }
+ }
+
+ public override Tokenizer Create(AttributeFactory factory, TextReader reader)
+ {
+ try
+ {
+ NLPSentenceDetectorOp sentenceOp = OpenNLPOpsFactory.GetSentenceDetector(sentenceModelFile);
+ NLPTokenizerOp tokenizerOp = OpenNLPOpsFactory.GetTokenizer(tokenizerModelFile);
+ return new OpenNLPTokenizer(factory, reader, sentenceOp, tokenizerOp);
+ }
+ catch (IOException e)
+ {
+ throw new Exception(e.ToString(), e);
+ }
+ }
+
+ public virtual void Inform(IResourceLoader loader)
+ {
+ // register models in cache with file/resource names
+ if (sentenceModelFile != null)
+ {
+ OpenNLPOpsFactory.GetSentenceModel(sentenceModelFile, loader);
+ }
+ if (tokenizerModelFile != null)
+ {
+ OpenNLPOpsFactory.GetTokenizerModel(tokenizerModelFile, loader);
+ }
+ }
+ }
+}
diff --git a/src/Lucene.Net.Analysis.OpenNLP/Tools/NLPChunkerOp.cs b/src/Lucene.Net.Analysis.OpenNLP/Tools/NLPChunkerOp.cs
new file mode 100644
index 0000000..1c3ecc8
--- /dev/null
+++ b/src/Lucene.Net.Analysis.OpenNLP/Tools/NLPChunkerOp.cs
@@ -0,0 +1,47 @@
+using opennlp.tools.chunker;
+
+
+namespace Lucene.Net.Analysis.OpenNlp.Tools
+{
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ /// <summary>
+ /// Supply OpenNLP Chunking tool
+ /// Requires binary models from OpenNLP project on SourceForge.
+ /// </summary>
+ public class NLPChunkerOp
+ {
+ private ChunkerME chunker = null;
+
+ public NLPChunkerOp(ChunkerModel chunkerModel)
+ {
+ chunker = new ChunkerME(chunkerModel);
+ }
+
+ public virtual string[] GetChunks(string[] words, string[] tags, double[] probs)
+ {
+ lock (this)
+ {
+ string[] chunks = chunker.chunk(words, tags);
+ if (probs != null)
+ chunker.probs(probs);
+ return chunks;
+ }
+ }
+ }
+}
diff --git a/src/Lucene.Net.Analysis.OpenNLP/Tools/NLPLemmatizerOp.cs b/src/Lucene.Net.Analysis.OpenNLP/Tools/NLPLemmatizerOp.cs
new file mode 100644
index 0000000..56b1d2b
--- /dev/null
+++ b/src/Lucene.Net.Analysis.OpenNLP/Tools/NLPLemmatizerOp.cs
@@ -0,0 +1,93 @@
+using opennlp.tools.lemmatizer;
+using System.Diagnostics;
+using System.IO;
+
+namespace Lucene.Net.Analysis.OpenNlp.Tools
+{
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ /// <summary>
+ /// Supply OpenNLP Lemmatizer tools.
+ /// <para/>
+ /// Both a dictionary-based lemmatizer and a MaxEnt lemmatizer are supported.
+ /// If both are configured, the dictionary-based lemmatizer is tried first,
+ /// and then the MaxEnt lemmatizer is consulted for out-of-vocabulary tokens.
+ /// <para/>
+ /// The MaxEnt implementation requires binary models from OpenNLP project on SourceForge.
+ /// </summary>
+ public class NLPLemmatizerOp
+ {
+ private readonly DictionaryLemmatizer dictionaryLemmatizer;
+ private readonly LemmatizerME lemmatizerME;
+
+ public NLPLemmatizerOp(Stream dictionary, LemmatizerModel lemmatizerModel)
+ {
+ Debug.Assert(dictionary != null || lemmatizerModel != null, "At least one parameter must be non-null");
+ dictionaryLemmatizer = dictionary == null ? null : new DictionaryLemmatizer(new ikvm.io.InputStreamWrapper(dictionary));
+ lemmatizerME = lemmatizerModel == null ? null : new LemmatizerME(lemmatizerModel);
+ }
+
+ public virtual string[] Lemmatize(string[] words, string[] postags)
+ {
+ string[] lemmas = null;
+ string[] maxEntLemmas = null;
+ if (dictionaryLemmatizer != null)
+ {
+ lemmas = dictionaryLemmatizer.lemmatize(words, postags);
+ for (int i = 0; i < lemmas.Length; ++i)
+ {
+ if (lemmas[i].Equals("O"))
+ { // this word is not in the dictionary
+ if (lemmatizerME != null)
+ { // fall back to the MaxEnt lemmatizer if it's enabled
+ if (maxEntLemmas == null)
+ {
+ maxEntLemmas = lemmatizerME.lemmatize(words, postags);
+ }
+ if ("_".Equals(maxEntLemmas[i]))
+ {
+ lemmas[i] = words[i]; // put back the original word if no lemma is found
+ }
+ else
+ {
+ lemmas[i] = maxEntLemmas[i];
+ }
+ }
+ else
+ { // there is no MaxEnt lemmatizer
+ lemmas[i] = words[i]; // put back the original word if no lemma is found
+ }
+ }
+ }
+ }
+ else
+ { // there is only a MaxEnt lemmatizer
+ maxEntLemmas = lemmatizerME.lemmatize(words, postags);
+ for (int i = 0; i < maxEntLemmas.Length; ++i)
+ {
+ if ("_".Equals(maxEntLemmas[i]))
+ {
+ maxEntLemmas[i] = words[i]; // put back the original word if no lemma is found
+ }
+ }
+ lemmas = maxEntLemmas;
+ }
+ return lemmas;
+ }
+ }
+}
diff --git a/src/Lucene.Net.Analysis.OpenNLP/Tools/NLPNERTaggerOp.cs b/src/Lucene.Net.Analysis.OpenNLP/Tools/NLPNERTaggerOp.cs
new file mode 100644
index 0000000..e4b43ac
--- /dev/null
+++ b/src/Lucene.Net.Analysis.OpenNLP/Tools/NLPNERTaggerOp.cs
@@ -0,0 +1,61 @@
+using opennlp.tools.namefind;
+using opennlp.tools.util;
+
+namespace Lucene.Net.Analysis.OpenNlp.Tools
+{
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ /// <summary>
+ /// Supply OpenNLP Named Entity Resolution tool
+ /// Requires binary models from OpenNLP project on SourceForge.
+ /// <para/>
+ /// Usage: from <a href="http://opennlp.apache.org/docs/1.8.3/manual/opennlp.html#tools.namefind.recognition.api">
+ /// the OpenNLP documentation</a>:
+ /// <para/>
+ /// "The NameFinderME class is not thread safe, it must only be called from one thread.
+ /// To use multiple threads multiple NameFinderME instances sharing the same model instance
+ /// can be created. The input text should be segmented into documents, sentences and tokens.
+ /// To perform entity detection an application calls the find method for every sentence in
+ /// the document. After every document clearAdaptiveData must be called to clear the adaptive
+ /// data in the feature generators. Not calling clearAdaptiveData can lead to a sharp drop
+ /// in the detection rate after a few documents."
+ /// </summary>
+ public class NLPNERTaggerOp
+ {
+ private readonly TokenNameFinder nameFinder;
+
+ public NLPNERTaggerOp(TokenNameFinderModel model)
+ {
+ this.nameFinder = new NameFinderME(model);
+ }
+
+ public virtual Span[] GetNames(string[] words)
+ {
+ Span[] names = nameFinder.find(words);
+ return names;
+ }
+
+ public virtual void Reset()
+ {
+ lock (this)
+ {
+ nameFinder.clearAdaptiveData();
+ }
+ }
+ }
+}
diff --git a/src/Lucene.Net.Analysis.OpenNLP/Tools/NLPPOSTaggerOp.cs b/src/Lucene.Net.Analysis.OpenNLP/Tools/NLPPOSTaggerOp.cs
new file mode 100644
index 0000000..a69a7d0
--- /dev/null
+++ b/src/Lucene.Net.Analysis.OpenNLP/Tools/NLPPOSTaggerOp.cs
@@ -0,0 +1,43 @@
+using opennlp.tools.postag;
+
+namespace Lucene.Net.Analysis.OpenNlp.Tools
+{
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ /// <summary>
+ /// Supply OpenNLP Parts-Of-Speech Tagging tool.
+ /// Requires binary models from OpenNLP project on SourceForge.
+ /// </summary>
+ public class NLPPOSTaggerOp
+ {
+ private POSTagger tagger = null;
+
+ public NLPPOSTaggerOp(POSModel model)
+ {
+ tagger = new POSTaggerME(model);
+ }
+
+ public virtual string[] GetPOSTags(string[] words)
+ {
+ lock (this)
+ {
+ return tagger.tag(words);
+ }
+ }
+ }
+}
diff --git a/src/Lucene.Net.Analysis.OpenNLP/Tools/NLPSentenceDetectorOp.cs b/src/Lucene.Net.Analysis.OpenNLP/Tools/NLPSentenceDetectorOp.cs
new file mode 100644
index 0000000..ebca1c5
--- /dev/null
+++ b/src/Lucene.Net.Analysis.OpenNLP/Tools/NLPSentenceDetectorOp.cs
@@ -0,0 +1,58 @@
+using opennlp.tools.sentdetect;
+using opennlp.tools.util;
+
+namespace Lucene.Net.Analysis.OpenNlp.Tools
+{
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ /// <summary>
+ /// Supply OpenNLP Sentence Detector tool.
+ /// Requires binary models from OpenNLP project on SourceForge.
+ /// </summary>
+ public class NLPSentenceDetectorOp
+ {
+ private readonly SentenceDetectorME sentenceSplitter;
+
+ public NLPSentenceDetectorOp(SentenceModel model)
+ {
+ sentenceSplitter = new SentenceDetectorME(model);
+ }
+
+ public NLPSentenceDetectorOp()
+ {
+ sentenceSplitter = null;
+ }
+
+ public virtual Span[] SplitSentences(string line)
+ {
+ lock (this)
+ {
+ if (sentenceSplitter != null)
+ {
+ return sentenceSplitter.sentPosDetect(line);
+ }
+ else
+ {
+ Span[] shorty = new Span[1];
+ shorty[0] = new Span(0, line.Length);
+ return shorty;
+ }
+ }
+ }
+ }
+}
diff --git a/src/Lucene.Net.Analysis.OpenNLP/Tools/NLPTokenizerOp.cs b/src/Lucene.Net.Analysis.OpenNLP/Tools/NLPTokenizerOp.cs
new file mode 100644
index 0000000..1e96163
--- /dev/null
+++ b/src/Lucene.Net.Analysis.OpenNLP/Tools/NLPTokenizerOp.cs
@@ -0,0 +1,55 @@
+using opennlp.tools.tokenize;
+using opennlp.tools.util;
+
+namespace Lucene.Net.Analysis.OpenNlp.Tools
+{
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ /// <summary>
+ /// Supply OpenNLP Sentence Tokenizer tool.
+ /// Requires binary models from OpenNLP project on SourceForge.
+ /// </summary>
+ public class NLPTokenizerOp
+ {
+ private readonly TokenizerME tokenizer;
+
+ public NLPTokenizerOp(TokenizerModel model)
+ {
+ tokenizer = new TokenizerME(model);
+ }
+
+ public NLPTokenizerOp()
+ {
+ tokenizer = null;
+ }
+
+ public virtual Span[] GetTerms(string sentence)
+ {
+ lock (this)
+ {
+ if (tokenizer == null)
+ {
+ Span[] span1 = new Span[1];
+ span1[0] = new Span(0, sentence.Length);
+ return span1;
+ }
+ return tokenizer.tokenizePos(sentence);
+ }
+ }
+ }
+}
diff --git a/src/Lucene.Net.Analysis.OpenNLP/Tools/OpenNLPOpsFactory.cs b/src/Lucene.Net.Analysis.OpenNLP/Tools/OpenNLPOpsFactory.cs
new file mode 100644
index 0000000..c49e429
--- /dev/null
+++ b/src/Lucene.Net.Analysis.OpenNLP/Tools/OpenNLPOpsFactory.cs
@@ -0,0 +1,225 @@
+using Lucene.Net.Analysis.Util;
+using opennlp.tools.chunker;
+using opennlp.tools.lemmatizer;
+using opennlp.tools.namefind;
+using opennlp.tools.postag;
+using opennlp.tools.sentdetect;
+using opennlp.tools.tokenize;
+using System.Collections.Concurrent;
+using System.Collections.Generic;
+using System.Diagnostics;
+using System.IO;
+using System.Text;
+
+namespace Lucene.Net.Analysis.OpenNlp.Tools
+{
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ /// <summary>
+ /// Supply OpenNLP Named Entity Recognizer.
+ /// Cache model file objects. Assumes model files are thread-safe.
+ /// </summary>
+ public static class OpenNLPOpsFactory // LUCENENET: Made static because all members are static
+ {
+ private static readonly IDictionary<string, SentenceModel> sentenceModels = new ConcurrentDictionary<string, SentenceModel>();
+ private static readonly ConcurrentDictionary<string, TokenizerModel> tokenizerModels = new ConcurrentDictionary<string, TokenizerModel>();
+ private static readonly ConcurrentDictionary<string, POSModel> posTaggerModels = new ConcurrentDictionary<string, POSModel>();
+ private static readonly ConcurrentDictionary<string, ChunkerModel> chunkerModels = new ConcurrentDictionary<string, ChunkerModel>();
+ private static readonly IDictionary<string, TokenNameFinderModel> nerModels = new ConcurrentDictionary<string, TokenNameFinderModel>();
+ private static readonly IDictionary<string, LemmatizerModel> lemmatizerModels = new ConcurrentDictionary<string, LemmatizerModel>();
+ private static readonly IDictionary<string, string> lemmaDictionaries = new ConcurrentDictionary<string, string>();
+
+ public static NLPSentenceDetectorOp GetSentenceDetector(string modelName)
+ {
+ if (modelName != null)
+ {
+ sentenceModels.TryGetValue(modelName, out SentenceModel model);
+ return new NLPSentenceDetectorOp(model);
+ }
+ else
+ {
+ return new NLPSentenceDetectorOp();
+ }
+ }
+
+ public static SentenceModel GetSentenceModel(string modelName, IResourceLoader loader)
+ {
+ //SentenceModel model = sentenceModels.get(modelName);
+ sentenceModels.TryGetValue(modelName, out SentenceModel model);
+ if (model == null)
+ {
+ using (Stream resource = loader.OpenResource(modelName))
+ {
+ model = new SentenceModel(new ikvm.io.InputStreamWrapper(resource));
+ }
+ sentenceModels[modelName] = model;
+ }
+ return model;
+ }
+
+ public static NLPTokenizerOp GetTokenizer(string modelName)
+ {
+ if (modelName == null)
+ {
+ return new NLPTokenizerOp();
+ }
+ else
+ {
+ TokenizerModel model = tokenizerModels[modelName];
+ return new NLPTokenizerOp(model);
+ }
+ }
+
+ public static TokenizerModel GetTokenizerModel(string modelName, IResourceLoader loader)
+ {
+ tokenizerModels.TryGetValue(modelName, out TokenizerModel model);
+ if (model == null)
+ {
+ using (Stream resource = loader.OpenResource(modelName))
+ {
+ model = new TokenizerModel(new ikvm.io.InputStreamWrapper(resource));
+ }
+ tokenizerModels[modelName] = model;
+ }
+ return model;
+ }
+
+ public static NLPPOSTaggerOp GetPOSTagger(string modelName)
+ {
+ posTaggerModels.TryGetValue(modelName, out POSModel model);
+ return new NLPPOSTaggerOp(model);
+ }
+
+ public static POSModel GetPOSTaggerModel(string modelName, IResourceLoader loader)
+ {
+ posTaggerModels.TryGetValue(modelName, out POSModel model);
+ if (model == null)
+ {
+ using (Stream resource = loader.OpenResource(modelName))
+ {
+ model = new POSModel(new ikvm.io.InputStreamWrapper(resource));
+ }
+ posTaggerModels[modelName] = model;
+ }
+ return model;
+ }
+
+ public static NLPChunkerOp GetChunker(string modelName)
+ {
+ chunkerModels.TryGetValue(modelName, out ChunkerModel model);
+ return new NLPChunkerOp(model);
+ }
+
+ public static ChunkerModel GetChunkerModel(string modelName, IResourceLoader loader)
+ {
+ chunkerModels.TryGetValue(modelName, out ChunkerModel model);
+ if (model == null)
+ {
+ using (Stream resource = loader.OpenResource(modelName))
+ {
+ model = new ChunkerModel(new ikvm.io.InputStreamWrapper(resource));
+ }
+ chunkerModels[modelName] = model;
+ }
+ return model;
+ }
+
+ public static NLPNERTaggerOp GetNERTagger(string modelName)
+ {
+ nerModels.TryGetValue(modelName, out TokenNameFinderModel model);
+ return new NLPNERTaggerOp(model);
+ }
+
+ public static TokenNameFinderModel GetNERTaggerModel(string modelName, IResourceLoader loader)
+ {
+ nerModels.TryGetValue(modelName, out TokenNameFinderModel model);
+ if (model == null)
+ {
+ using (Stream resource = loader.OpenResource(modelName))
+ {
+ model = new TokenNameFinderModel(new ikvm.io.InputStreamWrapper(resource));
+ }
+ nerModels[modelName] = model;
+ }
+ return model;
+ }
+
+ public static NLPLemmatizerOp GetLemmatizer(string dictionaryFile, string lemmatizerModelFile)
+ {
+ Debug.Assert(dictionaryFile != null || lemmatizerModelFile != null, "At least one parameter must be non-null");
+ Stream dictionaryInputStream = null;
+ if (dictionaryFile != null)
+ {
+ string dictionary = lemmaDictionaries[dictionaryFile];
+ dictionaryInputStream = new MemoryStream(Encoding.UTF8.GetBytes(dictionary));
+ }
+ LemmatizerModel lemmatizerModel = lemmatizerModelFile == null ? null : lemmatizerModels[lemmatizerModelFile];
+ return new NLPLemmatizerOp(dictionaryInputStream, lemmatizerModel);
+ }
+
+ public static string GetLemmatizerDictionary(string dictionaryFile, IResourceLoader loader)
+ {
+ lemmaDictionaries.TryGetValue(dictionaryFile, out string dictionary);
+ if (dictionary == null)
+ {
+ using (TextReader reader = new StreamReader(loader.OpenResource(dictionaryFile), Encoding.UTF8))
+ {
+ StringBuilder builder = new StringBuilder();
+ char[] chars = new char[8092];
+ int numRead = 0;
+ do
+ {
+ numRead = reader.Read(chars, 0, chars.Length);
+ if (numRead > 0)
+ {
+ builder.Append(chars, 0, numRead);
+ }
+ } while (numRead > 0);
+ dictionary = builder.ToString();
+ lemmaDictionaries[dictionaryFile] = dictionary;
+ }
+ }
+ return dictionary;
+ }
+
+ public static LemmatizerModel GetLemmatizerModel(string modelName, IResourceLoader loader)
+ {
+ lemmatizerModels.TryGetValue(modelName, out LemmatizerModel model);
+ if (model == null)
+ {
+ using (Stream resource = loader.OpenResource(modelName))
+ {
+ model = new LemmatizerModel(new ikvm.io.InputStreamWrapper(resource));
+ }
+ lemmatizerModels[modelName] = model;
+ }
+ return model;
+ }
+
+ // keeps unit test from blowing out memory
+ public static void ClearModels()
+ {
+ sentenceModels.Clear();
+ tokenizerModels.Clear();
+ posTaggerModels.Clear();
+ chunkerModels.Clear();
+ nerModels.Clear();
+ lemmaDictionaries.Clear();
+ }
+ }
+}
diff --git a/src/Lucene.Net.TestFramework/Analysis/BaseTokenStreamTestCase.cs b/src/Lucene.Net.TestFramework/Analysis/BaseTokenStreamTestCase.cs
index 483036d..dd0b059 100644
--- a/src/Lucene.Net.TestFramework/Analysis/BaseTokenStreamTestCase.cs
+++ b/src/Lucene.Net.TestFramework/Analysis/BaseTokenStreamTestCase.cs
@@ -11,9 +11,11 @@ using System.IO;
using System.Linq;
using System.Text;
using System.Threading;
+using AttributeFactory = Lucene.Net.Util.AttributeSource.AttributeFactory;
using Assert = Lucene.Net.TestFramework.Assert;
using AssertionError = Lucene.Net.Diagnostics.AssertionException;
using Console = Lucene.Net.Support.SystemConsole;
+using Lucene.Net.Util;
namespace Lucene.Net.Analysis
{
@@ -139,7 +141,7 @@ namespace Lucene.Net.Analysis
// arriving to pos Y have the same endOffset)
// - offsets only move forwards (startOffset >=
// lastStartOffset)
- public static void AssertTokenStreamContents(TokenStream ts, string[] output, int[] startOffsets, int[] endOffsets, string[] types, int[] posIncrements, int[] posLengths, int? finalOffset, int? finalPosInc, bool[] keywordAtts, bool offsetsAreCorrect)
+ public static void AssertTokenStreamContents(TokenStream ts, string[] output, int[] startOffsets, int[] endOffsets, string[] types, int[] posIncrements, int[] posLengths, int? finalOffset, int? finalPosInc, bool[] keywordAtts, bool offsetsAreCorrect, byte[][] payloads)
{
// LUCENENET: Bug fix: NUnit throws an exception when something fails.
// This causes Dispose() to be skipped and it pollutes other tests indicating false negatives.
@@ -192,6 +194,17 @@ namespace Lucene.Net.Analysis
keywordAtt = ts.GetAttribute<IKeywordAttribute>();
}
+ // *********** From Lucene 8.2.0 **************
+
+ IPayloadAttribute payloadAtt = null;
+ if (payloads != null)
+ {
+ Assert.IsTrue(ts.HasAttribute<IPayloadAttribute>(), "has no PayloadAttribute");
+ payloadAtt = ts.GetAttribute<IPayloadAttribute>();
+ }
+
+ // *********** End From Lucene 8.2.0 **************
+
// Maps position to the start/end offset:
IDictionary<int?, int?> posToStartOffset = new Dictionary<int?, int?>();
IDictionary<int?, int?> posToEndOffset = new Dictionary<int?, int?>();
@@ -224,6 +237,12 @@ namespace Lucene.Net.Analysis
{
keywordAtt.IsKeyword = (i & 1) == 0;
}
+ // *********** From Lucene 8.2.0 **************
+ if (payloadAtt != null)
+ {
+ payloadAtt.Payload = new BytesRef(new byte[] { 0x00, unchecked((byte)-0x21), 0x12, unchecked((byte)-0x43), 0x24 });
+ }
+ // *********** End From Lucene 8.2.0 **************
bool reset = checkClearAtt.GetAndResetClearCalled(); // reset it, because we called clearAttribute() before
Assert.IsTrue(ts.IncrementToken(), "token " + i + " does not exist");
@@ -254,6 +273,20 @@ namespace Lucene.Net.Analysis
{
Assert.AreEqual(keywordAtts[i], keywordAtt.IsKeyword, "keywordAtt " + i);
}
+ // *********** From Lucene 8.2.0 **************
+ if (payloads != null)
+ {
+ if (payloads[i] != null)
+ {
+ Assert.AreEqual(new BytesRef(payloads[i]), payloadAtt.Payload, "payloads " + i);
+ }
+ else
+ {
+ Assert.IsNull(payloads[i], "payloads " + i);
+ }
+ }
+ // *********** End From Lucene 8.2.0 **************
+
// we can enforce some basic things about a few attributes even if the caller doesn't check:
if (offsetAtt != null)
@@ -394,7 +427,7 @@ namespace Lucene.Net.Analysis
public static void AssertTokenStreamContents(TokenStream ts, string[] output, int[] startOffsets, int[] endOffsets, string[] types, int[] posIncrements, int[] posLengths, int? finalOffset, bool[] keywordAtts, bool offsetsAreCorrect)
{
- AssertTokenStreamContents(ts, output, startOffsets, endOffsets, types, posIncrements, posLengths, finalOffset, null, null, offsetsAreCorrect);
+ AssertTokenStreamContents(ts, output, startOffsets, endOffsets, types, posIncrements, posLengths, finalOffset, null, null, offsetsAreCorrect, null);
}
public static void AssertTokenStreamContents(TokenStream ts, string[] output, int[] startOffsets, int[] endOffsets, string[] types, int[] posIncrements, int[] posLengths, int? finalOffset, bool offsetsAreCorrect)
@@ -475,6 +508,13 @@ namespace Lucene.Net.Analysis
AssertTokenStreamContents(a.GetTokenStream("dummy", new StringReader(input)), output, startOffsets, endOffsets, types, posIncrements, posLengths, input.Length, offsetsAreCorrect);
}
+ // LUCENENET: Overload from Lucene 8.2.0
+ public static void AssertAnalyzesTo(Analyzer a, string input, string[] output, int[] startOffsets, int[] endOffsets, string[] types, int[] posIncrements, int[] posLengths, bool graphOffsetsAreCorrect, byte[][] payloads)
+ {
+ CheckResetException(a, input);
+ AssertTokenStreamContents(a.GetTokenStream("dummy", input), output, startOffsets, endOffsets, types, posIncrements, posLengths, input.Length, null, null, graphOffsetsAreCorrect, payloads);
+ }
+
public static void AssertAnalyzesTo(Analyzer a, string input, string[] output)
{
AssertAnalyzesTo(a, input, output, null, null, null, null, null);
@@ -1266,5 +1306,41 @@ namespace Lucene.Net.Analysis
}
return ret;
}
+
+ // *********** From Lucene 8.2.0 **************
+
+ /// <summary>Returns a random <see cref="AttributeFactory"/> impl</summary>
+ public static AttributeFactory NewAttributeFactory(Random random)
+ {
+ switch (random.nextInt(2))
+ {
+ case 0:
+ return Token.TOKEN_ATTRIBUTE_FACTORY;
+ case 1:
+ return AttributeFactory.DEFAULT_ATTRIBUTE_FACTORY;
+ default:
+ throw new AssertionError("Please fix the Random.nextInt() call above");
+ }
+
+ //switch (random.nextInt(3))
+ //{
+ // case 0:
+ // return TokenStream.DEFAULT_TOKEN_ATTRIBUTE_FACTORY;
+ // case 1:
+ // return Token.TOKEN_ATTRIBUTE_FACTORY;
+ // case 2:
+ // return AttributeFactory.DEFAULT_ATTRIBUTE_FACTORY;
+ // default:
+ // throw new AssertionError("Please fix the Random.nextInt() call above");
+ //}
+ }
+
+ /// <summary>Returns a random <see cref="AttributeFactory"/> impl</summary>
+ public static AttributeFactory NewAttributeFactory()
+ {
+ return NewAttributeFactory(Random);
+ }
+
+ // *********** End From Lucene 8.2.0 **************
}
}
\ No newline at end of file
diff --git a/src/Lucene.Net.Tests.Analysis.Common/Analysis/Core/TestStopFilter.cs b/src/Lucene.Net.Tests.Analysis.Common/Analysis/Core/TestStopFilter.cs
index 0a62869..8e10148 100644
--- a/src/Lucene.Net.Tests.Analysis.Common/Analysis/Core/TestStopFilter.cs
+++ b/src/Lucene.Net.Tests.Analysis.Common/Analysis/Core/TestStopFilter.cs
@@ -127,7 +127,7 @@ namespace Lucene.Net.Analysis.Core
{
CharArraySet stopSet = StopFilter.MakeStopSet(TEST_VERSION_CURRENT, "of");
StopFilter stpf = new StopFilter(TEST_VERSION_CURRENT, new MockTokenizer(new StringReader("test of"), MockTokenizer.WHITESPACE, false), stopSet);
- AssertTokenStreamContents(stpf, new string[] { "test" }, new int[] { 0 }, new int[] { 4 }, null, new int[] { 1 }, null, 7, 1, null, true);
+ AssertTokenStreamContents(stpf, new string[] { "test" }, new int[] { 0 }, new int[] { 4 }, null, new int[] { 1 }, null, 7, 1, null, true, null);
}
private void DoTestStopPositons(StopFilter stpf, bool enableIcrements)
diff --git a/src/Lucene.Net.Tests.Analysis.OpenNLP/Lucene.Net.Tests.Analysis.OpenNLP.csproj b/src/Lucene.Net.Tests.Analysis.OpenNLP/Lucene.Net.Tests.Analysis.OpenNLP.csproj
new file mode 100644
index 0000000..f72a2a7
--- /dev/null
+++ b/src/Lucene.Net.Tests.Analysis.OpenNLP/Lucene.Net.Tests.Analysis.OpenNLP.csproj
@@ -0,0 +1,71 @@
+<Project Sdk="Microsoft.NET.Sdk">
+
+ <Import Project="$(SolutionDir)TestTargetFramework.props" />
+
+ <PropertyGroup>
+ <TargetFrameworks></TargetFrameworks>
+ <TargetFramework>net451</TargetFramework>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+
+ <AssemblyTitle>Lucene.Net.Tests.Analysis.OpenNLP</AssemblyTitle>
+ <RootNamespace>Lucene.Net.Analysis.OpenNlp</RootNamespace>
+ <GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
+ <RuntimeIdentifiers>win7-x86;win7-x64</RuntimeIdentifiers>
+ </PropertyGroup>
+
+ <!-- OpenNLP is not strong-named -->
+ <PropertyGroup Label="Assembly Signing">
+ <SignAssembly>false</SignAssembly>
+ </PropertyGroup>
+
+ <ItemGroup>
+ <None Remove="en-test-chunker.bin" />
+ <None Remove="en-test-lemmas.dict" />
+ <None Remove="en-test-lemmatizer.bin" />
+ <None Remove="en-test-ner.bin" />
+ <None Remove="en-test-pos-maxent.bin" />
+ <None Remove="en-test-sent.bin" />
+ <None Remove="en-test-tokenizer.bin" />
+ </ItemGroup>
+
+ <ItemGroup>
+ <EmbeddedResource Include="en-test-chunker.bin" />
+ <EmbeddedResource Include="en-test-lemmas.dict" />
+ <EmbeddedResource Include="en-test-lemmatizer.bin" />
+ <EmbeddedResource Include="en-test-ner.bin" />
+ <EmbeddedResource Include="en-test-pos-maxent.bin" />
+ <EmbeddedResource Include="en-test-sent.bin" />
+ <EmbeddedResource Include="en-test-tokenizer.bin" />
+ </ItemGroup>
+
+ <ItemGroup>
+ <ProjectReference Include="..\Lucene.Net.Analysis.OpenNLP\Lucene.Net.Analysis.OpenNLP.csproj" />
+ <ProjectReference Include="..\Lucene.Net.TestFramework.NUnit\Lucene.Net.TestFramework.NUnit.csproj" />
+ </ItemGroup>
+
+ <Import Project="$(SolutionDir)build/TestReferences.Common.targets" />
+
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp2.1' ">
+ <DebugType>portable</DebugType>
+ </PropertyGroup>
+
+ <ItemGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PackageReference Include="System.Net.Primitives" Version="$(SystemNetPrimitivesPackageVersion)" />
+ </ItemGroup>
+
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <DefineConstants>$(DefineConstants);NETSTANDARD1_6</DefineConstants>
+ <DebugType>portable</DebugType>
+ </PropertyGroup>
+
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'net451' ">
+ <DefineConstants>$(DefineConstants);FEATURE_SERIALIZABLE_EXCEPTIONS</DefineConstants>
+ <DebugType>full</DebugType>
+ </PropertyGroup>
+
+ <ItemGroup Condition=" '$(TargetFramework)' == 'net451' ">
+ <Reference Include="System" />
+ <Reference Include="Microsoft.CSharp" />
+ </ItemGroup>
+
+</Project>
diff --git a/src/Lucene.Net.Tests.Analysis.OpenNLP/TestOpenNLPChunkerFilterFactory.cs b/src/Lucene.Net.Tests.Analysis.OpenNLP/TestOpenNLPChunkerFilterFactory.cs
new file mode 100644
index 0000000..0a0afc3
--- /dev/null
+++ b/src/Lucene.Net.Tests.Analysis.OpenNLP/TestOpenNLPChunkerFilterFactory.cs
@@ -0,0 +1,122 @@
+using Lucene.Net.Analysis;
+using Lucene.Net.Analysis.Payloads;
+using Lucene.Net.Analysis.Util;
+using NUnit.Framework;
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Text;
+
+namespace Lucene.Net.Analysis.OpenNlp
+{
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ /// <summary>
+ /// Needs the OpenNLP Tokenizer because it creates full streams of punctuation.
+ /// Needs the OpenNLP POS tagger for the POS tags.
+ /// <para/>
+ /// Tagging models are created from tiny test data in opennlp/tools/test-model-data/ and are not very accurate.
+ /// </summary>
+ public class TestOpenNLPChunkerFilterFactory : BaseTokenStreamTestCase
+ {
+ private const String SENTENCES = "Sentence number 1 has 6 words. Sentence number 2, 5 words.";
+ private static readonly String[] SENTENCES_punc
+ = { "Sentence", "number", "1", "has", "6", "words", ".", "Sentence", "number", "2", ",", "5", "words", "." };
+ private static readonly int[] SENTENCES_startOffsets = { 0, 9, 16, 18, 22, 24, 29, 31, 40, 47, 48, 50, 52, 57 };
+ private static readonly int[] SENTENCES_endOffsets = { 8, 15, 17, 21, 23, 29, 30, 39, 46, 48, 49, 51, 57, 58 };
+ private static readonly String[] SENTENCES_chunks
+ = { "B-NP", "I-NP", "I-NP", "B-VP", "B-NP", "I-NP", "O", "B-NP", "I-NP", "I-NP", "O", "B-NP", "I-NP", "O" };
+
+ private const String sentenceModelFile = "en-test-sent.bin";
+ private const String tokenizerModelFile = "en-test-tokenizer.bin";
+ private const String posTaggerModelFile = "en-test-pos-maxent.bin";
+ private const String chunkerModelFile = "en-test-chunker.bin";
+
+ private static byte[][] ToPayloads(params string[] strings)
+ {
+ return strings.Select(s => s == null ? null : Encoding.UTF8.GetBytes(s)).ToArray();
+ }
+
+ [Test]
+ public void TestBasic()
+ {
+ // CustomAnalyzer analyzer = CustomAnalyzer.builder(new ClasspathResourceLoader(getClass()))
+ //.withTokenizer("opennlp", "tokenizerModel", tokenizerModelFile, "sentenceModel", sentenceModelFile)
+ //.addTokenFilter("opennlpPOS", "posTaggerModel", posTaggerModelFile)
+ //.addTokenFilter("opennlpChunker", "chunkerModel", chunkerModelFile)
+ //.build();
+
+ Analyzer analyzer = Analyzer.NewAnonymous(createComponents: (fieldName, reader) =>
+ {
+ var loader = new ClasspathResourceLoader(GetType());
+
+ var opennlpFactory = new OpenNLPTokenizerFactory(new Dictionary<string, string> { { "tokenizerModel", tokenizerModelFile }, { "sentenceModel", sentenceModelFile } });
+ opennlpFactory.Inform(loader);
+ var opennlp = opennlpFactory.Create(NewAttributeFactory(), reader); //new OpenNLPTokenizer(reader, new Tools.NLPSentenceDetectorOp(sentenceModelFile), new Tools.NLPTokenizerOp(tokenizerModelFile));
+
+ var opennlpPOSFilterFactory = new OpenNLPPOSFilterFactory(new Dictionary<string, string> { { "posTaggerModel", posTaggerModelFile } });
+ opennlpPOSFilterFactory.Inform(loader);
+ var opennlpPOSFilter = opennlpPOSFilterFactory.Create(opennlp); //new OpenNLPPOSFilter(opennlp, new Tools.NLPPOSTaggerOp(posTaggerModelFile));
+
+ var opennlpChunkerFilterFactory = new OpenNLPChunkerFilterFactory(new Dictionary<string, string> { { "chunkerModel", chunkerModelFile } });
+ opennlpChunkerFilterFactory.Inform(loader);
+ var opennlpChunkerFilter = opennlpChunkerFilterFactory.Create(opennlpPOSFilter); //new OpenNLPChunkerFilter(filter1, new Tools.NLPChunkerOp(chunkerModelFile));
+
+ return new TokenStreamComponents(opennlp, opennlpChunkerFilter);
+ });
+
+ AssertAnalyzesTo(analyzer, SENTENCES, SENTENCES_punc, SENTENCES_startOffsets, SENTENCES_endOffsets,
+ SENTENCES_chunks, null, null, true);
+ }
+
+ [Test]
+ public void TestPayloads()
+ {
+ //CustomAnalyzer analyzer = CustomAnalyzer.builder(new ClasspathResourceLoader(getClass()))
+ //.withTokenizer("opennlp", "tokenizerModel", tokenizerModelFile, "sentenceModel", sentenceModelFile)
+ //.addTokenFilter("opennlpPOS", "posTaggerModel", posTaggerModelFile)
+ //.addTokenFilter("opennlpChunker", "chunkerModel", chunkerModelFile)
+ //.addTokenFilter(TypeAsPayloadTokenFilterFactory.class)
+ //.build();
+
+ Analyzer analyzer = Analyzer.NewAnonymous(createComponents: (fieldName, reader) =>
+ {
+ var loader = new ClasspathResourceLoader(GetType());
+
+ var opennlpFactory = new OpenNLPTokenizerFactory(new Dictionary<string, string> { { "tokenizerModel", tokenizerModelFile }, { "sentenceModel", sentenceModelFile } });
+ opennlpFactory.Inform(loader);
+ var opennlp = opennlpFactory.Create(NewAttributeFactory(), reader); //new OpenNLPTokenizer(reader, new Tools.NLPSentenceDetectorOp(sentenceModelFile), new Tools.NLPTokenizerOp(tokenizerModelFile));
+
+ var opennlpPOSFilterFactory = new OpenNLPPOSFilterFactory(new Dictionary<string, string> { { "posTaggerModel", posTaggerModelFile } });
+ opennlpPOSFilterFactory.Inform(loader);
+ var opennlpPOSFilter = opennlpPOSFilterFactory.Create(opennlp); //new OpenNLPPOSFilter(opennlp, new Tools.NLPPOSTaggerOp(posTaggerModelFile));
+
+ var opennlpChunkerFilterFactory = new OpenNLPChunkerFilterFactory(new Dictionary<string, string> { { "chunkerModel", chunkerModelFile } });
+ opennlpChunkerFilterFactory.Inform(loader);
+ var opennlpChunkerFilter = opennlpChunkerFilterFactory.Create(opennlpPOSFilter); //new OpenNLPChunkerFilter(filter1, new Tools.NLPChunkerOp(chunkerModelFile));
+
+ var typeAsPayloadFilterFactory = new TypeAsPayloadTokenFilterFactory(new Dictionary<string, string>());
+ var typeAsPayloadFilter = typeAsPayloadFilterFactory.Create(opennlpChunkerFilter);
+
+ return new TokenStreamComponents(opennlp, typeAsPayloadFilter);
+ });
+ AssertAnalyzesTo(analyzer, SENTENCES, SENTENCES_punc, SENTENCES_startOffsets, SENTENCES_endOffsets,
+ null, null, null, true, ToPayloads(SENTENCES_chunks));
+ }
+ }
+}
diff --git a/src/Lucene.Net.Tests.Analysis.OpenNLP/TestOpenNLPLemmatizerFilterFactory.cs b/src/Lucene.Net.Tests.Analysis.OpenNLP/TestOpenNLPLemmatizerFilterFactory.cs
new file mode 100644
index 0000000..f249131
--- /dev/null
+++ b/src/Lucene.Net.Tests.Analysis.OpenNLP/TestOpenNLPLemmatizerFilterFactory.cs
@@ -0,0 +1,372 @@
+using Lucene.Net.Analysis.Miscellaneous;
+using Lucene.Net.Analysis.Util;
+using NUnit.Framework;
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Text;
+using System.Threading.Tasks;
+
+namespace Lucene.Net.Analysis.OpenNlp
+{
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ public class TestOpenNLPLemmatizerFilterFactory : BaseTokenStreamTestCase
+ {
+ private const String SENTENCE = "They sent him running in the evening.";
+ private static readonly String[] SENTENCE_dict_punc = { "they", "send", "he", "run", "in", "the", "evening", "." };
+ private static readonly String[] SENTENCE_maxent_punc = { "they", "send", "he", "runn", "in", "the", "evening", "." };
+ private static readonly String[] SENTENCE_posTags = { "NNP", "VBD", "PRP", "VBG", "IN", "DT", "NN", "." };
+
+ private static readonly String SENTENCES = "They sent him running in the evening. He did not come back.";
+ private static readonly String[] SENTENCES_dict_punc
+ = { "they", "send", "he", "run", "in", "the", "evening", ".", "he", "do", "not", "come", "back", "." };
+ private static readonly String[] SENTENCES_maxent_punc
+ = { "they", "send", "he", "runn", "in", "the", "evening", ".", "he", "do", "not", "come", "back", "." };
+ private static readonly String[] SENTENCES_posTags
+ = { "NNP", "VBD", "PRP", "VBG", "IN", "DT", "NN", ".", "PRP", "VBD", "RB", "VB", "RB", "." };
+
+ private static readonly String SENTENCE_both = "Konstantin Kalashnitsov constantly caliphed.";
+ private static readonly String[] SENTENCE_both_punc
+ = { "konstantin", "kalashnitsov", "constantly", "caliph", "." };
+ private static readonly String[] SENTENCE_both_posTags
+ = { "IN", "JJ", "NN", "VBN", "." };
+
+ private const String SENTENCES_both = "Konstantin Kalashnitsov constantly caliphed. Coreena could care, completely.";
+ private static readonly String[] SENTENCES_both_punc
+ = { "konstantin", "kalashnitsov", "constantly", "caliph", ".", "coreena", "could", "care", ",", "completely", "." };
+ private static readonly String[] SENTENCES_both_posTags
+ = { "IN", "JJ", "NN", "VBN", ".", "NNP", "VBN", "NN", ",", "NN", "." };
+
+ private static readonly String[] SENTENCES_dict_keep_orig_punc
+ = { "They", "they", "sent", "send", "him", "he", "running", "run", "in", "the", "evening", ".", "He", "he", "did", "do", "not", "come", "back", "." };
+ private static readonly String[] SENTENCES_max_ent_keep_orig_punc
+ = { "They", "they", "sent", "send", "him", "he", "running", "runn", "in", "the", "evening", ".", "He", "he", "did", "do", "not", "come", "back", "." };
+ private static readonly String[] SENTENCES_keep_orig_posTags
+ = { "NNP", "NNP", "VBD", "VBD", "PRP", "PRP", "VBG", "VBG", "IN", "DT", "NN", ".", "PRP", "PRP", "VBD", "VBD", "RB", "VB", "RB", "." };
+
+ private static readonly String[] SENTENCES_both_keep_orig_punc
+ = { "Konstantin", "konstantin", "Kalashnitsov", "kalashnitsov", "constantly", "caliphed", "caliph", ".", "Coreena", "coreena", "could", "care", ",", "completely", "." };
+ private static readonly String[] SENTENCES_both_keep_orig_posTags
+ = { "IN", "IN", "JJ", "JJ", "NN", "VBN", "VBN", ".", "NNP", "NNP", "VBN", "NN", ",", "NN", "." };
+
+
+ private const String tokenizerModelFile = "en-test-tokenizer.bin";
+ private const String sentenceModelFile = "en-test-sent.bin";
+ private const String posTaggerModelFile = "en-test-pos-maxent.bin";
+ private const String lemmatizerModelFile = "en-test-lemmatizer.bin";
+ private const String lemmatizerDictFile = "en-test-lemmas.dict";
+
+ [Test]
+ public void Test1SentenceDictionaryOnly()
+ {
+ Analyzer analyzer = Analyzer.NewAnonymous(createComponents: (fieldname, reader) =>
+ {
+ var loader = new ClasspathResourceLoader(GetType());
+
+ var opennlpFactory = new OpenNLPTokenizerFactory(new Dictionary<string, string> { { "tokenizerModel", tokenizerModelFile }, { "sentenceModel", sentenceModelFile } });
+ opennlpFactory.Inform(loader);
+ var opennlp = opennlpFactory.Create(reader);
+
+ var opennlpPOSFilterFactory = new OpenNLPPOSFilterFactory(new Dictionary<string, string> { { "posTaggerModel", posTaggerModelFile } });
+ opennlpPOSFilterFactory.Inform(loader);
+ var opennlpPOSFilter = opennlpPOSFilterFactory.Create(opennlp);
+
+ var opennlpLemmatizerFilterFactory = new OpenNLPLemmatizerFilterFactory(new Dictionary<string, string> { { "dictionary", lemmatizerDictFile } });
+ opennlpLemmatizerFilterFactory.Inform(loader);
+ var opennlpLemmatizerFilter = opennlpLemmatizerFilterFactory.Create(opennlpPOSFilter);
+
+ return new TokenStreamComponents(opennlp, opennlpLemmatizerFilter);
+ });
+ //CustomAnalyzer analyzer = CustomAnalyzer.builder(new ClasspathResourceLoader(getClass()))
+ //.withTokenizer("opennlp", "tokenizerModel", tokenizerModelFile, "sentenceModel", sentenceModelFile)
+ //.addTokenFilter("opennlpPOS", "posTaggerModel", "en-test-pos-maxent.bin")
+ //.addTokenFilter("opennlplemmatizer", "dictionary", "en-test-lemmas.dict")
+ //.build();
+ AssertAnalyzesTo(analyzer, SENTENCE, SENTENCE_dict_punc, null, null,
+ SENTENCE_posTags, null, null, true);
+ }
+
+ [Test]
+ public void Test2SentencesDictionaryOnly()
+ {
+ Analyzer analyzer = Analyzer.NewAnonymous(createComponents: (fieldname, reader) =>
+ {
+ var loader = new ClasspathResourceLoader(GetType());
+
+ var opennlpFactory = new OpenNLPTokenizerFactory(new Dictionary<string, string> { { "tokenizerModel", tokenizerModelFile }, { "sentenceModel", sentenceModelFile } });
+ opennlpFactory.Inform(loader);
+ var opennlp = opennlpFactory.Create(reader);
+
+ var opennlpPOSFilterFactory = new OpenNLPPOSFilterFactory(new Dictionary<string, string> { { "posTaggerModel", posTaggerModelFile } });
+ opennlpPOSFilterFactory.Inform(loader);
+ var opennlpPOSFilter = opennlpPOSFilterFactory.Create(opennlp);
+
+ var opennlpLemmatizerFilterFactory = new OpenNLPLemmatizerFilterFactory(new Dictionary<string, string> { { "dictionary", lemmatizerDictFile } });
+ opennlpLemmatizerFilterFactory.Inform(loader);
+ var opennlpLemmatizerFilter = opennlpLemmatizerFilterFactory.Create(opennlpPOSFilter);
+
+ return new TokenStreamComponents(opennlp, opennlpLemmatizerFilter);
+ });
+ //CustomAnalyzer analyzer = CustomAnalyzer.builder(new ClasspathResourceLoader(getClass()))
+ //.withTokenizer("opennlp", "tokenizerModel", tokenizerModelFile, "sentenceModel", sentenceModelFile)
+ //.addTokenFilter("opennlpPOS", "posTaggerModel", posTaggerModelFile)
+ //.addTokenFilter("opennlplemmatizer", "dictionary", lemmatizerDictFile)
+ //.build();
+ AssertAnalyzesTo(analyzer, SENTENCES, SENTENCES_dict_punc, null, null,
+ SENTENCES_posTags, null, null, true);
+ }
+
+ [Test]
+ public void Test1SentenceMaxEntOnly()
+ {
+ Analyzer analyzer = Analyzer.NewAnonymous(createComponents: (fieldname, reader) =>
+ {
+ var loader = new ClasspathResourceLoader(GetType());
+
+ var opennlpFactory = new OpenNLPTokenizerFactory(new Dictionary<string, string> { { "tokenizerModel", tokenizerModelFile }, { "sentenceModel", sentenceModelFile } });
+ opennlpFactory.Inform(loader);
+ var opennlp = opennlpFactory.Create(reader);
+
+ var opennlpPOSFilterFactory = new OpenNLPPOSFilterFactory(new Dictionary<string, string> { { "posTaggerModel", posTaggerModelFile } });
+ opennlpPOSFilterFactory.Inform(loader);
+ var opennlpPOSFilter = opennlpPOSFilterFactory.Create(opennlp);
+
+ var opennlpLemmatizerFilterFactory = new OpenNLPLemmatizerFilterFactory(new Dictionary<string, string> { { "lemmatizerModel", lemmatizerModelFile } });
+ opennlpLemmatizerFilterFactory.Inform(loader);
+ var opennlpLemmatizerFilter = opennlpLemmatizerFilterFactory.Create(opennlpPOSFilter);
+
+ return new TokenStreamComponents(opennlp, opennlpLemmatizerFilter);
+ });
+ //CustomAnalyzer analyzer = CustomAnalyzer.builder(new ClasspathResourceLoader(getClass()))
+ // .withTokenizer("opennlp", "tokenizerModel", tokenizerModelFile, "sentenceModel", sentenceModelFile)
+ // .addTokenFilter("opennlpPOS", "posTaggerModel", posTaggerModelFile)
+ // .addTokenFilter("opennlplemmatizer", "lemmatizerModel", lemmatizerModelFile)
+ // .build();
+ AssertAnalyzesTo(analyzer, SENTENCE, SENTENCE_maxent_punc, null, null,
+ SENTENCE_posTags, null, null, true);
+ }
+
+ [Test]
+ public void Test2SentencesMaxEntOnly()
+ {
+ Analyzer analyzer = Analyzer.NewAnonymous(createComponents: (fieldname, reader) =>
+ {
+ var loader = new ClasspathResourceLoader(GetType());
+
+ var opennlpFactory = new OpenNLPTokenizerFactory(new Dictionary<string, string> { { "tokenizerModel", tokenizerModelFile }, { "sentenceModel", sentenceModelFile } });
+ opennlpFactory.Inform(loader);
+ var opennlp = opennlpFactory.Create(reader);
+
+ var opennlpPOSFilterFactory = new OpenNLPPOSFilterFactory(new Dictionary<string, string> { { "posTaggerModel", posTaggerModelFile } });
+ opennlpPOSFilterFactory.Inform(loader);
+ var opennlpPOSFilter = opennlpPOSFilterFactory.Create(opennlp);
+
+ var opennlpLemmatizerFilterFactory = new OpenNLPLemmatizerFilterFactory(new Dictionary<string, string> { { "lemmatizerModel", lemmatizerModelFile } });
+ opennlpLemmatizerFilterFactory.Inform(loader);
+ var opennlpLemmatizerFilter = opennlpLemmatizerFilterFactory.Create(opennlpPOSFilter);
+
+ return new TokenStreamComponents(opennlp, opennlpLemmatizerFilter);
+ });
+
+ //CustomAnalyzer analyzer = CustomAnalyzer.builder(new ClasspathResourceLoader(getClass()))
+ // .withTokenizer("opennlp", "tokenizerModel", tokenizerModelFile, "sentenceModel", sentenceModelFile)
+ // .addTokenFilter("opennlpPOS", "posTaggerModel", posTaggerModelFile)
+ // .addTokenFilter("OpenNLPLemmatizer", "lemmatizerModel", lemmatizerModelFile)
+ // .build();
+ AssertAnalyzesTo(analyzer, SENTENCES, SENTENCES_maxent_punc, null, null,
+ SENTENCES_posTags, null, null, true);
+ }
+
+ [Test]
+ public void Test1SentenceDictionaryAndMaxEnt()
+ {
+ Analyzer analyzer = Analyzer.NewAnonymous(createComponents: (fieldname, reader) =>
+ {
+ var loader = new ClasspathResourceLoader(GetType());
+
+ var opennlpFactory = new OpenNLPTokenizerFactory(new Dictionary<string, string> { { "tokenizerModel", tokenizerModelFile }, { "sentenceModel", sentenceModelFile } });
+ opennlpFactory.Inform(loader);
+ var opennlp = opennlpFactory.Create(reader);
+
+ var opennlpPOSFilterFactory = new OpenNLPPOSFilterFactory(new Dictionary<string, string> { { "posTaggerModel", posTaggerModelFile } });
+ opennlpPOSFilterFactory.Inform(loader);
+ var opennlpPOSFilter = opennlpPOSFilterFactory.Create(opennlp);
+
+ var opennlpLemmatizerFilterFactory = new OpenNLPLemmatizerFilterFactory(new Dictionary<string, string> { { "dictionary", lemmatizerDictFile }, { "lemmatizerModel", lemmatizerModelFile } });
+ opennlpLemmatizerFilterFactory.Inform(loader);
+ var opennlpLemmatizerFilter = opennlpLemmatizerFilterFactory.Create(opennlpPOSFilter);
+
+ return new TokenStreamComponents(opennlp, opennlpLemmatizerFilter);
+ });
+ //CustomAnalyzer analyzer = CustomAnalyzer.builder(new ClasspathResourceLoader(getClass()))
+ // .withTokenizer("opennlp", "tokenizerModel", tokenizerModelFile, "sentenceModel", sentenceModelFile)
+ // .addTokenFilter("opennlpPOS", "posTaggerModel", "en-test-pos-maxent.bin")
+ // .addTokenFilter("opennlplemmatizer", "dictionary", "en-test-lemmas.dict", "lemmatizerModel", lemmatizerModelFile)
+ // .build();
+ AssertAnalyzesTo(analyzer, SENTENCE_both, SENTENCE_both_punc, null, null,
+ SENTENCE_both_posTags, null, null, true);
+ }
+
+ [Test]
+ public void Test2SentencesDictionaryAndMaxEnt()
+ {
+ Analyzer analyzer = Analyzer.NewAnonymous(createComponents: (fieldname, reader) =>
+ {
+ var loader = new ClasspathResourceLoader(GetType());
+
+ var opennlpFactory = new OpenNLPTokenizerFactory(new Dictionary<string, string> { { "tokenizerModel", tokenizerModelFile }, { "sentenceModel", sentenceModelFile } });
+ opennlpFactory.Inform(loader);
+ var opennlp = opennlpFactory.Create(reader);
+
+ var opennlpPOSFilterFactory = new OpenNLPPOSFilterFactory(new Dictionary<string, string> { { "posTaggerModel", posTaggerModelFile } });
+ opennlpPOSFilterFactory.Inform(loader);
+ var opennlpPOSFilter = opennlpPOSFilterFactory.Create(opennlp);
+
+ var opennlpLemmatizerFilterFactory = new OpenNLPLemmatizerFilterFactory(new Dictionary<string, string> { { "dictionary", lemmatizerDictFile }, { "lemmatizerModel", lemmatizerModelFile } });
+ opennlpLemmatizerFilterFactory.Inform(loader);
+ var opennlpLemmatizerFilter = opennlpLemmatizerFilterFactory.Create(opennlpPOSFilter);
+
+ return new TokenStreamComponents(opennlp, opennlpLemmatizerFilter);
+ });
+
+ //CustomAnalyzer analyzer = CustomAnalyzer.builder(new ClasspathResourceLoader(getClass()))
+ // .withTokenizer("opennlp", "tokenizerModel", tokenizerModelFile, "sentenceModel", sentenceModelFile)
+ // .addTokenFilter("opennlpPOS", "posTaggerModel", posTaggerModelFile)
+ // .addTokenFilter("opennlplemmatizer", "dictionary", lemmatizerDictFile, "lemmatizerModel", lemmatizerModelFile)
+ // .build();
+ AssertAnalyzesTo(analyzer, SENTENCES_both, SENTENCES_both_punc, null, null,
+ SENTENCES_both_posTags, null, null, true);
+ }
+
+ [Test]
+ public void TestKeywordAttributeAwarenessDictionaryOnly()
+ {
+ Analyzer analyzer = Analyzer.NewAnonymous(createComponents: (fieldname, reader) =>
+ {
+ var loader = new ClasspathResourceLoader(GetType());
+
+ var opennlpFactory = new OpenNLPTokenizerFactory(new Dictionary<string, string> { { "tokenizerModel", tokenizerModelFile }, { "sentenceModel", sentenceModelFile } });
+ opennlpFactory.Inform(loader);
+ var opennlp = opennlpFactory.Create(reader);
+
+ var opennlpPOSFilterFactory = new OpenNLPPOSFilterFactory(new Dictionary<string, string> { { "posTaggerModel", posTaggerModelFile } });
+ opennlpPOSFilterFactory.Inform(loader);
+ var opennlpPOSFilter = opennlpPOSFilterFactory.Create(opennlp);
+
+ var keywordRepeatFilterFactory = new KeywordRepeatFilterFactory(new Dictionary<string, string>());
+ var keywordRepeatFilter = keywordRepeatFilterFactory.Create(opennlpPOSFilter);
+
+ var opennlpLemmatizerFilterFactory = new OpenNLPLemmatizerFilterFactory(new Dictionary<string, string> { { "dictionary", lemmatizerDictFile } });
+ opennlpLemmatizerFilterFactory.Inform(loader);
+ var opennlpLemmatizerFilter = opennlpLemmatizerFilterFactory.Create(keywordRepeatFilter);
+
+ var removeDuplicatesTokenFilterFactory = new RemoveDuplicatesTokenFilterFactory(new Dictionary<string, string>());
+ var removeDuplicatesTokenFilter = removeDuplicatesTokenFilterFactory.Create(opennlpLemmatizerFilter);
+
+ return new TokenStreamComponents(opennlp, removeDuplicatesTokenFilter);
+ });
+
+ //CustomAnalyzer analyzer = CustomAnalyzer.builder(new ClasspathResourceLoader(getClass()))
+ // .withTokenizer("opennlp", "tokenizerModel", tokenizerModelFile, "sentenceModel", sentenceModelFile)
+ // .addTokenFilter("opennlpPOS", "posTaggerModel", posTaggerModelFile)
+ // .addTokenFilter(KeywordRepeatFilterFactory.class)
+ // .addTokenFilter("opennlplemmatizer", "dictionary", lemmatizerDictFile)
+ // .addTokenFilter(RemoveDuplicatesTokenFilterFactory.class)
+ // .build();
+ AssertAnalyzesTo(analyzer, SENTENCES, SENTENCES_dict_keep_orig_punc, null, null,
+ SENTENCES_keep_orig_posTags, null, null, true);
+ }
+
+ [Test]
+ public void TestKeywordAttributeAwarenessMaxEntOnly()
+ {
+ Analyzer analyzer = Analyzer.NewAnonymous(createComponents: (fieldname, reader) =>
+ {
+ var loader = new ClasspathResourceLoader(GetType());
+
+ var opennlpFactory = new OpenNLPTokenizerFactory(new Dictionary<string, string> { { "tokenizerModel", tokenizerModelFile }, { "sentenceModel", sentenceModelFile } });
+ opennlpFactory.Inform(loader);
+ var opennlp = opennlpFactory.Create(reader);
+
+ var opennlpPOSFilterFactory = new OpenNLPPOSFilterFactory(new Dictionary<string, string> { { "posTaggerModel", posTaggerModelFile } });
+ opennlpPOSFilterFactory.Inform(loader);
+ var opennlpPOSFilter = opennlpPOSFilterFactory.Create(opennlp);
+
+ var keywordRepeatFilterFactory = new KeywordRepeatFilterFactory(new Dictionary<string, string>());
+ var keywordRepeatFilter = keywordRepeatFilterFactory.Create(opennlpPOSFilter);
+
+ var opennlpLemmatizerFilterFactory = new OpenNLPLemmatizerFilterFactory(new Dictionary<string, string> { { "lemmatizerModel", lemmatizerModelFile } });
+ opennlpLemmatizerFilterFactory.Inform(loader);
+ var opennlpLemmatizerFilter = opennlpLemmatizerFilterFactory.Create(keywordRepeatFilter);
+
+ var removeDuplicatesTokenFilterFactory = new RemoveDuplicatesTokenFilterFactory(new Dictionary<string, string>());
+ var removeDuplicatesTokenFilter = removeDuplicatesTokenFilterFactory.Create(opennlpLemmatizerFilter);
+
+ return new TokenStreamComponents(opennlp, removeDuplicatesTokenFilter);
+ });
+ //CustomAnalyzer analyzer = CustomAnalyzer.builder(new ClasspathResourceLoader(getClass()))
+ // .withTokenizer("opennlp", "tokenizerModel", tokenizerModelFile, "sentenceModel", sentenceModelFile)
+ // .addTokenFilter("opennlpPOS", "posTaggerModel", posTaggerModelFile)
+ // .addTokenFilter(KeywordRepeatFilterFactory.class)
+ // .addTokenFilter("opennlplemmatizer", "lemmatizerModel", lemmatizerModelFile)
+ // .addTokenFilter(RemoveDuplicatesTokenFilterFactory.class)
+ // .build();
+ AssertAnalyzesTo(analyzer, SENTENCES, SENTENCES_max_ent_keep_orig_punc, null, null,
+ SENTENCES_keep_orig_posTags, null, null, true);
+ }
+
+ [Test]
+ public void TestKeywordAttributeAwarenessDictionaryAndMaxEnt()
+ {
+ Analyzer analyzer = Analyzer.NewAnonymous(createComponents: (fieldname, reader) =>
+ {
+ var loader = new ClasspathResourceLoader(GetType());
+
+ var opennlpFactory = new OpenNLPTokenizerFactory(new Dictionary<string, string> { { "tokenizerModel", tokenizerModelFile }, { "sentenceModel", sentenceModelFile } });
+ opennlpFactory.Inform(loader);
+ var opennlp = opennlpFactory.Create(reader);
+
+ var opennlpPOSFilterFactory = new OpenNLPPOSFilterFactory(new Dictionary<string, string> { { "posTaggerModel", posTaggerModelFile } });
+ opennlpPOSFilterFactory.Inform(loader);
+ var opennlpPOSFilter = opennlpPOSFilterFactory.Create(opennlp);
+
+ var keywordRepeatFilterFactory = new KeywordRepeatFilterFactory(new Dictionary<string, string>());
+ var keywordRepeatFilter = keywordRepeatFilterFactory.Create(opennlpPOSFilter);
+
+ var opennlpLemmatizerFilterFactory = new OpenNLPLemmatizerFilterFactory(new Dictionary<string, string> { { "dictionary", lemmatizerDictFile }, { "lemmatizerModel", lemmatizerModelFile } });
+ opennlpLemmatizerFilterFactory.Inform(loader);
+ var opennlpLemmatizerFilter = opennlpLemmatizerFilterFactory.Create(keywordRepeatFilter);
+
+ var removeDuplicatesTokenFilterFactory = new RemoveDuplicatesTokenFilterFactory(new Dictionary<string, string>());
+ var removeDuplicatesTokenFilter = removeDuplicatesTokenFilterFactory.Create(opennlpLemmatizerFilter);
+
+ return new TokenStreamComponents(opennlp, removeDuplicatesTokenFilter);
+ });
+ //CustomAnalyzer analyzer = CustomAnalyzer.builder(new ClasspathResourceLoader(getClass()))
+ // .withTokenizer("opennlp", "tokenizerModel", tokenizerModelFile, "sentenceModel", sentenceModelFile)
+ // .addTokenFilter("opennlpPOS", "posTaggerModel", posTaggerModelFile)
+ // .addTokenFilter(KeywordRepeatFilterFactory.class)
+ // .addTokenFilter("opennlplemmatizer", "dictionary", lemmatizerDictFile, "lemmatizerModel", lemmatizerModelFile)
+ // .addTokenFilter(RemoveDuplicatesTokenFilterFactory.class)
+ // .build();
+ AssertAnalyzesTo(analyzer, SENTENCES_both, SENTENCES_both_keep_orig_punc, null, null,
+ SENTENCES_both_keep_orig_posTags, null, null, true);
+ }
+ }
+}
diff --git a/src/Lucene.Net.Tests.Analysis.OpenNLP/TestOpenNLPPOSFilterFactory.cs b/src/Lucene.Net.Tests.Analysis.OpenNLP/TestOpenNLPPOSFilterFactory.cs
new file mode 100644
index 0000000..0d54d49
--- /dev/null
+++ b/src/Lucene.Net.Tests.Analysis.OpenNLP/TestOpenNLPPOSFilterFactory.cs
@@ -0,0 +1,160 @@
+using Lucene.Net.Analysis.Payloads;
+using Lucene.Net.Analysis.Util;
+using NUnit.Framework;
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Text;
+using System.Threading.Tasks;
+
+namespace Lucene.Net.Analysis.OpenNlp
+{
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ /// <summary>
+ /// Needs the OpenNLP Tokenizer because it creates full streams of punctuation.
+ /// The POS model is based on this tokenization.
+ ///
+ /// <para/>Tagging models are created from tiny test data in opennlp/tools/test-model-data/ and are not very accurate.
+ /// </summary>
+ public class TestOpenNLPPOSFilterFactory : BaseTokenStreamTestCase
+ {
+ private const String SENTENCES = "Sentence number 1 has 6 words. Sentence number 2, 5 words.";
+ private static readonly String[] SENTENCES_punc
+ = { "Sentence", "number", "1", "has", "6", "words", ".", "Sentence", "number", "2", ",", "5", "words", "." };
+ private static readonly int[] SENTENCES_startOffsets = { 0, 9, 16, 18, 22, 24, 29, 31, 40, 47, 48, 50, 52, 57 };
+ private static readonly int[] SENTENCES_endOffsets = { 8, 15, 17, 21, 23, 29, 30, 39, 46, 48, 49, 51, 57, 58 };
+ private static readonly String[] SENTENCES_posTags
+ = { "NN", "NN", "CD", "VBZ", "CD", "NNS", ".", "NN", "NN", "CD", ",", "CD", "NNS", "." };
+
+ private const String NO_BREAK = "No period";
+ private static readonly String[] NO_BREAK_terms = { "No", "period" };
+ private static readonly int[] NO_BREAK_startOffsets = { 0, 3 };
+ private static readonly int[] NO_BREAK_endOffsets = { 2, 9 };
+
+ private const String sentenceModelFile = "en-test-sent.bin";
+ private const String tokenizerModelFile = "en-test-tokenizer.bin";
+ private const String posTaggerModelFile = "en-test-pos-maxent.bin";
+
+
+ private static byte[][] ToPayloads(params string[] strings)
+ {
+ return strings.Select(s => s == null ? null : Encoding.UTF8.GetBytes(s)).ToArray();
+ }
+
+ // private static byte[][] ToPayloads(params String[] strings)
+ // {
+ // return Arrays.stream(strings).map(s->s == null ? null : s.getBytes(StandardCharsets.UTF_8)).toArray(byte[][]::new);
+ // }
+
+ [Test]
+ public void TestBasic()
+ {
+ var loader = new ClasspathResourceLoader(GetType());
+ Analyzer analyzer = Analyzer.NewAnonymous(createComponents: (fieldname, reader) => {
+ var tokenizerFactory = new OpenNLPTokenizerFactory(new Dictionary<string, string> { { "tokenizerModel", tokenizerModelFile }, { "sentenceModel", sentenceModelFile } });
+ tokenizerFactory.Inform(loader);
+ var tokenizer = tokenizerFactory.Create(reader);
+
+ var filter1Factory = new OpenNLPPOSFilterFactory(new Dictionary<string, string> { { "posTaggerModel", posTaggerModelFile } });
+ filter1Factory.Inform(loader);
+ var filter1 = filter1Factory.Create(tokenizer);
+
+ return new TokenStreamComponents(tokenizer, filter1);
+ });
+ // CustomAnalyzer analyzer = CustomAnalyzer.builder(new ClasspathResourceLoader(GetType()))
+ //.withTokenizer("opennlp", "tokenizerModel", tokenizerModelFile, "sentenceModel", sentenceModelFile)
+ //.addTokenFilter("opennlpPOS", "posTaggerModel", posTaggerModelFile)
+ //.build();
+ AssertAnalyzesTo(analyzer, SENTENCES, SENTENCES_punc, SENTENCES_startOffsets, SENTENCES_endOffsets);
+ }
+
+ [Test]
+ public void TestPOS()
+ {
+ var loader = new ClasspathResourceLoader(GetType());
+ Analyzer analyzer = Analyzer.NewAnonymous(createComponents: (fieldname, reader) =>
+ {
+ var tokenizerFactory = new OpenNLPTokenizerFactory(new Dictionary<string, string> { { "tokenizerModel", tokenizerModelFile }, { "sentenceModel", sentenceModelFile } });
+ tokenizerFactory.Inform(loader);
+ var tokenizer = tokenizerFactory.Create(reader);
+
+ var filter1Factory = new OpenNLPPOSFilterFactory(new Dictionary<string, string> { { "posTaggerModel", posTaggerModelFile } });
+ filter1Factory.Inform(loader);
+ var filter1 = filter1Factory.Create(tokenizer);
+
+ return new TokenStreamComponents(tokenizer, filter1);
+ });
+ // CustomAnalyzer analyzer = CustomAnalyzer.builder(new ClasspathResourceLoader(GetType()))
+ // .withTokenizer("opennlp", "tokenizerModel", tokenizerModelFile, "sentenceModel", sentenceModelFile)
+ // .addTokenFilter("opennlpPOS", "posTaggerModel", posTaggerModelFile)
+ // .build();
+ AssertAnalyzesTo(analyzer, SENTENCES, SENTENCES_punc, SENTENCES_startOffsets, SENTENCES_endOffsets,
+ SENTENCES_posTags, null, null, true);
+
+ analyzer = Analyzer.NewAnonymous(createComponents: (fieldname, reader) =>
+ {
+ var tokenizerFactory = new OpenNLPTokenizerFactory(new Dictionary<string, string> { { "tokenizerModel", tokenizerModelFile }, { "sentenceModel", sentenceModelFile } });
+ tokenizerFactory.Inform(loader);
+ var tokenizer = tokenizerFactory.Create(reader);
+
+ var filter1Factory = new OpenNLPPOSFilterFactory(new Dictionary<string, string> { { "posTaggerModel", posTaggerModelFile } });
+ filter1Factory.Inform(loader);
+ var filter1 = filter1Factory.Create(tokenizer);
+
+ var filter2Factory = new TypeAsPayloadTokenFilterFactory(new Dictionary<string, string>());
+ var filter2 = filter2Factory.Create(filter1);
+
+ return new TokenStreamComponents(tokenizer, filter2);
+ });
+ //analyzer = CustomAnalyzer.builder(new ClasspathResourceLoader(GetType()))
+ // .withTokenizer("opennlp", "tokenizerModel", tokenizerModelFile, "sentenceModel", sentenceModelFile)
+ // .addTokenFilter("opennlpPOS", "posTaggerModel", posTaggerModelFile)
+ // .addTokenFilter(TypeAsPayloadTokenFilterFactory.class)
+ //.build();
+ AssertAnalyzesTo(analyzer, SENTENCES, SENTENCES_punc, SENTENCES_startOffsets, SENTENCES_endOffsets,
+ null, null, null, true, ToPayloads(SENTENCES_posTags));
+ }
+
+ [Test]
+ public void TestNoBreak()
+ {
+ var analyzer = Analyzer.NewAnonymous(createComponents: (fieldname, reader) =>
+ {
+ var loader = new ClasspathResourceLoader(GetType());
+
+ var tokenizerFactory = new OpenNLPTokenizerFactory(new Dictionary<string, string> { { "tokenizerModel", tokenizerModelFile }, { "sentenceModel", sentenceModelFile } });
+ tokenizerFactory.Inform(loader);
+ var tokenizer = tokenizerFactory.Create(reader);
+
+ var tokenFilterFactory = new OpenNLPPOSFilterFactory(new Dictionary<string, string> { { "posTaggerModel", posTaggerModelFile } });
+ tokenFilterFactory.Inform(loader);
+ var tokenFilter = tokenFilterFactory.Create(tokenizer);
+
+ return new TokenStreamComponents(tokenizer, tokenFilter);
+ });
+
+ //CustomAnalyzer analyzer = CustomAnalyzer.builder(new ClasspathResourceLoader(GetType()))
+ // .withTokenizer("opennlp", "tokenizerModel", tokenizerModelFile, "sentenceModel", sentenceModelFile)
+ // .addTokenFilter("opennlpPOS", "posTaggerModel", posTaggerModelFile)
+ // .build();
+ AssertAnalyzesTo(analyzer, NO_BREAK, NO_BREAK_terms, NO_BREAK_startOffsets, NO_BREAK_endOffsets,
+ null, null, null, true);
+ }
+ }
+}
diff --git a/src/Lucene.Net.Tests.Analysis.OpenNLP/TestOpenNLPSentenceBreakIterator.cs b/src/Lucene.Net.Tests.Analysis.OpenNLP/TestOpenNLPSentenceBreakIterator.cs
new file mode 100644
index 0000000..ca6c3cc
--- /dev/null
+++ b/src/Lucene.Net.Tests.Analysis.OpenNLP/TestOpenNLPSentenceBreakIterator.cs
@@ -0,0 +1,252 @@
+using ICU4N.Support.Text;
+using ICU4N.Text;
+using Lucene.Net.Analysis.OpenNlp.Tools;
+using Lucene.Net.Analysis.Util;
+using Lucene.Net.Util;
+using NUnit.Framework;
+using System;
+
+namespace Lucene.Net.Analysis.OpenNlp
+{
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ public class TestOpenNLPSentenceBreakIterator : LuceneTestCase
+ {
+ private const String TEXT
+ // 111
+ // 111111111122222222223333333333444444444455555555556666666666777777777788888888889999999999000
+ // 0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012
+ = "Sentence number 1 has 6 words. Sentence number 2, 5 words. And finally, sentence number 3 has 8 words.";
+ private static readonly String[] SENTENCES = new String[] {
+ "Sentence number 1 has 6 words. ", "Sentence number 2, 5 words. ", "And finally, sentence number 3 has 8 words." };
+ private static readonly String PADDING = " Word. Word. ";
+ private static readonly String sentenceModelFile = "en-test-sent.bin";
+
+ public override void BeforeClass()
+ {
+ base.BeforeClass();
+ PopulateCache();
+ }
+
+ public static void PopulateCache()
+ {
+ OpenNLPOpsFactory.GetSentenceModel(sentenceModelFile, new ClasspathResourceLoader(typeof(TestOpenNLPSentenceBreakIterator)));
+ }
+
+ [Test]
+ public void TestThreeSentences()
+ {
+ NLPSentenceDetectorOp sentenceDetectorOp = OpenNLPOpsFactory.GetSentenceDetector(sentenceModelFile);
+ BreakIterator bi = new OpenNLPSentenceBreakIterator(sentenceDetectorOp);
+ bi.SetText(TEXT); // String is converted to StringCharacterIterator
+
+ Do3SentenceTest(bi);
+
+ bi.SetText(GetCharArrayIterator(TEXT));
+ Do3SentenceTest(bi);
+ }
+
+ private CharacterIterator GetCharArrayIterator(String text)
+ {
+ return GetCharArrayIterator(text, 0, text.Length);
+ }
+
+ private class WorkaroundCharArrayIterator : CharArrayIterator
+ {
+ // Lie about all surrogates to the sentence tokenizer,
+ // instead we treat them all as SContinue so we won't break around them.
+ protected override char JreBugWorkaround(char ch)
+ {
+ return ch >= 0xD800 && ch <= 0xDFFF ? (char)0x002C : ch;
+ }
+ }
+
+ private CharacterIterator GetCharArrayIterator(String text, int start, int length)
+ {
+ // CharArrayIterator charArrayIterator = new CharArrayIterator() {
+ // // Lie about all surrogates to the sentence tokenizer,
+ // // instead we treat them all as SContinue so we won't break around them.
+ // protected override char JreBugWorkaround(char ch)
+ // {
+ // return ch >= 0xD800 && ch <= 0xDFFF ? 0x002C : ch;
+ // }
+ //};
+ CharArrayIterator charArrayIterator = new WorkaroundCharArrayIterator();
+ charArrayIterator.SetText(text.ToCharArray(), start, length);
+ return charArrayIterator;
+ }
+
+ private void Do3SentenceTest(BreakIterator bi) // LUCENENET NOTE: Refactored a bit because Substring in .NET requires some light math to match Java
+ {
+ assertEquals(0, bi.Current);
+ assertEquals(0, bi.First());
+ int current = bi.Current;
+ assertEquals(SENTENCES[0], TEXT.Substring(current, bi.Next() - current)); // LUCNENENET: Corrected 2nd parameter
+ current = bi.Current;
+ assertEquals(SENTENCES[1], TEXT.Substring(current, bi.Next() - current)); // LUCNENENET: Corrected 2nd parameter
+ current = bi.Current;
+ assertEquals(bi.Text.EndIndex, bi.Next());
+ int next = bi.Current;
+ assertEquals(SENTENCES[2], TEXT.Substring(current, next - current)); // LUCNENENET: Corrected 2nd parameter
+ assertEquals(BreakIterator.Done, bi.Next());
+
+ assertEquals(TEXT.Length, bi.Last());
+ int end = bi.Current;
+ int prev = bi.Previous();
+ assertEquals(SENTENCES[2], TEXT.Substring(prev, end - prev)); // LUCNENENET: Corrected 2nd parameter
+ end = bi.Current;
+ prev = bi.Previous();
+ assertEquals(SENTENCES[1], TEXT.Substring(prev, end - prev)); // LUCNENENET: Corrected 2nd parameter
+ end = bi.Current;
+ prev = bi.Previous();
+ assertEquals(SENTENCES[0], TEXT.Substring(prev, end - prev)); // LUCNENENET: Corrected 2nd parameter
+ assertEquals(BreakIterator.Done, bi.Previous());
+ assertEquals(0, bi.Current);
+
+ assertEquals(59, bi.Following(39));
+ assertEquals(59, bi.Following(31));
+ assertEquals(31, bi.Following(30));
+
+ assertEquals(0, bi.Preceding(57));
+ assertEquals(0, bi.Preceding(58));
+ assertEquals(31, bi.Preceding(59));
+
+ assertEquals(0, bi.First());
+ assertEquals(59, bi.Next(2));
+ assertEquals(0, bi.Next(-2));
+ }
+
+ [Test]
+ public void TestSingleSentence()
+ {
+ NLPSentenceDetectorOp sentenceDetectorOp = OpenNLPOpsFactory.GetSentenceDetector(sentenceModelFile);
+ BreakIterator bi = new OpenNLPSentenceBreakIterator(sentenceDetectorOp);
+ bi.SetText(GetCharArrayIterator(SENTENCES[0]));
+ Test1Sentence(bi, SENTENCES[0]);
+ }
+
+ private void Test1Sentence(BreakIterator bi, String text)
+ {
+ int start = bi.Text.BeginIndex;
+ assertEquals(start, bi.First());
+ int current = bi.Current;
+ assertEquals(bi.Text.EndIndex, bi.Next());
+ int end = bi.Current - start;
+ assertEquals(text, text.Substring(current - start, end - start));
+
+ assertEquals(text.Length, bi.Last() - start);
+ end = bi.Current;
+ bi.Previous();
+ assertEquals(BreakIterator.Done, bi.Previous());
+ int previous = bi.Current;
+ assertEquals(text, text.Substring(previous - start, end - start));
+ assertEquals(start, bi.Current);
+
+ assertEquals(BreakIterator.Done, bi.Following(bi.Last() / 2 + start));
+
+ assertEquals(BreakIterator.Done, bi.Preceding(bi.Last() / 2 + start));
+
+ assertEquals(start, bi.First());
+ assertEquals(BreakIterator.Done, bi.Next(13));
+ assertEquals(BreakIterator.Done, bi.Next(-8));
+ }
+
+ [Test]
+ public void TestSliceEnd()
+ {
+ NLPSentenceDetectorOp sentenceDetectorOp = OpenNLPOpsFactory.GetSentenceDetector(sentenceModelFile);
+ BreakIterator bi = new OpenNLPSentenceBreakIterator(sentenceDetectorOp);
+ bi.SetText(GetCharArrayIterator(SENTENCES[0] + PADDING, 0, SENTENCES[0].Length));
+
+ Test1Sentence(bi, SENTENCES[0]);
+ }
+
+ [Test]
+ public void TestSliceStart()
+ {
+ NLPSentenceDetectorOp sentenceDetectorOp = OpenNLPOpsFactory.GetSentenceDetector(sentenceModelFile);
+ BreakIterator bi = new OpenNLPSentenceBreakIterator(sentenceDetectorOp);
+ bi.SetText(GetCharArrayIterator(PADDING + SENTENCES[0], PADDING.Length, SENTENCES[0].Length));
+ Test1Sentence(bi, SENTENCES[0]);
+ }
+
+ [Test]
+ public void TestSliceMiddle()
+ {
+ NLPSentenceDetectorOp sentenceDetectorOp = OpenNLPOpsFactory.GetSentenceDetector(sentenceModelFile);
+ BreakIterator bi = new OpenNLPSentenceBreakIterator(sentenceDetectorOp);
+ bi.SetText(GetCharArrayIterator(PADDING + SENTENCES[0] + PADDING, PADDING.Length, SENTENCES[0].Length));
+
+ Test1Sentence(bi, SENTENCES[0]);
+ }
+
+ /** the current position must be ignored, initial position is always first() */
+ [Test]
+ public void TestFirstPosition()
+ {
+ NLPSentenceDetectorOp sentenceDetectorOp = OpenNLPOpsFactory.GetSentenceDetector(sentenceModelFile);
+ BreakIterator bi = new OpenNLPSentenceBreakIterator(sentenceDetectorOp);
+ bi.SetText(GetCharArrayIterator(SENTENCES[0]));
+ assertEquals(SENTENCES[0].Length, bi.Last()); // side-effect: set current position to last()
+ Test1Sentence(bi, SENTENCES[0]);
+ }
+
+ [Test]
+ public void TestWhitespaceOnly()
+ {
+ NLPSentenceDetectorOp sentenceDetectorOp = OpenNLPOpsFactory.GetSentenceDetector(sentenceModelFile);
+ BreakIterator bi = new OpenNLPSentenceBreakIterator(sentenceDetectorOp);
+ bi.SetText(" \n \n\n\r\n\t \n");
+ Test0Sentences(bi);
+ }
+
+ [Test]
+ public void TestEmptyString()
+ {
+ NLPSentenceDetectorOp sentenceDetectorOp = OpenNLPOpsFactory.GetSentenceDetector(sentenceModelFile);
+ BreakIterator bi = new OpenNLPSentenceBreakIterator(sentenceDetectorOp);
+ bi.SetText("");
+ Test0Sentences(bi);
+ }
+
+ private void Test0Sentences(BreakIterator bi)
+ {
+ assertEquals(0, bi.Current);
+ assertEquals(0, bi.First());
+ assertEquals(BreakIterator.Done, bi.Next());
+ assertEquals(0, bi.Last());
+ assertEquals(BreakIterator.Done, bi.Previous());
+ assertEquals(BreakIterator.Done, bi.Following(0));
+ assertEquals(BreakIterator.Done, bi.Preceding(0));
+ assertEquals(0, bi.First());
+ assertEquals(BreakIterator.Done, bi.Next(13));
+ assertEquals(BreakIterator.Done, bi.Next(-8));
+ }
+
+ internal static void assertEquals(object expected, object actual)
+ {
+ Assert.AreEqual(expected, actual);
+ }
+
+ internal static void assertEquals(long expected, long actual)
+ {
+ Assert.AreEqual(expected, actual);
+ }
+
+ }
+}
diff --git a/src/Lucene.Net.Tests.Analysis.OpenNLP/TestOpenNLPTokenizerFactory.cs b/src/Lucene.Net.Tests.Analysis.OpenNLP/TestOpenNLPTokenizerFactory.cs
new file mode 100644
index 0000000..51ae740
--- /dev/null
+++ b/src/Lucene.Net.Tests.Analysis.OpenNLP/TestOpenNLPTokenizerFactory.cs
@@ -0,0 +1,140 @@
+using Lucene.Net.Analysis.Util;
+using NUnit.Framework;
+using System;
+using System.Collections.Generic;
+using System.IO;
+
+namespace Lucene.Net.Analysis.OpenNlp
+{
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ /// <summary>
+ /// Tests the Tokenizer as well- the Tokenizer needs the OpenNLP model files,
+ /// which this can load from src/test-files/opennlp/solr/conf
+ /// </summary>
+ public class TestOpenNLPTokenizerFactory : BaseTokenStreamTestCase
+ {
+ private const String SENTENCES = "Sentence number 1 has 6 words. Sentence number 2, 5 words.";
+ private static String[] SENTENCES_punc = { "Sentence", "number", "1", "has", "6", "words", ".", "Sentence", "number", "2", ",", "5", "words", "." };
+ private static int[] SENTENCES_startOffsets = { 0, 9, 16, 18, 22, 24, 29, 31, 40, 47, 48, 50, 52, 57 };
+ private static int[] SENTENCES_endOffsets = { 8, 15, 17, 21, 23, 29, 30, 39, 46, 48, 49, 51, 57, 58 };
+
+ private const String SENTENCE1 = "Sentence number 1 has 6 words.";
+ private static String[] SENTENCE1_punc = { "Sentence", "number", "1", "has", "6", "words", "." };
+
+ [Test]
+ public void TestTokenizer()
+ {
+ Analyzer analyzer = Analyzer.NewAnonymous(createComponents: (fieldname, reader) =>
+ {
+ var tokenizerFactory = new OpenNLPTokenizerFactory(new Dictionary<string, string> { { "sentenceModel", "en-test-sent.bin" }, { "tokenizerModel", "en-test-tokenizer.bin" } });
+ var tokenizer = tokenizerFactory.Create(reader);
+ return new TokenStreamComponents(tokenizer);
+ });
+ //CustomAnalyzer analyzer = CustomAnalyzer.builder(new ClasspathResourceLoader(getClass()))
+ // .withTokenizer("opennlp", "sentenceModel", "en-test-sent.bin", "tokenizerModel", "en-test-tokenizer.bin")
+ // .build();
+ AssertAnalyzesTo(analyzer, SENTENCES, SENTENCES_punc, SENTENCES_startOffsets, SENTENCES_endOffsets);
+ AssertAnalyzesTo(analyzer, SENTENCE1, SENTENCE1_punc);
+ }
+
+ [Test]
+ public void TestTokenizerNoSentenceDetector()
+ {
+ var expected = Assert.Throws<ArgumentException>(() =>
+ {
+ Analyzer analyzer = Analyzer.NewAnonymous(createComponents: (fieldname, reader) =>
+ {
+ var tokenizerFactory = new OpenNLPTokenizerFactory(new Dictionary<string, string> { { "tokenizerModel", "en-test-tokenizer.bin" } });
+ var tokenizer = tokenizerFactory.Create(reader);
+ return new TokenStreamComponents(tokenizer);
+ });
+ analyzer.GetTokenStream("", "");
+ });
+
+ // IllegalArgumentException expected = expectThrows(IllegalArgumentException.class, () -> {
+ // CustomAnalyzer analyzer = CustomAnalyzer.builder(new ClasspathResourceLoader(getClass()))
+ // .withTokenizer("opennlp", "tokenizerModel", "en-test-tokenizer.bin")
+ // .build();
+ //});
+ assertTrue(expected.Message.Contains("Configuration Error: missing parameter 'sentenceModel'"));
+ }
+
+ [Test]
+ public void TestTokenizerNoTokenizer()
+ {
+ //Analyzer analyzer2 = Analyzer.NewAnonymous(createComponents: (fieldname, reader) =>
+ //{
+ // var tokenizerFactory = new OpenNLPTokenizerFactory(new Dictionary<string, string> { { "sentenceModel", "en-test-sent.bin" } });
+ // tokenizerFactory.Inform(new ClasspathResourceLoader(GetType()));
+ // var tokenizer = tokenizerFactory.Create(reader);
+ // return new TokenStreamComponents(tokenizer);
+ //});
+ //analyzer2.GetTokenStream("", "");
+
+ var expected = Assert.Throws<ArgumentException>(() =>
+ {
+ Analyzer analyzer = Analyzer.NewAnonymous(createComponents: (fieldname, reader) =>
+ {
+ var tokenizerFactory = new OpenNLPTokenizerFactory(new Dictionary<string, string> { { "sentenceModel", "en-test-sent.bin" } });
+ var tokenizer = tokenizerFactory.Create(reader);
+ return new TokenStreamComponents(tokenizer);
+ });
+ analyzer.GetTokenStream("", "");
+ });
+
+ // IllegalArgumentException expected = expectThrows(ArgumentException.class, () -> {
+ // CustomAnalyzer analyzer = CustomAnalyzer.builder(new ClasspathResourceLoader(getClass()))
+ // .withTokenizer("opennlp", "sentenceModel", "en-test-sent.bin")
+ // .build();
+ //});
+ assertTrue(expected.Message.Contains("Configuration Error: missing parameter 'tokenizerModel'"));
+ }
+
+ // test analyzer caching the tokenizer
+ [Test]
+ public void TestClose()
+ {
+ IDictionary<String, String> args = new Dictionary<String, String>()
+ {
+ { "sentenceModel", "en-test-sent.bin" },
+ { "tokenizerModel", "en-test-tokenizer.bin" }
+ };
+ OpenNLPTokenizerFactory factory = new OpenNLPTokenizerFactory(args);
+ factory.Inform(new ClasspathResourceLoader(GetType()));
+
+ Tokenizer ts = factory.Create(NewAttributeFactory(), new StringReader(SENTENCES));
+ //ts.SetReader(new StringReader(SENTENCES));
+
+ ts.Reset();
+ ts.Dispose();
+ ts.Reset();
+ ts.SetReader(new StringReader(SENTENCES));
+ AssertTokenStreamContents(ts, SENTENCES_punc);
+ ts.Dispose();
+ ts.Reset();
+ ts.SetReader(new StringReader(SENTENCES));
+ AssertTokenStreamContents(ts, SENTENCES_punc);
+ }
+
+ internal static void assertTrue(bool condition)
+ {
+ Assert.IsTrue(condition);
+ }
+ }
+}
diff --git a/src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-chunker.bin b/src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-chunker.bin
new file mode 100644
index 0000000..b9bfdb4
Binary files /dev/null and b/src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-chunker.bin differ
diff --git a/src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-lemmas.dict b/src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-lemmas.dict
new file mode 100644
index 0000000..d1d486c
--- /dev/null
+++ b/src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-lemmas.dict
@@ -0,0 +1,12 @@
+they NNP they
+sent VBD send
+him PRP he
+running VBG run
+in IN in
+the DT the
+evening NN evening
+he PRP he
+did VBD do
+not RB not
+come VB come
+back RB back
diff --git a/src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-lemmatizer.bin b/src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-lemmatizer.bin
new file mode 100644
index 0000000..aac6ce2
Binary files /dev/null and b/src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-lemmatizer.bin differ
diff --git a/src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-ner.bin b/src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-ner.bin
new file mode 100644
index 0000000..b4d8cdc
Binary files /dev/null and b/src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-ner.bin differ
diff --git a/src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-pos-maxent.bin b/src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-pos-maxent.bin
new file mode 100644
index 0000000..a5f1481
Binary files /dev/null and b/src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-pos-maxent.bin differ
diff --git a/src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-sent.bin b/src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-sent.bin
new file mode 100644
index 0000000..6e19e6b
Binary files /dev/null and b/src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-sent.bin differ
diff --git a/src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-tokenizer.bin b/src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-tokenizer.bin
new file mode 100644
index 0000000..796a744
Binary files /dev/null and b/src/Lucene.Net.Tests.Analysis.OpenNLP/en-test-tokenizer.bin differ
diff --git a/src/Lucene.Net.Tests.Analysis.Phonetic/TestOpenNLPChunkerFilterFactory.cs b/src/Lucene.Net.Tests.Analysis.Phonetic/TestOpenNLPChunkerFilterFactory.cs
new file mode 100644
index 0000000..d767d51
--- /dev/null
+++ b/src/Lucene.Net.Tests.Analysis.Phonetic/TestOpenNLPChunkerFilterFactory.cs
@@ -0,0 +1,86 @@
+//using Lucene.Net.Analysis;
+//using Lucene.Net.Analysis.Util;
+//using NUnit.Framework;
+//using System;
+//using System.Collections.Generic;
+//using System.Linq;
+//using System.Text;
+
+//namespace Lucene.Net.Analysis.OpenNlp
+//{
+// /// <summary>
+// /// Needs the OpenNLP Tokenizer because it creates full streams of punctuation.
+// /// Needs the OpenNLP POS tagger for the POS tags.
+// /// <para/>
+// /// Tagging models are created from tiny test data in opennlp/tools/test-model-data/ and are not very accurate.
+// /// </summary>
+// public class TestOpenNLPChunkerFilterFactory : BaseTokenStreamTestCase
+// {
+// private const String SENTENCES = "Sentence number 1 has 6 words. Sentence number 2, 5 words.";
+// private static readonly String[] SENTENCES_punc
+// = {"Sentence", "number", "1", "has", "6", "words", ".", "Sentence", "number", "2", ",", "5", "words", "."};
+// private static readonly int[] SENTENCES_startOffsets = { 0, 9, 16, 18, 22, 24, 29, 31, 40, 47, 48, 50, 52, 57 };
+// private static readonly int[] SENTENCES_endOffsets = { 8, 15, 17, 21, 23, 29, 30, 39, 46, 48, 49, 51, 57, 58 };
+// private static readonly String[] SENTENCES_chunks
+// = { "B-NP", "I-NP", "I-NP", "B-VP", "B-NP", "I-NP", "O", "B-NP", "I-NP", "I-NP", "O", "B-NP", "I-NP", "O" };
+
+// private const String sentenceModelFile = "en-test-sent.bin";
+// private const String tokenizerModelFile = "en-test-tokenizer.bin";
+// private const String posTaggerModelFile = "en-test-pos-maxent.bin";
+// private const String chunkerModelFile = "en-test-chunker.bin";
+
+
+// //private static byte[][] toPayloads(params string[] strings)
+// // {
+// // //return Arrays.stream(strings).map(s->s == null ? null : s.getBytes(StandardCharsets.UTF_8)).toArray(byte[][]::new);
+// // return strings.SelectMany(s => s == null ? null : Encoding.UTF8.GetBytes(s)).ToArray();
+// // }
+
+// [Test]
+// public void TestBasic()
+// {
+// // // CustomAnalyzer analyzer = CustomAnalyzer.builder(new ClasspathResourceLoader(getClass()))
+// // //.withTokenizer("opennlp", "tokenizerModel", tokenizerModelFile, "sentenceModel", sentenceModelFile)
+// // //.addTokenFilter("opennlpPOS", "posTaggerModel", posTaggerModelFile)
+// // //.addTokenFilter("opennlpChunker", "chunkerModel", chunkerModelFile)
+// // //.build();
+
+// Analyzer analyzer = Analyzer.NewAnonymous(createComponents: (fieldName, reader) =>
+// {
+// var loader = new ClasspathResourceLoader(GetType());
+
+// var opennlpFactory = new OpenNLPTokenizerFactory(new Dictionary<string, string> { { "tokenizerModel", tokenizerModelFile }, { "sentenceModel", sentenceModelFile } });
+// opennlpFactory.Inform(loader);
+// var opennlp = opennlpFactory.Create(AttributeFactory.Def //new OpenNLPTokenizer(reader, new Tools.NLPSentenceDetectorOp(sentenceModelFile), new Tools.NLPTokenizerOp(tokenizerModelFile));
+// var filter1 = new OpenNLPPOSFilter(opennlp, new Tools.NLPPOSTaggerOp(posTaggerModelFile));
+// var filter2 = new OpenNLPChunkerFilter(filter1, new Tools.NLPChunkerOp(chunkerModelFile));
+// return new TokenStreamComponents(opennlp, filter2);
+// });
+
+// AssertAnalyzesTo(analyzer, SENTENCES, SENTENCES_punc, SENTENCES_startOffsets, SENTENCES_endOffsets,
+// SENTENCES_chunks, null, null, true);
+// }
+
+// [Test]
+// public void TestPayloads()
+// {
+// // //CustomAnalyzer analyzer = CustomAnalyzer.builder(new ClasspathResourceLoader(getClass()))
+// // //.withTokenizer("opennlp", "tokenizerModel", tokenizerModelFile, "sentenceModel", sentenceModelFile)
+// // //.addTokenFilter("opennlpPOS", "posTaggerModel", posTaggerModelFile)
+// // //.addTokenFilter("opennlpChunker", "chunkerModel", chunkerModelFile)
+// // //.addTokenFilter(TypeAsPayloadTokenFilterFactory.class)
+// // //.build();
+
+// // Analyzer analyzer = Analyzer.NewAnonymous(createComponents: (fieldName, reader) =>
+// // {
+// // var opennlp = new OpenNLPTokenizer(reader, new Tools.NLPSentenceDetectorOp(sentenceModelFile), new Tools.NLPTokenizerOp(tokenizerModelFile));
+// // var filter1 = new OpenNLPPOSFilter(opennlp, new Tools.NLPPOSTaggerOp(posTaggerModelFile));
+// // var filter2 = new OpenNLPChunkerFilter(filter1, new Tools.NLPChunkerOp(chunkerModelFile));
+// // var filter3 = new Payloads.TypeAsPayloadTokenFilter(filter2);
+// // return new TokenStreamComponents(opennlp, filter3);
+// // });
+// // AssertAnalyzesTo(analyzer, SENTENCES, SENTENCES_punc, SENTENCES_startOffsets, SENTENCES_endOffsets,
+// //null, null, null, true, toPayloads(SENTENCES_chunks));
+//}
+// }
+//}
[lucenenet] 07/13: SWEEP: Changed test target framework from
netcoreapp1.0 to netcoreapp1.1
Posted by ni...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
nightowl888 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/lucenenet.git
commit 9bd6520d4fd40b1514f2c8a16f630663764069d7
Author: Shad Storhaug <sh...@shadstorhaug.com>
AuthorDate: Fri Oct 25 21:23:58 2019 +0700
SWEEP: Changed test target framework from netcoreapp1.0 to netcoreapp1.1
---
TestTargetFramework.props | 4 ++--
azure-pipelines.yml | 6 +++---
.../azure-templates/publish-test-results-for-target-frameworks.yml | 2 +-
build/build.ps1 | 2 +-
.../Lucene.Net.Tests.Analysis.Common.csproj | 4 ++--
.../Lucene.Net.Tests.Analysis.Kuromoji.csproj | 6 +++---
.../Lucene.Net.Tests.Analysis.Phonetic.csproj | 4 ++--
.../Lucene.Net.Tests.Analysis.SmartCn.csproj | 6 +++---
.../Lucene.Net.Tests.Analysis.Stempel.csproj | 4 ++--
src/Lucene.Net.Tests.Benchmark/Lucene.Net.Tests.Benchmark.csproj | 6 +++---
.../Lucene.Net.Tests.Classification.csproj | 4 ++--
src/Lucene.Net.Tests.Codecs/Lucene.Net.Tests.Codecs.csproj | 4 ++--
src/Lucene.Net.Tests.Demo/Lucene.Net.Tests.Demo.csproj | 4 ++--
.../Lucene.Net.Tests.Expressions.csproj | 4 ++--
src/Lucene.Net.Tests.Facet/Lucene.Net.Tests.Facet.csproj | 4 ++--
src/Lucene.Net.Tests.Grouping/Lucene.Net.Tests.Grouping.csproj | 4 ++--
.../Lucene.Net.Tests.Highlighter.csproj | 6 +++---
src/Lucene.Net.Tests.Join/Lucene.Net.Tests.Join.csproj | 4 ++--
src/Lucene.Net.Tests.Memory/Lucene.Net.Tests.Memory.csproj | 4 ++--
src/Lucene.Net.Tests.Misc/Lucene.Net.Tests.Misc.csproj | 4 ++--
src/Lucene.Net.Tests.Queries/Lucene.Net.Tests.Queries.csproj | 4 ++--
.../Lucene.Net.Tests.QueryParser.csproj | 4 ++--
src/Lucene.Net.Tests.Replicator/Lucene.Net.Tests.Replicator.csproj | 4 ++--
src/Lucene.Net.Tests.Sandbox/Lucene.Net.Tests.Sandbox.csproj | 4 ++--
src/Lucene.Net.Tests.Spatial/Lucene.Net.Tests.Spatial.csproj | 6 +++---
src/Lucene.Net.Tests.Suggest/Lucene.Net.Tests.Suggest.csproj | 4 ++--
.../Lucene.Net.Tests.TestFramework.NUnit.csproj | 4 ++--
.../Lucene.Net.Tests.TestFramework.xUnit.csproj | 4 ++--
src/Lucene.Net.Tests._A-D/Lucene.Net.Tests._A-D.csproj | 4 ++--
src/Lucene.Net.Tests._E-I/Lucene.Net.Tests._E-I.csproj | 4 ++--
src/Lucene.Net.Tests._J-S/Lucene.Net.Tests._J-S.csproj | 6 ++----
src/Lucene.Net.Tests._T-U/Lucene.Net.Tests._T-U.csproj | 4 ++--
src/Lucene.Net.Tests._U-Z/Lucene.Net.Tests._U-Z.csproj | 4 ++--
src/Lucene.Net.Tests/Support/TestLurchTableThreading.cs | 4 ++--
src/dotnet/Lucene.Net.Tests.ICU/Lucene.Net.Tests.ICU.csproj | 6 +++---
35 files changed, 75 insertions(+), 77 deletions(-)
diff --git a/TestTargetFramework.props b/TestTargetFramework.props
index f202cb1..1eadb7a 100644
--- a/TestTargetFramework.props
+++ b/TestTargetFramework.props
@@ -27,13 +27,13 @@
Note that the main libraries are multi-targeted, so this has no effect on how they are compiled,
this setting only affects the test projects. -->
<!--<TargetFramework>net451</TargetFramework>-->
- <!--<TargetFramework>netcoreapp1.0</TargetFramework>-->
+ <!--<TargetFramework>netcoreapp1.1</TargetFramework>-->
<TargetFramework>netcoreapp2.1</TargetFramework>
<!-- Allow the build script to pass in the test frameworks to build for.
This overrides the above TargetFramework setting.
LUCENENET TODO: Due to a parsing bug, we cannot pass a string with a ; to dotnet msbuild, so passing true as a workaround -->
- <TargetFrameworks Condition=" '$(TestFrameworks)' == 'true' ">netcoreapp2.1;netcoreapp1.0</TargetFrameworks>
+ <TargetFrameworks Condition=" '$(TestFrameworks)' == 'true' ">netcoreapp2.1;netcoreapp1.1</TargetFrameworks>
<TargetFrameworks Condition=" '$(TestFrameworks)' == 'true' AND $([MSBuild]::IsOsPlatform('Windows')) ">$(TargetFrameworks);net451</TargetFrameworks>
<TargetFramework Condition=" '$(TargetFrameworks)' != '' "></TargetFramework>
</PropertyGroup>
diff --git a/azure-pipelines.yml b/azure-pipelines.yml
index f57b108..4fd21d2 100644
--- a/azure-pipelines.yml
+++ b/azure-pipelines.yml
@@ -224,7 +224,7 @@ stages:
maximumParallelJobs: 7
maximumAllowedFailures: $(maximumAllowedFailures)
- - job: Test_netcoreapp1_0
+ - job: Test_netcoreapp1_1
condition: and(succeeded(), ne(variables['RunTests'], 'false'))
strategy:
matrix:
@@ -240,14 +240,14 @@ stages:
osName: 'macOS'
imageName: 'macOS-10.14'
maximumAllowedFailures: 4 # Maximum allowed failures for a successful build
- displayName: 'Test netcoreapp1.0 on'
+ displayName: 'Test netcoreapp1.1 on'
pool:
vmImage: $(imageName)
steps:
- template: 'build/azure-templates/run-tests-on-os.yml'
parameters:
osName: $(osName)
- testTargetFrameworks: 'netcoreapp1.0'
+ testTargetFrameworks: 'netcoreapp1.1'
testResultsArtifactName: '$(TestResultsArtifactName)'
publishedArtifactZipFileName: '$(PublishedArtifactZipFileName)'
maximumParallelJobs: 7
diff --git a/build/azure-templates/publish-test-results-for-target-frameworks.yml b/build/azure-templates/publish-test-results-for-target-frameworks.yml
index cc3af4f..009c44e 100644
--- a/build/azure-templates/publish-test-results-for-target-frameworks.yml
+++ b/build/azure-templates/publish-test-results-for-target-frameworks.yml
@@ -50,7 +50,7 @@ steps:
- template: publish-test-results.yml
parameters:
- framework: 'netcoreapp1.0'
+ framework: 'netcoreapp1.1'
testProjectName: '${{ parameters.testProjectName }}'
osName: '${{ parameters.osName }}'
testResultsFormat: '${{ parameters.testResultsFormat }}'
diff --git a/build/build.ps1 b/build/build.ps1
index 9693969..a3eec77 100644
--- a/build/build.ps1
+++ b/build/build.ps1
@@ -48,7 +48,7 @@ properties {
[int]$maximumParalellJobs = 8
#test paramters
- [string]$frameworks_to_test = "netcoreapp2.1,netcoreapp1.0,net451"
+ [string]$frameworks_to_test = "netcoreapp2.1,netcoreapp1.1,net451"
[string]$where = ""
}
diff --git a/src/Lucene.Net.Tests.Analysis.Common/Lucene.Net.Tests.Analysis.Common.csproj b/src/Lucene.Net.Tests.Analysis.Common/Lucene.Net.Tests.Analysis.Common.csproj
index dad8ffd..b759dde 100644
--- a/src/Lucene.Net.Tests.Analysis.Common/Lucene.Net.Tests.Analysis.Common.csproj
+++ b/src/Lucene.Net.Tests.Analysis.Common/Lucene.Net.Tests.Analysis.Common.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests.Analysis.Common</AssemblyTitle>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
@@ -47,7 +47,7 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD;NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests.Analysis.Kuromoji/Lucene.Net.Tests.Analysis.Kuromoji.csproj b/src/Lucene.Net.Tests.Analysis.Kuromoji/Lucene.Net.Tests.Analysis.Kuromoji.csproj
index c41db1b..a368e2a 100644
--- a/src/Lucene.Net.Tests.Analysis.Kuromoji/Lucene.Net.Tests.Analysis.Kuromoji.csproj
+++ b/src/Lucene.Net.Tests.Analysis.Kuromoji/Lucene.Net.Tests.Analysis.Kuromoji.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests.Analysis.Kuromoji</AssemblyTitle>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
@@ -46,11 +46,11 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <ItemGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <ItemGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<PackageReference Include="System.Net.Primitives" Version="$(SystemNetPrimitivesPackageVersion)" />
</ItemGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests.Analysis.Phonetic/Lucene.Net.Tests.Analysis.Phonetic.csproj b/src/Lucene.Net.Tests.Analysis.Phonetic/Lucene.Net.Tests.Analysis.Phonetic.csproj
index e7c97e2..b3b52a4 100644
--- a/src/Lucene.Net.Tests.Analysis.Phonetic/Lucene.Net.Tests.Analysis.Phonetic.csproj
+++ b/src/Lucene.Net.Tests.Analysis.Phonetic/Lucene.Net.Tests.Analysis.Phonetic.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests.Analysis.Phonetic</AssemblyTitle>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
@@ -43,7 +43,7 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD;NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests.Analysis.SmartCn/Lucene.Net.Tests.Analysis.SmartCn.csproj b/src/Lucene.Net.Tests.Analysis.SmartCn/Lucene.Net.Tests.Analysis.SmartCn.csproj
index 639ab03..8573c22 100644
--- a/src/Lucene.Net.Tests.Analysis.SmartCn/Lucene.Net.Tests.Analysis.SmartCn.csproj
+++ b/src/Lucene.Net.Tests.Analysis.SmartCn/Lucene.Net.Tests.Analysis.SmartCn.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests.Analysis.SmartCn</AssemblyTitle>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
@@ -46,11 +46,11 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <ItemGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <ItemGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<PackageReference Include="System.Net.Primitives" Version="$(SystemNetPrimitivesPackageVersion)" />
</ItemGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests.Analysis.Stempel/Lucene.Net.Tests.Analysis.Stempel.csproj b/src/Lucene.Net.Tests.Analysis.Stempel/Lucene.Net.Tests.Analysis.Stempel.csproj
index e691aae..c8a5754 100644
--- a/src/Lucene.Net.Tests.Analysis.Stempel/Lucene.Net.Tests.Analysis.Stempel.csproj
+++ b/src/Lucene.Net.Tests.Analysis.Stempel/Lucene.Net.Tests.Analysis.Stempel.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests.Analysis.Stempel</AssemblyTitle>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
@@ -46,7 +46,7 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests.Benchmark/Lucene.Net.Tests.Benchmark.csproj b/src/Lucene.Net.Tests.Benchmark/Lucene.Net.Tests.Benchmark.csproj
index eb82941..b8a1943 100644
--- a/src/Lucene.Net.Tests.Benchmark/Lucene.Net.Tests.Benchmark.csproj
+++ b/src/Lucene.Net.Tests.Benchmark/Lucene.Net.Tests.Benchmark.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests.Benchmark</AssemblyTitle>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
@@ -53,11 +53,11 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <ItemGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <ItemGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<PackageReference Include="System.Net.Primitives" Version="$(SystemNetPrimitivesPackageVersion)" />
</ItemGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests.Classification/Lucene.Net.Tests.Classification.csproj b/src/Lucene.Net.Tests.Classification/Lucene.Net.Tests.Classification.csproj
index a481236..135d98c 100644
--- a/src/Lucene.Net.Tests.Classification/Lucene.Net.Tests.Classification.csproj
+++ b/src/Lucene.Net.Tests.Classification/Lucene.Net.Tests.Classification.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests.Classification</AssemblyTitle>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
@@ -43,7 +43,7 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests.Codecs/Lucene.Net.Tests.Codecs.csproj b/src/Lucene.Net.Tests.Codecs/Lucene.Net.Tests.Codecs.csproj
index dd9e8c7..7fad164 100644
--- a/src/Lucene.Net.Tests.Codecs/Lucene.Net.Tests.Codecs.csproj
+++ b/src/Lucene.Net.Tests.Codecs/Lucene.Net.Tests.Codecs.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests.Codecs</AssemblyTitle>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
@@ -44,7 +44,7 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD;NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests.Demo/Lucene.Net.Tests.Demo.csproj b/src/Lucene.Net.Tests.Demo/Lucene.Net.Tests.Demo.csproj
index 44b2fc6..0669018 100644
--- a/src/Lucene.Net.Tests.Demo/Lucene.Net.Tests.Demo.csproj
+++ b/src/Lucene.Net.Tests.Demo/Lucene.Net.Tests.Demo.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests.Demo</AssemblyTitle>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
@@ -46,7 +46,7 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests.Expressions/Lucene.Net.Tests.Expressions.csproj b/src/Lucene.Net.Tests.Expressions/Lucene.Net.Tests.Expressions.csproj
index 07ced3e..0d087fe 100644
--- a/src/Lucene.Net.Tests.Expressions/Lucene.Net.Tests.Expressions.csproj
+++ b/src/Lucene.Net.Tests.Expressions/Lucene.Net.Tests.Expressions.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests.Expressions</AssemblyTitle>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
@@ -43,7 +43,7 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests.Facet/Lucene.Net.Tests.Facet.csproj b/src/Lucene.Net.Tests.Facet/Lucene.Net.Tests.Facet.csproj
index bae2a72..d291ae5 100644
--- a/src/Lucene.Net.Tests.Facet/Lucene.Net.Tests.Facet.csproj
+++ b/src/Lucene.Net.Tests.Facet/Lucene.Net.Tests.Facet.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests.Facet</AssemblyTitle>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
@@ -44,7 +44,7 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests.Grouping/Lucene.Net.Tests.Grouping.csproj b/src/Lucene.Net.Tests.Grouping/Lucene.Net.Tests.Grouping.csproj
index 0873c4b..f4b6c36 100644
--- a/src/Lucene.Net.Tests.Grouping/Lucene.Net.Tests.Grouping.csproj
+++ b/src/Lucene.Net.Tests.Grouping/Lucene.Net.Tests.Grouping.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests.Grouping</AssemblyTitle>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
@@ -43,7 +43,7 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests.Highlighter/Lucene.Net.Tests.Highlighter.csproj b/src/Lucene.Net.Tests.Highlighter/Lucene.Net.Tests.Highlighter.csproj
index eea47a0..80a5b53 100644
--- a/src/Lucene.Net.Tests.Highlighter/Lucene.Net.Tests.Highlighter.csproj
+++ b/src/Lucene.Net.Tests.Highlighter/Lucene.Net.Tests.Highlighter.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests.Highlighter</AssemblyTitle>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
@@ -47,11 +47,11 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <ItemGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <ItemGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<PackageReference Include="System.Xml.XmlDocument" Version="$(SystemXmlXmlDocumentPackageVersion)" />
</ItemGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests.Join/Lucene.Net.Tests.Join.csproj b/src/Lucene.Net.Tests.Join/Lucene.Net.Tests.Join.csproj
index a64d962..8988604 100644
--- a/src/Lucene.Net.Tests.Join/Lucene.Net.Tests.Join.csproj
+++ b/src/Lucene.Net.Tests.Join/Lucene.Net.Tests.Join.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests.Join</AssemblyTitle>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
@@ -43,7 +43,7 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests.Memory/Lucene.Net.Tests.Memory.csproj b/src/Lucene.Net.Tests.Memory/Lucene.Net.Tests.Memory.csproj
index cd8212b..d80ebc3 100644
--- a/src/Lucene.Net.Tests.Memory/Lucene.Net.Tests.Memory.csproj
+++ b/src/Lucene.Net.Tests.Memory/Lucene.Net.Tests.Memory.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests.Memory</AssemblyTitle>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
@@ -47,7 +47,7 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests.Misc/Lucene.Net.Tests.Misc.csproj b/src/Lucene.Net.Tests.Misc/Lucene.Net.Tests.Misc.csproj
index f36e9ef..8a0877d 100644
--- a/src/Lucene.Net.Tests.Misc/Lucene.Net.Tests.Misc.csproj
+++ b/src/Lucene.Net.Tests.Misc/Lucene.Net.Tests.Misc.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests.Misc</AssemblyTitle>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
@@ -42,7 +42,7 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests.Queries/Lucene.Net.Tests.Queries.csproj b/src/Lucene.Net.Tests.Queries/Lucene.Net.Tests.Queries.csproj
index 3fc9265..6aa1223 100644
--- a/src/Lucene.Net.Tests.Queries/Lucene.Net.Tests.Queries.csproj
+++ b/src/Lucene.Net.Tests.Queries/Lucene.Net.Tests.Queries.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests.Queries</AssemblyTitle>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
@@ -42,7 +42,7 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests.QueryParser/Lucene.Net.Tests.QueryParser.csproj b/src/Lucene.Net.Tests.QueryParser/Lucene.Net.Tests.QueryParser.csproj
index 5cb1cf4..f13a66b 100644
--- a/src/Lucene.Net.Tests.QueryParser/Lucene.Net.Tests.QueryParser.csproj
+++ b/src/Lucene.Net.Tests.QueryParser/Lucene.Net.Tests.QueryParser.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests.QueryParser</AssemblyTitle>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
@@ -47,7 +47,7 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests.Replicator/Lucene.Net.Tests.Replicator.csproj b/src/Lucene.Net.Tests.Replicator/Lucene.Net.Tests.Replicator.csproj
index 2c5fb3c..37c9fa5 100644
--- a/src/Lucene.Net.Tests.Replicator/Lucene.Net.Tests.Replicator.csproj
+++ b/src/Lucene.Net.Tests.Replicator/Lucene.Net.Tests.Replicator.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests.Replicator</AssemblyTitle>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
@@ -49,7 +49,7 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD;NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests.Sandbox/Lucene.Net.Tests.Sandbox.csproj b/src/Lucene.Net.Tests.Sandbox/Lucene.Net.Tests.Sandbox.csproj
index 0da4bcd..d9b3f88 100644
--- a/src/Lucene.Net.Tests.Sandbox/Lucene.Net.Tests.Sandbox.csproj
+++ b/src/Lucene.Net.Tests.Sandbox/Lucene.Net.Tests.Sandbox.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests.Sandbox</AssemblyTitle>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
@@ -47,7 +47,7 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests.Spatial/Lucene.Net.Tests.Spatial.csproj b/src/Lucene.Net.Tests.Spatial/Lucene.Net.Tests.Spatial.csproj
index af18cfa..786e0fe 100644
--- a/src/Lucene.Net.Tests.Spatial/Lucene.Net.Tests.Spatial.csproj
+++ b/src/Lucene.Net.Tests.Spatial/Lucene.Net.Tests.Spatial.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests.Spatial</AssemblyTitle>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
@@ -51,11 +51,11 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <ItemGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <ItemGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<PackageReference Include="System.Net.Primitives" Version="$(SystemNetPrimitivesPackageVersion)" />
</ItemGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests.Suggest/Lucene.Net.Tests.Suggest.csproj b/src/Lucene.Net.Tests.Suggest/Lucene.Net.Tests.Suggest.csproj
index 0b6ff79..5ecf694 100644
--- a/src/Lucene.Net.Tests.Suggest/Lucene.Net.Tests.Suggest.csproj
+++ b/src/Lucene.Net.Tests.Suggest/Lucene.Net.Tests.Suggest.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests.Suggest</AssemblyTitle>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
@@ -49,7 +49,7 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests.TestFramework.NUnit/Lucene.Net.Tests.TestFramework.NUnit.csproj b/src/Lucene.Net.Tests.TestFramework.NUnit/Lucene.Net.Tests.TestFramework.NUnit.csproj
index b9734d9..e875305 100644
--- a/src/Lucene.Net.Tests.TestFramework.NUnit/Lucene.Net.Tests.TestFramework.NUnit.csproj
+++ b/src/Lucene.Net.Tests.TestFramework.NUnit/Lucene.Net.Tests.TestFramework.NUnit.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests.TestFramework.NUnit</AssemblyTitle>
<RootNamespace>Lucene.Net</RootNamespace>
@@ -51,7 +51,7 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests.TestFramework.xUnit/Lucene.Net.Tests.TestFramework.xUnit.csproj b/src/Lucene.Net.Tests.TestFramework.xUnit/Lucene.Net.Tests.TestFramework.xUnit.csproj
index cab7dc7..cef9104 100644
--- a/src/Lucene.Net.Tests.TestFramework.xUnit/Lucene.Net.Tests.TestFramework.xUnit.csproj
+++ b/src/Lucene.Net.Tests.TestFramework.xUnit/Lucene.Net.Tests.TestFramework.xUnit.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests.TestFramework.xUnit</AssemblyTitle>
<RootNamespace>Lucene.Net</RootNamespace>
@@ -55,7 +55,7 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests._A-D/Lucene.Net.Tests._A-D.csproj b/src/Lucene.Net.Tests._A-D/Lucene.Net.Tests._A-D.csproj
index 8987bdf..9e6dd36 100644
--- a/src/Lucene.Net.Tests._A-D/Lucene.Net.Tests._A-D.csproj
+++ b/src/Lucene.Net.Tests._A-D/Lucene.Net.Tests._A-D.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests._A-D</AssemblyTitle>
<RootNamespace>Lucene.Net</RootNamespace>
@@ -58,7 +58,7 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD;NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests._E-I/Lucene.Net.Tests._E-I.csproj b/src/Lucene.Net.Tests._E-I/Lucene.Net.Tests._E-I.csproj
index 90d0940..a5d5f68 100644
--- a/src/Lucene.Net.Tests._E-I/Lucene.Net.Tests._E-I.csproj
+++ b/src/Lucene.Net.Tests._E-I/Lucene.Net.Tests._E-I.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests._E-I</AssemblyTitle>
<RootNamespace>Lucene.Net</RootNamespace>
@@ -56,7 +56,7 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD;NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests._J-S/Lucene.Net.Tests._J-S.csproj b/src/Lucene.Net.Tests._J-S/Lucene.Net.Tests._J-S.csproj
index 4c7236c..66b4484 100644
--- a/src/Lucene.Net.Tests._J-S/Lucene.Net.Tests._J-S.csproj
+++ b/src/Lucene.Net.Tests._J-S/Lucene.Net.Tests._J-S.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests._J-S</AssemblyTitle>
<RootNamespace>Lucene.Net</RootNamespace>
@@ -57,8 +57,6 @@
<DefineConstants>$(DefineConstants);NETSTANDARD;FEATURE_CONCURRENTMERGESCHEDULER;FEATURE_SERIALIZABLE</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
-
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
<PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD;NETSTANDARD1_6</DefineConstants>
@@ -70,7 +68,7 @@
<DebugType>full</DebugType>
</PropertyGroup>
- <ItemGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <ItemGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<Compile Remove="..\Lucene.Net.Tests\Support\TestCase.cs" />
</ItemGroup>
diff --git a/src/Lucene.Net.Tests._T-U/Lucene.Net.Tests._T-U.csproj b/src/Lucene.Net.Tests._T-U/Lucene.Net.Tests._T-U.csproj
index 5fabbd5..b73fa87 100644
--- a/src/Lucene.Net.Tests._T-U/Lucene.Net.Tests._T-U.csproj
+++ b/src/Lucene.Net.Tests._T-U/Lucene.Net.Tests._T-U.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests._T-U</AssemblyTitle>
<RootNamespace>Lucene.Net</RootNamespace>
@@ -57,7 +57,7 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD;NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests._U-Z/Lucene.Net.Tests._U-Z.csproj b/src/Lucene.Net.Tests._U-Z/Lucene.Net.Tests._U-Z.csproj
index e67536c..e227333 100644
--- a/src/Lucene.Net.Tests._U-Z/Lucene.Net.Tests._U-Z.csproj
+++ b/src/Lucene.Net.Tests._U-Z/Lucene.Net.Tests._U-Z.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests._U-Z</AssemblyTitle>
<RootNamespace>Lucene.Net</RootNamespace>
@@ -59,7 +59,7 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD;NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
diff --git a/src/Lucene.Net.Tests/Support/TestLurchTableThreading.cs b/src/Lucene.Net.Tests/Support/TestLurchTableThreading.cs
index 389696b..0e08523 100644
--- a/src/Lucene.Net.Tests/Support/TestLurchTableThreading.cs
+++ b/src/Lucene.Net.Tests/Support/TestLurchTableThreading.cs
@@ -77,7 +77,7 @@ namespace Lucene.Net.Support
// LUCENENET TODO: For some reason, this logic depends on the underlying
// implementation of Guid.NewGuid(), which has changed in .NET Core 2.0.
// But the functionality of LurchTable seems unaffected by this change.
-#if !NETCOREAPP2_1
+#if !NETCOREAPP
[Test, LuceneNetSpecific]
public void TestGuidHashCollision()
{
@@ -248,7 +248,7 @@ namespace Lucene.Net.Support
);
Guid result = new Guid(bytes);
-#if !NETCOREAPP2_1
+#if !NETCOREAPP
Assert.AreEqual(guid.GetHashCode(), result.GetHashCode());
#endif
return result;
diff --git a/src/dotnet/Lucene.Net.Tests.ICU/Lucene.Net.Tests.ICU.csproj b/src/dotnet/Lucene.Net.Tests.ICU/Lucene.Net.Tests.ICU.csproj
index e7de7c8..1efa553 100644
--- a/src/dotnet/Lucene.Net.Tests.ICU/Lucene.Net.Tests.ICU.csproj
+++ b/src/dotnet/Lucene.Net.Tests.ICU/Lucene.Net.Tests.ICU.csproj
@@ -24,7 +24,7 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
+ <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<AssemblyTitle>Lucene.Net.Tests.ICU</AssemblyTitle>
<DefineConstants>$(DefineConstants);FEATURE_BREAKITERATOR</DefineConstants>
@@ -68,7 +68,7 @@
<PackageReference Include="ICU4N.Transliterator" Version="$(ICU4NTransliteratorPackageVersion)" />
</ItemGroup>
- <ItemGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <ItemGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<PackageReference Include="System.Net.Primitives" Version="$(SystemNetPrimitivesPackageVersion)" />
<PackageReference Include="System.Xml.XmlDocument" Version="$(SystemXmlXmlDocumentPackageVersion)" />
</ItemGroup>
@@ -78,7 +78,7 @@
<DebugType>portable</DebugType>
</PropertyGroup>
- <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
<DefineConstants>$(DefineConstants);NETSTANDARD;NETSTANDARD1_6</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
[lucenenet] 04/13: Lucene.Net.TestFramework: Refactored Assert
classes to utilize FileSupport.IsFileAlreadyExistsException(). Deleted
Support/TestFramework/AssertCommon.cs.
Posted by ni...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
nightowl888 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/lucenenet.git
commit eecbcda0162e6dc0c8ec376858e80c5593616dea
Author: Shad Storhaug <sh...@shadstorhaug.com>
AuthorDate: Fri Oct 25 19:02:09 2019 +0700
Lucene.Net.TestFramework: Refactored Assert classes to utilize FileSupport.IsFileAlreadyExistsException(). Deleted Support/TestFramework/AssertCommon.cs.
---
.../Support/TestFramework/Assert.cs | 3 +-
.../Support/TestFramework/Assert.cs | 3 +-
.../Support/TestFramework/Assert.cs | 3 +-
.../Support/TestFramework/AssertCommon.cs | 41 ----------------------
4 files changed, 6 insertions(+), 44 deletions(-)
diff --git a/src/Lucene.Net.TestFramework.MSTest/Support/TestFramework/Assert.cs b/src/Lucene.Net.TestFramework.MSTest/Support/TestFramework/Assert.cs
index ce14510..a0c688c 100644
--- a/src/Lucene.Net.TestFramework.MSTest/Support/TestFramework/Assert.cs
+++ b/src/Lucene.Net.TestFramework.MSTest/Support/TestFramework/Assert.cs
@@ -1,4 +1,5 @@
using Lucene.Net.Support;
+using Lucene.Net.Support.IO;
using System;
using System.Collections;
using System.Collections.Generic;
@@ -584,7 +585,7 @@ namespace Lucene.Net.TestFramework
{
action();
}
- catch (Exception ex) when (!IsFileAlreadyExistsException(ex, filePath))
+ catch (Exception ex) when (!FileSupport.IsFileAlreadyExistsException(ex, filePath))
{
exception = ex;
}
diff --git a/src/Lucene.Net.TestFramework.NUnit/Support/TestFramework/Assert.cs b/src/Lucene.Net.TestFramework.NUnit/Support/TestFramework/Assert.cs
index ad0d733..42a1bed 100644
--- a/src/Lucene.Net.TestFramework.NUnit/Support/TestFramework/Assert.cs
+++ b/src/Lucene.Net.TestFramework.NUnit/Support/TestFramework/Assert.cs
@@ -1,4 +1,5 @@
using Lucene.Net.Support;
+using Lucene.Net.Support.IO;
using System;
using System.Collections;
using System.Collections.Generic;
@@ -562,7 +563,7 @@ namespace Lucene.Net.TestFramework
action();
throw new _NUnit.AssertionException($"{messagePrefix} <null>");
}
- catch (Exception ex) when (!IsFileAlreadyExistsException(ex, filePath))
+ catch (Exception ex) when (!FileSupport.IsFileAlreadyExistsException(ex, filePath))
{
throw new _NUnit.AssertionException($"{messagePrefix} {ex.GetType().FullName}", ex);
}
diff --git a/src/Lucene.Net.TestFramework.xUnit/Support/TestFramework/Assert.cs b/src/Lucene.Net.TestFramework.xUnit/Support/TestFramework/Assert.cs
index 49ffb16..e086675 100644
--- a/src/Lucene.Net.TestFramework.xUnit/Support/TestFramework/Assert.cs
+++ b/src/Lucene.Net.TestFramework.xUnit/Support/TestFramework/Assert.cs
@@ -1,4 +1,5 @@
using Lucene.Net.Support;
+using Lucene.Net.Support.IO;
using System;
using System.Collections;
using System.Collections.Generic;
@@ -567,7 +568,7 @@ namespace Lucene.Net.TestFramework
{
action();
}
- catch (Exception ex) when (!IsFileAlreadyExistsException(ex, filePath))
+ catch (Exception ex) when (!FileSupport.IsFileAlreadyExistsException(ex, filePath))
{
exception = ex;
}
diff --git a/src/Lucene.Net.TestFramework/Support/TestFramework/AssertCommon.cs b/src/Lucene.Net.TestFramework/Support/TestFramework/AssertCommon.cs
deleted file mode 100644
index c1c59bd..0000000
--- a/src/Lucene.Net.TestFramework/Support/TestFramework/AssertCommon.cs
+++ /dev/null
@@ -1,41 +0,0 @@
-using Lucene.Net.Util;
-using System;
-using System.IO;
-
-namespace Lucene.Net.TestFramework
-{
- /*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
- /// <summary>
- /// Common assertion code
- /// </summary>
- internal partial class Assert
- {
- private const int WARN_WIN32_FILE_EXISTS = unchecked((int)0x80070050);
-
- private static bool IsFileAlreadyExistsException(Exception ex, string filePath)
- {
- if (!typeof(IOException).Equals(ex))
- return false;
- else if (Constants.WINDOWS)
- return ex.HResult == WARN_WIN32_FILE_EXISTS;
- else
- return File.Exists(filePath);
- }
- }
-}
[lucenenet] 02/13:
Lucene.Net.TestFramework.Codecs.Lucene3x.PreFlexRWNormsConsumer: Changed
exceptions from InvalidOperationException to AssertionExeption (as they
were in Lucene)
Posted by ni...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
nightowl888 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/lucenenet.git
commit 2d29c1e17a881e9eae4210e8aa7ca89437ae1dc6
Author: Shad Storhaug <sh...@shadstorhaug.com>
AuthorDate: Sun Oct 27 00:32:50 2019 +0700
Lucene.Net.TestFramework.Codecs.Lucene3x.PreFlexRWNormsConsumer: Changed exceptions from InvalidOperationException to AssertionExeption (as they were in Lucene)
---
.../Codecs/Lucene3x/PreFlexRWNormsConsumer.cs | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/src/Lucene.Net.TestFramework/Codecs/Lucene3x/PreFlexRWNormsConsumer.cs b/src/Lucene.Net.TestFramework/Codecs/Lucene3x/PreFlexRWNormsConsumer.cs
index d18b95e..2e78a86 100644
--- a/src/Lucene.Net.TestFramework/Codecs/Lucene3x/PreFlexRWNormsConsumer.cs
+++ b/src/Lucene.Net.TestFramework/Codecs/Lucene3x/PreFlexRWNormsConsumer.cs
@@ -1,5 +1,6 @@
using System;
using Debug = Lucene.Net.Diagnostics.Debug; // LUCENENET NOTE: We cannot use System.Diagnostics.Debug because those calls will be optimized out of the release!
+using AssertionError = Lucene.Net.Diagnostics.AssertionException;
namespace Lucene.Net.Codecs.Lucene3x
{
@@ -100,17 +101,17 @@ namespace Lucene.Net.Codecs.Lucene3x
public override void AddBinaryField(FieldInfo field, IEnumerable<BytesRef> values)
{
- throw new InvalidOperationException(); // LUCENENET TODO: This should be AssertionError (AssertionException)
+ throw new AssertionError();
}
public override void AddSortedField(FieldInfo field, IEnumerable<BytesRef> values, IEnumerable<long?> docToOrd)
{
- throw new InvalidOperationException(); // LUCENENET TODO: This should be AssertionError (AssertionException)
+ throw new AssertionError();
}
public override void AddSortedSetField(FieldInfo field, IEnumerable<BytesRef> values, IEnumerable<long?> docToOrdCount, IEnumerable<long?> ords)
{
- throw new InvalidOperationException(); // LUCENENET TODO: This should be AssertionError (AssertionException)
+ throw new AssertionError();
}
}
}
\ No newline at end of file
[lucenenet] 10/13: run-tests-on-os.yml,
azure-pipelines.yml: Added test expression to filter tests,
filtered netcoreapp1.1 tests so long running tests are excluded. Changed VM
image to vs2017-win2016 because .NET Core support has been dropped from the
windows-2019 image.
Posted by ni...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
nightowl888 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/lucenenet.git
commit f93597b44b2e2a25b2277698e3ecf7023c2f7dbd
Author: Shad Storhaug <sh...@shadstorhaug.com>
AuthorDate: Sat Oct 26 00:14:36 2019 +0700
run-tests-on-os.yml, azure-pipelines.yml: Added test expression to filter tests, filtered netcoreapp1.1 tests so long running tests are excluded. Changed VM image to vs2017-win2016 because .NET Core support has been dropped from the windows-2019 image.
---
azure-pipelines.yml | 5 +++--
build/azure-templates/run-tests-on-os.yml | 6 ++++++
2 files changed, 9 insertions(+), 2 deletions(-)
diff --git a/azure-pipelines.yml b/azure-pipelines.yml
index 4fd21d2..fc0b895 100644
--- a/azure-pipelines.yml
+++ b/azure-pipelines.yml
@@ -230,7 +230,7 @@ stages:
matrix:
Windows:
osName: 'Windows'
- imageName: 'windows-2019'
+ imageName: 'vs2017-win2016'
maximumAllowedFailures: 4 # Maximum allowed failures for a successful build
Linux:
osName: 'Linux'
@@ -252,8 +252,9 @@ stages:
publishedArtifactZipFileName: '$(PublishedArtifactZipFileName)'
maximumParallelJobs: 7
maximumAllowedFailures: $(maximumAllowedFailures)
+ where: 'TestCategory!=LongRunningTest' # Skip long running tests on .NET Core 1.1
- - job: Test_net451
+ - job: Test_net451_Windows
condition: and(succeeded(), ne(variables['RunTests'], 'false'))
displayName: 'Test net451 on Windows'
pool:
diff --git a/build/azure-templates/run-tests-on-os.yml b/build/azure-templates/run-tests-on-os.yml
index 875cfdf..351c1cf 100644
--- a/build/azure-templates/run-tests-on-os.yml
+++ b/build/azure-templates/run-tests-on-os.yml
@@ -30,6 +30,7 @@ parameters:
publishedArtifactZipFileName: 'published.zip' # The name of the zip file (within the Azure DevOps build artifact binaryArtifactName) where the published test binaries can be extracted from. Default 'published.zip'.
maximumParallelJobs: 8
maximumAllowedFailures: 0
+ where: '' # A test filter expression, as defined by dotnet vstest
steps:
- powershell: |
@@ -75,6 +76,7 @@ steps:
$testBinaryFilesPattern = '${{ parameters.testBinaryFilesPattern }}'
$testResultsFileName = '${{ parameters.testResultsFileName }}'
$maximumParalellJobs = '${{ parameters.maximumParallelJobs }}'
+ $where = '${{ parameters.where }}'
$testTargetFrameworks = $testTargetFrameworksString.Split([char]',',[char]';')
function SeparateVersionDigits([string]$digits) {
@@ -145,6 +147,10 @@ steps:
" --logger:""console;verbosity=normal"" --logger:""trx;LogFileName=$testResultsFileName""" + `
" --ResultsDirectory:""$testResultDirectory"" --Blame"
+ if (![string]::IsNullOrEmpty($where)) {
+ $testExpression = "$testExpression --TestCaseFilter:""$where"""
+ }
+
Write-Host "Testing '$($testBinary.FullName)' on framework '$fwork' and outputting test results to '$testResultDirectory/$testResultsFileName'..."
Write-Host $testExpression -ForegroundColor Magenta
if ($maximumParalellJobs -le 1) {
[lucenenet] 05/13: build/TestReferences.Common.targets: Removed
TeamCity references and System.Runtime.InteropServices.RuntimeInformation
Posted by ni...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
nightowl888 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/lucenenet.git
commit b2352eaa22896edecfede61a97bc4473f3b0ac66
Author: Shad Storhaug <sh...@shadstorhaug.com>
AuthorDate: Fri Oct 25 23:44:05 2019 +0700
build/TestReferences.Common.targets: Removed TeamCity references and System.Runtime.InteropServices.RuntimeInformation
---
build/Dependencies.props | 3 ---
build/TestReferences.Common.targets | 4 ----
2 files changed, 7 deletions(-)
diff --git a/build/Dependencies.props b/build/Dependencies.props
index 50c3b54..625e6a7 100644
--- a/build/Dependencies.props
+++ b/build/Dependencies.props
@@ -50,7 +50,6 @@
<NewtonsoftJsonPackageVersion>10.0.3</NewtonsoftJsonPackageVersion>
<NUnit3TestAdapterPackageVersion>3.13.0</NUnit3TestAdapterPackageVersion>
<NUnitPackageVersion>3.9.0</NUnitPackageVersion>
- <NUnitExtensionTeamCityEventListenerPackageVersion>1.0.6</NUnitExtensionTeamCityEventListenerPackageVersion>
<SharpZipLibPackageVersion>1.1.0</SharpZipLibPackageVersion>
<SharpZipLibNETStandardPackageVersion>0.86.0.1</SharpZipLibNETStandardPackageVersion>
<Spatial4nCorePackageVersion>0.4.1</Spatial4nCorePackageVersion>
@@ -70,12 +69,10 @@
<SystemReflectionEmitPackageVersion>4.3.0</SystemReflectionEmitPackageVersion>
<SystemReflectionEmitILGenerationPackageVersion>4.3.0</SystemReflectionEmitILGenerationPackageVersion>
<SystemReflectionTypeExtensionsPackageVersion>4.3.0</SystemReflectionTypeExtensionsPackageVersion>
- <SystemRuntimeInteropServicesRuntimeInformationPackageVersion>4.3.0</SystemRuntimeInteropServicesRuntimeInformationPackageVersion>
<SystemTextEncodingCodePagesPackageVersion>4.3.0</SystemTextEncodingCodePagesPackageVersion>
<SystemThreadingThreadPackageVersion>4.3.0</SystemThreadingThreadPackageVersion>
<SystemThreadingThreadPoolPackageVersion>4.3.0</SystemThreadingThreadPoolPackageVersion>
<SystemXmlXmlDocumentPackageVersion>4.3.0</SystemXmlXmlDocumentPackageVersion>
- <TeamCityVSTestTestAdapterPackageVersion>1.0.15</TeamCityVSTestTestAdapterPackageVersion>
<XUnitPackageVersion>2.3.1</XUnitPackageVersion>
<XUnitRunnerVisualStudioPackageVersion>$(XUnitPackageVersion)</XUnitRunnerVisualStudioPackageVersion>
</PropertyGroup>
diff --git a/build/TestReferences.Common.targets b/build/TestReferences.Common.targets
index 6411a7b..4a6e513 100644
--- a/build/TestReferences.Common.targets
+++ b/build/TestReferences.Common.targets
@@ -23,9 +23,5 @@
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="$(MicrosoftNETTestSdkPackageVersion)" />
<PackageReference Include="NUnit" Version="$(NUnitPackageVersion)" />
<PackageReference Include="NUnit3TestAdapter" Version="$(NUnit3TestAdapterPackageVersion)" />
- <PackageReference Include="NUnit.Extension.TeamCityEventListener" Version="$(NUnitExtensionTeamCityEventListenerPackageVersion)" />
- <!-- Required by NUnit3TestAdapter on .NET Core -->
- <PackageReference Include="System.Runtime.InteropServices.RuntimeInformation" Version="$(SystemRuntimeInteropServicesRuntimeInformationPackageVersion)" />
- <PackageReference Include="TeamCity.VSTest.TestAdapter" Version="$(TeamCityVSTestTestAdapterPackageVersion)" />
</ItemGroup>
</Project>
\ No newline at end of file
[lucenenet] 09/13: build.ps1: Fixed where expression (dotnet vstest
uses --TestCaseFilter, not --filter, and must be separated with a colon)
Posted by ni...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
nightowl888 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/lucenenet.git
commit 4cc10c983d18c24851b44f88b4ec5815c601ad7f
Author: Shad Storhaug <sh...@shadstorhaug.com>
AuthorDate: Sat Oct 26 17:45:18 2019 +0700
build.ps1: Fixed where expression (dotnet vstest uses --TestCaseFilter, not --filter, and must be separated with a colon)
---
build/build.ps1 | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/build/build.ps1 b/build/build.ps1
index a3eec77..14513f3 100644
--- a/build/build.ps1
+++ b/build/build.ps1
@@ -323,8 +323,8 @@ task Test -depends InstallSDK, UpdateLocalSDKVersion, Restore -description "This
# doing release inspection and on the CI server.
$testExpression = "$testExpression --logger:""trx;LogFileName=TestResults.trx"""
- if ($where -ne $null -and (-Not [System.String]::IsNullOrEmpty($where))) {
- $testExpression = "$testExpression --filter $where"
+ if (![string]::IsNullOrEmpty($where)) {
+ $testExpression = "$testExpression --TestCaseFilter:""$where"""
}
Write-Host $testExpression -ForegroundColor Magenta
[lucenenet] 11/13: azure-pipelines.yml: Increased
maximumParallelJobs on net451 from 7 to 8
Posted by ni...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
nightowl888 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/lucenenet.git
commit 9a235297e262d063f8458de3597c8e675f3346ad
Author: Shad Storhaug <sh...@shadstorhaug.com>
AuthorDate: Sat Oct 26 18:45:09 2019 +0700
azure-pipelines.yml: Increased maximumParallelJobs on net451 from 7 to 8
---
azure-pipelines.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/azure-pipelines.yml b/azure-pipelines.yml
index fc0b895..f01a1d2 100644
--- a/azure-pipelines.yml
+++ b/azure-pipelines.yml
@@ -266,7 +266,7 @@ stages:
testTargetFrameworks: 'net451'
testResultsArtifactName: '$(TestResultsArtifactName)'
publishedArtifactZipFileName: '$(PublishedArtifactZipFileName)'
- maximumParallelJobs: 7
+ maximumParallelJobs: 8
maximumAllowedFailures: 4 # Maximum allowed failures for a successful build