You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pylucene-dev@lucene.apache.org by Peter Brown <pe...@griffithuni.edu.au> on 2017/05/05 01:29:11 UTC
JCC SimilarityBase
Hi all,
I was just wondering if anybody else has run into a similar issue before.
I’m trying to create a wrapper class for the org.apache.lucene.search.similarities.SimilarityBase class within PyLucene (similar to the existing wrapper for PythonClassicSimilarity).
However, upon re-building and re-installing PyLucene, when trying to use the wrapper class it fails with the following exception:
java.lang.UnsatisfiedLinkError: org.apache.pylucene.search.similarities.PythonSimilarityBase.score(Lorg/apache/lucene/search/similarities/BasicStats;FF)F
Upon examination of the wrap0*.cpp files, and comparing with PythonClassicSimilarity, I found that the score method is not being generated. E.g.
static PyMethodDef t_PythonSimilarityBase__methods_[] = {
DECLARE_METHOD(t_PythonSimilarityBase, cast_, METH_O | METH_CLASS),
DECLARE_METHOD(t_PythonSimilarityBase, instance_, METH_O | METH_CLASS),
DECLARE_METHOD(t_PythonSimilarityBase, finalize, METH_NOARGS),
DECLARE_METHOD(t_PythonSimilarityBase, pythonExtension, METH_VARARGS),
{ NULL, NULL, 0, NULL }
};
and
JNINativeMethod methods[] = {
{ "pythonDecRef", "()V", (void *) t_PythonSimilarityBase_pythonDecRef0 },
{ "toString", "()Ljava/lang/String;", (void *) t_PythonSimilarityBase_toString1 },
};
But it is present in the header file enum…
class PythonSimilarityBase : public ::org::apache::lucene::search::similarities::SimilarityBase {
public:
enum {
mid_init$_7353aa4eb69a6f70,
mid_finalize_7353aa4eb69a6f70,
mid_pythonDecRef_7353aa4eb69a6f70,
mid_pythonExtension_aae05472994e7446,
mid_pythonExtension_f41cd3dc7c7d5d78,
mid_toString_5df7257cf25bf439,
mid_score_a6c8466c4349b376,
max_mid
};
But not in the header file methods.
PythonSimilarityBase();
void finalize() const;
void pythonDecRef() const;
jlong pythonExtension() const;
void pythonExtension(jlong) const;
::java::lang::String toString() const;
Compared to
static PyMethodDef t_PythonClassicSimilarity__methods_[] = {
DECLARE_METHOD(t_PythonClassicSimilarity, cast_, METH_O | METH_CLASS),
DECLARE_METHOD(t_PythonClassicSimilarity, instance_, METH_O | METH_CLASS),
DECLARE_METHOD(t_PythonClassicSimilarity, coord, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, finalize, METH_NOARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, idf, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, idfExplain, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, lengthNorm, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, pythonExtension, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, queryNorm, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, sloppyFreq, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, tf, METH_VARARGS),
{ NULL, NULL, 0, NULL }
};
and
JNINativeMethod methods[] = {
{ "coord", "(II)F", (void *) t_PythonClassicSimilarity_coord0 },
{ "idf", "(JJ)F", (void *) t_PythonClassicSimilarity_idf1 },
{ "idfExplain", "(Lorg/apache/lucene/search/CollectionStatistics;[Lorg/apache/lucene/search/TermStatistics;)Lorg/apache/lucene/search/Explanation;", (void *) t_PythonClassicSimilarity_idfExplain2 },
{ "lengthNorm", "(Lorg/apache/lucene/index/FieldInvertState;)F", (void *) t_PythonClassicSimilarity_lengthNorm3 },
{ "pythonDecRef", "()V", (void *) t_PythonClassicSimilarity_pythonDecRef4 },
{ "queryNorm", "(F)F", (void *) t_PythonClassicSimilarity_queryNorm5 },
{ "sloppyFreq", "(I)F", (void *) t_PythonClassicSimilarity_sloppyFreq6 },
{ "tf", "(F)F", (void *) t_PythonClassicSimilarity_tf7 },
};
enum {
mid_init$_7353aa4eb69a6f70,
mid_coord_78a3b70e2a43b9c7,
mid_finalize_7353aa4eb69a6f70,
mid_idf_0af4b58f982c0dd1,
mid_idfExplain_08e4e4733921c6f1,
mid_lengthNorm_a7d67c4cd2e3b6a2,
mid_pythonDecRef_7353aa4eb69a6f70,
mid_pythonExtension_aae05472994e7446,
mid_pythonExtension_f41cd3dc7c7d5d78,
mid_queryNorm_eecff9f314592ce9,
mid_sloppyFreq_6b1faadec3ed3278,
mid_tf_eecff9f314592ce9,
max_mid
};
and
jfloat coord(jint, jint) const;
void finalize() const;
jfloat idf(jlong, jlong) const;
::org::apache::lucene::search::Explanation idfExplain(const ::org::apache::lucene::search::CollectionStatistics &, const JArray< ::org::apache::lucene::search::TermStatistics > &) const;
jfloat lengthNorm(const ::org::apache::lucene::index::FieldInvertState &) const;
void pythonDecRef() const;
jlong pythonExtension() const;
void pythonExtension(jlong) const;
jfloat queryNorm(jfloat) const;
jfloat sloppyFreq(jint) const;
jfloat tf(jfloat) const;
Here is the code for PythonSimilarityBase.java (stored in the same directory as PythonClassicSimilarity) which compiles without issues:
package org.apache.pylucene.search.similarities;
import org.apache.lucene.search.similarities.BasicStats;
import org.apache.lucene.search.similarities.SimilarityBase;
public class PythonSimilarityBase extends SimilarityBase {
private long pythonObject;
public PythonSimilarityBase()
{
}
public void pythonExtension(long pythonObject)
{
this.pythonObject = pythonObject;
}
public long pythonExtension()
{
return this.pythonObject;
}
public void finalize()
throws Throwable
{
pythonDecRef();
}
public native void pythonDecRef();
@Override
protected native float score(BasicStats stats, float freq, float docLen);
@Override
public native String toString();
}
I’ve also tried variations of adding the BasicStats class, the Similarity$SimWeight class and the whole similarities package into the JCC command, for example
JCC=$(PYTHON) -m jcc.__main__ --shared --arch x86_64 --package org.apache.lucene.search.similarities
(Trying to follow advice in https://issues.apache.org/jira/browse/PYLUCENE-21 <https://issues.apache.org/jira/browse/PYLUCENE-21>)
My feeling is that it has something to do with that BasicStats object…
If anyone is able to assist it would be greatly appreciated!!
Thanks,
Peter Brown
School of Information and Communication Technology,
Griffith University - Gold Coast Campus
Re: JCC SimilarityBase
Posted by Peter Brown <pe...@griffithuni.edu.au>.
Andi, you are a champion! :)
Thanks so much for your timely and helpful response.
Unfortunately, I had no luck adding the score function to the makefile.
However, I was able to resolve the issue by defining the method as public.
Peter Brown - s2623892
School of Information and Communication Technology,
Griffith University - Gold Coast Campus
> On 5 May 2017, at 11:58 am, Andi Vajda <va...@apache.org> wrote:
>
>
> On Fri, 5 May 2017, Peter Brown wrote:
>
>> Hi all,
>>
>> I was just wondering if anybody else has run into a similar issue before.
>> I?m trying to create a wrapper class for the org.apache.lucene.search.similarities.SimilarityBase class within PyLucene (similar to the existing wrapper for PythonClassicSimilarity).
>>
>> However, upon re-building and re-installing PyLucene, when trying to use the wrapper class it fails with the following exception:
>>
>> java.lang.UnsatisfiedLinkError: org.apache.pylucene.search.similarities.PythonSimilarityBase.score(Lorg/apache/lucene/search/similarities/BasicStats;FF)F
>>
>> Upon examination of the wrap0*.cpp files, and comparing with PythonClassicSimilarity, I found that the score method is not being generated. E.g.
>>
>> static PyMethodDef t_PythonSimilarityBase__methods_[] = {
>> DECLARE_METHOD(t_PythonSimilarityBase, cast_, METH_O | METH_CLASS),
>> DECLARE_METHOD(t_PythonSimilarityBase, instance_, METH_O | METH_CLASS),
>> DECLARE_METHOD(t_PythonSimilarityBase, finalize, METH_NOARGS),
>> DECLARE_METHOD(t_PythonSimilarityBase, pythonExtension, METH_VARARGS),
>> { NULL, NULL, 0, NULL }
>> };
>>
>> and
>>
>> JNINativeMethod methods[] = {
>> { "pythonDecRef", "()V", (void *) t_PythonSimilarityBase_pythonDecRef0 },
>> { "toString", "()Ljava/lang/String;", (void *) t_PythonSimilarityBase_toString1 },
>> };
>>
>> But it is present in the header file enum?
>> class PythonSimilarityBase : public ::org::apache::lucene::search::similarities::SimilarityBase {
>> public:
>> enum {
>> mid_init$_7353aa4eb69a6f70,
>> mid_finalize_7353aa4eb69a6f70,
>> mid_pythonDecRef_7353aa4eb69a6f70,
>> mid_pythonExtension_aae05472994e7446,
>> mid_pythonExtension_f41cd3dc7c7d5d78,
>> mid_toString_5df7257cf25bf439,
>> mid_score_a6c8466c4349b376,
>> max_mid
>> };
>>
>> But not in the header file methods.
>> PythonSimilarityBase();
>>
>> void finalize() const;
>> void pythonDecRef() const;
>> jlong pythonExtension() const;
>> void pythonExtension(jlong) const;
>> ::java::lang::String toString() const;
>>
>>
>> Compared to
>> static PyMethodDef t_PythonClassicSimilarity__methods_[] = {
>> DECLARE_METHOD(t_PythonClassicSimilarity, cast_, METH_O | METH_CLASS),
>> DECLARE_METHOD(t_PythonClassicSimilarity, instance_, METH_O | METH_CLASS),
>> DECLARE_METHOD(t_PythonClassicSimilarity, coord, METH_VARARGS),
>> DECLARE_METHOD(t_PythonClassicSimilarity, finalize, METH_NOARGS),
>> DECLARE_METHOD(t_PythonClassicSimilarity, idf, METH_VARARGS),
>> DECLARE_METHOD(t_PythonClassicSimilarity, idfExplain, METH_VARARGS),
>> DECLARE_METHOD(t_PythonClassicSimilarity, lengthNorm, METH_VARARGS),
>> DECLARE_METHOD(t_PythonClassicSimilarity, pythonExtension, METH_VARARGS),
>> DECLARE_METHOD(t_PythonClassicSimilarity, queryNorm, METH_VARARGS),
>> DECLARE_METHOD(t_PythonClassicSimilarity, sloppyFreq, METH_VARARGS),
>> DECLARE_METHOD(t_PythonClassicSimilarity, tf, METH_VARARGS),
>> { NULL, NULL, 0, NULL }
>> };
>>
>> and
>>
>> JNINativeMethod methods[] = {
>> { "coord", "(II)F", (void *) t_PythonClassicSimilarity_coord0 },
>> { "idf", "(JJ)F", (void *) t_PythonClassicSimilarity_idf1 },
>> { "idfExplain", "(Lorg/apache/lucene/search/CollectionStatistics;[Lorg/apache/lucene/search/TermStatistics;)Lorg/apache/lucene/search/Explanation;", (void *) t_PythonClassicSimilarity_idfExplain2 },
>> { "lengthNorm", "(Lorg/apache/lucene/index/FieldInvertState;)F", (void *) t_PythonClassicSimilarity_lengthNorm3 },
>> { "pythonDecRef", "()V", (void *) t_PythonClassicSimilarity_pythonDecRef4 },
>> { "queryNorm", "(F)F", (void *) t_PythonClassicSimilarity_queryNorm5 },
>> { "sloppyFreq", "(I)F", (void *) t_PythonClassicSimilarity_sloppyFreq6 },
>> { "tf", "(F)F", (void *) t_PythonClassicSimilarity_tf7 },
>> };
>>
>>
>> enum {
>> mid_init$_7353aa4eb69a6f70,
>> mid_coord_78a3b70e2a43b9c7,
>> mid_finalize_7353aa4eb69a6f70,
>> mid_idf_0af4b58f982c0dd1,
>> mid_idfExplain_08e4e4733921c6f1,
>> mid_lengthNorm_a7d67c4cd2e3b6a2,
>> mid_pythonDecRef_7353aa4eb69a6f70,
>> mid_pythonExtension_aae05472994e7446,
>> mid_pythonExtension_f41cd3dc7c7d5d78,
>> mid_queryNorm_eecff9f314592ce9,
>> mid_sloppyFreq_6b1faadec3ed3278,
>> mid_tf_eecff9f314592ce9,
>> max_mid
>> };
>>
>> and
>>
>> jfloat coord(jint, jint) const;
>> void finalize() const;
>> jfloat idf(jlong, jlong) const;
>> ::org::apache::lucene::search::Explanation idfExplain(const ::org::apache::lucene::search::CollectionStatistics &, const JArray< ::org::apache::lucene::search::TermStatistics > &) const;
>> jfloat lengthNorm(const ::org::apache::lucene::index::FieldInvertState &) const;
>> void pythonDecRef() const;
>> jlong pythonExtension() const;
>> void pythonExtension(jlong) const;
>> jfloat queryNorm(jfloat) const;
>> jfloat sloppyFreq(jint) const;
>> jfloat tf(jfloat) const;
>>
>> Here is the code for PythonSimilarityBase.java (stored in the same directory as PythonClassicSimilarity) which compiles without issues:
>> package org.apache.pylucene.search.similarities;
>>
>> import org.apache.lucene.search.similarities.BasicStats;
>> import org.apache.lucene.search.similarities.SimilarityBase;
>>
>>
>>
>> public class PythonSimilarityBase extends SimilarityBase {
>>
>> private long pythonObject;
>>
>> public PythonSimilarityBase()
>> {
>> }
>>
>> public void pythonExtension(long pythonObject)
>> {
>> this.pythonObject = pythonObject;
>> }
>> public long pythonExtension()
>> {
>> return this.pythonObject;
>> }
>>
>> public void finalize()
>> throws Throwable
>> {
>> pythonDecRef();
>> }
>>
>> public native void pythonDecRef();
>>
>>
>> @Override
>> protected native float score(BasicStats stats, float freq, float docLen);
>>
>> @Override
>> public native String toString();
>> }
>>
>>
>> I?ve also tried variations of adding the BasicStats class, the Similarity$SimWeight class and the whole similarities package into the JCC command, for example
>> JCC=$(PYTHON) -m jcc.__main__ --shared --arch x86_64 --package org.apache.lucene.search.similarities
>> (Trying to follow advice in https://issues.apache.org/jira/browse/PYLUCENE-21 <https://issues.apache.org/jira/browse/PYLUCENE-21> <https://issues.apache.org/jira/browse/PYLUCENE-21 <https://issues.apache.org/jira/browse/PYLUCENE-21>>)
>>
>>
>> My feeling is that it has something to do with that BasicStats object?
>>
>>
>> If anyone is able to assist it would be greatly appreciated!!
>
> The 'score' method you're trying to get wrapped is marked 'protected' in the source code you showed above.
>
> By default, JCC only wraps public methods and fields.
>
> You can either make it public or force a wrapper for it by explicitely listing it on the JCC command line.
>
> At line 326 in the pylucene Makefile you can see two such non public things being force-wrapped already via listing:
> org.apache.lucene.index.IndexWriter:getReader \
> org.apache.lucene.analysis.Tokenizer:input \
> Just add your score method to the list there...
>
> Andi..
>
>>
>> Thanks,
>> Peter Brown
>> School of Information and Communication Technology,
>> Griffith University - Gold Coast Campus
Re: JCC SimilarityBase
Posted by Andi Vajda <va...@apache.org>.
On Fri, 5 May 2017, Peter Brown wrote:
> Hi all,
>
> I was just wondering if anybody else has run into a similar issue before.
> I?m trying to create a wrapper class for the org.apache.lucene.search.similarities.SimilarityBase class within PyLucene (similar to the existing wrapper for PythonClassicSimilarity).
>
> However, upon re-building and re-installing PyLucene, when trying to use the wrapper class it fails with the following exception:
>
> java.lang.UnsatisfiedLinkError: org.apache.pylucene.search.similarities.PythonSimilarityBase.score(Lorg/apache/lucene/search/similarities/BasicStats;FF)F
>
> Upon examination of the wrap0*.cpp files, and comparing with PythonClassicSimilarity, I found that the score method is not being generated. E.g.
>
> static PyMethodDef t_PythonSimilarityBase__methods_[] = {
> DECLARE_METHOD(t_PythonSimilarityBase, cast_, METH_O | METH_CLASS),
> DECLARE_METHOD(t_PythonSimilarityBase, instance_, METH_O | METH_CLASS),
> DECLARE_METHOD(t_PythonSimilarityBase, finalize, METH_NOARGS),
> DECLARE_METHOD(t_PythonSimilarityBase, pythonExtension, METH_VARARGS),
> { NULL, NULL, 0, NULL }
> };
>
> and
>
> JNINativeMethod methods[] = {
> { "pythonDecRef", "()V", (void *) t_PythonSimilarityBase_pythonDecRef0 },
> { "toString", "()Ljava/lang/String;", (void *) t_PythonSimilarityBase_toString1 },
> };
>
> But it is present in the header file enum?
> class PythonSimilarityBase : public ::org::apache::lucene::search::similarities::SimilarityBase {
> public:
> enum {
> mid_init$_7353aa4eb69a6f70,
> mid_finalize_7353aa4eb69a6f70,
> mid_pythonDecRef_7353aa4eb69a6f70,
> mid_pythonExtension_aae05472994e7446,
> mid_pythonExtension_f41cd3dc7c7d5d78,
> mid_toString_5df7257cf25bf439,
> mid_score_a6c8466c4349b376,
> max_mid
> };
>
> But not in the header file methods.
> PythonSimilarityBase();
>
> void finalize() const;
> void pythonDecRef() const;
> jlong pythonExtension() const;
> void pythonExtension(jlong) const;
> ::java::lang::String toString() const;
>
>
> Compared to
> static PyMethodDef t_PythonClassicSimilarity__methods_[] = {
> DECLARE_METHOD(t_PythonClassicSimilarity, cast_, METH_O | METH_CLASS),
> DECLARE_METHOD(t_PythonClassicSimilarity, instance_, METH_O | METH_CLASS),
> DECLARE_METHOD(t_PythonClassicSimilarity, coord, METH_VARARGS),
> DECLARE_METHOD(t_PythonClassicSimilarity, finalize, METH_NOARGS),
> DECLARE_METHOD(t_PythonClassicSimilarity, idf, METH_VARARGS),
> DECLARE_METHOD(t_PythonClassicSimilarity, idfExplain, METH_VARARGS),
> DECLARE_METHOD(t_PythonClassicSimilarity, lengthNorm, METH_VARARGS),
> DECLARE_METHOD(t_PythonClassicSimilarity, pythonExtension, METH_VARARGS),
> DECLARE_METHOD(t_PythonClassicSimilarity, queryNorm, METH_VARARGS),
> DECLARE_METHOD(t_PythonClassicSimilarity, sloppyFreq, METH_VARARGS),
> DECLARE_METHOD(t_PythonClassicSimilarity, tf, METH_VARARGS),
> { NULL, NULL, 0, NULL }
> };
>
> and
>
> JNINativeMethod methods[] = {
> { "coord", "(II)F", (void *) t_PythonClassicSimilarity_coord0 },
> { "idf", "(JJ)F", (void *) t_PythonClassicSimilarity_idf1 },
> { "idfExplain", "(Lorg/apache/lucene/search/CollectionStatistics;[Lorg/apache/lucene/search/TermStatistics;)Lorg/apache/lucene/search/Explanation;", (void *) t_PythonClassicSimilarity_idfExplain2 },
> { "lengthNorm", "(Lorg/apache/lucene/index/FieldInvertState;)F", (void *) t_PythonClassicSimilarity_lengthNorm3 },
> { "pythonDecRef", "()V", (void *) t_PythonClassicSimilarity_pythonDecRef4 },
> { "queryNorm", "(F)F", (void *) t_PythonClassicSimilarity_queryNorm5 },
> { "sloppyFreq", "(I)F", (void *) t_PythonClassicSimilarity_sloppyFreq6 },
> { "tf", "(F)F", (void *) t_PythonClassicSimilarity_tf7 },
> };
>
>
> enum {
> mid_init$_7353aa4eb69a6f70,
> mid_coord_78a3b70e2a43b9c7,
> mid_finalize_7353aa4eb69a6f70,
> mid_idf_0af4b58f982c0dd1,
> mid_idfExplain_08e4e4733921c6f1,
> mid_lengthNorm_a7d67c4cd2e3b6a2,
> mid_pythonDecRef_7353aa4eb69a6f70,
> mid_pythonExtension_aae05472994e7446,
> mid_pythonExtension_f41cd3dc7c7d5d78,
> mid_queryNorm_eecff9f314592ce9,
> mid_sloppyFreq_6b1faadec3ed3278,
> mid_tf_eecff9f314592ce9,
> max_mid
> };
>
> and
>
> jfloat coord(jint, jint) const;
> void finalize() const;
> jfloat idf(jlong, jlong) const;
> ::org::apache::lucene::search::Explanation idfExplain(const ::org::apache::lucene::search::CollectionStatistics &, const JArray< ::org::apache::lucene::search::TermStatistics > &) const;
> jfloat lengthNorm(const ::org::apache::lucene::index::FieldInvertState &) const;
> void pythonDecRef() const;
> jlong pythonExtension() const;
> void pythonExtension(jlong) const;
> jfloat queryNorm(jfloat) const;
> jfloat sloppyFreq(jint) const;
> jfloat tf(jfloat) const;
>
> Here is the code for PythonSimilarityBase.java (stored in the same directory as PythonClassicSimilarity) which compiles without issues:
> package org.apache.pylucene.search.similarities;
>
> import org.apache.lucene.search.similarities.BasicStats;
> import org.apache.lucene.search.similarities.SimilarityBase;
>
>
>
> public class PythonSimilarityBase extends SimilarityBase {
>
> private long pythonObject;
>
> public PythonSimilarityBase()
> {
> }
>
> public void pythonExtension(long pythonObject)
> {
> this.pythonObject = pythonObject;
> }
> public long pythonExtension()
> {
> return this.pythonObject;
> }
>
> public void finalize()
> throws Throwable
> {
> pythonDecRef();
> }
>
> public native void pythonDecRef();
>
>
> @Override
> protected native float score(BasicStats stats, float freq, float docLen);
>
> @Override
> public native String toString();
> }
>
>
> I?ve also tried variations of adding the BasicStats class, the Similarity$SimWeight class and the whole similarities package into the JCC command, for example
> JCC=$(PYTHON) -m jcc.__main__ --shared --arch x86_64 --package org.apache.lucene.search.similarities
> (Trying to follow advice in https://issues.apache.org/jira/browse/PYLUCENE-21 <https://issues.apache.org/jira/browse/PYLUCENE-21>)
>
>
> My feeling is that it has something to do with that BasicStats object?
>
>
> If anyone is able to assist it would be greatly appreciated!!
The 'score' method you're trying to get wrapped is marked 'protected' in
the source code you showed above.
By default, JCC only wraps public methods and fields.
You can either make it public or force a wrapper for it by explicitely
listing it on the JCC command line.
At line 326 in the pylucene Makefile you can see two such non public things
being force-wrapped already via listing:
org.apache.lucene.index.IndexWriter:getReader \
org.apache.lucene.analysis.Tokenizer:input \
Just add your score method to the list there...
Andi..
>
> Thanks,
> Peter Brown
> School of Information and Communication Technology,
> Griffith University - Gold Coast Campus
>
>