You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@manifoldcf.apache.org by Karl Wright <da...@gmail.com> on 2016/09/01 06:51:05 UTC
Re: Has anyone successfully populated Elastic Search properties from
custom SharePoint Meta data fields?
Hi Mike,
The code all now builds, and the unit tests pass. I don't know if it works
as a plugin, however. And I haven't changed the jar packaging yet to
include the properties file. I will try to look at that today at some
point.
Note well: I am building this new plugin against ES 2.0.2. The reason I
chose this version is because I'm hoping to limit the number of plugin
versions we need to release, and this might be compatible with all of 2.x,
with luck. If you can tell me which version of ES you want to use, I can
try building against that as a trial when the packaging work is done.
Note also that I've needed to make *no* functional changes to the
authorization code -- all the work has been in making the code function as
a proper plugin under 2.0.2. If the plugin API changes between each
version of ES 2.x are as significant as the changes between 1.7 and 2.0,
and the release schedule of ES continues to be essentially one release
every couple of months, there is no way that ManifoldCF can possibly keep
up. I think if that turns out to be the reality I'd recommend that we just
go back to the original toolkit and have people do their own plugin
wrappers in the future.
Thanks,
Karl
On Wed, Aug 31, 2016 at 5:09 AM, Karl Wright <da...@gmail.com> wrote:
> Ok, I made some progress disentangling what we truly need to get this
> plugin to build and what's just noise.
>
> There are two main problems.
>
> The first main problem is that I can't see how you're supposed to register
> a REST action now. See MCFAuthorizerPlugin.java, and note the fact that
> onModule() for RestModule objects doesn't exist, and yet that's what the
> few examples I've found say you're supposed to do.
>
> The second problem is that the way the plugin overrides RestSearchAction
> is no longer viable; the way requests are processed in the base class is in
> place into member variables and not into a return variable. That's going
> to require some structural changes to address. It also looks to me like
> the code that was confusing me before was just modified code from the
> original RestSearchAction.parse() method, so I can just lift the 2.0.2 code
> and modify it in a similar manner. Anyway, I'll push at that one later
> today.
>
> Karl
>
>
> On Tue, Aug 30, 2016 at 7:43 PM, Karl Wright <da...@gmail.com> wrote:
>
>> I tinkered around with this further a bit but the API indeed changed
>> dramatically from 1.x to 2.x. The MCFAuthorizerRestSearchAction class is
>> the one that needs serious help -- it looks like a lot of specialty code
>> was needed in 1.x to build a RestSearchAction class that is quite different
>> in 2.x. If you can find an example of a RestSearchAction class written
>> against the 2.0.x API then I might be able to use that as a basis for
>> completing the plugin; otherwise I don't know where to begin.
>>
>> Karl
>>
>>
>> On Tue, Aug 30, 2016 at 5:31 PM, Karl Wright <da...@gmail.com> wrote:
>>
>>> I set up an ES-2.0 plugin and pointed it at ES 2.0.2. Build fails
>>> pretty dramatically. I'd say that pretty much all the API's changed. I
>>> committed the code if you want to try for yourself.
>>>
>>> You can find the plugin at:
>>>
>>> https://svn.apache.org/repos/asf/manifoldcf/integration/elas
>>> ticsearch-2.0/trunk
>>>
>>> Karl
>>>
>>>
>>> On Tue, Aug 30, 2016 at 3:27 PM, Holtz, Michael (IT Consultant) <
>>> MHoltz@consultantemail.com> wrote:
>>>
>>>> Ah got it. I was wondering if it was something like that.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Mike H.
>>>>
>>>>
>>>>
>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>> *Sent:* Tuesday, August 30, 2016 2:55 PM
>>>>
>>>> *To:* user@manifoldcf.apache.org
>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>> properties from custom SharePoint Meta data fields?
>>>>
>>>>
>>>>
>>>> Hi Mike,
>>>>
>>>>
>>>>
>>>> There are TWO versions for the plugin: (1) the version of ES it was
>>>> written for (1.5), and the functional compatibility with MCF itself (that's
>>>> where the 2.1 comes from).
>>>>
>>>>
>>>>
>>>> There is no version of the ES plugin that we built for ES 2.x. That's
>>>> what we're trying to create right now.
>>>>
>>>>
>>>>
>>>> So when you see:
>>>>
>>>>
>>>>
>>>> apache-manifoldcf-elasticsearch-1.5-plugin-2.2
>>>>
>>>>
>>>>
>>>> ... the 1.5 means the ES version and the 2.2 is the actual version of
>>>> the ES-1.5 plugin.
>>>>
>>>>
>>>>
>>>> Karl
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Aug 30, 2016 at 1:25 PM, Holtz, Michael (IT Consultant) <
>>>> MHoltz@consultantemail.com> wrote:
>>>>
>>>> So the plugin install fails in Elastic Search 2.1.0 with the same error:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> C:\APPS\elasticsearch-2.1.0>bin\plugin install
>>>> file:///C:/Users/holtzm/Downloads/apache-manifoldcf-elastics
>>>> earch-plugin-2.1-bin.zip
>>>>
>>>> -> Installing from file:/C:/Users/holtzm/Download
>>>> s/apache-manifoldcf-elasticsearch-plugin-2.1-bin.zip...
>>>>
>>>> Trying file:/C:/Users/holtzm/Downloads/apache-manifoldcf-elasticsearch-plugin-2.1-bin.zip
>>>> ...
>>>>
>>>> Downloading .DONE
>>>>
>>>> Verifying file:/C:/Users/holtzm/Downloads/apache-manifoldcf-elasticsearch-plugin-2.1-bin.zip
>>>> checksums if available ...
>>>>
>>>> NOTE: Unable to verify checksum for downloaded plugin (unable to find
>>>> .sha1 or .md5 file to verify)
>>>>
>>>> ERROR: Could not find plugin descriptor 'plugin-descriptor.properties'
>>>> in plugin zip
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> The documentation says for 2.1 though so I’m confused at this point.
>>>>
>>>>
>>>>
>>>> I will try adding the ‘plugin-descriptor.properties ‘ file
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Mike H.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *From:* Holtz, Michael (IT Consultant)
>>>> *Sent:* Tuesday, August 30, 2016 1:03 PM
>>>>
>>>>
>>>> *To:* user@manifoldcf.apache.org
>>>> *Subject:* RE: Has anyone successfully populated Elastic Search
>>>> properties from custom SharePoint Meta data fields?
>>>>
>>>>
>>>>
>>>> I think I’ll switch to Elastic Search version 2.1, just to see it work
>>>> first:
>>>>
>>>>
>>>>
>>>> *Plugin for Elastic Search (toolkit only)*
>>>>
>>>> Latest release (Apache ManifoldCF Plugin for Elastic Search, version
>>>> 2.1, 2015 Jul 25)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> BTW I was able to see my first use case work. It turns out that I
>>>> needed to explicitly include the custom meta data fields, “include all
>>>> metadata” didn’t work.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> That leaves me with #4
>>>>
>>>>
>>>>
>>>> 1. Index custom meta data fields from SharePoint into the
>>>> ElasticSearch index *done*
>>>>
>>>> 2. Full text indexing of SharePoint content in ElasticSearch
>>>> *done*
>>>>
>>>> 3. SharePoint security information indexed in ElasticSearch*
>>>> done*
>>>>
>>>> 4. ElasticSearch Plugin to enforce SharePoint security settings
>>>> for search *todo*
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Mike H.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *From:* Karl Wright [mailto:daddywri@gmail.com <da...@gmail.com>]
>>>> *Sent:* Tuesday, August 30, 2016 12:34 PM
>>>> *To:* user@manifoldcf.apache.org
>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>> properties from custom SharePoint Meta data fields?
>>>>
>>>>
>>>>
>>>> We'll probably need to build against ES 2.x, and will need a whole new
>>>> plugin.
>>>>
>>>>
>>>>
>>>> Karl
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Aug 30, 2016 at 12:29 PM, Holtz, Michael (IT Consultant) <
>>>> MHoltz@consultantemail.com> wrote:
>>>>
>>>> Actually I see a classpath exception when starting elastic search so it
>>>> doesn’t appear to be that simple.
>>>>
>>>>
>>>>
>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>> *Sent:* Tuesday, August 30, 2016 11:55 AM
>>>>
>>>>
>>>> *To:* user@manifoldcf.apache.org
>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>> properties from custom SharePoint Meta data fields?
>>>>
>>>>
>>>>
>>>> Ok, you just included this in the jar and all was OK? It should be
>>>> straightforward to set that up if so...
>>>>
>>>>
>>>>
>>>> Thanks!
>>>>
>>>> Karl
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Aug 30, 2016 at 11:27 AM, Holtz, Michael (IT Consultant) <
>>>> MHoltz@consultantemail.com> wrote:
>>>>
>>>> Karl,
>>>>
>>>>
>>>>
>>>> Ok, I will see what I can do.
>>>>
>>>>
>>>>
>>>> I just created the attached file for the 1.5 version of the plugin and
>>>> was able to install it. Let me know if you see any issues with it.
>>>>
>>>>
>>>>
>>>> I’m going to start with a clean version of the new 2.5 release for my
>>>> use cases just using the example setup, let me know if you don’t think that
>>>> is a good idea.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Mike H.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>> *Sent:* Tuesday, August 30, 2016 11:17 AM
>>>>
>>>>
>>>> *To:* user@manifoldcf.apache.org
>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>> properties from custom SharePoint Meta data fields?
>>>>
>>>>
>>>>
>>>> The 2.0 spec has clearly changed from that needed for the 1.5-level
>>>> plugins.
>>>>
>>>>
>>>>
>>>> The plugin code is likely to be pretty much unchanged but the packaging
>>>> will need to change. Now, I did not develop the 1.5-level plugin, and
>>>> don't know that much about ES in the first place, so if you want to
>>>> undertake the plugin transformation and contribute it, we'd be very
>>>> grateful.
>>>>
>>>>
>>>>
>>>> Karl
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Aug 30, 2016 at 11:09 AM, Holtz, Michael (IT Consultant) <
>>>> MHoltz@consultantemail.com> wrote:
>>>>
>>>> It looks like a ‘plugin-descriptor.properties’ file needs to be added
>>>> to the jar:
>>>>
>>>>
>>>>
>>>> https://www.elastic.co/guide/en/elasticsearch/plugins/2.0/pl
>>>> ugin-authors.html
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *From:* Holtz, Michael (IT Consultant)
>>>> *Sent:* Tuesday, August 30, 2016 10:53 AM
>>>>
>>>>
>>>> *To:* user@manifoldcf.apache.org
>>>> *Subject:* RE: Has anyone successfully populated Elastic Search
>>>> properties from custom SharePoint Meta data fields?
>>>>
>>>>
>>>>
>>>> Karl,
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> FYI, Now I’m seeing this:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> C:\APPS\elasticsearch-2.3.3>bin\plugin install
>>>> file:///C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist
>>>> /plugins/elasticsearch/elasticsearch-1.5/target/elasticsearc
>>>> h-1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar
>>>>
>>>> -> Installing from file:/C:/Users/holtzm/workspac
>>>> e/apache-manifoldcf-2.4/dist/plugins/elasticsearch/elasticse
>>>> arch-1.5/target/elasticsearch-1.5-plugin-mcf-2.0.1-jar-with-
>>>> dependencies.jar...
>>>>
>>>> Trying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/p
>>>> lugins/elasticsearch/elasticsearch-1.5/target/elasticsearch-
>>>> 1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar ...
>>>>
>>>> Downloading ...........................DONE
>>>>
>>>> Verifying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/p
>>>> lugins/elasticsearch/elasticsearch-1.5/target/elasticsearch-
>>>> 1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar checksums if available
>>>> ...
>>>>
>>>> NOTE: Unable to verify checksum for downloaded plugin (unable to find
>>>> .sha1 or .md5 file to verify)
>>>>
>>>> ERROR: Could not find plugin descriptor 'plugin-descriptor.properties'
>>>> in plugin zip
>>>>
>>>>
>>>>
>>>> Mike H.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *From:* Holtz, Michael (IT Consultant)
>>>> *Sent:* Tuesday, August 30, 2016 10:48 AM
>>>> *To:* user@manifoldcf.apache.org
>>>> *Subject:* RE: Has anyone successfully populated Elastic Search
>>>> properties from custom SharePoint Meta data fields?
>>>>
>>>>
>>>>
>>>> Karl,
>>>>
>>>>
>>>>
>>>> I got what your’re saying, I should use the 1.5 plugin. Got it.
>>>>
>>>>
>>>>
>>>> I will try that
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Mike H.
>>>>
>>>>
>>>>
>>>> *From:* Holtz, Michael (IT Consultant)
>>>> *Sent:* Tuesday, August 30, 2016 10:47 AM
>>>> *To:* user@manifoldcf.apache.org
>>>> *Subject:* RE: Has anyone successfully populated Elastic Search
>>>> properties from custom SharePoint Meta data fields?
>>>>
>>>>
>>>>
>>>> Hmm that’s confusing, here is the documentation:
>>>>
>>>>
>>>>
>>>> *Apache ManifoldCF ElasticSearch plugin compatibility*
>>>>
>>>> *ManifoldCF versions*
>>>>
>>>> *Plugin version*
>>>>
>>>> 0.1.x-1.4.x
>>>>
>>>> 0.x
>>>>
>>>> 1.5.x
>>>>
>>>> 1.x
>>>>
>>>> >=1.6.x
>>>>
>>>> 2.x
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> I’m running version 2.4 of manifold so based on this I went with the
>>>> 2.1 plugin.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Where can I get the 1.0 plugin? I don’t see it on the site or in the
>>>> 2.4 build.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Mike H.
>>>>
>>>>
>>>>
>>>> *From:* Karl Wright [mailto:daddywri@gmail.com <da...@gmail.com>]
>>>> *Sent:* Tuesday, August 30, 2016 10:38 AM
>>>> *To:* user@manifoldcf.apache.org
>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>> properties from custom SharePoint Meta data fields?
>>>>
>>>>
>>>>
>>>> You need the ES-1.5 plugin. That's the older (deprecated) 1.0 plugin.
>>>>
>>>>
>>>>
>>>> Karl
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Aug 30, 2016 at 10:36 AM, Holtz, Michael (IT Consultant) <
>>>> MHoltz@consultantemail.com> wrote:
>>>>
>>>> Elasticsearch 2.3.3
>>>>
>>>> Plugin 2.1
>>>>
>>>>
>>>>
>>>> Here is the current error
>>>>
>>>>
>>>>
>>>> C:\APPS\elasticsearch-2.3.3>bin\plugin install
>>>> file:///C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist
>>>> /plugins/elasticsearch/elasticsearch/elasticsearch-plugin-mcf-2.1.jar
>>>>
>>>> -> Installing from file:/C:/Users/holtzm/workspac
>>>> e/apache-manifoldcf-2.4/dist/plugins/elasticsearch/elasticse
>>>> arch/elasticsearch-plugin-mcf-2.1.jar...
>>>>
>>>> Trying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/p
>>>> lugins/elasticsearch/elasticsearch/elasticsearch-plugin-mcf-2.1.jar ...
>>>>
>>>> Downloading .DONE
>>>>
>>>> Verifying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/p
>>>> lugins/elasticsearch/elasticsearch/elasticsearch-plugin-mcf-2.1.jar
>>>> checksums if available ...
>>>>
>>>> NOTE: Unable to verify checksum for downloaded plugin (unable to find
>>>> .sha1 or .md5 file to verify)
>>>>
>>>> ERROR: Could not find plugin descriptor 'plugin-descriptor.properties'
>>>> in plugin zip
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>> *Sent:* Tuesday, August 30, 2016 10:22 AM
>>>>
>>>>
>>>> *To:* user@manifoldcf.apache.org
>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>> properties from custom SharePoint Meta data fields?
>>>>
>>>>
>>>>
>>>> Hi Mike,
>>>>
>>>>
>>>>
>>>> If you are just trying things out you can use the single-process
>>>> example and it will make your life easier for experimentation.
>>>>
>>>>
>>>>
>>>> As far as the ES plugin, there are two: one for ES up to 1.5, and one
>>>> for 1.5 onward. Which did you try to install?
>>>>
>>>>
>>>>
>>>> Karl
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Aug 30, 2016 at 9:56 AM, Holtz, Michael (IT Consultant) <
>>>> MHoltz@consultantemail.com> wrote:
>>>>
>>>> Karl,
>>>>
>>>>
>>>>
>>>> You are correct.
>>>>
>>>>
>>>>
>>>> Before I setup zookeeper I would like to prove out a few use cases.
>>>>
>>>>
>>>>
>>>> 1. Index custom meta data fields from SharePoint into the
>>>> ElasticSearch index
>>>>
>>>> 2. Full text indexing of SharePoint content in ElasticSearch
>>>>
>>>> 3. SharePoint security information indexed in ElasticSearch
>>>>
>>>> 4. ElasticSearch Plugin to enforce SharePoint security settings
>>>> for search
>>>>
>>>>
>>>>
>>>> At this point I’ve only managed to see #2 and #3 work
>>>>
>>>>
>>>>
>>>> For #4 I was getting an error when installing the ElasticSearch plugin.
>>>>
>>>> I would like to see #1 work before I do anything else. Are there any
>>>> example configurations that show how to do this?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Mike H.
>>>>
>>>>
>>>>
>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>> *Sent:* Tuesday, August 30, 2016 9:49 AM
>>>>
>>>>
>>>> *To:* user@manifoldcf.apache.org
>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>> properties from custom SharePoint Meta data fields?
>>>>
>>>>
>>>>
>>>> Hi Mike,
>>>>
>>>>
>>>>
>>>> This sounds like you're having a lock problem.
>>>>
>>>>
>>>>
>>>> Let me guess: you are using the multiprocess example with file-based
>>>> locking? If so, you need to stop everything and do the "lock clean"
>>>> procedure. I would also strongly recommend that you switch to the
>>>> zookeeper multiprocess example.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Karl
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Aug 30, 2016 at 9:44 AM, Holtz, Michael (IT Consultant) <
>>>> MHoltz@consultantemail.com> wrote:
>>>>
>>>> I currently cannot view the job. The UI is hanging when I click the
>>>> “view” link for the job. I’m wondering if that has anything to do with the
>>>> “Metadata Adjuster” I added?
>>>>
>>>>
>>>>
>>>> I did not schedule the job.
>>>>
>>>>
>>>>
>>>> At this point the job has finished running and the Document Status and
>>>> Queue status reports. I do not see any errors reported there.
>>>>
>>>>
>>>>
>>>> Also I still do not see my custom meta data fields from SharePoint in
>>>> the ElasticSearch index.
>>>>
>>>>
>>>>
>>>> We are hoping to use this in production, but at this point it is not
>>>> looking too good.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Mike H.
>>>>
>>>>
>>>>
>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>> *Sent:* Tuesday, August 30, 2016 9:09 AM
>>>>
>>>>
>>>> *To:* user@manifoldcf.apache.org
>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>> properties from custom SharePoint Meta data fields?
>>>>
>>>>
>>>>
>>>> It is also important to know whether you have your job set up as
>>>> "continuous". If so, it will run forever.
>>>>
>>>>
>>>>
>>>> Karl
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Aug 30, 2016 at 8:48 AM, Furkan KAMACI <fu...@gmail.com>
>>>> wrote:
>>>>
>>>> Hi Mike,
>>>>
>>>>
>>>>
>>>> Could you tell us what "Document Status" and "Queue Status" says as
>>>> Karl mentioned?
>>>>
>>>>
>>>>
>>>> Kind Regards,
>>>>
>>>> Furkan KAMACI
>>>>
>>>>
>>>>
>>>> On Tue, Aug 30, 2016 at 3:39 PM, Holtz, Michael (IT Consultant) <
>>>> MHoltz@consultantemail.com> wrote:
>>>>
>>>> I’m looking at manifoldcf.log and I see no errors.
>>>>
>>>>
>>>>
>>>> I have only 13 files and they are all very tiny < 15kb .docx files.
>>>>
>>>>
>>>>
>>>> Mike H.
>>>>
>>>>
>>>>
>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>> *Sent:* Monday, August 29, 2016 5:26 PM
>>>>
>>>>
>>>> *To:* user@manifoldcf.apache.org
>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>> properties from custom SharePoint Meta data fields?
>>>>
>>>>
>>>>
>>>> You should be looking at the ManifoldCF.log file, and the examples have
>>>> logging for errors at least as being the default.
>>>>
>>>>
>>>>
>>>> The only other possibility is that you have a very large document that
>>>> is taking a long time to transfer/index.
>>>>
>>>>
>>>>
>>>> Also, you can generally see what MCF is up to using the "document
>>>> status" and "queue status" reports.
>>>>
>>>>
>>>>
>>>> Karl
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Aug 29, 2016 at 5:22 PM, Holtz, Michael (IT Consultant) <
>>>> MHoltz@consultantemail.com> wrote:
>>>>
>>>> Karl,
>>>>
>>>>
>>>>
>>>> I see no errors in the log.
>>>>
>>>>
>>>>
>>>> I probably need to configure the logging I think.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Mike H.
>>>>
>>>>
>>>>
>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>> *Sent:* Monday, August 29, 2016 5:20 PM
>>>>
>>>>
>>>> *To:* user@manifoldcf.apache.org
>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>> properties from custom SharePoint Meta data fields?
>>>>
>>>>
>>>>
>>>> Hi Mike,
>>>>
>>>>
>>>>
>>>> Have a look in the log. If there are errors there please send them to
>>>> us. Generally, "hangs" are due to MCF retrying documents that have errors.
>>>>
>>>>
>>>>
>>>> Karl
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Aug 29, 2016 at 5:18 PM, Holtz, Michael (IT Consultant) <
>>>> MHoltz@consultantemail.com> wrote:
>>>>
>>>> Karl,
>>>>
>>>>
>>>>
>>>> Yes, it shows up. I include all meta-data.
>>>>
>>>>
>>>>
>>>> I’m no trying to use a MetaData adjuster like this
>>>>
>>>>
>>>>
>>>> Parameter
>>>>
>>>> field1 “${Field 1}”
>>>>
>>>>
>>>>
>>>> And now I see “field1”=”” in Elastic Search.
>>>>
>>>>
>>>>
>>>> Also it seems each time I make a change the job will hang and I need to
>>>> restart everything including ElasticSearch to get it to complete.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Mike H.
>>>>
>>>>
>>>>
>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>> *Sent:* Monday, August 29, 2016 5:13 PM
>>>> *To:* user@manifoldcf.apache.org
>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>> properties from custom SharePoint Meta data fields?
>>>>
>>>>
>>>>
>>>> Hi Mike,
>>>>
>>>>
>>>>
>>>> If I were you I'd try to independently verify that the custom field
>>>> shows up. Do you see it appear in the pulldown for selecting metadata? If
>>>> not, it probably is not accessible through the inspection methods that give
>>>> us access into SharePoint fields.
>>>>
>>>>
>>>>
>>>> If it appears there then we should look at what ES gets sent. It's
>>>> possible it appears but under a different field name.
>>>>
>>>>
>>>>
>>>> Karl
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Aug 29, 2016 at 5:01 PM, Holtz, Michael (IT Consultant) <
>>>> MHoltz@consultantemail.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>>
>>>>
>>>> I have created a custom content type with a custom field “Field 1”.
>>>>
>>>>
>>>>
>>>> I have configured the job to include all meta data.
>>>>
>>>>
>>>>
>>>> I do not see “Field 1” being populated in the Elastic Search index.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Does anyone have any experience with this?
>>>>
>>>>
>>>>
>>>> Thanks in advance,
>>>>
>>>> Mike H
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>
Re: Has anyone successfully populated Elastic Search properties from
custom SharePoint Meta data fields?
Posted by Karl Wright <da...@gmail.com>.
Ok, the packaging should now be updated as well.
Karl
On Thu, Sep 1, 2016 at 2:51 AM, Karl Wright <da...@gmail.com> wrote:
> Hi Mike,
>
> The code all now builds, and the unit tests pass. I don't know if it
> works as a plugin, however. And I haven't changed the jar packaging yet to
> include the properties file. I will try to look at that today at some
> point.
>
> Note well: I am building this new plugin against ES 2.0.2. The reason I
> chose this version is because I'm hoping to limit the number of plugin
> versions we need to release, and this might be compatible with all of 2.x,
> with luck. If you can tell me which version of ES you want to use, I can
> try building against that as a trial when the packaging work is done.
>
> Note also that I've needed to make *no* functional changes to the
> authorization code -- all the work has been in making the code function as
> a proper plugin under 2.0.2. If the plugin API changes between each
> version of ES 2.x are as significant as the changes between 1.7 and 2.0,
> and the release schedule of ES continues to be essentially one release
> every couple of months, there is no way that ManifoldCF can possibly keep
> up. I think if that turns out to be the reality I'd recommend that we just
> go back to the original toolkit and have people do their own plugin
> wrappers in the future.
>
> Thanks,
> Karl
>
>
> On Wed, Aug 31, 2016 at 5:09 AM, Karl Wright <da...@gmail.com> wrote:
>
>> Ok, I made some progress disentangling what we truly need to get this
>> plugin to build and what's just noise.
>>
>> There are two main problems.
>>
>> The first main problem is that I can't see how you're supposed to
>> register a REST action now. See MCFAuthorizerPlugin.java, and note the
>> fact that onModule() for RestModule objects doesn't exist, and yet that's
>> what the few examples I've found say you're supposed to do.
>>
>> The second problem is that the way the plugin overrides RestSearchAction
>> is no longer viable; the way requests are processed in the base class is in
>> place into member variables and not into a return variable. That's going
>> to require some structural changes to address. It also looks to me like
>> the code that was confusing me before was just modified code from the
>> original RestSearchAction.parse() method, so I can just lift the 2.0.2 code
>> and modify it in a similar manner. Anyway, I'll push at that one later
>> today.
>>
>> Karl
>>
>>
>> On Tue, Aug 30, 2016 at 7:43 PM, Karl Wright <da...@gmail.com> wrote:
>>
>>> I tinkered around with this further a bit but the API indeed changed
>>> dramatically from 1.x to 2.x. The MCFAuthorizerRestSearchAction class is
>>> the one that needs serious help -- it looks like a lot of specialty code
>>> was needed in 1.x to build a RestSearchAction class that is quite different
>>> in 2.x. If you can find an example of a RestSearchAction class written
>>> against the 2.0.x API then I might be able to use that as a basis for
>>> completing the plugin; otherwise I don't know where to begin.
>>>
>>> Karl
>>>
>>>
>>> On Tue, Aug 30, 2016 at 5:31 PM, Karl Wright <da...@gmail.com> wrote:
>>>
>>>> I set up an ES-2.0 plugin and pointed it at ES 2.0.2. Build fails
>>>> pretty dramatically. I'd say that pretty much all the API's changed. I
>>>> committed the code if you want to try for yourself.
>>>>
>>>> You can find the plugin at:
>>>>
>>>> https://svn.apache.org/repos/asf/manifoldcf/integration/elas
>>>> ticsearch-2.0/trunk
>>>>
>>>> Karl
>>>>
>>>>
>>>> On Tue, Aug 30, 2016 at 3:27 PM, Holtz, Michael (IT Consultant) <
>>>> MHoltz@consultantemail.com> wrote:
>>>>
>>>>> Ah got it. I was wondering if it was something like that.
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mike H.
>>>>>
>>>>>
>>>>>
>>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>>> *Sent:* Tuesday, August 30, 2016 2:55 PM
>>>>>
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> Hi Mike,
>>>>>
>>>>>
>>>>>
>>>>> There are TWO versions for the plugin: (1) the version of ES it was
>>>>> written for (1.5), and the functional compatibility with MCF itself (that's
>>>>> where the 2.1 comes from).
>>>>>
>>>>>
>>>>>
>>>>> There is no version of the ES plugin that we built for ES 2.x. That's
>>>>> what we're trying to create right now.
>>>>>
>>>>>
>>>>>
>>>>> So when you see:
>>>>>
>>>>>
>>>>>
>>>>> apache-manifoldcf-elasticsearch-1.5-plugin-2.2
>>>>>
>>>>>
>>>>>
>>>>> ... the 1.5 means the ES version and the 2.2 is the actual version of
>>>>> the ES-1.5 plugin.
>>>>>
>>>>>
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 30, 2016 at 1:25 PM, Holtz, Michael (IT Consultant) <
>>>>> MHoltz@consultantemail.com> wrote:
>>>>>
>>>>> So the plugin install fails in Elastic Search 2.1.0 with the same
>>>>> error:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> C:\APPS\elasticsearch-2.1.0>bin\plugin install
>>>>> file:///C:/Users/holtzm/Downloads/apache-manifoldcf-elastics
>>>>> earch-plugin-2.1-bin.zip
>>>>>
>>>>> -> Installing from file:/C:/Users/holtzm/Download
>>>>> s/apache-manifoldcf-elasticsearch-plugin-2.1-bin.zip...
>>>>>
>>>>> Trying file:/C:/Users/holtzm/Downloads/apache-manifoldcf-elasticsearch-plugin-2.1-bin.zip
>>>>> ...
>>>>>
>>>>> Downloading .DONE
>>>>>
>>>>> Verifying file:/C:/Users/holtzm/Downloads/apache-manifoldcf-elasticsearch-plugin-2.1-bin.zip
>>>>> checksums if available ...
>>>>>
>>>>> NOTE: Unable to verify checksum for downloaded plugin (unable to find
>>>>> .sha1 or .md5 file to verify)
>>>>>
>>>>> ERROR: Could not find plugin descriptor 'plugin-descriptor.properties'
>>>>> in plugin zip
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> The documentation says for 2.1 though so I’m confused at this point.
>>>>>
>>>>>
>>>>>
>>>>> I will try adding the ‘plugin-descriptor.properties ‘ file
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mike H.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *From:* Holtz, Michael (IT Consultant)
>>>>> *Sent:* Tuesday, August 30, 2016 1:03 PM
>>>>>
>>>>>
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* RE: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> I think I’ll switch to Elastic Search version 2.1, just to see it work
>>>>> first:
>>>>>
>>>>>
>>>>>
>>>>> *Plugin for Elastic Search (toolkit only)*
>>>>>
>>>>> Latest release (Apache ManifoldCF Plugin for Elastic Search, version
>>>>> 2.1, 2015 Jul 25)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> BTW I was able to see my first use case work. It turns out that I
>>>>> needed to explicitly include the custom meta data fields, “include all
>>>>> metadata” didn’t work.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> That leaves me with #4
>>>>>
>>>>>
>>>>>
>>>>> 1. Index custom meta data fields from SharePoint into the
>>>>> ElasticSearch index *done*
>>>>>
>>>>> 2. Full text indexing of SharePoint content in ElasticSearch
>>>>> *done*
>>>>>
>>>>> 3. SharePoint security information indexed in ElasticSearch*
>>>>> done*
>>>>>
>>>>> 4. ElasticSearch Plugin to enforce SharePoint security settings
>>>>> for search *todo*
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mike H.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *From:* Karl Wright [mailto:daddywri@gmail.com <da...@gmail.com>]
>>>>> *Sent:* Tuesday, August 30, 2016 12:34 PM
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> We'll probably need to build against ES 2.x, and will need a whole new
>>>>> plugin.
>>>>>
>>>>>
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 30, 2016 at 12:29 PM, Holtz, Michael (IT Consultant) <
>>>>> MHoltz@consultantemail.com> wrote:
>>>>>
>>>>> Actually I see a classpath exception when starting elastic search so
>>>>> it doesn’t appear to be that simple.
>>>>>
>>>>>
>>>>>
>>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>>> *Sent:* Tuesday, August 30, 2016 11:55 AM
>>>>>
>>>>>
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> Ok, you just included this in the jar and all was OK? It should be
>>>>> straightforward to set that up if so...
>>>>>
>>>>>
>>>>>
>>>>> Thanks!
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 30, 2016 at 11:27 AM, Holtz, Michael (IT Consultant) <
>>>>> MHoltz@consultantemail.com> wrote:
>>>>>
>>>>> Karl,
>>>>>
>>>>>
>>>>>
>>>>> Ok, I will see what I can do.
>>>>>
>>>>>
>>>>>
>>>>> I just created the attached file for the 1.5 version of the plugin and
>>>>> was able to install it. Let me know if you see any issues with it.
>>>>>
>>>>>
>>>>>
>>>>> I’m going to start with a clean version of the new 2.5 release for my
>>>>> use cases just using the example setup, let me know if you don’t think that
>>>>> is a good idea.
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mike H.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>>> *Sent:* Tuesday, August 30, 2016 11:17 AM
>>>>>
>>>>>
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> The 2.0 spec has clearly changed from that needed for the 1.5-level
>>>>> plugins.
>>>>>
>>>>>
>>>>>
>>>>> The plugin code is likely to be pretty much unchanged but the
>>>>> packaging will need to change. Now, I did not develop the 1.5-level
>>>>> plugin, and don't know that much about ES in the first place, so if you
>>>>> want to undertake the plugin transformation and contribute it, we'd be very
>>>>> grateful.
>>>>>
>>>>>
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 30, 2016 at 11:09 AM, Holtz, Michael (IT Consultant) <
>>>>> MHoltz@consultantemail.com> wrote:
>>>>>
>>>>> It looks like a ‘plugin-descriptor.properties’ file needs to be added
>>>>> to the jar:
>>>>>
>>>>>
>>>>>
>>>>> https://www.elastic.co/guide/en/elasticsearch/plugins/2.0/pl
>>>>> ugin-authors.html
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *From:* Holtz, Michael (IT Consultant)
>>>>> *Sent:* Tuesday, August 30, 2016 10:53 AM
>>>>>
>>>>>
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* RE: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> Karl,
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> FYI, Now I’m seeing this:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> C:\APPS\elasticsearch-2.3.3>bin\plugin install
>>>>> file:///C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist
>>>>> /plugins/elasticsearch/elasticsearch-1.5/target/elasticsearc
>>>>> h-1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar
>>>>>
>>>>> -> Installing from file:/C:/Users/holtzm/workspac
>>>>> e/apache-manifoldcf-2.4/dist/plugins/elasticsearch/elasticse
>>>>> arch-1.5/target/elasticsearch-1.5-plugin-mcf-2.0.1-jar-with-
>>>>> dependencies.jar...
>>>>>
>>>>> Trying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/p
>>>>> lugins/elasticsearch/elasticsearch-1.5/target/elasticsearch-
>>>>> 1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar ...
>>>>>
>>>>> Downloading ...........................DONE
>>>>>
>>>>> Verifying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/p
>>>>> lugins/elasticsearch/elasticsearch-1.5/target/elasticsearch-
>>>>> 1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar checksums if available
>>>>> ...
>>>>>
>>>>> NOTE: Unable to verify checksum for downloaded plugin (unable to find
>>>>> .sha1 or .md5 file to verify)
>>>>>
>>>>> ERROR: Could not find plugin descriptor 'plugin-descriptor.properties'
>>>>> in plugin zip
>>>>>
>>>>>
>>>>>
>>>>> Mike H.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *From:* Holtz, Michael (IT Consultant)
>>>>> *Sent:* Tuesday, August 30, 2016 10:48 AM
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* RE: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> Karl,
>>>>>
>>>>>
>>>>>
>>>>> I got what your’re saying, I should use the 1.5 plugin. Got it.
>>>>>
>>>>>
>>>>>
>>>>> I will try that
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mike H.
>>>>>
>>>>>
>>>>>
>>>>> *From:* Holtz, Michael (IT Consultant)
>>>>> *Sent:* Tuesday, August 30, 2016 10:47 AM
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* RE: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> Hmm that’s confusing, here is the documentation:
>>>>>
>>>>>
>>>>>
>>>>> *Apache ManifoldCF ElasticSearch plugin compatibility*
>>>>>
>>>>> *ManifoldCF versions*
>>>>>
>>>>> *Plugin version*
>>>>>
>>>>> 0.1.x-1.4.x
>>>>>
>>>>> 0.x
>>>>>
>>>>> 1.5.x
>>>>>
>>>>> 1.x
>>>>>
>>>>> >=1.6.x
>>>>>
>>>>> 2.x
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> I’m running version 2.4 of manifold so based on this I went with the
>>>>> 2.1 plugin.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Where can I get the 1.0 plugin? I don’t see it on the site or in the
>>>>> 2.4 build.
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mike H.
>>>>>
>>>>>
>>>>>
>>>>> *From:* Karl Wright [mailto:daddywri@gmail.com <da...@gmail.com>]
>>>>> *Sent:* Tuesday, August 30, 2016 10:38 AM
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> You need the ES-1.5 plugin. That's the older (deprecated) 1.0 plugin.
>>>>>
>>>>>
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 30, 2016 at 10:36 AM, Holtz, Michael (IT Consultant) <
>>>>> MHoltz@consultantemail.com> wrote:
>>>>>
>>>>> Elasticsearch 2.3.3
>>>>>
>>>>> Plugin 2.1
>>>>>
>>>>>
>>>>>
>>>>> Here is the current error
>>>>>
>>>>>
>>>>>
>>>>> C:\APPS\elasticsearch-2.3.3>bin\plugin install
>>>>> file:///C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist
>>>>> /plugins/elasticsearch/elasticsearch/elasticsearch-plugin-mcf-2.1.jar
>>>>>
>>>>> -> Installing from file:/C:/Users/holtzm/workspac
>>>>> e/apache-manifoldcf-2.4/dist/plugins/elasticsearch/elasticse
>>>>> arch/elasticsearch-plugin-mcf-2.1.jar...
>>>>>
>>>>> Trying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/p
>>>>> lugins/elasticsearch/elasticsearch/elasticsearch-plugin-mcf-2.1.jar
>>>>> ...
>>>>>
>>>>> Downloading .DONE
>>>>>
>>>>> Verifying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/p
>>>>> lugins/elasticsearch/elasticsearch/elasticsearch-plugin-mcf-2.1.jar
>>>>> checksums if available ...
>>>>>
>>>>> NOTE: Unable to verify checksum for downloaded plugin (unable to find
>>>>> .sha1 or .md5 file to verify)
>>>>>
>>>>> ERROR: Could not find plugin descriptor 'plugin-descriptor.properties'
>>>>> in plugin zip
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>>> *Sent:* Tuesday, August 30, 2016 10:22 AM
>>>>>
>>>>>
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> Hi Mike,
>>>>>
>>>>>
>>>>>
>>>>> If you are just trying things out you can use the single-process
>>>>> example and it will make your life easier for experimentation.
>>>>>
>>>>>
>>>>>
>>>>> As far as the ES plugin, there are two: one for ES up to 1.5, and one
>>>>> for 1.5 onward. Which did you try to install?
>>>>>
>>>>>
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 30, 2016 at 9:56 AM, Holtz, Michael (IT Consultant) <
>>>>> MHoltz@consultantemail.com> wrote:
>>>>>
>>>>> Karl,
>>>>>
>>>>>
>>>>>
>>>>> You are correct.
>>>>>
>>>>>
>>>>>
>>>>> Before I setup zookeeper I would like to prove out a few use cases.
>>>>>
>>>>>
>>>>>
>>>>> 1. Index custom meta data fields from SharePoint into the
>>>>> ElasticSearch index
>>>>>
>>>>> 2. Full text indexing of SharePoint content in ElasticSearch
>>>>>
>>>>> 3. SharePoint security information indexed in ElasticSearch
>>>>>
>>>>> 4. ElasticSearch Plugin to enforce SharePoint security settings
>>>>> for search
>>>>>
>>>>>
>>>>>
>>>>> At this point I’ve only managed to see #2 and #3 work
>>>>>
>>>>>
>>>>>
>>>>> For #4 I was getting an error when installing the ElasticSearch plugin.
>>>>>
>>>>> I would like to see #1 work before I do anything else. Are there any
>>>>> example configurations that show how to do this?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mike H.
>>>>>
>>>>>
>>>>>
>>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>>> *Sent:* Tuesday, August 30, 2016 9:49 AM
>>>>>
>>>>>
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> Hi Mike,
>>>>>
>>>>>
>>>>>
>>>>> This sounds like you're having a lock problem.
>>>>>
>>>>>
>>>>>
>>>>> Let me guess: you are using the multiprocess example with file-based
>>>>> locking? If so, you need to stop everything and do the "lock clean"
>>>>> procedure. I would also strongly recommend that you switch to the
>>>>> zookeeper multiprocess example.
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 30, 2016 at 9:44 AM, Holtz, Michael (IT Consultant) <
>>>>> MHoltz@consultantemail.com> wrote:
>>>>>
>>>>> I currently cannot view the job. The UI is hanging when I click the
>>>>> “view” link for the job. I’m wondering if that has anything to do with the
>>>>> “Metadata Adjuster” I added?
>>>>>
>>>>>
>>>>>
>>>>> I did not schedule the job.
>>>>>
>>>>>
>>>>>
>>>>> At this point the job has finished running and the Document Status and
>>>>> Queue status reports. I do not see any errors reported there.
>>>>>
>>>>>
>>>>>
>>>>> Also I still do not see my custom meta data fields from SharePoint in
>>>>> the ElasticSearch index.
>>>>>
>>>>>
>>>>>
>>>>> We are hoping to use this in production, but at this point it is not
>>>>> looking too good.
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mike H.
>>>>>
>>>>>
>>>>>
>>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>>> *Sent:* Tuesday, August 30, 2016 9:09 AM
>>>>>
>>>>>
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> It is also important to know whether you have your job set up as
>>>>> "continuous". If so, it will run forever.
>>>>>
>>>>>
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 30, 2016 at 8:48 AM, Furkan KAMACI <fu...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi Mike,
>>>>>
>>>>>
>>>>>
>>>>> Could you tell us what "Document Status" and "Queue Status" says as
>>>>> Karl mentioned?
>>>>>
>>>>>
>>>>>
>>>>> Kind Regards,
>>>>>
>>>>> Furkan KAMACI
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 30, 2016 at 3:39 PM, Holtz, Michael (IT Consultant) <
>>>>> MHoltz@consultantemail.com> wrote:
>>>>>
>>>>> I’m looking at manifoldcf.log and I see no errors.
>>>>>
>>>>>
>>>>>
>>>>> I have only 13 files and they are all very tiny < 15kb .docx files.
>>>>>
>>>>>
>>>>>
>>>>> Mike H.
>>>>>
>>>>>
>>>>>
>>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>>> *Sent:* Monday, August 29, 2016 5:26 PM
>>>>>
>>>>>
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> You should be looking at the ManifoldCF.log file, and the examples
>>>>> have logging for errors at least as being the default.
>>>>>
>>>>>
>>>>>
>>>>> The only other possibility is that you have a very large document that
>>>>> is taking a long time to transfer/index.
>>>>>
>>>>>
>>>>>
>>>>> Also, you can generally see what MCF is up to using the "document
>>>>> status" and "queue status" reports.
>>>>>
>>>>>
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Aug 29, 2016 at 5:22 PM, Holtz, Michael (IT Consultant) <
>>>>> MHoltz@consultantemail.com> wrote:
>>>>>
>>>>> Karl,
>>>>>
>>>>>
>>>>>
>>>>> I see no errors in the log.
>>>>>
>>>>>
>>>>>
>>>>> I probably need to configure the logging I think.
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mike H.
>>>>>
>>>>>
>>>>>
>>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>>> *Sent:* Monday, August 29, 2016 5:20 PM
>>>>>
>>>>>
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> Hi Mike,
>>>>>
>>>>>
>>>>>
>>>>> Have a look in the log. If there are errors there please send them to
>>>>> us. Generally, "hangs" are due to MCF retrying documents that have errors.
>>>>>
>>>>>
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Aug 29, 2016 at 5:18 PM, Holtz, Michael (IT Consultant) <
>>>>> MHoltz@consultantemail.com> wrote:
>>>>>
>>>>> Karl,
>>>>>
>>>>>
>>>>>
>>>>> Yes, it shows up. I include all meta-data.
>>>>>
>>>>>
>>>>>
>>>>> I’m no trying to use a MetaData adjuster like this
>>>>>
>>>>>
>>>>>
>>>>> Parameter
>>>>>
>>>>> field1 “${Field 1}”
>>>>>
>>>>>
>>>>>
>>>>> And now I see “field1”=”” in Elastic Search.
>>>>>
>>>>>
>>>>>
>>>>> Also it seems each time I make a change the job will hang and I need
>>>>> to restart everything including ElasticSearch to get it to complete.
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mike H.
>>>>>
>>>>>
>>>>>
>>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>>> *Sent:* Monday, August 29, 2016 5:13 PM
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> Hi Mike,
>>>>>
>>>>>
>>>>>
>>>>> If I were you I'd try to independently verify that the custom field
>>>>> shows up. Do you see it appear in the pulldown for selecting metadata? If
>>>>> not, it probably is not accessible through the inspection methods that give
>>>>> us access into SharePoint fields.
>>>>>
>>>>>
>>>>>
>>>>> If it appears there then we should look at what ES gets sent. It's
>>>>> possible it appears but under a different field name.
>>>>>
>>>>>
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Aug 29, 2016 at 5:01 PM, Holtz, Michael (IT Consultant) <
>>>>> MHoltz@consultantemail.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>>
>>>>> I have created a custom content type with a custom field “Field 1”.
>>>>>
>>>>>
>>>>>
>>>>> I have configured the job to include all meta data.
>>>>>
>>>>>
>>>>>
>>>>> I do not see “Field 1” being populated in the Elastic Search index.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Does anyone have any experience with this?
>>>>>
>>>>>
>>>>>
>>>>> Thanks in advance,
>>>>>
>>>>> Mike H
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>