You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@manifoldcf.apache.org by Nikita Ahuja <ni...@smartshore.nl> on 2018/01/25 06:49:47 UTC
Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
Hello,
I am using ManifoldCF 2.8.1 and trying to access the files from FileShare
configured with the Active Directory and get the data in elastic search
index.
For this steps followed are:
- An Authority Group is created under which Authority Connection and
Repository connection are created.
[image: Inline image 3]
- And the elastic search is created as an Output Connector and a job is
created for the connection in which a transformation and metadata connector
is also linked in the
- Which is giving all the output except the access token values.
- [image: Inline image 1]
Please guide to take appropriate steps for it so that I can get the access
tokens in the elastic search index. Hoping for the solution asap.
Thanks and Regards,
Nikita
Fwd: Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
Posted by Nikita Ahuja <ni...@smartshore.nl>.
---------- Forwarded message ----------
From: Nikita Ahuja <ni...@smartshore.nl>
Date: Thu, Jan 25, 2018 at 12:19 PM
Subject: Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
To: user@manifoldcf.apache.org
Hello,
I am using ManifoldCF 2.8.1 and trying to access the files from FileShare
configured with the Active Directory and get the data in elastic search
index.
For this steps followed are:
- An Authority Group is created under which Authority Connection and
Repository connection are created.
[image: Inline image 3]
- And the elastic search is created as an Output Connector and a job is
created for the connection in which a transformation and metadata connector
is also linked in the
- Which is giving all the output except the access token values.
- [image: Inline image 1]
Please guide to take appropriate steps for it so that I can get the access
tokens in the elastic search index. Hoping for the solution asap.
Thanks and Regards,
Nikita
Fwd: Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
Posted by Nikita Ahuja <ni...@smartshore.nl>.
---------- Forwarded message ----------
From: Nikita Ahuja <ni...@smartshore.nl>
Date: Thu, Jan 25, 2018 at 12:19 PM
Subject: Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
To: user@manifoldcf.apache.org
Hello,
I am using ManifoldCF 2.8.1 and trying to access the files from FileShare
configured with the Active Directory and get the data in elastic search
index.
For this steps followed are:
- An Authority Group is created under which Authority Connection and
Repository connection are created.
[image: Inline image 3]
- And the elastic search is created as an Output Connector and a job is
created for the connection in which a transformation and metadata connector
is also linked in the
- Which is giving all the output except the access token values.
- [image: Inline image 1]
Please guide to take appropriate steps for it so that I can get the access
tokens in the elastic search index. Hoping for the solution asap.
Thanks and Regards,
Nikita
Re: Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
Posted by Karl Wright <da...@gmail.com>.
I just noticed something -- useActiveDirectory is set to "true". That
means that for every user ACL ManifoldCF finds, it will try to convert it
to a SID. It's supposed to fail hard if it can't do that but that code is
possibly not correct. I suspect this is why you are getting no tokens
back.
Is there any reason you can't run things in full claims-based mode? That
is, configure your repository connector for "claims based" and then use
both the SharepointADAuthority and the SharepointNativeAuthority in the
same authority group, which is how you are supposed to set things up with
claims-based auth?
Karl
On Tue, Feb 6, 2018 at 10:03 AM, Karl Wright <da...@gmail.com> wrote:
> Can you clarify what you mean by:
>
> " And in the log file there are ACLs in the the "share_document " but am
> not able to get the access token in the elastic search index."
>
> Can you provide the full log message where you see the ACLs? I need to
> see where it is being logged.
>
> Karl
>
> On Tue, Feb 6, 2018 at 8:00 AM, Nikita Ahuja <ni...@smartshore.nl> wrote:
>
>> Hi Karl,
>>
>>
>> I have checked that the document on the Sharepoint and it is not
>> "Public".
>>
>> And in the log file there are ACLs in the the "share_document " but am
>> not able to get the access token in the elastic search index.
>>
>> Output in the ManifoldCF log file:
>>
>> DEBUG 2018-02-06T18:24:22,443 (Worker thread '8') - SharePoint: In
>> getACLs; site='', guid='{E145B91E-2890-42E6-BCDB-1AE393DD3F9C}',
>> activeDirectoryAuthority=true
>> DEBUG 2018-02-06T18:24:22,472 (Worker thread '8') - SharePoint: getACLs
>> xml response: <ns1:GetPermissionCollection xmlns:ns1="http://schemas.micr
>> osoft.com/sharepoint/soap/directory/
>> <https://www.google.com/url?q=http://schemas.microsoft.com/sharepoint/soap/directory/&sa=D&ust=1518007947929000&usg=AFQjCNFvqyAit3q3uRJHM8ZexlNKv_8OlA>
>> "><ns1:Permissions><ns1:Permission MemberID="1" Mask="-1"
>> MemberIsUser="True" MemberGlobal="False" UserLogin="i:0#.w|smartshore\administrator"/><ns1:Permission
>> MemberID="3" Mask="138612801" MemberIsUser="False" MemberGlobal="True"
>> GroupName="Excel Services Viewers"/><ns1:Permission MemberID="4"
>> Mask="134287360" MemberIsUser="True" MemberGlobal="False"
>> UserLogin="c:0(.s|true"/><ns1:Permission MemberID="5" Mask="-1"
>> MemberIsUser="False" MemberGlobal="True" GroupName="NikitaSite
>> Owners"/><ns1:Permission MemberID="6" Mask="138612833" MemberIsUser="False"
>> MemberGlobal="True" GroupName="NikitaSite Visitors"/><ns1:Permission
>> MemberID="7" Mask="1011030767" MemberIsUser="False" MemberGlobal="True"
>> GroupName="NikitaSite Members"/><ns1:Permission MemberID="8"
>> Mask="134287360" MemberIsUser="True" MemberGlobal="False"
>> UserLogin="i:0#.w|smartshore\nikita"/><ns1:Permission MemberID="9"
>> Mask="-1" MemberIsUser="True" MemberGlobal="False"
>> UserLogin="i:0#.w|smartshore\manjinder"/></ns1:Permissions><
>> /ns1:GetPermissionCollection>
>> DEBUG 2018-02-06T18:24:22,639 (Worker thread '8') - SharePoint: In
>> getChildren; site='', guid='{E145B91E-2890-42E6-BCDB-1AE393DD3F9C}',
>> dspStsWorks=false
>>
>>
>>
>> What is the actual problem behind this as the MCPermission plugin is
>> still returning the values here I guess.
>>
>>
>> Thanks and Regards,
>> Nikita
>>
>>
>> On Mon, Feb 5, 2018 at 7:56 PM, Karl Wright <da...@gmail.com> wrote:
>>
>>> No, everything looks properly configured.
>>>
>>> First, please verify that the document for which you are seeing
>>> "__nosecurity__" tokens being indexed is NOT a public document in
>>> Sharepoint. If it is public, then the security tokens being indexed are
>>> indeed correct.
>>>
>>> Once you have verified that, then the next step is to turn on wire
>>> debugging. This will log all SOAP traffic back and forth between
>>> ManifoldCF and the server. We really need the logs for only one document
>>> being indexed -- the goal is to see whether the response we're getting back
>>> from the MCPermissions call actually returns any ACLs. It seems like it is
>>> silently failing to do so -- very bad, because that means that security is
>>> not being enforced. Turning on wire debugging involves making changes to
>>> the logging.xml file; you will need to add appropriate loggers for
>>> HttpComponents/HttpClient. The instructions are here:
>>>
>>> https://cwiki.apache.org/confluence/display/CONNECTORS/Debug
>>> ging+Connections
>>>
>>> Once you have redacted whatever sensitive information you feel
>>> necessary, please post a link to a google doc that includes the chunk of
>>> log corresponding to one document that is not public.
>>>
>>> Thanks!
>>>
>>> Karl
>>>
>>>
>>>
>>>
>>> On Mon, Feb 5, 2018 at 9:09 AM, Nikita Ahuja <ni...@smartshore.nl>
>>> wrote:
>>>
>>>> Hi Karl,
>>>>
>>>>
>>>> This is Sharepoint 2013 version being used. and it uses the Claim Based
>>>> Authentication technique only.
>>>> And the repository connection created is for Sharepoint 2013 version
>>>> only
>>>> [image: Inline image 1]
>>>>
>>>>
>>>>
>>>> [image: Inline image 2]
>>>>
>>>>
>>>> [image: Inline image 3]
>>>>
>>>> Is there anything which is being missed to add in the connector??
>>>>
>>>>
>>>>
>>>> Thanks and Regards,
>>>> Nikita
>>>>
>>>>
>>>> On Mon, Feb 5, 2018 at 7:33 PM, Karl Wright <da...@gmail.com> wrote:
>>>>
>>>>> The "__nosecurity__" values, for Elastic Search as an output, come
>>>>> from ManifoldCF. So you are definitely getting the access tokens from
>>>>> ManifoldCF to ElasticSearch.
>>>>>
>>>>> The configuration for the Security tab also looks correct.
>>>>>
>>>>> So at the moment I cannot explain why you are not getting security
>>>>> tokens out of SharePoint. About the only possibility left is whether the
>>>>> wrong version of Sharepoint is specified. Can you describe:
>>>>> - What version of Sharepoint this is;
>>>>> - Whether it is configured to use Claims Based authorization
>>>>>
>>>>> And please then take a screen shot when you view the repository
>>>>> connection. Thanks!
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>> On Mon, Feb 5, 2018 at 3:21 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>>> wrote:
>>>>>
>>>>>> Hi Karl,
>>>>>>
>>>>>> I have done the configuration of the job as per the *End User
>>>>>> Documentation*. Still, I am getting "no security" value in the
>>>>>> access tokens in the elastic search index (output of my job).
>>>>>>
>>>>>> [image: Inline image 2]
>>>>>>
>>>>>>
>>>>>> This is the Output connector for Elastic Search. Is there anything
>>>>>> needs to added here?
>>>>>>
>>>>>> [image: Inline image 1]
>>>>>>
>>>>>>
>>>>>>
>>>>>> Also, the security is also enabled.
>>>>>>
>>>>>> [image: Inline image 3]
>>>>>>
>>>>>>
>>>>>> I am not able to understand where is the actual problem?
>>>>>>
>>>>>>
>>>>>> Thanks and Regards,
>>>>>> Nikita
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Feb 2, 2018 at 7:53 PM, Karl Wright <da...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Nikita,
>>>>>>>
>>>>>>> If you are seeing *no* tokens indexed then I would check your output
>>>>>>> connection and your index configuration. Custom fields need to be added to
>>>>>>> Solr for these tokens to be indexed, if you are using Solr. There are
>>>>>>> instructions in the online documentation on what to add and how to
>>>>>>> configure them.
>>>>>>>
>>>>>>> The other possibility is that you have disabled security for you
>>>>>>> Sharepoint job in the "Security" tab.
>>>>>>>
>>>>>>> Karl
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Feb 2, 2018 at 8:48 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Karl,
>>>>>>>>
>>>>>>>>
>>>>>>>> Yes, you are right I am able to fetch the files but the access
>>>>>>>> tokens indexed with the files are not included what is required. And I am
>>>>>>>> using the ManifoldCF security model not any custom. Everything is fine but
>>>>>>>> only problem is with the token.
>>>>>>>> And am not sure what is the reason behind that: is it because the
>>>>>>>> Sharepoint site which is used is not with *HTTPS* protocol or any
>>>>>>>> other reason?
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks and Regards,
>>>>>>>> Nikita
>>>>>>>>
>>>>>>>> On Fri, Feb 2, 2018 at 7:05 PM, Karl Wright <da...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I am not sure I understand -- are you saying that you can fetch
>>>>>>>>> the files but the access tokens indexed with the files do not include what
>>>>>>>>> you require? And that the reason you require it done a certain way is
>>>>>>>>> because you intend to apply security in a manner of your own design?
>>>>>>>>>
>>>>>>>>> The access tokens that are indexed with files in SharePoint are
>>>>>>>>> what are needed to enforce security using the ManifoldCF Sharepoint
>>>>>>>>> authorities, no more and no less. The authorities map an individual user
>>>>>>>>> to the access tokens needed for that user, which include tokens for the
>>>>>>>>> groups as well as for the user name. The tokens indexed with the files do
>>>>>>>>> NOT include tokens generated by mapping groups to individual users -- the
>>>>>>>>> only tokens indexed for a file are for the groups and for any individual
>>>>>>>>> users specified explicitly.
>>>>>>>>>
>>>>>>>>> I strongly suggest you attempt to apply security fully using the
>>>>>>>>> ManifoldCF model rather than trying to do something custom.
>>>>>>>>>
>>>>>>>>> Karl
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Feb 2, 2018 at 8:04 AM, Nikita Ahuja <nikita@smartshore.nl
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> Hi Karl,
>>>>>>>>>>
>>>>>>>>>> Thanks a lot for your help.It is working fine now.. I am able to
>>>>>>>>>> fetch all the documents in the path but there is a requirement to get the
>>>>>>>>>> access tokens for the files. But,it shows the authority group name not any
>>>>>>>>>> details related to the users who all have access for the document accessed.
>>>>>>>>>>
>>>>>>>>>> Please guide for that.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks and Regards,
>>>>>>>>>> Nikita
>>>>>>>>>>
>>>>>>>>>> On Wed, Jan 31, 2018 at 5:55 PM, Karl Wright <da...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Clearly you are unable to fetch document ACLs from SharePoint.
>>>>>>>>>>> This has nothing to do with whether your Active Directory is properly
>>>>>>>>>>> configured, and has more to do with the MCPermissions plugin.
>>>>>>>>>>>
>>>>>>>>>>> When you view the SharePoint repository connection, does it say
>>>>>>>>>>> "Connection working"? If not, you need the figure out why.
>>>>>>>>>>>
>>>>>>>>>>> Here are some possibilities:
>>>>>>>>>>> (1) The plugin was properly installed but with the wrong
>>>>>>>>>>> permissions. You must be logged in as an Admin to install it properly. If
>>>>>>>>>>> that's not done it will mostly work but not be able to retrieve the
>>>>>>>>>>> permissions.
>>>>>>>>>>> (2) You may have selected the wrong version of Sharepoint in the
>>>>>>>>>>> repository connection. Make sure you select the right one or, once again,
>>>>>>>>>>> you won't be able to retrieve permissions.
>>>>>>>>>>>
>>>>>>>>>>> The documentation says this: "Select your SharePoint server
>>>>>>>>>>> version from the pulldown. If you do not select the correct server version,
>>>>>>>>>>> your documents may either be indexed with insufficient security protection,
>>>>>>>>>>> or you may not be able to index any documents. Check with your SharePoint
>>>>>>>>>>> system administrator if you are not sure what to select."
>>>>>>>>>>>
>>>>>>>>>>> Please let me know whether this helps. Thanks!
>>>>>>>>>>> Karl
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jan 31, 2018 at 6:51 AM, Nikita Ahuja <
>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Karl,
>>>>>>>>>>>>
>>>>>>>>>>>> I have checked the log Files but there is issue with the ACLs
>>>>>>>>>>>> but there is Active Directory already connected with Sharepoint site. And
>>>>>>>>>>>> it is not possible to pass the Access tokens in the connector. Please
>>>>>>>>>>>> provide the suggestion for the problem.
>>>>>>>>>>>>
>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>> Nikita
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jan 29, 2018 at 6:55 PM, Karl Wright <
>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I don't see any path rules there. That will clearly not work.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If you want confirmation of this, you can turn on connector
>>>>>>>>>>>>> debugging (set properties.xml property org.apache.manifoldcf.connectors
>>>>>>>>>>>>> to "DEBUG"), look at the log output. Every time the SharePoint connector
>>>>>>>>>>>>> makes a decision to not include something, it logs it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Jan 29, 2018 at 8:07 AM, Nikita Ahuja <
>>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have checked the documentation and the path configuration
>>>>>>>>>>>>>> as well. But still not able to get any output.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> and the metadata path given is:
>>>>>>>>>>>>>> [image: Inline image 2]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Jan 29, 2018 at 4:45 PM, Karl Wright <
>>>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think your path configuration is incorrect. See the
>>>>>>>>>>>>>>> end-user documentation. It's a little tricky because you must specify a
>>>>>>>>>>>>>>> matching rule for both files and paths.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Jan 29, 2018 at 4:42 AM, Nikita Ahuja <
>>>>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Hi Karl,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> There is another query related to Sharepoint 2013
>>>>>>>>>>>>>>>> Connector. I am trying to crawl the files of Sharepoint On Premise site.
>>>>>>>>>>>>>>>> But I am not able to get anything in the output either in ElasticSearch or
>>>>>>>>>>>>>>>> in FileShare.
>>>>>>>>>>>>>>>> I have followed given steps:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Created an Authority group for the Active Directory
>>>>>>>>>>>>>>>> and same is used in Sharepoint Repository Connection.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [image: Inline image 2]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Installed the Sharepoint 2013 MCPermission Plugin and
>>>>>>>>>>>>>>>> is accessible on the site.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [image: Inline image 3]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Still not able to access any document from the
>>>>>>>>>>>>>>>> selected folder of Sharepoint Site.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Please guide me accordingly.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 6:46 PM, Nikita Ahuja <
>>>>>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks Karl,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It is working fine now and Also I am able to get the
>>>>>>>>>>>>>>>>> access tokens in the elastic search index.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks a lot for your valuable suggestions.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> With Regards,
>>>>>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 4:42 PM, Karl Wright <
>>>>>>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The __nosecurity__ token is generated by the Elastic
>>>>>>>>>>>>>>>>>> Search Connector. It is produced (correctly) when no acls reach the
>>>>>>>>>>>>>>>>>> connector. This is why I asked you about the repository connection,
>>>>>>>>>>>>>>>>>> because that connection is not producing any ACLs.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> If you are crawling files in a file system, you must use
>>>>>>>>>>>>>>>>>> the JCIFS connector if you want ACLs to be generated. The FileSystem
>>>>>>>>>>>>>>>>>> connector does not generate ACLs.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 3:56 AM, Nikita Ahuja <
>>>>>>>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Karl,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I have tried for ElasticSearch version 1.3.1 and 5.6.1
>>>>>>>>>>>>>>>>>>> but still getting this " __nosecurity__" token and for
>>>>>>>>>>>>>>>>>>> Repository connection File Sytem is used and for Transformation "Tika
>>>>>>>>>>>>>>>>>>> Content Extractor" and Metadata adjuster Connector is
>>>>>>>>>>>>>>>>>>> used.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Is there any other thing which I need to perform??
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 2:13 PM, Karl Wright <
>>>>>>>>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Nikita,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> You are getting the __nosecurity__ token value
>>>>>>>>>>>>>>>>>>>> transmitted and stored, which means your Elastic Search setup is probably
>>>>>>>>>>>>>>>>>>>> reasonable.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Can you give us details about your pipeline? What
>>>>>>>>>>>>>>>>>>>> repository connector is this?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 1:49 AM, Nikita Ahuja <
>>>>>>>>>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I am using ManifoldCF 2.8.1 and trying to access the
>>>>>>>>>>>>>>>>>>>>> files from FileShare configured with the Active Directory and get the data
>>>>>>>>>>>>>>>>>>>>> in elastic search index.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> For this steps followed are:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> - An Authority Group is created under which
>>>>>>>>>>>>>>>>>>>>> Authority Connection and Repository connection are created.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> [image: Inline image 3]
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> - And the elastic search is created as an Output
>>>>>>>>>>>>>>>>>>>>> Connector and a job is created for the connection in which a transformation
>>>>>>>>>>>>>>>>>>>>> and metadata connector is also linked in the
>>>>>>>>>>>>>>>>>>>>> - Which is giving all the output except the access
>>>>>>>>>>>>>>>>>>>>> token values.
>>>>>>>>>>>>>>>>>>>>> - [image: Inline image 1]
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Please guide to take appropriate steps for it so that
>>>>>>>>>>>>>>>>>>>>> I can get the access tokens in the elastic search index. Hoping for the
>>>>>>>>>>>>>>>>>>>>> solution asap.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
Posted by Karl Wright <da...@gmail.com>.
Can you clarify what you mean by:
" And in the log file there are ACLs in the the "share_document " but am
not able to get the access token in the elastic search index."
Can you provide the full log message where you see the ACLs? I need to see
where it is being logged.
Karl
On Tue, Feb 6, 2018 at 8:00 AM, Nikita Ahuja <ni...@smartshore.nl> wrote:
> Hi Karl,
>
>
> I have checked that the document on the Sharepoint and it is not "Public".
>
> And in the log file there are ACLs in the the "share_document " but am not
> able to get the access token in the elastic search index.
>
> Output in the ManifoldCF log file:
>
> DEBUG 2018-02-06T18:24:22,443 (Worker thread '8') - SharePoint: In
> getACLs; site='', guid='{E145B91E-2890-42E6-BCDB-1AE393DD3F9C}',
> activeDirectoryAuthority=true
> DEBUG 2018-02-06T18:24:22,472 (Worker thread '8') - SharePoint: getACLs
> xml response: <ns1:GetPermissionCollection xmlns:ns1="http://schemas.
> microsoft.com/sharepoint/soap/directory/
> <https://www.google.com/url?q=http://schemas.microsoft.com/sharepoint/soap/directory/&sa=D&ust=1518007947929000&usg=AFQjCNFvqyAit3q3uRJHM8ZexlNKv_8OlA>
> "><ns1:Permissions><ns1:Permission MemberID="1" Mask="-1"
> MemberIsUser="True" MemberGlobal="False" UserLogin="i:0#.w|smartshore\
> administrator"/><ns1:Permission MemberID="3" Mask="138612801"
> MemberIsUser="False" MemberGlobal="True" GroupName="Excel Services
> Viewers"/><ns1:Permission MemberID="4" Mask="134287360" MemberIsUser="True"
> MemberGlobal="False" UserLogin="c:0(.s|true"/><ns1:Permission
> MemberID="5" Mask="-1" MemberIsUser="False" MemberGlobal="True"
> GroupName="NikitaSite Owners"/><ns1:Permission MemberID="6"
> Mask="138612833" MemberIsUser="False" MemberGlobal="True"
> GroupName="NikitaSite Visitors"/><ns1:Permission MemberID="7"
> Mask="1011030767" MemberIsUser="False" MemberGlobal="True"
> GroupName="NikitaSite Members"/><ns1:Permission MemberID="8"
> Mask="134287360" MemberIsUser="True" MemberGlobal="False"
> UserLogin="i:0#.w|smartshore\nikita"/><ns1:Permission MemberID="9"
> Mask="-1" MemberIsUser="True" MemberGlobal="False"
> UserLogin="i:0#.w|smartshore\manjinder"/></ns1:Permissions>
> </ns1:GetPermissionCollection>
> DEBUG 2018-02-06T18:24:22,639 (Worker thread '8') - SharePoint: In
> getChildren; site='', guid='{E145B91E-2890-42E6-BCDB-1AE393DD3F9C}',
> dspStsWorks=false
>
>
>
> What is the actual problem behind this as the MCPermission plugin is still
> returning the values here I guess.
>
>
> Thanks and Regards,
> Nikita
>
>
> On Mon, Feb 5, 2018 at 7:56 PM, Karl Wright <da...@gmail.com> wrote:
>
>> No, everything looks properly configured.
>>
>> First, please verify that the document for which you are seeing
>> "__nosecurity__" tokens being indexed is NOT a public document in
>> Sharepoint. If it is public, then the security tokens being indexed are
>> indeed correct.
>>
>> Once you have verified that, then the next step is to turn on wire
>> debugging. This will log all SOAP traffic back and forth between
>> ManifoldCF and the server. We really need the logs for only one document
>> being indexed -- the goal is to see whether the response we're getting back
>> from the MCPermissions call actually returns any ACLs. It seems like it is
>> silently failing to do so -- very bad, because that means that security is
>> not being enforced. Turning on wire debugging involves making changes to
>> the logging.xml file; you will need to add appropriate loggers for
>> HttpComponents/HttpClient. The instructions are here:
>>
>> https://cwiki.apache.org/confluence/display/CONNECTORS/Debug
>> ging+Connections
>>
>> Once you have redacted whatever sensitive information you feel necessary,
>> please post a link to a google doc that includes the chunk of log
>> corresponding to one document that is not public.
>>
>> Thanks!
>>
>> Karl
>>
>>
>>
>>
>> On Mon, Feb 5, 2018 at 9:09 AM, Nikita Ahuja <ni...@smartshore.nl>
>> wrote:
>>
>>> Hi Karl,
>>>
>>>
>>> This is Sharepoint 2013 version being used. and it uses the Claim Based
>>> Authentication technique only.
>>> And the repository connection created is for Sharepoint 2013 version
>>> only
>>> [image: Inline image 1]
>>>
>>>
>>>
>>> [image: Inline image 2]
>>>
>>>
>>> [image: Inline image 3]
>>>
>>> Is there anything which is being missed to add in the connector??
>>>
>>>
>>>
>>> Thanks and Regards,
>>> Nikita
>>>
>>>
>>> On Mon, Feb 5, 2018 at 7:33 PM, Karl Wright <da...@gmail.com> wrote:
>>>
>>>> The "__nosecurity__" values, for Elastic Search as an output, come from
>>>> ManifoldCF. So you are definitely getting the access tokens from
>>>> ManifoldCF to ElasticSearch.
>>>>
>>>> The configuration for the Security tab also looks correct.
>>>>
>>>> So at the moment I cannot explain why you are not getting security
>>>> tokens out of SharePoint. About the only possibility left is whether the
>>>> wrong version of Sharepoint is specified. Can you describe:
>>>> - What version of Sharepoint this is;
>>>> - Whether it is configured to use Claims Based authorization
>>>>
>>>> And please then take a screen shot when you view the repository
>>>> connection. Thanks!
>>>>
>>>> Karl
>>>>
>>>>
>>>> On Mon, Feb 5, 2018 at 3:21 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>> wrote:
>>>>
>>>>> Hi Karl,
>>>>>
>>>>> I have done the configuration of the job as per the *End User
>>>>> Documentation*. Still, I am getting "no security" value in the access
>>>>> tokens in the elastic search index (output of my job).
>>>>>
>>>>> [image: Inline image 2]
>>>>>
>>>>>
>>>>> This is the Output connector for Elastic Search. Is there anything
>>>>> needs to added here?
>>>>>
>>>>> [image: Inline image 1]
>>>>>
>>>>>
>>>>>
>>>>> Also, the security is also enabled.
>>>>>
>>>>> [image: Inline image 3]
>>>>>
>>>>>
>>>>> I am not able to understand where is the actual problem?
>>>>>
>>>>>
>>>>> Thanks and Regards,
>>>>> Nikita
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Feb 2, 2018 at 7:53 PM, Karl Wright <da...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Nikita,
>>>>>>
>>>>>> If you are seeing *no* tokens indexed then I would check your output
>>>>>> connection and your index configuration. Custom fields need to be added to
>>>>>> Solr for these tokens to be indexed, if you are using Solr. There are
>>>>>> instructions in the online documentation on what to add and how to
>>>>>> configure them.
>>>>>>
>>>>>> The other possibility is that you have disabled security for you
>>>>>> Sharepoint job in the "Security" tab.
>>>>>>
>>>>>> Karl
>>>>>>
>>>>>>
>>>>>> On Fri, Feb 2, 2018 at 8:48 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Karl,
>>>>>>>
>>>>>>>
>>>>>>> Yes, you are right I am able to fetch the files but the access
>>>>>>> tokens indexed with the files are not included what is required. And I am
>>>>>>> using the ManifoldCF security model not any custom. Everything is fine but
>>>>>>> only problem is with the token.
>>>>>>> And am not sure what is the reason behind that: is it because the
>>>>>>> Sharepoint site which is used is not with *HTTPS* protocol or any
>>>>>>> other reason?
>>>>>>>
>>>>>>>
>>>>>>> Thanks and Regards,
>>>>>>> Nikita
>>>>>>>
>>>>>>> On Fri, Feb 2, 2018 at 7:05 PM, Karl Wright <da...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I am not sure I understand -- are you saying that you can fetch the
>>>>>>>> files but the access tokens indexed with the files do not include what you
>>>>>>>> require? And that the reason you require it done a certain way is because
>>>>>>>> you intend to apply security in a manner of your own design?
>>>>>>>>
>>>>>>>> The access tokens that are indexed with files in SharePoint are
>>>>>>>> what are needed to enforce security using the ManifoldCF Sharepoint
>>>>>>>> authorities, no more and no less. The authorities map an individual user
>>>>>>>> to the access tokens needed for that user, which include tokens for the
>>>>>>>> groups as well as for the user name. The tokens indexed with the files do
>>>>>>>> NOT include tokens generated by mapping groups to individual users -- the
>>>>>>>> only tokens indexed for a file are for the groups and for any individual
>>>>>>>> users specified explicitly.
>>>>>>>>
>>>>>>>> I strongly suggest you attempt to apply security fully using the
>>>>>>>> ManifoldCF model rather than trying to do something custom.
>>>>>>>>
>>>>>>>> Karl
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Feb 2, 2018 at 8:04 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Karl,
>>>>>>>>>
>>>>>>>>> Thanks a lot for your help.It is working fine now.. I am able to
>>>>>>>>> fetch all the documents in the path but there is a requirement to get the
>>>>>>>>> access tokens for the files. But,it shows the authority group name not any
>>>>>>>>> details related to the users who all have access for the document accessed.
>>>>>>>>>
>>>>>>>>> Please guide for that.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [image: Inline image 1]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks and Regards,
>>>>>>>>> Nikita
>>>>>>>>>
>>>>>>>>> On Wed, Jan 31, 2018 at 5:55 PM, Karl Wright <da...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Clearly you are unable to fetch document ACLs from SharePoint.
>>>>>>>>>> This has nothing to do with whether your Active Directory is properly
>>>>>>>>>> configured, and has more to do with the MCPermissions plugin.
>>>>>>>>>>
>>>>>>>>>> When you view the SharePoint repository connection, does it say
>>>>>>>>>> "Connection working"? If not, you need the figure out why.
>>>>>>>>>>
>>>>>>>>>> Here are some possibilities:
>>>>>>>>>> (1) The plugin was properly installed but with the wrong
>>>>>>>>>> permissions. You must be logged in as an Admin to install it properly. If
>>>>>>>>>> that's not done it will mostly work but not be able to retrieve the
>>>>>>>>>> permissions.
>>>>>>>>>> (2) You may have selected the wrong version of Sharepoint in the
>>>>>>>>>> repository connection. Make sure you select the right one or, once again,
>>>>>>>>>> you won't be able to retrieve permissions.
>>>>>>>>>>
>>>>>>>>>> The documentation says this: "Select your SharePoint server
>>>>>>>>>> version from the pulldown. If you do not select the correct server version,
>>>>>>>>>> your documents may either be indexed with insufficient security protection,
>>>>>>>>>> or you may not be able to index any documents. Check with your SharePoint
>>>>>>>>>> system administrator if you are not sure what to select."
>>>>>>>>>>
>>>>>>>>>> Please let me know whether this helps. Thanks!
>>>>>>>>>> Karl
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Jan 31, 2018 at 6:51 AM, Nikita Ahuja <
>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Karl,
>>>>>>>>>>>
>>>>>>>>>>> I have checked the log Files but there is issue with the ACLs
>>>>>>>>>>> but there is Active Directory already connected with Sharepoint site. And
>>>>>>>>>>> it is not possible to pass the Access tokens in the connector. Please
>>>>>>>>>>> provide the suggestion for the problem.
>>>>>>>>>>>
>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>> Nikita
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jan 29, 2018 at 6:55 PM, Karl Wright <daddywri@gmail.com
>>>>>>>>>>> > wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I don't see any path rules there. That will clearly not work.
>>>>>>>>>>>>
>>>>>>>>>>>> If you want confirmation of this, you can turn on connector
>>>>>>>>>>>> debugging (set properties.xml property org.apache.manifoldcf.connectors
>>>>>>>>>>>> to "DEBUG"), look at the log output. Every time the SharePoint connector
>>>>>>>>>>>> makes a decision to not include something, it logs it.
>>>>>>>>>>>>
>>>>>>>>>>>> Karl
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jan 29, 2018 at 8:07 AM, Nikita Ahuja <
>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have checked the documentation and the path configuration as
>>>>>>>>>>>>> well. But still not able to get any output.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> and the metadata path given is:
>>>>>>>>>>>>> [image: Inline image 2]
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Jan 29, 2018 at 4:45 PM, Karl Wright <
>>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think your path configuration is incorrect. See the
>>>>>>>>>>>>>> end-user documentation. It's a little tricky because you must specify a
>>>>>>>>>>>>>> matching rule for both files and paths.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Jan 29, 2018 at 4:42 AM, Nikita Ahuja <
>>>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Hi Karl,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> There is another query related to Sharepoint 2013 Connector.
>>>>>>>>>>>>>>> I am trying to crawl the files of Sharepoint On Premise site. But I am not
>>>>>>>>>>>>>>> able to get anything in the output either in ElasticSearch or in FileShare.
>>>>>>>>>>>>>>> I have followed given steps:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Created an Authority group for the Active Directory
>>>>>>>>>>>>>>> and same is used in Sharepoint Repository Connection.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [image: Inline image 2]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Installed the Sharepoint 2013 MCPermission Plugin and
>>>>>>>>>>>>>>> is accessible on the site.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [image: Inline image 3]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Still not able to access any document from the
>>>>>>>>>>>>>>> selected folder of Sharepoint Site.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Please guide me accordingly.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 6:46 PM, Nikita Ahuja <
>>>>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks Karl,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It is working fine now and Also I am able to get the access
>>>>>>>>>>>>>>>> tokens in the elastic search index.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks a lot for your valuable suggestions.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> With Regards,
>>>>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 4:42 PM, Karl Wright <
>>>>>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The __nosecurity__ token is generated by the Elastic
>>>>>>>>>>>>>>>>> Search Connector. It is produced (correctly) when no acls reach the
>>>>>>>>>>>>>>>>> connector. This is why I asked you about the repository connection,
>>>>>>>>>>>>>>>>> because that connection is not producing any ACLs.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If you are crawling files in a file system, you must use
>>>>>>>>>>>>>>>>> the JCIFS connector if you want ACLs to be generated. The FileSystem
>>>>>>>>>>>>>>>>> connector does not generate ACLs.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 3:56 AM, Nikita Ahuja <
>>>>>>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Karl,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I have tried for ElasticSearch version 1.3.1 and 5.6.1
>>>>>>>>>>>>>>>>>> but still getting this " __nosecurity__" token and for
>>>>>>>>>>>>>>>>>> Repository connection File Sytem is used and for Transformation "Tika
>>>>>>>>>>>>>>>>>> Content Extractor" and Metadata adjuster Connector is
>>>>>>>>>>>>>>>>>> used.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Is there any other thing which I need to perform??
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 2:13 PM, Karl Wright <
>>>>>>>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Nikita,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> You are getting the __nosecurity__ token value
>>>>>>>>>>>>>>>>>>> transmitted and stored, which means your Elastic Search setup is probably
>>>>>>>>>>>>>>>>>>> reasonable.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Can you give us details about your pipeline? What
>>>>>>>>>>>>>>>>>>> repository connector is this?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 1:49 AM, Nikita Ahuja <
>>>>>>>>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I am using ManifoldCF 2.8.1 and trying to access the
>>>>>>>>>>>>>>>>>>>> files from FileShare configured with the Active Directory and get the data
>>>>>>>>>>>>>>>>>>>> in elastic search index.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> For this steps followed are:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> - An Authority Group is created under which
>>>>>>>>>>>>>>>>>>>> Authority Connection and Repository connection are created.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [image: Inline image 3]
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> - And the elastic search is created as an Output
>>>>>>>>>>>>>>>>>>>> Connector and a job is created for the connection in which a transformation
>>>>>>>>>>>>>>>>>>>> and metadata connector is also linked in the
>>>>>>>>>>>>>>>>>>>> - Which is giving all the output except the access
>>>>>>>>>>>>>>>>>>>> token values.
>>>>>>>>>>>>>>>>>>>> - [image: Inline image 1]
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Please guide to take appropriate steps for it so that I
>>>>>>>>>>>>>>>>>>>> can get the access tokens in the elastic search index. Hoping for the
>>>>>>>>>>>>>>>>>>>> solution asap.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
Posted by Nikita Ahuja <ni...@smartshore.nl>.
Hi Karl,
I have checked that the document on the Sharepoint and it is not "Public".
And in the log file there are ACLs in the the "share_document " but am not
able to get the access token in the elastic search index.
Output in the ManifoldCF log file:
DEBUG 2018-02-06T18:24:22,443 (Worker thread '8') - SharePoint: In getACLs;
site='', guid='{E145B91E-2890-42E6-BCDB-1AE393DD3F9C}',
activeDirectoryAuthority=true
DEBUG 2018-02-06T18:24:22,472 (Worker thread '8') - SharePoint: getACLs xml
response: <ns1:GetPermissionCollection xmlns:ns1="
http://schemas.microsoft.com/sharepoint/soap/directory/
<https://www.google.com/url?q=http://schemas.microsoft.com/sharepoint/soap/directory/&sa=D&ust=1518007947929000&usg=AFQjCNFvqyAit3q3uRJHM8ZexlNKv_8OlA>"><ns1:Permissions><ns1:Permission
MemberID="1" Mask="-1" MemberIsUser="True" MemberGlobal="False"
UserLogin="i:0#.w|smartshore\administrator"/><ns1:Permission MemberID="3"
Mask="138612801" MemberIsUser="False" MemberGlobal="True" GroupName="Excel
Services Viewers"/><ns1:Permission MemberID="4" Mask="134287360"
MemberIsUser="True" MemberGlobal="False"
UserLogin="c:0(.s|true"/><ns1:Permission MemberID="5" Mask="-1"
MemberIsUser="False" MemberGlobal="True" GroupName="NikitaSite
Owners"/><ns1:Permission MemberID="6" Mask="138612833" MemberIsUser="False"
MemberGlobal="True" GroupName="NikitaSite Visitors"/><ns1:Permission
MemberID="7" Mask="1011030767" MemberIsUser="False" MemberGlobal="True"
GroupName="NikitaSite Members"/><ns1:Permission MemberID="8"
Mask="134287360" MemberIsUser="True" MemberGlobal="False"
UserLogin="i:0#.w|smartshore\nikita"/><ns1:Permission MemberID="9"
Mask="-1" MemberIsUser="True" MemberGlobal="False"
UserLogin="i:0#.w|smartshore\manjinder"/></ns1:Permissions></ns1:GetPermissionCollection>
DEBUG 2018-02-06T18:24:22,639 (Worker thread '8') - SharePoint: In
getChildren; site='', guid='{E145B91E-2890-42E6-BCDB-1AE393DD3F9C}',
dspStsWorks=false
What is the actual problem behind this as the MCPermission plugin is still
returning the values here I guess.
Thanks and Regards,
Nikita
On Mon, Feb 5, 2018 at 7:56 PM, Karl Wright <da...@gmail.com> wrote:
> No, everything looks properly configured.
>
> First, please verify that the document for which you are seeing
> "__nosecurity__" tokens being indexed is NOT a public document in
> Sharepoint. If it is public, then the security tokens being indexed are
> indeed correct.
>
> Once you have verified that, then the next step is to turn on wire
> debugging. This will log all SOAP traffic back and forth between
> ManifoldCF and the server. We really need the logs for only one document
> being indexed -- the goal is to see whether the response we're getting back
> from the MCPermissions call actually returns any ACLs. It seems like it is
> silently failing to do so -- very bad, because that means that security is
> not being enforced. Turning on wire debugging involves making changes to
> the logging.xml file; you will need to add appropriate loggers for
> HttpComponents/HttpClient. The instructions are here:
>
> https://cwiki.apache.org/confluence/display/CONNECTORS/
> Debugging+Connections
>
> Once you have redacted whatever sensitive information you feel necessary,
> please post a link to a google doc that includes the chunk of log
> corresponding to one document that is not public.
>
> Thanks!
>
> Karl
>
>
>
>
> On Mon, Feb 5, 2018 at 9:09 AM, Nikita Ahuja <ni...@smartshore.nl> wrote:
>
>> Hi Karl,
>>
>>
>> This is Sharepoint 2013 version being used. and it uses the Claim Based
>> Authentication technique only.
>> And the repository connection created is for Sharepoint 2013 version
>> only
>> [image: Inline image 1]
>>
>>
>>
>> [image: Inline image 2]
>>
>>
>> [image: Inline image 3]
>>
>> Is there anything which is being missed to add in the connector??
>>
>>
>>
>> Thanks and Regards,
>> Nikita
>>
>>
>> On Mon, Feb 5, 2018 at 7:33 PM, Karl Wright <da...@gmail.com> wrote:
>>
>>> The "__nosecurity__" values, for Elastic Search as an output, come from
>>> ManifoldCF. So you are definitely getting the access tokens from
>>> ManifoldCF to ElasticSearch.
>>>
>>> The configuration for the Security tab also looks correct.
>>>
>>> So at the moment I cannot explain why you are not getting security
>>> tokens out of SharePoint. About the only possibility left is whether the
>>> wrong version of Sharepoint is specified. Can you describe:
>>> - What version of Sharepoint this is;
>>> - Whether it is configured to use Claims Based authorization
>>>
>>> And please then take a screen shot when you view the repository
>>> connection. Thanks!
>>>
>>> Karl
>>>
>>>
>>> On Mon, Feb 5, 2018 at 3:21 AM, Nikita Ahuja <ni...@smartshore.nl>
>>> wrote:
>>>
>>>> Hi Karl,
>>>>
>>>> I have done the configuration of the job as per the *End User
>>>> Documentation*. Still, I am getting "no security" value in the access
>>>> tokens in the elastic search index (output of my job).
>>>>
>>>> [image: Inline image 2]
>>>>
>>>>
>>>> This is the Output connector for Elastic Search. Is there anything
>>>> needs to added here?
>>>>
>>>> [image: Inline image 1]
>>>>
>>>>
>>>>
>>>> Also, the security is also enabled.
>>>>
>>>> [image: Inline image 3]
>>>>
>>>>
>>>> I am not able to understand where is the actual problem?
>>>>
>>>>
>>>> Thanks and Regards,
>>>> Nikita
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Feb 2, 2018 at 7:53 PM, Karl Wright <da...@gmail.com> wrote:
>>>>
>>>>> Hi Nikita,
>>>>>
>>>>> If you are seeing *no* tokens indexed then I would check your output
>>>>> connection and your index configuration. Custom fields need to be added to
>>>>> Solr for these tokens to be indexed, if you are using Solr. There are
>>>>> instructions in the online documentation on what to add and how to
>>>>> configure them.
>>>>>
>>>>> The other possibility is that you have disabled security for you
>>>>> Sharepoint job in the "Security" tab.
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>> On Fri, Feb 2, 2018 at 8:48 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>>> wrote:
>>>>>
>>>>>> Hi Karl,
>>>>>>
>>>>>>
>>>>>> Yes, you are right I am able to fetch the files but the access
>>>>>> tokens indexed with the files are not included what is required. And I am
>>>>>> using the ManifoldCF security model not any custom. Everything is fine but
>>>>>> only problem is with the token.
>>>>>> And am not sure what is the reason behind that: is it because the
>>>>>> Sharepoint site which is used is not with *HTTPS* protocol or any
>>>>>> other reason?
>>>>>>
>>>>>>
>>>>>> Thanks and Regards,
>>>>>> Nikita
>>>>>>
>>>>>> On Fri, Feb 2, 2018 at 7:05 PM, Karl Wright <da...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I am not sure I understand -- are you saying that you can fetch the
>>>>>>> files but the access tokens indexed with the files do not include what you
>>>>>>> require? And that the reason you require it done a certain way is because
>>>>>>> you intend to apply security in a manner of your own design?
>>>>>>>
>>>>>>> The access tokens that are indexed with files in SharePoint are what
>>>>>>> are needed to enforce security using the ManifoldCF Sharepoint authorities,
>>>>>>> no more and no less. The authorities map an individual user to the access
>>>>>>> tokens needed for that user, which include tokens for the groups as well as
>>>>>>> for the user name. The tokens indexed with the files do NOT include tokens
>>>>>>> generated by mapping groups to individual users -- the only tokens indexed
>>>>>>> for a file are for the groups and for any individual users specified
>>>>>>> explicitly.
>>>>>>>
>>>>>>> I strongly suggest you attempt to apply security fully using the
>>>>>>> ManifoldCF model rather than trying to do something custom.
>>>>>>>
>>>>>>> Karl
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Feb 2, 2018 at 8:04 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Karl,
>>>>>>>>
>>>>>>>> Thanks a lot for your help.It is working fine now.. I am able to
>>>>>>>> fetch all the documents in the path but there is a requirement to get the
>>>>>>>> access tokens for the files. But,it shows the authority group name not any
>>>>>>>> details related to the users who all have access for the document accessed.
>>>>>>>>
>>>>>>>> Please guide for that.
>>>>>>>>
>>>>>>>>
>>>>>>>> [image: Inline image 1]
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks and Regards,
>>>>>>>> Nikita
>>>>>>>>
>>>>>>>> On Wed, Jan 31, 2018 at 5:55 PM, Karl Wright <da...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Clearly you are unable to fetch document ACLs from SharePoint.
>>>>>>>>> This has nothing to do with whether your Active Directory is properly
>>>>>>>>> configured, and has more to do with the MCPermissions plugin.
>>>>>>>>>
>>>>>>>>> When you view the SharePoint repository connection, does it say
>>>>>>>>> "Connection working"? If not, you need the figure out why.
>>>>>>>>>
>>>>>>>>> Here are some possibilities:
>>>>>>>>> (1) The plugin was properly installed but with the wrong
>>>>>>>>> permissions. You must be logged in as an Admin to install it properly. If
>>>>>>>>> that's not done it will mostly work but not be able to retrieve the
>>>>>>>>> permissions.
>>>>>>>>> (2) You may have selected the wrong version of Sharepoint in the
>>>>>>>>> repository connection. Make sure you select the right one or, once again,
>>>>>>>>> you won't be able to retrieve permissions.
>>>>>>>>>
>>>>>>>>> The documentation says this: "Select your SharePoint server
>>>>>>>>> version from the pulldown. If you do not select the correct server version,
>>>>>>>>> your documents may either be indexed with insufficient security protection,
>>>>>>>>> or you may not be able to index any documents. Check with your SharePoint
>>>>>>>>> system administrator if you are not sure what to select."
>>>>>>>>>
>>>>>>>>> Please let me know whether this helps. Thanks!
>>>>>>>>> Karl
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Jan 31, 2018 at 6:51 AM, Nikita Ahuja <
>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Karl,
>>>>>>>>>>
>>>>>>>>>> I have checked the log Files but there is issue with the ACLs but
>>>>>>>>>> there is Active Directory already connected with Sharepoint site. And it is
>>>>>>>>>> not possible to pass the Access tokens in the connector. Please provide the
>>>>>>>>>> suggestion for the problem.
>>>>>>>>>>
>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks and Regards,
>>>>>>>>>> Nikita
>>>>>>>>>>
>>>>>>>>>> On Mon, Jan 29, 2018 at 6:55 PM, Karl Wright <da...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> I don't see any path rules there. That will clearly not work.
>>>>>>>>>>>
>>>>>>>>>>> If you want confirmation of this, you can turn on connector
>>>>>>>>>>> debugging (set properties.xml property org.apache.manifoldcf.connectors
>>>>>>>>>>> to "DEBUG"), look at the log output. Every time the SharePoint connector
>>>>>>>>>>> makes a decision to not include something, it logs it.
>>>>>>>>>>>
>>>>>>>>>>> Karl
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jan 29, 2018 at 8:07 AM, Nikita Ahuja <
>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I have checked the documentation and the path configuration as
>>>>>>>>>>>> well. But still not able to get any output.
>>>>>>>>>>>>
>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> and the metadata path given is:
>>>>>>>>>>>> [image: Inline image 2]
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>> Nikita
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jan 29, 2018 at 4:45 PM, Karl Wright <
>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think your path configuration is incorrect. See the
>>>>>>>>>>>>> end-user documentation. It's a little tricky because you must specify a
>>>>>>>>>>>>> matching rule for both files and paths.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Jan 29, 2018 at 4:42 AM, Nikita Ahuja <
>>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Hi Karl,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> There is another query related to Sharepoint 2013 Connector.
>>>>>>>>>>>>>> I am trying to crawl the files of Sharepoint On Premise site. But I am not
>>>>>>>>>>>>>> able to get anything in the output either in ElasticSearch or in FileShare.
>>>>>>>>>>>>>> I have followed given steps:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Created an Authority group for the Active Directory and
>>>>>>>>>>>>>> same is used in Sharepoint Repository Connection.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [image: Inline image 2]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Installed the Sharepoint 2013 MCPermission Plugin and
>>>>>>>>>>>>>> is accessible on the site.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [image: Inline image 3]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Still not able to access any document from the selected
>>>>>>>>>>>>>> folder of Sharepoint Site.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Please guide me accordingly.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 6:46 PM, Nikita Ahuja <
>>>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks Karl,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It is working fine now and Also I am able to get the access
>>>>>>>>>>>>>>> tokens in the elastic search index.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks a lot for your valuable suggestions.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> With Regards,
>>>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 4:42 PM, Karl Wright <
>>>>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The __nosecurity__ token is generated by the Elastic Search
>>>>>>>>>>>>>>>> Connector. It is produced (correctly) when no acls reach the connector.
>>>>>>>>>>>>>>>> This is why I asked you about the repository connection, because that
>>>>>>>>>>>>>>>> connection is not producing any ACLs.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If you are crawling files in a file system, you must use
>>>>>>>>>>>>>>>> the JCIFS connector if you want ACLs to be generated. The FileSystem
>>>>>>>>>>>>>>>> connector does not generate ACLs.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 3:56 AM, Nikita Ahuja <
>>>>>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Karl,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I have tried for ElasticSearch version 1.3.1 and 5.6.1 but
>>>>>>>>>>>>>>>>> still getting this " __nosecurity__" token and for
>>>>>>>>>>>>>>>>> Repository connection File Sytem is used and for Transformation "Tika
>>>>>>>>>>>>>>>>> Content Extractor" and Metadata adjuster Connector is
>>>>>>>>>>>>>>>>> used.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Is there any other thing which I need to perform??
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 2:13 PM, Karl Wright <
>>>>>>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Nikita,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> You are getting the __nosecurity__ token value
>>>>>>>>>>>>>>>>>> transmitted and stored, which means your Elastic Search setup is probably
>>>>>>>>>>>>>>>>>> reasonable.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Can you give us details about your pipeline? What
>>>>>>>>>>>>>>>>>> repository connector is this?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 1:49 AM, Nikita Ahuja <
>>>>>>>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I am using ManifoldCF 2.8.1 and trying to access the
>>>>>>>>>>>>>>>>>>> files from FileShare configured with the Active Directory and get the data
>>>>>>>>>>>>>>>>>>> in elastic search index.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> For this steps followed are:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> - An Authority Group is created under which
>>>>>>>>>>>>>>>>>>> Authority Connection and Repository connection are created.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [image: Inline image 3]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> - And the elastic search is created as an Output
>>>>>>>>>>>>>>>>>>> Connector and a job is created for the connection in which a transformation
>>>>>>>>>>>>>>>>>>> and metadata connector is also linked in the
>>>>>>>>>>>>>>>>>>> - Which is giving all the output except the access
>>>>>>>>>>>>>>>>>>> token values.
>>>>>>>>>>>>>>>>>>> - [image: Inline image 1]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Please guide to take appropriate steps for it so that I
>>>>>>>>>>>>>>>>>>> can get the access tokens in the elastic search index. Hoping for the
>>>>>>>>>>>>>>>>>>> solution asap.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
Posted by Karl Wright <da...@gmail.com>.
No, everything looks properly configured.
First, please verify that the document for which you are seeing
"__nosecurity__" tokens being indexed is NOT a public document in
Sharepoint. If it is public, then the security tokens being indexed are
indeed correct.
Once you have verified that, then the next step is to turn on wire
debugging. This will log all SOAP traffic back and forth between
ManifoldCF and the server. We really need the logs for only one document
being indexed -- the goal is to see whether the response we're getting back
from the MCPermissions call actually returns any ACLs. It seems like it is
silently failing to do so -- very bad, because that means that security is
not being enforced. Turning on wire debugging involves making changes to
the logging.xml file; you will need to add appropriate loggers for
HttpComponents/HttpClient. The instructions are here:
https://cwiki.apache.org/confluence/display/CONNECTORS/Debugging+Connections
Once you have redacted whatever sensitive information you feel necessary,
please post a link to a google doc that includes the chunk of log
corresponding to one document that is not public.
Thanks!
Karl
On Mon, Feb 5, 2018 at 9:09 AM, Nikita Ahuja <ni...@smartshore.nl> wrote:
> Hi Karl,
>
>
> This is Sharepoint 2013 version being used. and it uses the Claim Based
> Authentication technique only.
> And the repository connection created is for Sharepoint 2013 version only
> [image: Inline image 1]
>
>
>
> [image: Inline image 2]
>
>
> [image: Inline image 3]
>
> Is there anything which is being missed to add in the connector??
>
>
>
> Thanks and Regards,
> Nikita
>
>
> On Mon, Feb 5, 2018 at 7:33 PM, Karl Wright <da...@gmail.com> wrote:
>
>> The "__nosecurity__" values, for Elastic Search as an output, come from
>> ManifoldCF. So you are definitely getting the access tokens from
>> ManifoldCF to ElasticSearch.
>>
>> The configuration for the Security tab also looks correct.
>>
>> So at the moment I cannot explain why you are not getting security tokens
>> out of SharePoint. About the only possibility left is whether the wrong
>> version of Sharepoint is specified. Can you describe:
>> - What version of Sharepoint this is;
>> - Whether it is configured to use Claims Based authorization
>>
>> And please then take a screen shot when you view the repository
>> connection. Thanks!
>>
>> Karl
>>
>>
>> On Mon, Feb 5, 2018 at 3:21 AM, Nikita Ahuja <ni...@smartshore.nl>
>> wrote:
>>
>>> Hi Karl,
>>>
>>> I have done the configuration of the job as per the *End User
>>> Documentation*. Still, I am getting "no security" value in the access
>>> tokens in the elastic search index (output of my job).
>>>
>>> [image: Inline image 2]
>>>
>>>
>>> This is the Output connector for Elastic Search. Is there anything needs
>>> to added here?
>>>
>>> [image: Inline image 1]
>>>
>>>
>>>
>>> Also, the security is also enabled.
>>>
>>> [image: Inline image 3]
>>>
>>>
>>> I am not able to understand where is the actual problem?
>>>
>>>
>>> Thanks and Regards,
>>> Nikita
>>>
>>>
>>>
>>>
>>> On Fri, Feb 2, 2018 at 7:53 PM, Karl Wright <da...@gmail.com> wrote:
>>>
>>>> Hi Nikita,
>>>>
>>>> If you are seeing *no* tokens indexed then I would check your output
>>>> connection and your index configuration. Custom fields need to be added to
>>>> Solr for these tokens to be indexed, if you are using Solr. There are
>>>> instructions in the online documentation on what to add and how to
>>>> configure them.
>>>>
>>>> The other possibility is that you have disabled security for you
>>>> Sharepoint job in the "Security" tab.
>>>>
>>>> Karl
>>>>
>>>>
>>>> On Fri, Feb 2, 2018 at 8:48 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>> wrote:
>>>>
>>>>> Hi Karl,
>>>>>
>>>>>
>>>>> Yes, you are right I am able to fetch the files but the access
>>>>> tokens indexed with the files are not included what is required. And I am
>>>>> using the ManifoldCF security model not any custom. Everything is fine but
>>>>> only problem is with the token.
>>>>> And am not sure what is the reason behind that: is it because the
>>>>> Sharepoint site which is used is not with *HTTPS* protocol or any
>>>>> other reason?
>>>>>
>>>>>
>>>>> Thanks and Regards,
>>>>> Nikita
>>>>>
>>>>> On Fri, Feb 2, 2018 at 7:05 PM, Karl Wright <da...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I am not sure I understand -- are you saying that you can fetch the
>>>>>> files but the access tokens indexed with the files do not include what you
>>>>>> require? And that the reason you require it done a certain way is because
>>>>>> you intend to apply security in a manner of your own design?
>>>>>>
>>>>>> The access tokens that are indexed with files in SharePoint are what
>>>>>> are needed to enforce security using the ManifoldCF Sharepoint authorities,
>>>>>> no more and no less. The authorities map an individual user to the access
>>>>>> tokens needed for that user, which include tokens for the groups as well as
>>>>>> for the user name. The tokens indexed with the files do NOT include tokens
>>>>>> generated by mapping groups to individual users -- the only tokens indexed
>>>>>> for a file are for the groups and for any individual users specified
>>>>>> explicitly.
>>>>>>
>>>>>> I strongly suggest you attempt to apply security fully using the
>>>>>> ManifoldCF model rather than trying to do something custom.
>>>>>>
>>>>>> Karl
>>>>>>
>>>>>>
>>>>>> On Fri, Feb 2, 2018 at 8:04 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Karl,
>>>>>>>
>>>>>>> Thanks a lot for your help.It is working fine now.. I am able to
>>>>>>> fetch all the documents in the path but there is a requirement to get the
>>>>>>> access tokens for the files. But,it shows the authority group name not any
>>>>>>> details related to the users who all have access for the document accessed.
>>>>>>>
>>>>>>> Please guide for that.
>>>>>>>
>>>>>>>
>>>>>>> [image: Inline image 1]
>>>>>>>
>>>>>>>
>>>>>>> Thanks and Regards,
>>>>>>> Nikita
>>>>>>>
>>>>>>> On Wed, Jan 31, 2018 at 5:55 PM, Karl Wright <da...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Clearly you are unable to fetch document ACLs from SharePoint.
>>>>>>>> This has nothing to do with whether your Active Directory is properly
>>>>>>>> configured, and has more to do with the MCPermissions plugin.
>>>>>>>>
>>>>>>>> When you view the SharePoint repository connection, does it say
>>>>>>>> "Connection working"? If not, you need the figure out why.
>>>>>>>>
>>>>>>>> Here are some possibilities:
>>>>>>>> (1) The plugin was properly installed but with the wrong
>>>>>>>> permissions. You must be logged in as an Admin to install it properly. If
>>>>>>>> that's not done it will mostly work but not be able to retrieve the
>>>>>>>> permissions.
>>>>>>>> (2) You may have selected the wrong version of Sharepoint in the
>>>>>>>> repository connection. Make sure you select the right one or, once again,
>>>>>>>> you won't be able to retrieve permissions.
>>>>>>>>
>>>>>>>> The documentation says this: "Select your SharePoint server
>>>>>>>> version from the pulldown. If you do not select the correct server version,
>>>>>>>> your documents may either be indexed with insufficient security protection,
>>>>>>>> or you may not be able to index any documents. Check with your SharePoint
>>>>>>>> system administrator if you are not sure what to select."
>>>>>>>>
>>>>>>>> Please let me know whether this helps. Thanks!
>>>>>>>> Karl
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Jan 31, 2018 at 6:51 AM, Nikita Ahuja <nikita@smartshore.nl
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Hi Karl,
>>>>>>>>>
>>>>>>>>> I have checked the log Files but there is issue with the ACLs but
>>>>>>>>> there is Active Directory already connected with Sharepoint site. And it is
>>>>>>>>> not possible to pass the Access tokens in the connector. Please provide the
>>>>>>>>> suggestion for the problem.
>>>>>>>>>
>>>>>>>>> [image: Inline image 1]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks and Regards,
>>>>>>>>> Nikita
>>>>>>>>>
>>>>>>>>> On Mon, Jan 29, 2018 at 6:55 PM, Karl Wright <da...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I don't see any path rules there. That will clearly not work.
>>>>>>>>>>
>>>>>>>>>> If you want confirmation of this, you can turn on connector
>>>>>>>>>> debugging (set properties.xml property org.apache.manifoldcf.connectors
>>>>>>>>>> to "DEBUG"), look at the log output. Every time the SharePoint connector
>>>>>>>>>> makes a decision to not include something, it logs it.
>>>>>>>>>>
>>>>>>>>>> Karl
>>>>>>>>>>
>>>>>>>>>> On Mon, Jan 29, 2018 at 8:07 AM, Nikita Ahuja <
>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I have checked the documentation and the path configuration as
>>>>>>>>>>> well. But still not able to get any output.
>>>>>>>>>>>
>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> and the metadata path given is:
>>>>>>>>>>> [image: Inline image 2]
>>>>>>>>>>>
>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>> Nikita
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jan 29, 2018 at 4:45 PM, Karl Wright <daddywri@gmail.com
>>>>>>>>>>> > wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I think your path configuration is incorrect. See the end-user
>>>>>>>>>>>> documentation. It's a little tricky because you must specify a matching
>>>>>>>>>>>> rule for both files and paths.
>>>>>>>>>>>>
>>>>>>>>>>>> Karl
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jan 29, 2018 at 4:42 AM, Nikita Ahuja <
>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Hi Karl,
>>>>>>>>>>>>>
>>>>>>>>>>>>> There is another query related to Sharepoint 2013 Connector. I
>>>>>>>>>>>>> am trying to crawl the files of Sharepoint On Premise site. But I am not
>>>>>>>>>>>>> able to get anything in the output either in ElasticSearch or in FileShare.
>>>>>>>>>>>>> I have followed given steps:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Created an Authority group for the Active Directory and
>>>>>>>>>>>>> same is used in Sharepoint Repository Connection.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [image: Inline image 2]
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Installed the Sharepoint 2013 MCPermission Plugin and is
>>>>>>>>>>>>> accessible on the site.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [image: Inline image 3]
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Still not able to access any document from the selected
>>>>>>>>>>>>> folder of Sharepoint Site.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please guide me accordingly.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 6:46 PM, Nikita Ahuja <
>>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks Karl,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It is working fine now and Also I am able to get the access
>>>>>>>>>>>>>> tokens in the elastic search index.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks a lot for your valuable suggestions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> With Regards,
>>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 4:42 PM, Karl Wright <
>>>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The __nosecurity__ token is generated by the Elastic Search
>>>>>>>>>>>>>>> Connector. It is produced (correctly) when no acls reach the connector.
>>>>>>>>>>>>>>> This is why I asked you about the repository connection, because that
>>>>>>>>>>>>>>> connection is not producing any ACLs.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If you are crawling files in a file system, you must use the
>>>>>>>>>>>>>>> JCIFS connector if you want ACLs to be generated. The FileSystem connector
>>>>>>>>>>>>>>> does not generate ACLs.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 3:56 AM, Nikita Ahuja <
>>>>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Karl,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have tried for ElasticSearch version 1.3.1 and 5.6.1 but
>>>>>>>>>>>>>>>> still getting this " __nosecurity__" token and for
>>>>>>>>>>>>>>>> Repository connection File Sytem is used and for Transformation "Tika
>>>>>>>>>>>>>>>> Content Extractor" and Metadata adjuster Connector is used.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Is there any other thing which I need to perform??
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 2:13 PM, Karl Wright <
>>>>>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Nikita,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> You are getting the __nosecurity__ token value transmitted
>>>>>>>>>>>>>>>>> and stored, which means your Elastic Search setup is probably reasonable.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Can you give us details about your pipeline? What
>>>>>>>>>>>>>>>>> repository connector is this?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 1:49 AM, Nikita Ahuja <
>>>>>>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I am using ManifoldCF 2.8.1 and trying to access the
>>>>>>>>>>>>>>>>>> files from FileShare configured with the Active Directory and get the data
>>>>>>>>>>>>>>>>>> in elastic search index.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> For this steps followed are:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> - An Authority Group is created under which Authority
>>>>>>>>>>>>>>>>>> Connection and Repository connection are created.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [image: Inline image 3]
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> - And the elastic search is created as an Output
>>>>>>>>>>>>>>>>>> Connector and a job is created for the connection in which a transformation
>>>>>>>>>>>>>>>>>> and metadata connector is also linked in the
>>>>>>>>>>>>>>>>>> - Which is giving all the output except the access
>>>>>>>>>>>>>>>>>> token values.
>>>>>>>>>>>>>>>>>> - [image: Inline image 1]
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Please guide to take appropriate steps for it so that I
>>>>>>>>>>>>>>>>>> can get the access tokens in the elastic search index. Hoping for the
>>>>>>>>>>>>>>>>>> solution asap.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
Posted by Nikita Ahuja <ni...@smartshore.nl>.
Hi Karl,
This is Sharepoint 2013 version being used. and it uses the Claim Based
Authentication technique only.
And the repository connection created is for Sharepoint 2013 version only
[image: Inline image 1]
[image: Inline image 2]
[image: Inline image 3]
Is there anything which is being missed to add in the connector??
Thanks and Regards,
Nikita
On Mon, Feb 5, 2018 at 7:33 PM, Karl Wright <da...@gmail.com> wrote:
> The "__nosecurity__" values, for Elastic Search as an output, come from
> ManifoldCF. So you are definitely getting the access tokens from
> ManifoldCF to ElasticSearch.
>
> The configuration for the Security tab also looks correct.
>
> So at the moment I cannot explain why you are not getting security tokens
> out of SharePoint. About the only possibility left is whether the wrong
> version of Sharepoint is specified. Can you describe:
> - What version of Sharepoint this is;
> - Whether it is configured to use Claims Based authorization
>
> And please then take a screen shot when you view the repository
> connection. Thanks!
>
> Karl
>
>
> On Mon, Feb 5, 2018 at 3:21 AM, Nikita Ahuja <ni...@smartshore.nl> wrote:
>
>> Hi Karl,
>>
>> I have done the configuration of the job as per the *End User
>> Documentation*. Still, I am getting "no security" value in the access
>> tokens in the elastic search index (output of my job).
>>
>> [image: Inline image 2]
>>
>>
>> This is the Output connector for Elastic Search. Is there anything needs
>> to added here?
>>
>> [image: Inline image 1]
>>
>>
>>
>> Also, the security is also enabled.
>>
>> [image: Inline image 3]
>>
>>
>> I am not able to understand where is the actual problem?
>>
>>
>> Thanks and Regards,
>> Nikita
>>
>>
>>
>>
>> On Fri, Feb 2, 2018 at 7:53 PM, Karl Wright <da...@gmail.com> wrote:
>>
>>> Hi Nikita,
>>>
>>> If you are seeing *no* tokens indexed then I would check your output
>>> connection and your index configuration. Custom fields need to be added to
>>> Solr for these tokens to be indexed, if you are using Solr. There are
>>> instructions in the online documentation on what to add and how to
>>> configure them.
>>>
>>> The other possibility is that you have disabled security for you
>>> Sharepoint job in the "Security" tab.
>>>
>>> Karl
>>>
>>>
>>> On Fri, Feb 2, 2018 at 8:48 AM, Nikita Ahuja <ni...@smartshore.nl>
>>> wrote:
>>>
>>>> Hi Karl,
>>>>
>>>>
>>>> Yes, you are right I am able to fetch the files but the access tokens
>>>> indexed with the files are not included what is required. And I am using
>>>> the ManifoldCF security model not any custom. Everything is fine but only
>>>> problem is with the token.
>>>> And am not sure what is the reason behind that: is it because the
>>>> Sharepoint site which is used is not with *HTTPS* protocol or any
>>>> other reason?
>>>>
>>>>
>>>> Thanks and Regards,
>>>> Nikita
>>>>
>>>> On Fri, Feb 2, 2018 at 7:05 PM, Karl Wright <da...@gmail.com> wrote:
>>>>
>>>>> I am not sure I understand -- are you saying that you can fetch the
>>>>> files but the access tokens indexed with the files do not include what you
>>>>> require? And that the reason you require it done a certain way is because
>>>>> you intend to apply security in a manner of your own design?
>>>>>
>>>>> The access tokens that are indexed with files in SharePoint are what
>>>>> are needed to enforce security using the ManifoldCF Sharepoint authorities,
>>>>> no more and no less. The authorities map an individual user to the access
>>>>> tokens needed for that user, which include tokens for the groups as well as
>>>>> for the user name. The tokens indexed with the files do NOT include tokens
>>>>> generated by mapping groups to individual users -- the only tokens indexed
>>>>> for a file are for the groups and for any individual users specified
>>>>> explicitly.
>>>>>
>>>>> I strongly suggest you attempt to apply security fully using the
>>>>> ManifoldCF model rather than trying to do something custom.
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>> On Fri, Feb 2, 2018 at 8:04 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>>> wrote:
>>>>>
>>>>>> Hi Karl,
>>>>>>
>>>>>> Thanks a lot for your help.It is working fine now.. I am able to
>>>>>> fetch all the documents in the path but there is a requirement to get the
>>>>>> access tokens for the files. But,it shows the authority group name not any
>>>>>> details related to the users who all have access for the document accessed.
>>>>>>
>>>>>> Please guide for that.
>>>>>>
>>>>>>
>>>>>> [image: Inline image 1]
>>>>>>
>>>>>>
>>>>>> Thanks and Regards,
>>>>>> Nikita
>>>>>>
>>>>>> On Wed, Jan 31, 2018 at 5:55 PM, Karl Wright <da...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Clearly you are unable to fetch document ACLs from SharePoint. This
>>>>>>> has nothing to do with whether your Active Directory is properly
>>>>>>> configured, and has more to do with the MCPermissions plugin.
>>>>>>>
>>>>>>> When you view the SharePoint repository connection, does it say
>>>>>>> "Connection working"? If not, you need the figure out why.
>>>>>>>
>>>>>>> Here are some possibilities:
>>>>>>> (1) The plugin was properly installed but with the wrong
>>>>>>> permissions. You must be logged in as an Admin to install it properly. If
>>>>>>> that's not done it will mostly work but not be able to retrieve the
>>>>>>> permissions.
>>>>>>> (2) You may have selected the wrong version of Sharepoint in the
>>>>>>> repository connection. Make sure you select the right one or, once again,
>>>>>>> you won't be able to retrieve permissions.
>>>>>>>
>>>>>>> The documentation says this: "Select your SharePoint server version
>>>>>>> from the pulldown. If you do not select the correct server version, your
>>>>>>> documents may either be indexed with insufficient security protection, or
>>>>>>> you may not be able to index any documents. Check with your SharePoint
>>>>>>> system administrator if you are not sure what to select."
>>>>>>>
>>>>>>> Please let me know whether this helps. Thanks!
>>>>>>> Karl
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jan 31, 2018 at 6:51 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Karl,
>>>>>>>>
>>>>>>>> I have checked the log Files but there is issue with the ACLs but
>>>>>>>> there is Active Directory already connected with Sharepoint site. And it is
>>>>>>>> not possible to pass the Access tokens in the connector. Please provide the
>>>>>>>> suggestion for the problem.
>>>>>>>>
>>>>>>>> [image: Inline image 1]
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks and Regards,
>>>>>>>> Nikita
>>>>>>>>
>>>>>>>> On Mon, Jan 29, 2018 at 6:55 PM, Karl Wright <da...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I don't see any path rules there. That will clearly not work.
>>>>>>>>>
>>>>>>>>> If you want confirmation of this, you can turn on connector
>>>>>>>>> debugging (set properties.xml property org.apache.manifoldcf.connectors
>>>>>>>>> to "DEBUG"), look at the log output. Every time the SharePoint connector
>>>>>>>>> makes a decision to not include something, it logs it.
>>>>>>>>>
>>>>>>>>> Karl
>>>>>>>>>
>>>>>>>>> On Mon, Jan 29, 2018 at 8:07 AM, Nikita Ahuja <
>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I have checked the documentation and the path configuration as
>>>>>>>>>> well. But still not able to get any output.
>>>>>>>>>>
>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> and the metadata path given is:
>>>>>>>>>> [image: Inline image 2]
>>>>>>>>>>
>>>>>>>>>> Thanks and Regards,
>>>>>>>>>> Nikita
>>>>>>>>>>
>>>>>>>>>> On Mon, Jan 29, 2018 at 4:45 PM, Karl Wright <da...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I think your path configuration is incorrect. See the end-user
>>>>>>>>>>> documentation. It's a little tricky because you must specify a matching
>>>>>>>>>>> rule for both files and paths.
>>>>>>>>>>>
>>>>>>>>>>> Karl
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jan 29, 2018 at 4:42 AM, Nikita Ahuja <
>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> - Hi Karl,
>>>>>>>>>>>>
>>>>>>>>>>>> There is another query related to Sharepoint 2013 Connector. I
>>>>>>>>>>>> am trying to crawl the files of Sharepoint On Premise site. But I am not
>>>>>>>>>>>> able to get anything in the output either in ElasticSearch or in FileShare.
>>>>>>>>>>>> I have followed given steps:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> - Created an Authority group for the Active Directory and
>>>>>>>>>>>> same is used in Sharepoint Repository Connection.
>>>>>>>>>>>>
>>>>>>>>>>>> [image: Inline image 2]
>>>>>>>>>>>>
>>>>>>>>>>>> - Installed the Sharepoint 2013 MCPermission Plugin and is
>>>>>>>>>>>> accessible on the site.
>>>>>>>>>>>>
>>>>>>>>>>>> [image: Inline image 3]
>>>>>>>>>>>>
>>>>>>>>>>>> - Still not able to access any document from the selected
>>>>>>>>>>>> folder of Sharepoint Site.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Please guide me accordingly.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>> Nikita
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Jan 25, 2018 at 6:46 PM, Nikita Ahuja <
>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks Karl,
>>>>>>>>>>>>>
>>>>>>>>>>>>> It is working fine now and Also I am able to get the access
>>>>>>>>>>>>> tokens in the elastic search index.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks a lot for your valuable suggestions.
>>>>>>>>>>>>>
>>>>>>>>>>>>> With Regards,
>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 4:42 PM, Karl Wright <
>>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> The __nosecurity__ token is generated by the Elastic Search
>>>>>>>>>>>>>> Connector. It is produced (correctly) when no acls reach the connector.
>>>>>>>>>>>>>> This is why I asked you about the repository connection, because that
>>>>>>>>>>>>>> connection is not producing any ACLs.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If you are crawling files in a file system, you must use the
>>>>>>>>>>>>>> JCIFS connector if you want ACLs to be generated. The FileSystem connector
>>>>>>>>>>>>>> does not generate ACLs.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 3:56 AM, Nikita Ahuja <
>>>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Karl,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have tried for ElasticSearch version 1.3.1 and 5.6.1 but
>>>>>>>>>>>>>>> still getting this " __nosecurity__" token and for
>>>>>>>>>>>>>>> Repository connection File Sytem is used and for Transformation "Tika
>>>>>>>>>>>>>>> Content Extractor" and Metadata adjuster Connector is used.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Is there any other thing which I need to perform??
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 2:13 PM, Karl Wright <
>>>>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Nikita,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> You are getting the __nosecurity__ token value transmitted
>>>>>>>>>>>>>>>> and stored, which means your Elastic Search setup is probably reasonable.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can you give us details about your pipeline? What
>>>>>>>>>>>>>>>> repository connector is this?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 1:49 AM, Nikita Ahuja <
>>>>>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I am using ManifoldCF 2.8.1 and trying to access the files
>>>>>>>>>>>>>>>>> from FileShare configured with the Active Directory and get the data in
>>>>>>>>>>>>>>>>> elastic search index.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> For this steps followed are:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - An Authority Group is created under which Authority
>>>>>>>>>>>>>>>>> Connection and Repository connection are created.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [image: Inline image 3]
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - And the elastic search is created as an Output
>>>>>>>>>>>>>>>>> Connector and a job is created for the connection in which a transformation
>>>>>>>>>>>>>>>>> and metadata connector is also linked in the
>>>>>>>>>>>>>>>>> - Which is giving all the output except the access
>>>>>>>>>>>>>>>>> token values.
>>>>>>>>>>>>>>>>> - [image: Inline image 1]
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Please guide to take appropriate steps for it so that I
>>>>>>>>>>>>>>>>> can get the access tokens in the elastic search index. Hoping for the
>>>>>>>>>>>>>>>>> solution asap.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
Posted by Karl Wright <da...@gmail.com>.
The "__nosecurity__" values, for Elastic Search as an output, come from
ManifoldCF. So you are definitely getting the access tokens from
ManifoldCF to ElasticSearch.
The configuration for the Security tab also looks correct.
So at the moment I cannot explain why you are not getting security tokens
out of SharePoint. About the only possibility left is whether the wrong
version of Sharepoint is specified. Can you describe:
- What version of Sharepoint this is;
- Whether it is configured to use Claims Based authorization
And please then take a screen shot when you view the repository
connection. Thanks!
Karl
On Mon, Feb 5, 2018 at 3:21 AM, Nikita Ahuja <ni...@smartshore.nl> wrote:
> Hi Karl,
>
> I have done the configuration of the job as per the *End User
> Documentation*. Still, I am getting "no security" value in the access
> tokens in the elastic search index (output of my job).
>
> [image: Inline image 2]
>
>
> This is the Output connector for Elastic Search. Is there anything needs
> to added here?
>
> [image: Inline image 1]
>
>
>
> Also, the security is also enabled.
>
> [image: Inline image 3]
>
>
> I am not able to understand where is the actual problem?
>
>
> Thanks and Regards,
> Nikita
>
>
>
>
> On Fri, Feb 2, 2018 at 7:53 PM, Karl Wright <da...@gmail.com> wrote:
>
>> Hi Nikita,
>>
>> If you are seeing *no* tokens indexed then I would check your output
>> connection and your index configuration. Custom fields need to be added to
>> Solr for these tokens to be indexed, if you are using Solr. There are
>> instructions in the online documentation on what to add and how to
>> configure them.
>>
>> The other possibility is that you have disabled security for you
>> Sharepoint job in the "Security" tab.
>>
>> Karl
>>
>>
>> On Fri, Feb 2, 2018 at 8:48 AM, Nikita Ahuja <ni...@smartshore.nl>
>> wrote:
>>
>>> Hi Karl,
>>>
>>>
>>> Yes, you are right I am able to fetch the files but the access tokens
>>> indexed with the files are not included what is required. And I am using
>>> the ManifoldCF security model not any custom. Everything is fine but only
>>> problem is with the token.
>>> And am not sure what is the reason behind that: is it because the
>>> Sharepoint site which is used is not with *HTTPS* protocol or any other
>>> reason?
>>>
>>>
>>> Thanks and Regards,
>>> Nikita
>>>
>>> On Fri, Feb 2, 2018 at 7:05 PM, Karl Wright <da...@gmail.com> wrote:
>>>
>>>> I am not sure I understand -- are you saying that you can fetch the
>>>> files but the access tokens indexed with the files do not include what you
>>>> require? And that the reason you require it done a certain way is because
>>>> you intend to apply security in a manner of your own design?
>>>>
>>>> The access tokens that are indexed with files in SharePoint are what
>>>> are needed to enforce security using the ManifoldCF Sharepoint authorities,
>>>> no more and no less. The authorities map an individual user to the access
>>>> tokens needed for that user, which include tokens for the groups as well as
>>>> for the user name. The tokens indexed with the files do NOT include tokens
>>>> generated by mapping groups to individual users -- the only tokens indexed
>>>> for a file are for the groups and for any individual users specified
>>>> explicitly.
>>>>
>>>> I strongly suggest you attempt to apply security fully using the
>>>> ManifoldCF model rather than trying to do something custom.
>>>>
>>>> Karl
>>>>
>>>>
>>>> On Fri, Feb 2, 2018 at 8:04 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>> wrote:
>>>>
>>>>> Hi Karl,
>>>>>
>>>>> Thanks a lot for your help.It is working fine now.. I am able to fetch
>>>>> all the documents in the path but there is a requirement to get the access
>>>>> tokens for the files. But,it shows the authority group name not any details
>>>>> related to the users who all have access for the document accessed.
>>>>>
>>>>> Please guide for that.
>>>>>
>>>>>
>>>>> [image: Inline image 1]
>>>>>
>>>>>
>>>>> Thanks and Regards,
>>>>> Nikita
>>>>>
>>>>> On Wed, Jan 31, 2018 at 5:55 PM, Karl Wright <da...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Clearly you are unable to fetch document ACLs from SharePoint. This
>>>>>> has nothing to do with whether your Active Directory is properly
>>>>>> configured, and has more to do with the MCPermissions plugin.
>>>>>>
>>>>>> When you view the SharePoint repository connection, does it say
>>>>>> "Connection working"? If not, you need the figure out why.
>>>>>>
>>>>>> Here are some possibilities:
>>>>>> (1) The plugin was properly installed but with the wrong
>>>>>> permissions. You must be logged in as an Admin to install it properly. If
>>>>>> that's not done it will mostly work but not be able to retrieve the
>>>>>> permissions.
>>>>>> (2) You may have selected the wrong version of Sharepoint in the
>>>>>> repository connection. Make sure you select the right one or, once again,
>>>>>> you won't be able to retrieve permissions.
>>>>>>
>>>>>> The documentation says this: "Select your SharePoint server version
>>>>>> from the pulldown. If you do not select the correct server version, your
>>>>>> documents may either be indexed with insufficient security protection, or
>>>>>> you may not be able to index any documents. Check with your SharePoint
>>>>>> system administrator if you are not sure what to select."
>>>>>>
>>>>>> Please let me know whether this helps. Thanks!
>>>>>> Karl
>>>>>>
>>>>>>
>>>>>> On Wed, Jan 31, 2018 at 6:51 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Karl,
>>>>>>>
>>>>>>> I have checked the log Files but there is issue with the ACLs but
>>>>>>> there is Active Directory already connected with Sharepoint site. And it is
>>>>>>> not possible to pass the Access tokens in the connector. Please provide the
>>>>>>> suggestion for the problem.
>>>>>>>
>>>>>>> [image: Inline image 1]
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks and Regards,
>>>>>>> Nikita
>>>>>>>
>>>>>>> On Mon, Jan 29, 2018 at 6:55 PM, Karl Wright <da...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I don't see any path rules there. That will clearly not work.
>>>>>>>>
>>>>>>>> If you want confirmation of this, you can turn on connector
>>>>>>>> debugging (set properties.xml property org.apache.manifoldcf.connectors
>>>>>>>> to "DEBUG"), look at the log output. Every time the SharePoint connector
>>>>>>>> makes a decision to not include something, it logs it.
>>>>>>>>
>>>>>>>> Karl
>>>>>>>>
>>>>>>>> On Mon, Jan 29, 2018 at 8:07 AM, Nikita Ahuja <nikita@smartshore.nl
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I have checked the documentation and the path configuration as
>>>>>>>>> well. But still not able to get any output.
>>>>>>>>>
>>>>>>>>> [image: Inline image 1]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> and the metadata path given is:
>>>>>>>>> [image: Inline image 2]
>>>>>>>>>
>>>>>>>>> Thanks and Regards,
>>>>>>>>> Nikita
>>>>>>>>>
>>>>>>>>> On Mon, Jan 29, 2018 at 4:45 PM, Karl Wright <da...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I think your path configuration is incorrect. See the end-user
>>>>>>>>>> documentation. It's a little tricky because you must specify a matching
>>>>>>>>>> rule for both files and paths.
>>>>>>>>>>
>>>>>>>>>> Karl
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Jan 29, 2018 at 4:42 AM, Nikita Ahuja <
>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> - Hi Karl,
>>>>>>>>>>>
>>>>>>>>>>> There is another query related to Sharepoint 2013 Connector. I
>>>>>>>>>>> am trying to crawl the files of Sharepoint On Premise site. But I am not
>>>>>>>>>>> able to get anything in the output either in ElasticSearch or in FileShare.
>>>>>>>>>>> I have followed given steps:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> - Created an Authority group for the Active Directory and
>>>>>>>>>>> same is used in Sharepoint Repository Connection.
>>>>>>>>>>>
>>>>>>>>>>> [image: Inline image 2]
>>>>>>>>>>>
>>>>>>>>>>> - Installed the Sharepoint 2013 MCPermission Plugin and is
>>>>>>>>>>> accessible on the site.
>>>>>>>>>>>
>>>>>>>>>>> [image: Inline image 3]
>>>>>>>>>>>
>>>>>>>>>>> - Still not able to access any document from the selected
>>>>>>>>>>> folder of Sharepoint Site.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Please guide me accordingly.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>> Nikita
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jan 25, 2018 at 6:46 PM, Nikita Ahuja <
>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks Karl,
>>>>>>>>>>>>
>>>>>>>>>>>> It is working fine now and Also I am able to get the access
>>>>>>>>>>>> tokens in the elastic search index.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks a lot for your valuable suggestions.
>>>>>>>>>>>>
>>>>>>>>>>>> With Regards,
>>>>>>>>>>>> Nikita
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Jan 25, 2018 at 4:42 PM, Karl Wright <
>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> The __nosecurity__ token is generated by the Elastic Search
>>>>>>>>>>>>> Connector. It is produced (correctly) when no acls reach the connector.
>>>>>>>>>>>>> This is why I asked you about the repository connection, because that
>>>>>>>>>>>>> connection is not producing any ACLs.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If you are crawling files in a file system, you must use the
>>>>>>>>>>>>> JCIFS connector if you want ACLs to be generated. The FileSystem connector
>>>>>>>>>>>>> does not generate ACLs.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 3:56 AM, Nikita Ahuja <
>>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Karl,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have tried for ElasticSearch version 1.3.1 and 5.6.1 but
>>>>>>>>>>>>>> still getting this " __nosecurity__" token and for
>>>>>>>>>>>>>> Repository connection File Sytem is used and for Transformation "Tika
>>>>>>>>>>>>>> Content Extractor" and Metadata adjuster Connector is used.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is there any other thing which I need to perform??
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 2:13 PM, Karl Wright <
>>>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Nikita,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> You are getting the __nosecurity__ token value transmitted
>>>>>>>>>>>>>>> and stored, which means your Elastic Search setup is probably reasonable.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Can you give us details about your pipeline? What
>>>>>>>>>>>>>>> repository connector is this?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 1:49 AM, Nikita Ahuja <
>>>>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I am using ManifoldCF 2.8.1 and trying to access the files
>>>>>>>>>>>>>>>> from FileShare configured with the Active Directory and get the data in
>>>>>>>>>>>>>>>> elastic search index.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> For this steps followed are:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - An Authority Group is created under which Authority
>>>>>>>>>>>>>>>> Connection and Repository connection are created.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [image: Inline image 3]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - And the elastic search is created as an Output
>>>>>>>>>>>>>>>> Connector and a job is created for the connection in which a transformation
>>>>>>>>>>>>>>>> and metadata connector is also linked in the
>>>>>>>>>>>>>>>> - Which is giving all the output except the access
>>>>>>>>>>>>>>>> token values.
>>>>>>>>>>>>>>>> - [image: Inline image 1]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Please guide to take appropriate steps for it so that I can
>>>>>>>>>>>>>>>> get the access tokens in the elastic search index. Hoping for the solution
>>>>>>>>>>>>>>>> asap.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
Posted by Nikita Ahuja <ni...@smartshore.nl>.
Hi Karl,
I have done the configuration of the job as per the *End User Documentation*.
Still, I am getting "no security" value in the access tokens in the elastic
search index (output of my job).
[image: Inline image 2]
This is the Output connector for Elastic Search. Is there anything needs to
added here?
[image: Inline image 1]
Also, the security is also enabled.
[image: Inline image 3]
I am not able to understand where is the actual problem?
Thanks and Regards,
Nikita
On Fri, Feb 2, 2018 at 7:53 PM, Karl Wright <da...@gmail.com> wrote:
> Hi Nikita,
>
> If you are seeing *no* tokens indexed then I would check your output
> connection and your index configuration. Custom fields need to be added to
> Solr for these tokens to be indexed, if you are using Solr. There are
> instructions in the online documentation on what to add and how to
> configure them.
>
> The other possibility is that you have disabled security for you
> Sharepoint job in the "Security" tab.
>
> Karl
>
>
> On Fri, Feb 2, 2018 at 8:48 AM, Nikita Ahuja <ni...@smartshore.nl> wrote:
>
>> Hi Karl,
>>
>>
>> Yes, you are right I am able to fetch the files but the access tokens
>> indexed with the files are not included what is required. And I am using
>> the ManifoldCF security model not any custom. Everything is fine but only
>> problem is with the token.
>> And am not sure what is the reason behind that: is it because the
>> Sharepoint site which is used is not with *HTTPS* protocol or any other
>> reason?
>>
>>
>> Thanks and Regards,
>> Nikita
>>
>> On Fri, Feb 2, 2018 at 7:05 PM, Karl Wright <da...@gmail.com> wrote:
>>
>>> I am not sure I understand -- are you saying that you can fetch the
>>> files but the access tokens indexed with the files do not include what you
>>> require? And that the reason you require it done a certain way is because
>>> you intend to apply security in a manner of your own design?
>>>
>>> The access tokens that are indexed with files in SharePoint are what are
>>> needed to enforce security using the ManifoldCF Sharepoint authorities, no
>>> more and no less. The authorities map an individual user to the access
>>> tokens needed for that user, which include tokens for the groups as well as
>>> for the user name. The tokens indexed with the files do NOT include tokens
>>> generated by mapping groups to individual users -- the only tokens indexed
>>> for a file are for the groups and for any individual users specified
>>> explicitly.
>>>
>>> I strongly suggest you attempt to apply security fully using the
>>> ManifoldCF model rather than trying to do something custom.
>>>
>>> Karl
>>>
>>>
>>> On Fri, Feb 2, 2018 at 8:04 AM, Nikita Ahuja <ni...@smartshore.nl>
>>> wrote:
>>>
>>>> Hi Karl,
>>>>
>>>> Thanks a lot for your help.It is working fine now.. I am able to fetch
>>>> all the documents in the path but there is a requirement to get the access
>>>> tokens for the files. But,it shows the authority group name not any details
>>>> related to the users who all have access for the document accessed.
>>>>
>>>> Please guide for that.
>>>>
>>>>
>>>> [image: Inline image 1]
>>>>
>>>>
>>>> Thanks and Regards,
>>>> Nikita
>>>>
>>>> On Wed, Jan 31, 2018 at 5:55 PM, Karl Wright <da...@gmail.com>
>>>> wrote:
>>>>
>>>>> Clearly you are unable to fetch document ACLs from SharePoint. This
>>>>> has nothing to do with whether your Active Directory is properly
>>>>> configured, and has more to do with the MCPermissions plugin.
>>>>>
>>>>> When you view the SharePoint repository connection, does it say
>>>>> "Connection working"? If not, you need the figure out why.
>>>>>
>>>>> Here are some possibilities:
>>>>> (1) The plugin was properly installed but with the wrong permissions.
>>>>> You must be logged in as an Admin to install it properly. If that's not
>>>>> done it will mostly work but not be able to retrieve the permissions.
>>>>> (2) You may have selected the wrong version of Sharepoint in the
>>>>> repository connection. Make sure you select the right one or, once again,
>>>>> you won't be able to retrieve permissions.
>>>>>
>>>>> The documentation says this: "Select your SharePoint server version
>>>>> from the pulldown. If you do not select the correct server version, your
>>>>> documents may either be indexed with insufficient security protection, or
>>>>> you may not be able to index any documents. Check with your SharePoint
>>>>> system administrator if you are not sure what to select."
>>>>>
>>>>> Please let me know whether this helps. Thanks!
>>>>> Karl
>>>>>
>>>>>
>>>>> On Wed, Jan 31, 2018 at 6:51 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>>> wrote:
>>>>>
>>>>>> Hi Karl,
>>>>>>
>>>>>> I have checked the log Files but there is issue with the ACLs but
>>>>>> there is Active Directory already connected with Sharepoint site. And it is
>>>>>> not possible to pass the Access tokens in the connector. Please provide the
>>>>>> suggestion for the problem.
>>>>>>
>>>>>> [image: Inline image 1]
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks and Regards,
>>>>>> Nikita
>>>>>>
>>>>>> On Mon, Jan 29, 2018 at 6:55 PM, Karl Wright <da...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I don't see any path rules there. That will clearly not work.
>>>>>>>
>>>>>>> If you want confirmation of this, you can turn on connector
>>>>>>> debugging (set properties.xml property org.apache.manifoldcf.connectors
>>>>>>> to "DEBUG"), look at the log output. Every time the SharePoint connector
>>>>>>> makes a decision to not include something, it logs it.
>>>>>>>
>>>>>>> Karl
>>>>>>>
>>>>>>> On Mon, Jan 29, 2018 at 8:07 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I have checked the documentation and the path configuration as
>>>>>>>> well. But still not able to get any output.
>>>>>>>>
>>>>>>>> [image: Inline image 1]
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> and the metadata path given is:
>>>>>>>> [image: Inline image 2]
>>>>>>>>
>>>>>>>> Thanks and Regards,
>>>>>>>> Nikita
>>>>>>>>
>>>>>>>> On Mon, Jan 29, 2018 at 4:45 PM, Karl Wright <da...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I think your path configuration is incorrect. See the end-user
>>>>>>>>> documentation. It's a little tricky because you must specify a matching
>>>>>>>>> rule for both files and paths.
>>>>>>>>>
>>>>>>>>> Karl
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jan 29, 2018 at 4:42 AM, Nikita Ahuja <
>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> - Hi Karl,
>>>>>>>>>>
>>>>>>>>>> There is another query related to Sharepoint 2013 Connector. I am
>>>>>>>>>> trying to crawl the files of Sharepoint On Premise site. But I am not able
>>>>>>>>>> to get anything in the output either in ElasticSearch or in FileShare.
>>>>>>>>>> I have followed given steps:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> - Created an Authority group for the Active Directory and
>>>>>>>>>> same is used in Sharepoint Repository Connection.
>>>>>>>>>>
>>>>>>>>>> [image: Inline image 2]
>>>>>>>>>>
>>>>>>>>>> - Installed the Sharepoint 2013 MCPermission Plugin and is
>>>>>>>>>> accessible on the site.
>>>>>>>>>>
>>>>>>>>>> [image: Inline image 3]
>>>>>>>>>>
>>>>>>>>>> - Still not able to access any document from the selected
>>>>>>>>>> folder of Sharepoint Site.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Please guide me accordingly.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks and Regards,
>>>>>>>>>> Nikita
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Jan 25, 2018 at 6:46 PM, Nikita Ahuja <
>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks Karl,
>>>>>>>>>>>
>>>>>>>>>>> It is working fine now and Also I am able to get the access
>>>>>>>>>>> tokens in the elastic search index.
>>>>>>>>>>>
>>>>>>>>>>> Thanks a lot for your valuable suggestions.
>>>>>>>>>>>
>>>>>>>>>>> With Regards,
>>>>>>>>>>> Nikita
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jan 25, 2018 at 4:42 PM, Karl Wright <daddywri@gmail.com
>>>>>>>>>>> > wrote:
>>>>>>>>>>>
>>>>>>>>>>>> The __nosecurity__ token is generated by the Elastic Search
>>>>>>>>>>>> Connector. It is produced (correctly) when no acls reach the connector.
>>>>>>>>>>>> This is why I asked you about the repository connection, because that
>>>>>>>>>>>> connection is not producing any ACLs.
>>>>>>>>>>>>
>>>>>>>>>>>> If you are crawling files in a file system, you must use the
>>>>>>>>>>>> JCIFS connector if you want ACLs to be generated. The FileSystem connector
>>>>>>>>>>>> does not generate ACLs.
>>>>>>>>>>>>
>>>>>>>>>>>> Karl
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Jan 25, 2018 at 3:56 AM, Nikita Ahuja <
>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Karl,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have tried for ElasticSearch version 1.3.1 and 5.6.1 but
>>>>>>>>>>>>> still getting this " __nosecurity__" token and for Repository
>>>>>>>>>>>>> connection File Sytem is used and for Transformation "Tika Content
>>>>>>>>>>>>> Extractor" and Metadata adjuster Connector is used.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is there any other thing which I need to perform??
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 2:13 PM, Karl Wright <
>>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Nikita,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You are getting the __nosecurity__ token value transmitted
>>>>>>>>>>>>>> and stored, which means your Elastic Search setup is probably reasonable.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can you give us details about your pipeline? What repository
>>>>>>>>>>>>>> connector is this?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 1:49 AM, Nikita Ahuja <
>>>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I am using ManifoldCF 2.8.1 and trying to access the files
>>>>>>>>>>>>>>> from FileShare configured with the Active Directory and get the data in
>>>>>>>>>>>>>>> elastic search index.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For this steps followed are:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - An Authority Group is created under which Authority
>>>>>>>>>>>>>>> Connection and Repository connection are created.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [image: Inline image 3]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - And the elastic search is created as an Output
>>>>>>>>>>>>>>> Connector and a job is created for the connection in which a transformation
>>>>>>>>>>>>>>> and metadata connector is also linked in the
>>>>>>>>>>>>>>> - Which is giving all the output except the access token
>>>>>>>>>>>>>>> values.
>>>>>>>>>>>>>>> - [image: Inline image 1]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Please guide to take appropriate steps for it so that I can
>>>>>>>>>>>>>>> get the access tokens in the elastic search index. Hoping for the solution
>>>>>>>>>>>>>>> asap.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
Posted by Karl Wright <da...@gmail.com>.
Hi Nikita,
If you are seeing *no* tokens indexed then I would check your output
connection and your index configuration. Custom fields need to be added to
Solr for these tokens to be indexed, if you are using Solr. There are
instructions in the online documentation on what to add and how to
configure them.
The other possibility is that you have disabled security for you Sharepoint
job in the "Security" tab.
Karl
On Fri, Feb 2, 2018 at 8:48 AM, Nikita Ahuja <ni...@smartshore.nl> wrote:
> Hi Karl,
>
>
> Yes, you are right I am able to fetch the files but the access tokens
> indexed with the files are not included what is required. And I am using
> the ManifoldCF security model not any custom. Everything is fine but only
> problem is with the token.
> And am not sure what is the reason behind that: is it because the
> Sharepoint site which is used is not with *HTTPS* protocol or any other
> reason?
>
>
> Thanks and Regards,
> Nikita
>
> On Fri, Feb 2, 2018 at 7:05 PM, Karl Wright <da...@gmail.com> wrote:
>
>> I am not sure I understand -- are you saying that you can fetch the files
>> but the access tokens indexed with the files do not include what you
>> require? And that the reason you require it done a certain way is because
>> you intend to apply security in a manner of your own design?
>>
>> The access tokens that are indexed with files in SharePoint are what are
>> needed to enforce security using the ManifoldCF Sharepoint authorities, no
>> more and no less. The authorities map an individual user to the access
>> tokens needed for that user, which include tokens for the groups as well as
>> for the user name. The tokens indexed with the files do NOT include tokens
>> generated by mapping groups to individual users -- the only tokens indexed
>> for a file are for the groups and for any individual users specified
>> explicitly.
>>
>> I strongly suggest you attempt to apply security fully using the
>> ManifoldCF model rather than trying to do something custom.
>>
>> Karl
>>
>>
>> On Fri, Feb 2, 2018 at 8:04 AM, Nikita Ahuja <ni...@smartshore.nl>
>> wrote:
>>
>>> Hi Karl,
>>>
>>> Thanks a lot for your help.It is working fine now.. I am able to fetch
>>> all the documents in the path but there is a requirement to get the access
>>> tokens for the files. But,it shows the authority group name not any details
>>> related to the users who all have access for the document accessed.
>>>
>>> Please guide for that.
>>>
>>>
>>> [image: Inline image 1]
>>>
>>>
>>> Thanks and Regards,
>>> Nikita
>>>
>>> On Wed, Jan 31, 2018 at 5:55 PM, Karl Wright <da...@gmail.com> wrote:
>>>
>>>> Clearly you are unable to fetch document ACLs from SharePoint. This
>>>> has nothing to do with whether your Active Directory is properly
>>>> configured, and has more to do with the MCPermissions plugin.
>>>>
>>>> When you view the SharePoint repository connection, does it say
>>>> "Connection working"? If not, you need the figure out why.
>>>>
>>>> Here are some possibilities:
>>>> (1) The plugin was properly installed but with the wrong permissions.
>>>> You must be logged in as an Admin to install it properly. If that's not
>>>> done it will mostly work but not be able to retrieve the permissions.
>>>> (2) You may have selected the wrong version of Sharepoint in the
>>>> repository connection. Make sure you select the right one or, once again,
>>>> you won't be able to retrieve permissions.
>>>>
>>>> The documentation says this: "Select your SharePoint server version
>>>> from the pulldown. If you do not select the correct server version, your
>>>> documents may either be indexed with insufficient security protection, or
>>>> you may not be able to index any documents. Check with your SharePoint
>>>> system administrator if you are not sure what to select."
>>>>
>>>> Please let me know whether this helps. Thanks!
>>>> Karl
>>>>
>>>>
>>>> On Wed, Jan 31, 2018 at 6:51 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>> wrote:
>>>>
>>>>> Hi Karl,
>>>>>
>>>>> I have checked the log Files but there is issue with the ACLs but
>>>>> there is Active Directory already connected with Sharepoint site. And it is
>>>>> not possible to pass the Access tokens in the connector. Please provide the
>>>>> suggestion for the problem.
>>>>>
>>>>> [image: Inline image 1]
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks and Regards,
>>>>> Nikita
>>>>>
>>>>> On Mon, Jan 29, 2018 at 6:55 PM, Karl Wright <da...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I don't see any path rules there. That will clearly not work.
>>>>>>
>>>>>> If you want confirmation of this, you can turn on connector debugging
>>>>>> (set properties.xml property org.apache.manifoldcf.connectors to
>>>>>> "DEBUG"), look at the log output. Every time the SharePoint connector
>>>>>> makes a decision to not include something, it logs it.
>>>>>>
>>>>>> Karl
>>>>>>
>>>>>> On Mon, Jan 29, 2018 at 8:07 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I have checked the documentation and the path configuration as well.
>>>>>>> But still not able to get any output.
>>>>>>>
>>>>>>> [image: Inline image 1]
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> and the metadata path given is:
>>>>>>> [image: Inline image 2]
>>>>>>>
>>>>>>> Thanks and Regards,
>>>>>>> Nikita
>>>>>>>
>>>>>>> On Mon, Jan 29, 2018 at 4:45 PM, Karl Wright <da...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I think your path configuration is incorrect. See the end-user
>>>>>>>> documentation. It's a little tricky because you must specify a matching
>>>>>>>> rule for both files and paths.
>>>>>>>>
>>>>>>>> Karl
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jan 29, 2018 at 4:42 AM, Nikita Ahuja <nikita@smartshore.nl
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> - Hi Karl,
>>>>>>>>>
>>>>>>>>> There is another query related to Sharepoint 2013 Connector. I am
>>>>>>>>> trying to crawl the files of Sharepoint On Premise site. But I am not able
>>>>>>>>> to get anything in the output either in ElasticSearch or in FileShare.
>>>>>>>>> I have followed given steps:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> - Created an Authority group for the Active Directory and same
>>>>>>>>> is used in Sharepoint Repository Connection.
>>>>>>>>>
>>>>>>>>> [image: Inline image 2]
>>>>>>>>>
>>>>>>>>> - Installed the Sharepoint 2013 MCPermission Plugin and is
>>>>>>>>> accessible on the site.
>>>>>>>>>
>>>>>>>>> [image: Inline image 3]
>>>>>>>>>
>>>>>>>>> - Still not able to access any document from the selected
>>>>>>>>> folder of Sharepoint Site.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Please guide me accordingly.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks and Regards,
>>>>>>>>> Nikita
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jan 25, 2018 at 6:46 PM, Nikita Ahuja <
>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks Karl,
>>>>>>>>>>
>>>>>>>>>> It is working fine now and Also I am able to get the access
>>>>>>>>>> tokens in the elastic search index.
>>>>>>>>>>
>>>>>>>>>> Thanks a lot for your valuable suggestions.
>>>>>>>>>>
>>>>>>>>>> With Regards,
>>>>>>>>>> Nikita
>>>>>>>>>>
>>>>>>>>>> On Thu, Jan 25, 2018 at 4:42 PM, Karl Wright <da...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> The __nosecurity__ token is generated by the Elastic Search
>>>>>>>>>>> Connector. It is produced (correctly) when no acls reach the connector.
>>>>>>>>>>> This is why I asked you about the repository connection, because that
>>>>>>>>>>> connection is not producing any ACLs.
>>>>>>>>>>>
>>>>>>>>>>> If you are crawling files in a file system, you must use the
>>>>>>>>>>> JCIFS connector if you want ACLs to be generated. The FileSystem connector
>>>>>>>>>>> does not generate ACLs.
>>>>>>>>>>>
>>>>>>>>>>> Karl
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jan 25, 2018 at 3:56 AM, Nikita Ahuja <
>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Karl,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I have tried for ElasticSearch version 1.3.1 and 5.6.1 but
>>>>>>>>>>>> still getting this " __nosecurity__" token and for Repository
>>>>>>>>>>>> connection File Sytem is used and for Transformation "Tika Content
>>>>>>>>>>>> Extractor" and Metadata adjuster Connector is used.
>>>>>>>>>>>>
>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Is there any other thing which I need to perform??
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>> Nikita
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Jan 25, 2018 at 2:13 PM, Karl Wright <
>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Nikita,
>>>>>>>>>>>>>
>>>>>>>>>>>>> You are getting the __nosecurity__ token value transmitted and
>>>>>>>>>>>>> stored, which means your Elastic Search setup is probably reasonable.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can you give us details about your pipeline? What repository
>>>>>>>>>>>>> connector is this?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 1:49 AM, Nikita Ahuja <
>>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am using ManifoldCF 2.8.1 and trying to access the files
>>>>>>>>>>>>>> from FileShare configured with the Active Directory and get the data in
>>>>>>>>>>>>>> elastic search index.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For this steps followed are:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - An Authority Group is created under which Authority
>>>>>>>>>>>>>> Connection and Repository connection are created.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [image: Inline image 3]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - And the elastic search is created as an Output
>>>>>>>>>>>>>> Connector and a job is created for the connection in which a transformation
>>>>>>>>>>>>>> and metadata connector is also linked in the
>>>>>>>>>>>>>> - Which is giving all the output except the access token
>>>>>>>>>>>>>> values.
>>>>>>>>>>>>>> - [image: Inline image 1]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Please guide to take appropriate steps for it so that I can
>>>>>>>>>>>>>> get the access tokens in the elastic search index. Hoping for the solution
>>>>>>>>>>>>>> asap.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
Posted by Nikita Ahuja <ni...@smartshore.nl>.
Hi Karl,
Yes, you are right I am able to fetch the files but the access tokens
indexed with the files are not included what is required. And I am using
the ManifoldCF security model not any custom. Everything is fine but only
problem is with the token.
And am not sure what is the reason behind that: is it because the
Sharepoint site which is used is not with *HTTPS* protocol or any other
reason?
Thanks and Regards,
Nikita
On Fri, Feb 2, 2018 at 7:05 PM, Karl Wright <da...@gmail.com> wrote:
> I am not sure I understand -- are you saying that you can fetch the files
> but the access tokens indexed with the files do not include what you
> require? And that the reason you require it done a certain way is because
> you intend to apply security in a manner of your own design?
>
> The access tokens that are indexed with files in SharePoint are what are
> needed to enforce security using the ManifoldCF Sharepoint authorities, no
> more and no less. The authorities map an individual user to the access
> tokens needed for that user, which include tokens for the groups as well as
> for the user name. The tokens indexed with the files do NOT include tokens
> generated by mapping groups to individual users -- the only tokens indexed
> for a file are for the groups and for any individual users specified
> explicitly.
>
> I strongly suggest you attempt to apply security fully using the
> ManifoldCF model rather than trying to do something custom.
>
> Karl
>
>
> On Fri, Feb 2, 2018 at 8:04 AM, Nikita Ahuja <ni...@smartshore.nl> wrote:
>
>> Hi Karl,
>>
>> Thanks a lot for your help.It is working fine now.. I am able to fetch
>> all the documents in the path but there is a requirement to get the access
>> tokens for the files. But,it shows the authority group name not any details
>> related to the users who all have access for the document accessed.
>>
>> Please guide for that.
>>
>>
>> [image: Inline image 1]
>>
>>
>> Thanks and Regards,
>> Nikita
>>
>> On Wed, Jan 31, 2018 at 5:55 PM, Karl Wright <da...@gmail.com> wrote:
>>
>>> Clearly you are unable to fetch document ACLs from SharePoint. This has
>>> nothing to do with whether your Active Directory is properly configured,
>>> and has more to do with the MCPermissions plugin.
>>>
>>> When you view the SharePoint repository connection, does it say
>>> "Connection working"? If not, you need the figure out why.
>>>
>>> Here are some possibilities:
>>> (1) The plugin was properly installed but with the wrong permissions.
>>> You must be logged in as an Admin to install it properly. If that's not
>>> done it will mostly work but not be able to retrieve the permissions.
>>> (2) You may have selected the wrong version of Sharepoint in the
>>> repository connection. Make sure you select the right one or, once again,
>>> you won't be able to retrieve permissions.
>>>
>>> The documentation says this: "Select your SharePoint server version
>>> from the pulldown. If you do not select the correct server version, your
>>> documents may either be indexed with insufficient security protection, or
>>> you may not be able to index any documents. Check with your SharePoint
>>> system administrator if you are not sure what to select."
>>>
>>> Please let me know whether this helps. Thanks!
>>> Karl
>>>
>>>
>>> On Wed, Jan 31, 2018 at 6:51 AM, Nikita Ahuja <ni...@smartshore.nl>
>>> wrote:
>>>
>>>> Hi Karl,
>>>>
>>>> I have checked the log Files but there is issue with the ACLs but there
>>>> is Active Directory already connected with Sharepoint site. And it is not
>>>> possible to pass the Access tokens in the connector. Please provide the
>>>> suggestion for the problem.
>>>>
>>>> [image: Inline image 1]
>>>>
>>>>
>>>>
>>>>
>>>> Thanks and Regards,
>>>> Nikita
>>>>
>>>> On Mon, Jan 29, 2018 at 6:55 PM, Karl Wright <da...@gmail.com>
>>>> wrote:
>>>>
>>>>> I don't see any path rules there. That will clearly not work.
>>>>>
>>>>> If you want confirmation of this, you can turn on connector debugging
>>>>> (set properties.xml property org.apache.manifoldcf.connectors to
>>>>> "DEBUG"), look at the log output. Every time the SharePoint connector
>>>>> makes a decision to not include something, it logs it.
>>>>>
>>>>> Karl
>>>>>
>>>>> On Mon, Jan 29, 2018 at 8:07 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have checked the documentation and the path configuration as well.
>>>>>> But still not able to get any output.
>>>>>>
>>>>>> [image: Inline image 1]
>>>>>>
>>>>>>
>>>>>>
>>>>>> and the metadata path given is:
>>>>>> [image: Inline image 2]
>>>>>>
>>>>>> Thanks and Regards,
>>>>>> Nikita
>>>>>>
>>>>>> On Mon, Jan 29, 2018 at 4:45 PM, Karl Wright <da...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I think your path configuration is incorrect. See the end-user
>>>>>>> documentation. It's a little tricky because you must specify a matching
>>>>>>> rule for both files and paths.
>>>>>>>
>>>>>>> Karl
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jan 29, 2018 at 4:42 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> - Hi Karl,
>>>>>>>>
>>>>>>>> There is another query related to Sharepoint 2013 Connector. I am
>>>>>>>> trying to crawl the files of Sharepoint On Premise site. But I am not able
>>>>>>>> to get anything in the output either in ElasticSearch or in FileShare.
>>>>>>>> I have followed given steps:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> - Created an Authority group for the Active Directory and same
>>>>>>>> is used in Sharepoint Repository Connection.
>>>>>>>>
>>>>>>>> [image: Inline image 2]
>>>>>>>>
>>>>>>>> - Installed the Sharepoint 2013 MCPermission Plugin and is
>>>>>>>> accessible on the site.
>>>>>>>>
>>>>>>>> [image: Inline image 3]
>>>>>>>>
>>>>>>>> - Still not able to access any document from the selected
>>>>>>>> folder of Sharepoint Site.
>>>>>>>>
>>>>>>>>
>>>>>>>> Please guide me accordingly.
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks and Regards,
>>>>>>>> Nikita
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jan 25, 2018 at 6:46 PM, Nikita Ahuja <nikita@smartshore.nl
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Thanks Karl,
>>>>>>>>>
>>>>>>>>> It is working fine now and Also I am able to get the access tokens
>>>>>>>>> in the elastic search index.
>>>>>>>>>
>>>>>>>>> Thanks a lot for your valuable suggestions.
>>>>>>>>>
>>>>>>>>> With Regards,
>>>>>>>>> Nikita
>>>>>>>>>
>>>>>>>>> On Thu, Jan 25, 2018 at 4:42 PM, Karl Wright <da...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> The __nosecurity__ token is generated by the Elastic Search
>>>>>>>>>> Connector. It is produced (correctly) when no acls reach the connector.
>>>>>>>>>> This is why I asked you about the repository connection, because that
>>>>>>>>>> connection is not producing any ACLs.
>>>>>>>>>>
>>>>>>>>>> If you are crawling files in a file system, you must use the
>>>>>>>>>> JCIFS connector if you want ACLs to be generated. The FileSystem connector
>>>>>>>>>> does not generate ACLs.
>>>>>>>>>>
>>>>>>>>>> Karl
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Jan 25, 2018 at 3:56 AM, Nikita Ahuja <
>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Karl,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I have tried for ElasticSearch version 1.3.1 and 5.6.1 but still
>>>>>>>>>>> getting this " __nosecurity__" token and for Repository
>>>>>>>>>>> connection File Sytem is used and for Transformation "Tika Content
>>>>>>>>>>> Extractor" and Metadata adjuster Connector is used.
>>>>>>>>>>>
>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Is there any other thing which I need to perform??
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>> Nikita
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jan 25, 2018 at 2:13 PM, Karl Wright <daddywri@gmail.com
>>>>>>>>>>> > wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Nikita,
>>>>>>>>>>>>
>>>>>>>>>>>> You are getting the __nosecurity__ token value transmitted and
>>>>>>>>>>>> stored, which means your Elastic Search setup is probably reasonable.
>>>>>>>>>>>>
>>>>>>>>>>>> Can you give us details about your pipeline? What repository
>>>>>>>>>>>> connector is this?
>>>>>>>>>>>>
>>>>>>>>>>>> Karl
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Jan 25, 2018 at 1:49 AM, Nikita Ahuja <
>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am using ManifoldCF 2.8.1 and trying to access the files
>>>>>>>>>>>>> from FileShare configured with the Active Directory and get the data in
>>>>>>>>>>>>> elastic search index.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For this steps followed are:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> - An Authority Group is created under which Authority
>>>>>>>>>>>>> Connection and Repository connection are created.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [image: Inline image 3]
>>>>>>>>>>>>>
>>>>>>>>>>>>> - And the elastic search is created as an Output Connector
>>>>>>>>>>>>> and a job is created for the connection in which a transformation and
>>>>>>>>>>>>> metadata connector is also linked in the
>>>>>>>>>>>>> - Which is giving all the output except the access token
>>>>>>>>>>>>> values.
>>>>>>>>>>>>> - [image: Inline image 1]
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please guide to take appropriate steps for it so that I can
>>>>>>>>>>>>> get the access tokens in the elastic search index. Hoping for the solution
>>>>>>>>>>>>> asap.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
Posted by Karl Wright <da...@gmail.com>.
I am not sure I understand -- are you saying that you can fetch the files
but the access tokens indexed with the files do not include what you
require? And that the reason you require it done a certain way is because
you intend to apply security in a manner of your own design?
The access tokens that are indexed with files in SharePoint are what are
needed to enforce security using the ManifoldCF Sharepoint authorities, no
more and no less. The authorities map an individual user to the access
tokens needed for that user, which include tokens for the groups as well as
for the user name. The tokens indexed with the files do NOT include tokens
generated by mapping groups to individual users -- the only tokens indexed
for a file are for the groups and for any individual users specified
explicitly.
I strongly suggest you attempt to apply security fully using the ManifoldCF
model rather than trying to do something custom.
Karl
On Fri, Feb 2, 2018 at 8:04 AM, Nikita Ahuja <ni...@smartshore.nl> wrote:
> Hi Karl,
>
> Thanks a lot for your help.It is working fine now.. I am able to fetch all
> the documents in the path but there is a requirement to get the access
> tokens for the files. But,it shows the authority group name not any details
> related to the users who all have access for the document accessed.
>
> Please guide for that.
>
>
> [image: Inline image 1]
>
>
> Thanks and Regards,
> Nikita
>
> On Wed, Jan 31, 2018 at 5:55 PM, Karl Wright <da...@gmail.com> wrote:
>
>> Clearly you are unable to fetch document ACLs from SharePoint. This has
>> nothing to do with whether your Active Directory is properly configured,
>> and has more to do with the MCPermissions plugin.
>>
>> When you view the SharePoint repository connection, does it say
>> "Connection working"? If not, you need the figure out why.
>>
>> Here are some possibilities:
>> (1) The plugin was properly installed but with the wrong permissions.
>> You must be logged in as an Admin to install it properly. If that's not
>> done it will mostly work but not be able to retrieve the permissions.
>> (2) You may have selected the wrong version of Sharepoint in the
>> repository connection. Make sure you select the right one or, once again,
>> you won't be able to retrieve permissions.
>>
>> The documentation says this: "Select your SharePoint server version from
>> the pulldown. If you do not select the correct server version, your
>> documents may either be indexed with insufficient security protection, or
>> you may not be able to index any documents. Check with your SharePoint
>> system administrator if you are not sure what to select."
>>
>> Please let me know whether this helps. Thanks!
>> Karl
>>
>>
>> On Wed, Jan 31, 2018 at 6:51 AM, Nikita Ahuja <ni...@smartshore.nl>
>> wrote:
>>
>>> Hi Karl,
>>>
>>> I have checked the log Files but there is issue with the ACLs but there
>>> is Active Directory already connected with Sharepoint site. And it is not
>>> possible to pass the Access tokens in the connector. Please provide the
>>> suggestion for the problem.
>>>
>>> [image: Inline image 1]
>>>
>>>
>>>
>>>
>>> Thanks and Regards,
>>> Nikita
>>>
>>> On Mon, Jan 29, 2018 at 6:55 PM, Karl Wright <da...@gmail.com> wrote:
>>>
>>>> I don't see any path rules there. That will clearly not work.
>>>>
>>>> If you want confirmation of this, you can turn on connector debugging
>>>> (set properties.xml property org.apache.manifoldcf.connectors to
>>>> "DEBUG"), look at the log output. Every time the SharePoint connector
>>>> makes a decision to not include something, it logs it.
>>>>
>>>> Karl
>>>>
>>>> On Mon, Jan 29, 2018 at 8:07 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have checked the documentation and the path configuration as well.
>>>>> But still not able to get any output.
>>>>>
>>>>> [image: Inline image 1]
>>>>>
>>>>>
>>>>>
>>>>> and the metadata path given is:
>>>>> [image: Inline image 2]
>>>>>
>>>>> Thanks and Regards,
>>>>> Nikita
>>>>>
>>>>> On Mon, Jan 29, 2018 at 4:45 PM, Karl Wright <da...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I think your path configuration is incorrect. See the end-user
>>>>>> documentation. It's a little tricky because you must specify a matching
>>>>>> rule for both files and paths.
>>>>>>
>>>>>> Karl
>>>>>>
>>>>>>
>>>>>> On Mon, Jan 29, 2018 at 4:42 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> - Hi Karl,
>>>>>>>
>>>>>>> There is another query related to Sharepoint 2013 Connector. I am
>>>>>>> trying to crawl the files of Sharepoint On Premise site. But I am not able
>>>>>>> to get anything in the output either in ElasticSearch or in FileShare.
>>>>>>> I have followed given steps:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> - Created an Authority group for the Active Directory and same
>>>>>>> is used in Sharepoint Repository Connection.
>>>>>>>
>>>>>>> [image: Inline image 2]
>>>>>>>
>>>>>>> - Installed the Sharepoint 2013 MCPermission Plugin and is
>>>>>>> accessible on the site.
>>>>>>>
>>>>>>> [image: Inline image 3]
>>>>>>>
>>>>>>> - Still not able to access any document from the selected folder
>>>>>>> of Sharepoint Site.
>>>>>>>
>>>>>>>
>>>>>>> Please guide me accordingly.
>>>>>>>
>>>>>>>
>>>>>>> Thanks and Regards,
>>>>>>> Nikita
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jan 25, 2018 at 6:46 PM, Nikita Ahuja <ni...@smartshore.nl>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks Karl,
>>>>>>>>
>>>>>>>> It is working fine now and Also I am able to get the access tokens
>>>>>>>> in the elastic search index.
>>>>>>>>
>>>>>>>> Thanks a lot for your valuable suggestions.
>>>>>>>>
>>>>>>>> With Regards,
>>>>>>>> Nikita
>>>>>>>>
>>>>>>>> On Thu, Jan 25, 2018 at 4:42 PM, Karl Wright <da...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> The __nosecurity__ token is generated by the Elastic Search
>>>>>>>>> Connector. It is produced (correctly) when no acls reach the connector.
>>>>>>>>> This is why I asked you about the repository connection, because that
>>>>>>>>> connection is not producing any ACLs.
>>>>>>>>>
>>>>>>>>> If you are crawling files in a file system, you must use the JCIFS
>>>>>>>>> connector if you want ACLs to be generated. The FileSystem connector does
>>>>>>>>> not generate ACLs.
>>>>>>>>>
>>>>>>>>> Karl
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jan 25, 2018 at 3:56 AM, Nikita Ahuja <
>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Karl,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I have tried for ElasticSearch version 1.3.1 and 5.6.1 but still
>>>>>>>>>> getting this " __nosecurity__" token and for Repository
>>>>>>>>>> connection File Sytem is used and for Transformation "Tika Content
>>>>>>>>>> Extractor" and Metadata adjuster Connector is used.
>>>>>>>>>>
>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Is there any other thing which I need to perform??
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks and Regards,
>>>>>>>>>> Nikita
>>>>>>>>>>
>>>>>>>>>> On Thu, Jan 25, 2018 at 2:13 PM, Karl Wright <da...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Nikita,
>>>>>>>>>>>
>>>>>>>>>>> You are getting the __nosecurity__ token value transmitted and
>>>>>>>>>>> stored, which means your Elastic Search setup is probably reasonable.
>>>>>>>>>>>
>>>>>>>>>>> Can you give us details about your pipeline? What repository
>>>>>>>>>>> connector is this?
>>>>>>>>>>>
>>>>>>>>>>> Karl
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jan 25, 2018 at 1:49 AM, Nikita Ahuja <
>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>> I am using ManifoldCF 2.8.1 and trying to access the files from
>>>>>>>>>>>> FileShare configured with the Active Directory and get the data in elastic
>>>>>>>>>>>> search index.
>>>>>>>>>>>>
>>>>>>>>>>>> For this steps followed are:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> - An Authority Group is created under which Authority
>>>>>>>>>>>> Connection and Repository connection are created.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [image: Inline image 3]
>>>>>>>>>>>>
>>>>>>>>>>>> - And the elastic search is created as an Output Connector
>>>>>>>>>>>> and a job is created for the connection in which a transformation and
>>>>>>>>>>>> metadata connector is also linked in the
>>>>>>>>>>>> - Which is giving all the output except the access token
>>>>>>>>>>>> values.
>>>>>>>>>>>> - [image: Inline image 1]
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Please guide to take appropriate steps for it so that I can get
>>>>>>>>>>>> the access tokens in the elastic search index. Hoping for the solution asap.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>> Nikita
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
Posted by Nikita Ahuja <ni...@smartshore.nl>.
Hi Karl,
Thanks a lot for your help.It is working fine now.. I am able to fetch all
the documents in the path but there is a requirement to get the access
tokens for the files. But,it shows the authority group name not any details
related to the users who all have access for the document accessed.
Please guide for that.
[image: Inline image 1]
Thanks and Regards,
Nikita
On Wed, Jan 31, 2018 at 5:55 PM, Karl Wright <da...@gmail.com> wrote:
> Clearly you are unable to fetch document ACLs from SharePoint. This has
> nothing to do with whether your Active Directory is properly configured,
> and has more to do with the MCPermissions plugin.
>
> When you view the SharePoint repository connection, does it say
> "Connection working"? If not, you need the figure out why.
>
> Here are some possibilities:
> (1) The plugin was properly installed but with the wrong permissions. You
> must be logged in as an Admin to install it properly. If that's not done
> it will mostly work but not be able to retrieve the permissions.
> (2) You may have selected the wrong version of Sharepoint in the
> repository connection. Make sure you select the right one or, once again,
> you won't be able to retrieve permissions.
>
> The documentation says this: "Select your SharePoint server version from
> the pulldown. If you do not select the correct server version, your
> documents may either be indexed with insufficient security protection, or
> you may not be able to index any documents. Check with your SharePoint
> system administrator if you are not sure what to select."
>
> Please let me know whether this helps. Thanks!
> Karl
>
>
> On Wed, Jan 31, 2018 at 6:51 AM, Nikita Ahuja <ni...@smartshore.nl>
> wrote:
>
>> Hi Karl,
>>
>> I have checked the log Files but there is issue with the ACLs but there
>> is Active Directory already connected with Sharepoint site. And it is not
>> possible to pass the Access tokens in the connector. Please provide the
>> suggestion for the problem.
>>
>> [image: Inline image 1]
>>
>>
>>
>>
>> Thanks and Regards,
>> Nikita
>>
>> On Mon, Jan 29, 2018 at 6:55 PM, Karl Wright <da...@gmail.com> wrote:
>>
>>> I don't see any path rules there. That will clearly not work.
>>>
>>> If you want confirmation of this, you can turn on connector debugging
>>> (set properties.xml property org.apache.manifoldcf.connectors to
>>> "DEBUG"), look at the log output. Every time the SharePoint connector
>>> makes a decision to not include something, it logs it.
>>>
>>> Karl
>>>
>>> On Mon, Jan 29, 2018 at 8:07 AM, Nikita Ahuja <ni...@smartshore.nl>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have checked the documentation and the path configuration as well.
>>>> But still not able to get any output.
>>>>
>>>> [image: Inline image 1]
>>>>
>>>>
>>>>
>>>> and the metadata path given is:
>>>> [image: Inline image 2]
>>>>
>>>> Thanks and Regards,
>>>> Nikita
>>>>
>>>> On Mon, Jan 29, 2018 at 4:45 PM, Karl Wright <da...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I think your path configuration is incorrect. See the end-user
>>>>> documentation. It's a little tricky because you must specify a matching
>>>>> rule for both files and paths.
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>> On Mon, Jan 29, 2018 at 4:42 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> - Hi Karl,
>>>>>>
>>>>>> There is another query related to Sharepoint 2013 Connector. I am
>>>>>> trying to crawl the files of Sharepoint On Premise site. But I am not able
>>>>>> to get anything in the output either in ElasticSearch or in FileShare.
>>>>>> I have followed given steps:
>>>>>>
>>>>>>
>>>>>>
>>>>>> - Created an Authority group for the Active Directory and same is
>>>>>> used in Sharepoint Repository Connection.
>>>>>>
>>>>>> [image: Inline image 2]
>>>>>>
>>>>>> - Installed the Sharepoint 2013 MCPermission Plugin and is
>>>>>> accessible on the site.
>>>>>>
>>>>>> [image: Inline image 3]
>>>>>>
>>>>>> - Still not able to access any document from the selected folder
>>>>>> of Sharepoint Site.
>>>>>>
>>>>>>
>>>>>> Please guide me accordingly.
>>>>>>
>>>>>>
>>>>>> Thanks and Regards,
>>>>>> Nikita
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Jan 25, 2018 at 6:46 PM, Nikita Ahuja <ni...@smartshore.nl>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks Karl,
>>>>>>>
>>>>>>> It is working fine now and Also I am able to get the access tokens
>>>>>>> in the elastic search index.
>>>>>>>
>>>>>>> Thanks a lot for your valuable suggestions.
>>>>>>>
>>>>>>> With Regards,
>>>>>>> Nikita
>>>>>>>
>>>>>>> On Thu, Jan 25, 2018 at 4:42 PM, Karl Wright <da...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> The __nosecurity__ token is generated by the Elastic Search
>>>>>>>> Connector. It is produced (correctly) when no acls reach the connector.
>>>>>>>> This is why I asked you about the repository connection, because that
>>>>>>>> connection is not producing any ACLs.
>>>>>>>>
>>>>>>>> If you are crawling files in a file system, you must use the JCIFS
>>>>>>>> connector if you want ACLs to be generated. The FileSystem connector does
>>>>>>>> not generate ACLs.
>>>>>>>>
>>>>>>>> Karl
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jan 25, 2018 at 3:56 AM, Nikita Ahuja <nikita@smartshore.nl
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Hi Karl,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have tried for ElasticSearch version 1.3.1 and 5.6.1 but still
>>>>>>>>> getting this " __nosecurity__" token and for Repository
>>>>>>>>> connection File Sytem is used and for Transformation "Tika Content
>>>>>>>>> Extractor" and Metadata adjuster Connector is used.
>>>>>>>>>
>>>>>>>>> [image: Inline image 1]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Is there any other thing which I need to perform??
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks and Regards,
>>>>>>>>> Nikita
>>>>>>>>>
>>>>>>>>> On Thu, Jan 25, 2018 at 2:13 PM, Karl Wright <da...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Nikita,
>>>>>>>>>>
>>>>>>>>>> You are getting the __nosecurity__ token value transmitted and
>>>>>>>>>> stored, which means your Elastic Search setup is probably reasonable.
>>>>>>>>>>
>>>>>>>>>> Can you give us details about your pipeline? What repository
>>>>>>>>>> connector is this?
>>>>>>>>>>
>>>>>>>>>> Karl
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Jan 25, 2018 at 1:49 AM, Nikita Ahuja <
>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> I am using ManifoldCF 2.8.1 and trying to access the files from
>>>>>>>>>>> FileShare configured with the Active Directory and get the data in elastic
>>>>>>>>>>> search index.
>>>>>>>>>>>
>>>>>>>>>>> For this steps followed are:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> - An Authority Group is created under which Authority
>>>>>>>>>>> Connection and Repository connection are created.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> [image: Inline image 3]
>>>>>>>>>>>
>>>>>>>>>>> - And the elastic search is created as an Output Connector
>>>>>>>>>>> and a job is created for the connection in which a transformation and
>>>>>>>>>>> metadata connector is also linked in the
>>>>>>>>>>> - Which is giving all the output except the access token
>>>>>>>>>>> values.
>>>>>>>>>>> - [image: Inline image 1]
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Please guide to take appropriate steps for it so that I can get
>>>>>>>>>>> the access tokens in the elastic search index. Hoping for the solution asap.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>> Nikita
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
Posted by Karl Wright <da...@gmail.com>.
Clearly you are unable to fetch document ACLs from SharePoint. This has
nothing to do with whether your Active Directory is properly configured,
and has more to do with the MCPermissions plugin.
When you view the SharePoint repository connection, does it say "Connection
working"? If not, you need the figure out why.
Here are some possibilities:
(1) The plugin was properly installed but with the wrong permissions. You
must be logged in as an Admin to install it properly. If that's not done
it will mostly work but not be able to retrieve the permissions.
(2) You may have selected the wrong version of Sharepoint in the repository
connection. Make sure you select the right one or, once again, you won't
be able to retrieve permissions.
The documentation says this: "Select your SharePoint server version from
the pulldown. If you do not select the correct server version, your
documents may either be indexed with insufficient security protection, or
you may not be able to index any documents. Check with your SharePoint
system administrator if you are not sure what to select."
Please let me know whether this helps. Thanks!
Karl
On Wed, Jan 31, 2018 at 6:51 AM, Nikita Ahuja <ni...@smartshore.nl> wrote:
> Hi Karl,
>
> I have checked the log Files but there is issue with the ACLs but there is
> Active Directory already connected with Sharepoint site. And it is not
> possible to pass the Access tokens in the connector. Please provide the
> suggestion for the problem.
>
> [image: Inline image 1]
>
>
>
>
> Thanks and Regards,
> Nikita
>
> On Mon, Jan 29, 2018 at 6:55 PM, Karl Wright <da...@gmail.com> wrote:
>
>> I don't see any path rules there. That will clearly not work.
>>
>> If you want confirmation of this, you can turn on connector debugging
>> (set properties.xml property org.apache.manifoldcf.connectors to
>> "DEBUG"), look at the log output. Every time the SharePoint connector
>> makes a decision to not include something, it logs it.
>>
>> Karl
>>
>> On Mon, Jan 29, 2018 at 8:07 AM, Nikita Ahuja <ni...@smartshore.nl>
>> wrote:
>>
>>> Hi,
>>>
>>> I have checked the documentation and the path configuration as well. But
>>> still not able to get any output.
>>>
>>> [image: Inline image 1]
>>>
>>>
>>>
>>> and the metadata path given is:
>>> [image: Inline image 2]
>>>
>>> Thanks and Regards,
>>> Nikita
>>>
>>> On Mon, Jan 29, 2018 at 4:45 PM, Karl Wright <da...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I think your path configuration is incorrect. See the end-user
>>>> documentation. It's a little tricky because you must specify a matching
>>>> rule for both files and paths.
>>>>
>>>> Karl
>>>>
>>>>
>>>> On Mon, Jan 29, 2018 at 4:42 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>> wrote:
>>>>
>>>>>
>>>>> - Hi Karl,
>>>>>
>>>>> There is another query related to Sharepoint 2013 Connector. I am
>>>>> trying to crawl the files of Sharepoint On Premise site. But I am not able
>>>>> to get anything in the output either in ElasticSearch or in FileShare.
>>>>> I have followed given steps:
>>>>>
>>>>>
>>>>>
>>>>> - Created an Authority group for the Active Directory and same is
>>>>> used in Sharepoint Repository Connection.
>>>>>
>>>>> [image: Inline image 2]
>>>>>
>>>>> - Installed the Sharepoint 2013 MCPermission Plugin and is
>>>>> accessible on the site.
>>>>>
>>>>> [image: Inline image 3]
>>>>>
>>>>> - Still not able to access any document from the selected folder
>>>>> of Sharepoint Site.
>>>>>
>>>>>
>>>>> Please guide me accordingly.
>>>>>
>>>>>
>>>>> Thanks and Regards,
>>>>> Nikita
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jan 25, 2018 at 6:46 PM, Nikita Ahuja <ni...@smartshore.nl>
>>>>> wrote:
>>>>>
>>>>>> Thanks Karl,
>>>>>>
>>>>>> It is working fine now and Also I am able to get the access tokens in
>>>>>> the elastic search index.
>>>>>>
>>>>>> Thanks a lot for your valuable suggestions.
>>>>>>
>>>>>> With Regards,
>>>>>> Nikita
>>>>>>
>>>>>> On Thu, Jan 25, 2018 at 4:42 PM, Karl Wright <da...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> The __nosecurity__ token is generated by the Elastic Search
>>>>>>> Connector. It is produced (correctly) when no acls reach the connector.
>>>>>>> This is why I asked you about the repository connection, because that
>>>>>>> connection is not producing any ACLs.
>>>>>>>
>>>>>>> If you are crawling files in a file system, you must use the JCIFS
>>>>>>> connector if you want ACLs to be generated. The FileSystem connector does
>>>>>>> not generate ACLs.
>>>>>>>
>>>>>>> Karl
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jan 25, 2018 at 3:56 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Karl,
>>>>>>>>
>>>>>>>>
>>>>>>>> I have tried for ElasticSearch version 1.3.1 and 5.6.1 but still
>>>>>>>> getting this " __nosecurity__" token and for Repository
>>>>>>>> connection File Sytem is used and for Transformation "Tika Content
>>>>>>>> Extractor" and Metadata adjuster Connector is used.
>>>>>>>>
>>>>>>>> [image: Inline image 1]
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Is there any other thing which I need to perform??
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks and Regards,
>>>>>>>> Nikita
>>>>>>>>
>>>>>>>> On Thu, Jan 25, 2018 at 2:13 PM, Karl Wright <da...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Nikita,
>>>>>>>>>
>>>>>>>>> You are getting the __nosecurity__ token value transmitted and
>>>>>>>>> stored, which means your Elastic Search setup is probably reasonable.
>>>>>>>>>
>>>>>>>>> Can you give us details about your pipeline? What repository
>>>>>>>>> connector is this?
>>>>>>>>>
>>>>>>>>> Karl
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jan 25, 2018 at 1:49 AM, Nikita Ahuja <
>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> I am using ManifoldCF 2.8.1 and trying to access the files from
>>>>>>>>>> FileShare configured with the Active Directory and get the data in elastic
>>>>>>>>>> search index.
>>>>>>>>>>
>>>>>>>>>> For this steps followed are:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> - An Authority Group is created under which Authority
>>>>>>>>>> Connection and Repository connection are created.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [image: Inline image 3]
>>>>>>>>>>
>>>>>>>>>> - And the elastic search is created as an Output Connector
>>>>>>>>>> and a job is created for the connection in which a transformation and
>>>>>>>>>> metadata connector is also linked in the
>>>>>>>>>> - Which is giving all the output except the access token
>>>>>>>>>> values.
>>>>>>>>>> - [image: Inline image 1]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Please guide to take appropriate steps for it so that I can get
>>>>>>>>>> the access tokens in the elastic search index. Hoping for the solution asap.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks and Regards,
>>>>>>>>>> Nikita
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
Posted by Nikita Ahuja <ni...@smartshore.nl>.
Hi Karl,
I have checked the log Files but there is issue with the ACLs but there is
Active Directory already connected with Sharepoint site. And it is not
possible to pass the Access tokens in the connector. Please provide the
suggestion for the problem.
[image: Inline image 1]
Thanks and Regards,
Nikita
On Mon, Jan 29, 2018 at 6:55 PM, Karl Wright <da...@gmail.com> wrote:
> I don't see any path rules there. That will clearly not work.
>
> If you want confirmation of this, you can turn on connector debugging (set
> properties.xml property org.apache.manifoldcf.connectors to "DEBUG"),
> look at the log output. Every time the SharePoint connector makes a
> decision to not include something, it logs it.
>
> Karl
>
> On Mon, Jan 29, 2018 at 8:07 AM, Nikita Ahuja <ni...@smartshore.nl>
> wrote:
>
>> Hi,
>>
>> I have checked the documentation and the path configuration as well. But
>> still not able to get any output.
>>
>> [image: Inline image 1]
>>
>>
>>
>> and the metadata path given is:
>> [image: Inline image 2]
>>
>> Thanks and Regards,
>> Nikita
>>
>> On Mon, Jan 29, 2018 at 4:45 PM, Karl Wright <da...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I think your path configuration is incorrect. See the end-user
>>> documentation. It's a little tricky because you must specify a matching
>>> rule for both files and paths.
>>>
>>> Karl
>>>
>>>
>>> On Mon, Jan 29, 2018 at 4:42 AM, Nikita Ahuja <ni...@smartshore.nl>
>>> wrote:
>>>
>>>>
>>>> - Hi Karl,
>>>>
>>>> There is another query related to Sharepoint 2013 Connector. I am
>>>> trying to crawl the files of Sharepoint On Premise site. But I am not able
>>>> to get anything in the output either in ElasticSearch or in FileShare.
>>>> I have followed given steps:
>>>>
>>>>
>>>>
>>>> - Created an Authority group for the Active Directory and same is
>>>> used in Sharepoint Repository Connection.
>>>>
>>>> [image: Inline image 2]
>>>>
>>>> - Installed the Sharepoint 2013 MCPermission Plugin and is
>>>> accessible on the site.
>>>>
>>>> [image: Inline image 3]
>>>>
>>>> - Still not able to access any document from the selected folder of
>>>> Sharepoint Site.
>>>>
>>>>
>>>> Please guide me accordingly.
>>>>
>>>>
>>>> Thanks and Regards,
>>>> Nikita
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Jan 25, 2018 at 6:46 PM, Nikita Ahuja <ni...@smartshore.nl>
>>>> wrote:
>>>>
>>>>> Thanks Karl,
>>>>>
>>>>> It is working fine now and Also I am able to get the access tokens in
>>>>> the elastic search index.
>>>>>
>>>>> Thanks a lot for your valuable suggestions.
>>>>>
>>>>> With Regards,
>>>>> Nikita
>>>>>
>>>>> On Thu, Jan 25, 2018 at 4:42 PM, Karl Wright <da...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> The __nosecurity__ token is generated by the Elastic Search
>>>>>> Connector. It is produced (correctly) when no acls reach the connector.
>>>>>> This is why I asked you about the repository connection, because that
>>>>>> connection is not producing any ACLs.
>>>>>>
>>>>>> If you are crawling files in a file system, you must use the JCIFS
>>>>>> connector if you want ACLs to be generated. The FileSystem connector does
>>>>>> not generate ACLs.
>>>>>>
>>>>>> Karl
>>>>>>
>>>>>>
>>>>>> On Thu, Jan 25, 2018 at 3:56 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Karl,
>>>>>>>
>>>>>>>
>>>>>>> I have tried for ElasticSearch version 1.3.1 and 5.6.1 but still
>>>>>>> getting this " __nosecurity__" token and for Repository connection
>>>>>>> File Sytem is used and for Transformation "Tika Content Extractor" and Metadata
>>>>>>> adjuster Connector is used.
>>>>>>>
>>>>>>> [image: Inline image 1]
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Is there any other thing which I need to perform??
>>>>>>>
>>>>>>>
>>>>>>> Thanks and Regards,
>>>>>>> Nikita
>>>>>>>
>>>>>>> On Thu, Jan 25, 2018 at 2:13 PM, Karl Wright <da...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Nikita,
>>>>>>>>
>>>>>>>> You are getting the __nosecurity__ token value transmitted and
>>>>>>>> stored, which means your Elastic Search setup is probably reasonable.
>>>>>>>>
>>>>>>>> Can you give us details about your pipeline? What repository
>>>>>>>> connector is this?
>>>>>>>>
>>>>>>>> Karl
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jan 25, 2018 at 1:49 AM, Nikita Ahuja <nikita@smartshore.nl
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I am using ManifoldCF 2.8.1 and trying to access the files from
>>>>>>>>> FileShare configured with the Active Directory and get the data in elastic
>>>>>>>>> search index.
>>>>>>>>>
>>>>>>>>> For this steps followed are:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> - An Authority Group is created under which Authority
>>>>>>>>> Connection and Repository connection are created.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [image: Inline image 3]
>>>>>>>>>
>>>>>>>>> - And the elastic search is created as an Output Connector and
>>>>>>>>> a job is created for the connection in which a transformation and metadata
>>>>>>>>> connector is also linked in the
>>>>>>>>> - Which is giving all the output except the access token
>>>>>>>>> values.
>>>>>>>>> - [image: Inline image 1]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Please guide to take appropriate steps for it so that I can get
>>>>>>>>> the access tokens in the elastic search index. Hoping for the solution asap.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks and Regards,
>>>>>>>>> Nikita
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
Posted by Karl Wright <da...@gmail.com>.
I don't see any path rules there. That will clearly not work.
If you want confirmation of this, you can turn on connector debugging (set
properties.xml property org.apache.manifoldcf.connectors to "DEBUG"), look
at the log output. Every time the SharePoint connector makes a decision to
not include something, it logs it.
Karl
On Mon, Jan 29, 2018 at 8:07 AM, Nikita Ahuja <ni...@smartshore.nl> wrote:
> Hi,
>
> I have checked the documentation and the path configuration as well. But
> still not able to get any output.
>
> [image: Inline image 1]
>
>
>
> and the metadata path given is:
> [image: Inline image 2]
>
> Thanks and Regards,
> Nikita
>
> On Mon, Jan 29, 2018 at 4:45 PM, Karl Wright <da...@gmail.com> wrote:
>
>> Hi,
>>
>> I think your path configuration is incorrect. See the end-user
>> documentation. It's a little tricky because you must specify a matching
>> rule for both files and paths.
>>
>> Karl
>>
>>
>> On Mon, Jan 29, 2018 at 4:42 AM, Nikita Ahuja <ni...@smartshore.nl>
>> wrote:
>>
>>>
>>> - Hi Karl,
>>>
>>> There is another query related to Sharepoint 2013 Connector. I am trying
>>> to crawl the files of Sharepoint On Premise site. But I am not able to get
>>> anything in the output either in ElasticSearch or in FileShare.
>>> I have followed given steps:
>>>
>>>
>>>
>>> - Created an Authority group for the Active Directory and same is
>>> used in Sharepoint Repository Connection.
>>>
>>> [image: Inline image 2]
>>>
>>> - Installed the Sharepoint 2013 MCPermission Plugin and is
>>> accessible on the site.
>>>
>>> [image: Inline image 3]
>>>
>>> - Still not able to access any document from the selected folder of
>>> Sharepoint Site.
>>>
>>>
>>> Please guide me accordingly.
>>>
>>>
>>> Thanks and Regards,
>>> Nikita
>>>
>>>
>>>
>>>
>>> On Thu, Jan 25, 2018 at 6:46 PM, Nikita Ahuja <ni...@smartshore.nl>
>>> wrote:
>>>
>>>> Thanks Karl,
>>>>
>>>> It is working fine now and Also I am able to get the access tokens in
>>>> the elastic search index.
>>>>
>>>> Thanks a lot for your valuable suggestions.
>>>>
>>>> With Regards,
>>>> Nikita
>>>>
>>>> On Thu, Jan 25, 2018 at 4:42 PM, Karl Wright <da...@gmail.com>
>>>> wrote:
>>>>
>>>>> The __nosecurity__ token is generated by the Elastic Search
>>>>> Connector. It is produced (correctly) when no acls reach the connector.
>>>>> This is why I asked you about the repository connection, because that
>>>>> connection is not producing any ACLs.
>>>>>
>>>>> If you are crawling files in a file system, you must use the JCIFS
>>>>> connector if you want ACLs to be generated. The FileSystem connector does
>>>>> not generate ACLs.
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>> On Thu, Jan 25, 2018 at 3:56 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>>> wrote:
>>>>>
>>>>>> Hi Karl,
>>>>>>
>>>>>>
>>>>>> I have tried for ElasticSearch version 1.3.1 and 5.6.1 but still
>>>>>> getting this " __nosecurity__" token and for Repository connection
>>>>>> File Sytem is used and for Transformation "Tika Content Extractor" and Metadata
>>>>>> adjuster Connector is used.
>>>>>>
>>>>>> [image: Inline image 1]
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Is there any other thing which I need to perform??
>>>>>>
>>>>>>
>>>>>> Thanks and Regards,
>>>>>> Nikita
>>>>>>
>>>>>> On Thu, Jan 25, 2018 at 2:13 PM, Karl Wright <da...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Nikita,
>>>>>>>
>>>>>>> You are getting the __nosecurity__ token value transmitted and
>>>>>>> stored, which means your Elastic Search setup is probably reasonable.
>>>>>>>
>>>>>>> Can you give us details about your pipeline? What repository
>>>>>>> connector is this?
>>>>>>>
>>>>>>> Karl
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jan 25, 2018 at 1:49 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I am using ManifoldCF 2.8.1 and trying to access the files from
>>>>>>>> FileShare configured with the Active Directory and get the data in elastic
>>>>>>>> search index.
>>>>>>>>
>>>>>>>> For this steps followed are:
>>>>>>>>
>>>>>>>>
>>>>>>>> - An Authority Group is created under which Authority
>>>>>>>> Connection and Repository connection are created.
>>>>>>>>
>>>>>>>>
>>>>>>>> [image: Inline image 3]
>>>>>>>>
>>>>>>>> - And the elastic search is created as an Output Connector and
>>>>>>>> a job is created for the connection in which a transformation and metadata
>>>>>>>> connector is also linked in the
>>>>>>>> - Which is giving all the output except the access token
>>>>>>>> values.
>>>>>>>> - [image: Inline image 1]
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Please guide to take appropriate steps for it so that I can get the
>>>>>>>> access tokens in the elastic search index. Hoping for the solution asap.
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks and Regards,
>>>>>>>> Nikita
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
Posted by Nikita Ahuja <ni...@smartshore.nl>.
Hi,
I have checked the documentation and the path configuration as well. But
still not able to get any output.
[image: Inline image 1]
and the metadata path given is:
[image: Inline image 2]
Thanks and Regards,
Nikita
On Mon, Jan 29, 2018 at 4:45 PM, Karl Wright <da...@gmail.com> wrote:
> Hi,
>
> I think your path configuration is incorrect. See the end-user
> documentation. It's a little tricky because you must specify a matching
> rule for both files and paths.
>
> Karl
>
>
> On Mon, Jan 29, 2018 at 4:42 AM, Nikita Ahuja <ni...@smartshore.nl>
> wrote:
>
>>
>> - Hi Karl,
>>
>> There is another query related to Sharepoint 2013 Connector. I am trying
>> to crawl the files of Sharepoint On Premise site. But I am not able to get
>> anything in the output either in ElasticSearch or in FileShare.
>> I have followed given steps:
>>
>>
>>
>> - Created an Authority group for the Active Directory and same is
>> used in Sharepoint Repository Connection.
>>
>> [image: Inline image 2]
>>
>> - Installed the Sharepoint 2013 MCPermission Plugin and is accessible
>> on the site.
>>
>> [image: Inline image 3]
>>
>> - Still not able to access any document from the selected folder of
>> Sharepoint Site.
>>
>>
>> Please guide me accordingly.
>>
>>
>> Thanks and Regards,
>> Nikita
>>
>>
>>
>>
>> On Thu, Jan 25, 2018 at 6:46 PM, Nikita Ahuja <ni...@smartshore.nl>
>> wrote:
>>
>>> Thanks Karl,
>>>
>>> It is working fine now and Also I am able to get the access tokens in
>>> the elastic search index.
>>>
>>> Thanks a lot for your valuable suggestions.
>>>
>>> With Regards,
>>> Nikita
>>>
>>> On Thu, Jan 25, 2018 at 4:42 PM, Karl Wright <da...@gmail.com> wrote:
>>>
>>>> The __nosecurity__ token is generated by the Elastic Search Connector.
>>>> It is produced (correctly) when no acls reach the connector. This is why I
>>>> asked you about the repository connection, because that connection is not
>>>> producing any ACLs.
>>>>
>>>> If you are crawling files in a file system, you must use the JCIFS
>>>> connector if you want ACLs to be generated. The FileSystem connector does
>>>> not generate ACLs.
>>>>
>>>> Karl
>>>>
>>>>
>>>> On Thu, Jan 25, 2018 at 3:56 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>> wrote:
>>>>
>>>>> Hi Karl,
>>>>>
>>>>>
>>>>> I have tried for ElasticSearch version 1.3.1 and 5.6.1 but still
>>>>> getting this " __nosecurity__" token and for Repository connection
>>>>> File Sytem is used and for Transformation "Tika Content Extractor" and Metadata
>>>>> adjuster Connector is used.
>>>>>
>>>>> [image: Inline image 1]
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Is there any other thing which I need to perform??
>>>>>
>>>>>
>>>>> Thanks and Regards,
>>>>> Nikita
>>>>>
>>>>> On Thu, Jan 25, 2018 at 2:13 PM, Karl Wright <da...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Nikita,
>>>>>>
>>>>>> You are getting the __nosecurity__ token value transmitted and
>>>>>> stored, which means your Elastic Search setup is probably reasonable.
>>>>>>
>>>>>> Can you give us details about your pipeline? What repository
>>>>>> connector is this?
>>>>>>
>>>>>> Karl
>>>>>>
>>>>>>
>>>>>> On Thu, Jan 25, 2018 at 1:49 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I am using ManifoldCF 2.8.1 and trying to access the files from
>>>>>>> FileShare configured with the Active Directory and get the data in elastic
>>>>>>> search index.
>>>>>>>
>>>>>>> For this steps followed are:
>>>>>>>
>>>>>>>
>>>>>>> - An Authority Group is created under which Authority Connection
>>>>>>> and Repository connection are created.
>>>>>>>
>>>>>>>
>>>>>>> [image: Inline image 3]
>>>>>>>
>>>>>>> - And the elastic search is created as an Output Connector and a
>>>>>>> job is created for the connection in which a transformation and metadata
>>>>>>> connector is also linked in the
>>>>>>> - Which is giving all the output except the access token values.
>>>>>>> - [image: Inline image 1]
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Please guide to take appropriate steps for it so that I can get the
>>>>>>> access tokens in the elastic search index. Hoping for the solution asap.
>>>>>>>
>>>>>>>
>>>>>>> Thanks and Regards,
>>>>>>> Nikita
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
Posted by Karl Wright <da...@gmail.com>.
Hi,
I think your path configuration is incorrect. See the end-user
documentation. It's a little tricky because you must specify a matching
rule for both files and paths.
Karl
On Mon, Jan 29, 2018 at 4:42 AM, Nikita Ahuja <ni...@smartshore.nl> wrote:
>
> - Hi Karl,
>
> There is another query related to Sharepoint 2013 Connector. I am trying
> to crawl the files of Sharepoint On Premise site. But I am not able to get
> anything in the output either in ElasticSearch or in FileShare.
> I have followed given steps:
>
>
>
> - Created an Authority group for the Active Directory and same is used
> in Sharepoint Repository Connection.
>
> [image: Inline image 2]
>
> - Installed the Sharepoint 2013 MCPermission Plugin and is accessible
> on the site.
>
> [image: Inline image 3]
>
> - Still not able to access any document from the selected folder of
> Sharepoint Site.
>
>
> Please guide me accordingly.
>
>
> Thanks and Regards,
> Nikita
>
>
>
>
> On Thu, Jan 25, 2018 at 6:46 PM, Nikita Ahuja <ni...@smartshore.nl>
> wrote:
>
>> Thanks Karl,
>>
>> It is working fine now and Also I am able to get the access tokens in the
>> elastic search index.
>>
>> Thanks a lot for your valuable suggestions.
>>
>> With Regards,
>> Nikita
>>
>> On Thu, Jan 25, 2018 at 4:42 PM, Karl Wright <da...@gmail.com> wrote:
>>
>>> The __nosecurity__ token is generated by the Elastic Search Connector.
>>> It is produced (correctly) when no acls reach the connector. This is why I
>>> asked you about the repository connection, because that connection is not
>>> producing any ACLs.
>>>
>>> If you are crawling files in a file system, you must use the JCIFS
>>> connector if you want ACLs to be generated. The FileSystem connector does
>>> not generate ACLs.
>>>
>>> Karl
>>>
>>>
>>> On Thu, Jan 25, 2018 at 3:56 AM, Nikita Ahuja <ni...@smartshore.nl>
>>> wrote:
>>>
>>>> Hi Karl,
>>>>
>>>>
>>>> I have tried for ElasticSearch version 1.3.1 and 5.6.1 but still
>>>> getting this " __nosecurity__" token and for Repository connection
>>>> File Sytem is used and for Transformation "Tika Content Extractor" and Metadata
>>>> adjuster Connector is used.
>>>>
>>>> [image: Inline image 1]
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Is there any other thing which I need to perform??
>>>>
>>>>
>>>> Thanks and Regards,
>>>> Nikita
>>>>
>>>> On Thu, Jan 25, 2018 at 2:13 PM, Karl Wright <da...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Nikita,
>>>>>
>>>>> You are getting the __nosecurity__ token value transmitted and stored,
>>>>> which means your Elastic Search setup is probably reasonable.
>>>>>
>>>>> Can you give us details about your pipeline? What repository
>>>>> connector is this?
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>> On Thu, Jan 25, 2018 at 1:49 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>>> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I am using ManifoldCF 2.8.1 and trying to access the files from
>>>>>> FileShare configured with the Active Directory and get the data in elastic
>>>>>> search index.
>>>>>>
>>>>>> For this steps followed are:
>>>>>>
>>>>>>
>>>>>> - An Authority Group is created under which Authority Connection
>>>>>> and Repository connection are created.
>>>>>>
>>>>>>
>>>>>> [image: Inline image 3]
>>>>>>
>>>>>> - And the elastic search is created as an Output Connector and a
>>>>>> job is created for the connection in which a transformation and metadata
>>>>>> connector is also linked in the
>>>>>> - Which is giving all the output except the access token values.
>>>>>> - [image: Inline image 1]
>>>>>>
>>>>>>
>>>>>>
>>>>>> Please guide to take appropriate steps for it so that I can get the
>>>>>> access tokens in the elastic search index. Hoping for the solution asap.
>>>>>>
>>>>>>
>>>>>> Thanks and Regards,
>>>>>> Nikita
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
Posted by Nikita Ahuja <ni...@smartshore.nl>.
- Hi Karl,
There is another query related to Sharepoint 2013 Connector. I am trying to
crawl the files of Sharepoint On Premise site. But I am not able to get
anything in the output either in ElasticSearch or in FileShare.
I have followed given steps:
- Created an Authority group for the Active Directory and same is used
in Sharepoint Repository Connection.
[image: Inline image 2]
- Installed the Sharepoint 2013 MCPermission Plugin and is accessible on
the site.
[image: Inline image 3]
- Still not able to access any document from the selected folder of
Sharepoint Site.
Please guide me accordingly.
Thanks and Regards,
Nikita
On Thu, Jan 25, 2018 at 6:46 PM, Nikita Ahuja <ni...@smartshore.nl> wrote:
> Thanks Karl,
>
> It is working fine now and Also I am able to get the access tokens in the
> elastic search index.
>
> Thanks a lot for your valuable suggestions.
>
> With Regards,
> Nikita
>
> On Thu, Jan 25, 2018 at 4:42 PM, Karl Wright <da...@gmail.com> wrote:
>
>> The __nosecurity__ token is generated by the Elastic Search Connector.
>> It is produced (correctly) when no acls reach the connector. This is why I
>> asked you about the repository connection, because that connection is not
>> producing any ACLs.
>>
>> If you are crawling files in a file system, you must use the JCIFS
>> connector if you want ACLs to be generated. The FileSystem connector does
>> not generate ACLs.
>>
>> Karl
>>
>>
>> On Thu, Jan 25, 2018 at 3:56 AM, Nikita Ahuja <ni...@smartshore.nl>
>> wrote:
>>
>>> Hi Karl,
>>>
>>>
>>> I have tried for ElasticSearch version 1.3.1 and 5.6.1 but still getting
>>> this " __nosecurity__" token and for Repository connection File Sytem
>>> is used and for Transformation "Tika Content Extractor" and Metadata
>>> adjuster Connector is used.
>>>
>>> [image: Inline image 1]
>>>
>>>
>>>
>>>
>>>
>>>
>>> Is there any other thing which I need to perform??
>>>
>>>
>>> Thanks and Regards,
>>> Nikita
>>>
>>> On Thu, Jan 25, 2018 at 2:13 PM, Karl Wright <da...@gmail.com> wrote:
>>>
>>>> Hi Nikita,
>>>>
>>>> You are getting the __nosecurity__ token value transmitted and stored,
>>>> which means your Elastic Search setup is probably reasonable.
>>>>
>>>> Can you give us details about your pipeline? What repository connector
>>>> is this?
>>>>
>>>> Karl
>>>>
>>>>
>>>> On Thu, Jan 25, 2018 at 1:49 AM, Nikita Ahuja <ni...@smartshore.nl>
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I am using ManifoldCF 2.8.1 and trying to access the files from
>>>>> FileShare configured with the Active Directory and get the data in elastic
>>>>> search index.
>>>>>
>>>>> For this steps followed are:
>>>>>
>>>>>
>>>>> - An Authority Group is created under which Authority Connection
>>>>> and Repository connection are created.
>>>>>
>>>>>
>>>>> [image: Inline image 3]
>>>>>
>>>>> - And the elastic search is created as an Output Connector and a
>>>>> job is created for the connection in which a transformation and metadata
>>>>> connector is also linked in the
>>>>> - Which is giving all the output except the access token values.
>>>>> - [image: Inline image 1]
>>>>>
>>>>>
>>>>>
>>>>> Please guide to take appropriate steps for it so that I can get the
>>>>> access tokens in the elastic search index. Hoping for the solution asap.
>>>>>
>>>>>
>>>>> Thanks and Regards,
>>>>> Nikita
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
Posted by Nikita Ahuja <ni...@smartshore.nl>.
Thanks Karl,
It is working fine now and Also I am able to get the access tokens in the
elastic search index.
Thanks a lot for your valuable suggestions.
With Regards,
Nikita
On Thu, Jan 25, 2018 at 4:42 PM, Karl Wright <da...@gmail.com> wrote:
> The __nosecurity__ token is generated by the Elastic Search Connector. It
> is produced (correctly) when no acls reach the connector. This is why I
> asked you about the repository connection, because that connection is not
> producing any ACLs.
>
> If you are crawling files in a file system, you must use the JCIFS
> connector if you want ACLs to be generated. The FileSystem connector does
> not generate ACLs.
>
> Karl
>
>
> On Thu, Jan 25, 2018 at 3:56 AM, Nikita Ahuja <ni...@smartshore.nl>
> wrote:
>
>> Hi Karl,
>>
>>
>> I have tried for ElasticSearch version 1.3.1 and 5.6.1 but still getting
>> this " __nosecurity__" token and for Repository connection File Sytem
>> is used and for Transformation "Tika Content Extractor" and Metadata
>> adjuster Connector is used.
>>
>> [image: Inline image 1]
>>
>>
>>
>>
>>
>>
>> Is there any other thing which I need to perform??
>>
>>
>> Thanks and Regards,
>> Nikita
>>
>> On Thu, Jan 25, 2018 at 2:13 PM, Karl Wright <da...@gmail.com> wrote:
>>
>>> Hi Nikita,
>>>
>>> You are getting the __nosecurity__ token value transmitted and stored,
>>> which means your Elastic Search setup is probably reasonable.
>>>
>>> Can you give us details about your pipeline? What repository connector
>>> is this?
>>>
>>> Karl
>>>
>>>
>>> On Thu, Jan 25, 2018 at 1:49 AM, Nikita Ahuja <ni...@smartshore.nl>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> I am using ManifoldCF 2.8.1 and trying to access the files from
>>>> FileShare configured with the Active Directory and get the data in elastic
>>>> search index.
>>>>
>>>> For this steps followed are:
>>>>
>>>>
>>>> - An Authority Group is created under which Authority Connection
>>>> and Repository connection are created.
>>>>
>>>>
>>>> [image: Inline image 3]
>>>>
>>>> - And the elastic search is created as an Output Connector and a
>>>> job is created for the connection in which a transformation and metadata
>>>> connector is also linked in the
>>>> - Which is giving all the output except the access token values.
>>>> - [image: Inline image 1]
>>>>
>>>>
>>>>
>>>> Please guide to take appropriate steps for it so that I can get the
>>>> access tokens in the elastic search index. Hoping for the solution asap.
>>>>
>>>>
>>>> Thanks and Regards,
>>>> Nikita
>>>>
>>>
>>>
>>
>
Re: Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
Posted by Karl Wright <da...@gmail.com>.
The __nosecurity__ token is generated by the Elastic Search Connector. It
is produced (correctly) when no acls reach the connector. This is why I
asked you about the repository connection, because that connection is not
producing any ACLs.
If you are crawling files in a file system, you must use the JCIFS
connector if you want ACLs to be generated. The FileSystem connector does
not generate ACLs.
Karl
On Thu, Jan 25, 2018 at 3:56 AM, Nikita Ahuja <ni...@smartshore.nl> wrote:
> Hi Karl,
>
>
> I have tried for ElasticSearch version 1.3.1 and 5.6.1 but still getting
> this " __nosecurity__" token and for Repository connection File Sytem is
> used and for Transformation "Tika Content Extractor" and Metadata
> adjuster Connector is used.
>
> [image: Inline image 1]
>
>
>
>
>
>
> Is there any other thing which I need to perform??
>
>
> Thanks and Regards,
> Nikita
>
> On Thu, Jan 25, 2018 at 2:13 PM, Karl Wright <da...@gmail.com> wrote:
>
>> Hi Nikita,
>>
>> You are getting the __nosecurity__ token value transmitted and stored,
>> which means your Elastic Search setup is probably reasonable.
>>
>> Can you give us details about your pipeline? What repository connector
>> is this?
>>
>> Karl
>>
>>
>> On Thu, Jan 25, 2018 at 1:49 AM, Nikita Ahuja <ni...@smartshore.nl>
>> wrote:
>>
>>> Hello,
>>>
>>> I am using ManifoldCF 2.8.1 and trying to access the files from
>>> FileShare configured with the Active Directory and get the data in elastic
>>> search index.
>>>
>>> For this steps followed are:
>>>
>>>
>>> - An Authority Group is created under which Authority Connection and
>>> Repository connection are created.
>>>
>>>
>>> [image: Inline image 3]
>>>
>>> - And the elastic search is created as an Output Connector and a job
>>> is created for the connection in which a transformation and metadata
>>> connector is also linked in the
>>> - Which is giving all the output except the access token values.
>>> - [image: Inline image 1]
>>>
>>>
>>>
>>> Please guide to take appropriate steps for it so that I can get the
>>> access tokens in the elastic search index. Hoping for the solution asap.
>>>
>>>
>>> Thanks and Regards,
>>> Nikita
>>>
>>
>>
>
Re: Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
Posted by Nikita Ahuja <ni...@smartshore.nl>.
Hi Karl,
I have tried for ElasticSearch version 1.3.1 and 5.6.1 but still getting
this " __nosecurity__" token and for Repository connection File Sytem is
used and for Transformation "Tika Content Extractor" and Metadata
adjuster Connector
is used.
[image: Inline image 1]
Is there any other thing which I need to perform??
Thanks and Regards,
Nikita
On Thu, Jan 25, 2018 at 2:13 PM, Karl Wright <da...@gmail.com> wrote:
> Hi Nikita,
>
> You are getting the __nosecurity__ token value transmitted and stored,
> which means your Elastic Search setup is probably reasonable.
>
> Can you give us details about your pipeline? What repository connector is
> this?
>
> Karl
>
>
> On Thu, Jan 25, 2018 at 1:49 AM, Nikita Ahuja <ni...@smartshore.nl>
> wrote:
>
>> Hello,
>>
>> I am using ManifoldCF 2.8.1 and trying to access the files from FileShare
>> configured with the Active Directory and get the data in elastic search
>> index.
>>
>> For this steps followed are:
>>
>>
>> - An Authority Group is created under which Authority Connection and
>> Repository connection are created.
>>
>>
>> [image: Inline image 3]
>>
>> - And the elastic search is created as an Output Connector and a job
>> is created for the connection in which a transformation and metadata
>> connector is also linked in the
>> - Which is giving all the output except the access token values.
>> - [image: Inline image 1]
>>
>>
>>
>> Please guide to take appropriate steps for it so that I can get the
>> access tokens in the elastic search index. Hoping for the solution asap.
>>
>>
>> Thanks and Regards,
>> Nikita
>>
>
>
Re: Problem in fetching the access tokens from Active Directory in
Elastic Search index for FileSystem Connector
Posted by Karl Wright <da...@gmail.com>.
Hi Nikita,
You are getting the __nosecurity__ token value transmitted and stored,
which means your Elastic Search setup is probably reasonable.
Can you give us details about your pipeline? What repository connector is
this?
Karl
On Thu, Jan 25, 2018 at 1:49 AM, Nikita Ahuja <ni...@smartshore.nl> wrote:
> Hello,
>
> I am using ManifoldCF 2.8.1 and trying to access the files from FileShare
> configured with the Active Directory and get the data in elastic search
> index.
>
> For this steps followed are:
>
>
> - An Authority Group is created under which Authority Connection and
> Repository connection are created.
>
>
> [image: Inline image 3]
>
> - And the elastic search is created as an Output Connector and a job
> is created for the connection in which a transformation and metadata
> connector is also linked in the
> - Which is giving all the output except the access token values.
> - [image: Inline image 1]
>
>
>
> Please guide to take appropriate steps for it so that I can get the access
> tokens in the elastic search index. Hoping for the solution asap.
>
>
> Thanks and Regards,
> Nikita
>