You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@manifoldcf.apache.org by Alejandro Calbazana <ac...@gmail.com> on 2014/10/30 17:37:30 UTC
JDBC Repository Connector + Access Token Question
Hello,
I'm getting started with ManifoldCF and I'm scratching my head a little
with how to get access tokens into my indexed content using a JDBC
repository. Here is my setup:
- JDBC repository connector, indicating which authority group to use
- Solr output connector
- Auth connector, specifying user query and access token query
- Job tying the connectors together indicating the queries for content IDs
and content data
Where do I instruct Manifold on how where obtain access tokens for the
content that it is crawling? I see that there is a "security" tab in the
job setup that allows me to specify access tokens, but this is static. My
content will have different tokens associated with it per row. How do I
accomplish this? Is this a place where I need to add custom code to fetch
the authorization detail and apply it to content?
Any guidance is greatly appreciated!
Thanks,
Alejandro
Re: JDBC Repository Connector + Access Token Question
Posted by Alejandro Calbazana <ac...@gmail.com>.
Hi Karl,
Thanks for the response and the patch! I pulled this down, built,
deployed, configured, and ran it. I see the new fields for the ACL query
and supplied some test data. Unfortunately, I don't see anything in the
output document. My setup is similar to my earlier post with the exception
of the added access token query. I tried adding a static token to the job,
and that does not show up when I use the access token query. I did define
a new job to test static tokens, and that seems to work okay, so I'm not
sure of the issue.
Your help is appreciated!
Thanks,
Alejandro
On Fri, Oct 31, 2014 at 8:55 AM, Karl Wright <da...@gmail.com> wrote:
> Hi Alejandro,
>
> I've completed development of this feature. Remember that this work is
> currently based on unreleased MCF 2.0 code, so you will want to try it out
> in a way that does not impact any existing instances of MCF you may have
> running. To try it out, please do the following:
>
> (1) Check out the branch:
>
> svn co
> https://svn.apache.org/repos/asf/manifoldcf/branches/CONNECTORS-1089
>
> (2) Build the branch:
>
> cd CONNECTORS-1089
> ant make-core-deps build
>
> (3) Start the simple example:
>
> cd dist/example
> start.bat (or start.sh)
>
> (4) Go to the UI and create your connections and jobs. Then crawl.
>
> Please let me know if you find any problems. If everything looks good, I
> will merge this branch back to trunk and to the dev_1x branch.
>
>
> Thanks!
> Karl
>
>
> On Thu, Oct 30, 2014 at 1:38 PM, Alejandro Calbazana <acalbazana@gmail.com
> > wrote:
>
>> Hi Karl,
>>
>> Thanks for the quick reply.
>>
>> (a) What authority(s) do you hope to use to secure JDBC-based document
>> content?
>> I was hoping to tie together 3 authorities in one group. One for user
>> level ACLs, one for roles, and another for "group" level authorization
>> (although its possible that group could be a variation of a role in my
>> context). All of these would be JDBC bound. Please let me know if I missed
>> your question.
>>
>> (b) Do you need support for both "allow" and "deny" ACLs?
>> No. Allow meets my use case.
>>
>> (c) Are there multiple levels of ACL required (e.g. document, folder,
>> share)?
>> No. Just documents.
>>
>> (d) What is your preferred schema for access tokens in the database? a
>> joined table, or a delimiter-separated list?
>> A joined table would be preferable.
>>
>> Thanks,
>>
>> Alejandro
>>
>>
>> On Thu, Oct 30, 2014 at 12:55 PM, Karl Wright <da...@gmail.com> wrote:
>>
>>> Hi Alejandro,
>>>
>>> The reason you are confused about document security in JDBC is because,
>>> heretofore, nobody has implemented document security using JDBC. We've
>>> been waiting for quite a while for a user who needs this functionality, so
>>> that we develop it properly.
>>>
>>> If you are hoping to get security tokens from each document row in the
>>> database, then we should talk further and I will develop a patch for you
>>> that works in a way that is consistent with your problem. I've created a
>>> ticket, CONNECTORS-1089, for tracking work on this feature.
>>>
>>> For a start, can you let me know the following:
>>>
>>> (a) What authority(s) do you hope to use to secure JDBC-based document
>>> content?
>>> (b) Do you need support for both "allow" and "deny" ACLs?
>>> (c) Are there multiple levels of ACL required (e.g. document, folder,
>>> share)?
>>> (d) What is your preferred schema for access tokens in the database? a
>>> joined table, or a delimiter-separated list?
>>>
>>> Thanks,
>>> Karl
>>>
>>>
>>> On Thu, Oct 30, 2014 at 12:37 PM, Alejandro Calbazana <
>>> acalbazana@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I'm getting started with ManifoldCF and I'm scratching my head a little
>>>> with how to get access tokens into my indexed content using a JDBC
>>>> repository. Here is my setup:
>>>>
>>>> - JDBC repository connector, indicating which authority group to use
>>>> - Solr output connector
>>>> - Auth connector, specifying user query and access token query
>>>> - Job tying the connectors together indicating the queries for content
>>>> IDs and content data
>>>>
>>>> Where do I instruct Manifold on how where obtain access tokens for the
>>>> content that it is crawling? I see that there is a "security" tab in the
>>>> job setup that allows me to specify access tokens, but this is static. My
>>>> content will have different tokens associated with it per row. How do I
>>>> accomplish this? Is this a place where I need to add custom code to fetch
>>>> the authorization detail and apply it to content?
>>>>
>>>> Any guidance is greatly appreciated!
>>>>
>>>> Thanks,
>>>>
>>>> Alejandro
>>>>
>>>>
>>>>
>>>
>>
>
Re: JDBC Repository Connector + Access Token Question
Posted by Karl Wright <da...@gmail.com>.
Hi Alejandro,
I've completed development of this feature. Remember that this work is
currently based on unreleased MCF 2.0 code, so you will want to try it out
in a way that does not impact any existing instances of MCF you may have
running. To try it out, please do the following:
(1) Check out the branch:
svn co https://svn.apache.org/repos/asf/manifoldcf/branches/CONNECTORS-1089
(2) Build the branch:
cd CONNECTORS-1089
ant make-core-deps build
(3) Start the simple example:
cd dist/example
start.bat (or start.sh)
(4) Go to the UI and create your connections and jobs. Then crawl.
Please let me know if you find any problems. If everything looks good, I
will merge this branch back to trunk and to the dev_1x branch.
Thanks!
Karl
On Thu, Oct 30, 2014 at 1:38 PM, Alejandro Calbazana <ac...@gmail.com>
wrote:
> Hi Karl,
>
> Thanks for the quick reply.
>
> (a) What authority(s) do you hope to use to secure JDBC-based document
> content?
> I was hoping to tie together 3 authorities in one group. One for user
> level ACLs, one for roles, and another for "group" level authorization
> (although its possible that group could be a variation of a role in my
> context). All of these would be JDBC bound. Please let me know if I missed
> your question.
>
> (b) Do you need support for both "allow" and "deny" ACLs?
> No. Allow meets my use case.
>
> (c) Are there multiple levels of ACL required (e.g. document, folder,
> share)?
> No. Just documents.
>
> (d) What is your preferred schema for access tokens in the database? a
> joined table, or a delimiter-separated list?
> A joined table would be preferable.
>
> Thanks,
>
> Alejandro
>
>
> On Thu, Oct 30, 2014 at 12:55 PM, Karl Wright <da...@gmail.com> wrote:
>
>> Hi Alejandro,
>>
>> The reason you are confused about document security in JDBC is because,
>> heretofore, nobody has implemented document security using JDBC. We've
>> been waiting for quite a while for a user who needs this functionality, so
>> that we develop it properly.
>>
>> If you are hoping to get security tokens from each document row in the
>> database, then we should talk further and I will develop a patch for you
>> that works in a way that is consistent with your problem. I've created a
>> ticket, CONNECTORS-1089, for tracking work on this feature.
>>
>> For a start, can you let me know the following:
>>
>> (a) What authority(s) do you hope to use to secure JDBC-based document
>> content?
>> (b) Do you need support for both "allow" and "deny" ACLs?
>> (c) Are there multiple levels of ACL required (e.g. document, folder,
>> share)?
>> (d) What is your preferred schema for access tokens in the database? a
>> joined table, or a delimiter-separated list?
>>
>> Thanks,
>> Karl
>>
>>
>> On Thu, Oct 30, 2014 at 12:37 PM, Alejandro Calbazana <
>> acalbazana@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I'm getting started with ManifoldCF and I'm scratching my head a little
>>> with how to get access tokens into my indexed content using a JDBC
>>> repository. Here is my setup:
>>>
>>> - JDBC repository connector, indicating which authority group to use
>>> - Solr output connector
>>> - Auth connector, specifying user query and access token query
>>> - Job tying the connectors together indicating the queries for content
>>> IDs and content data
>>>
>>> Where do I instruct Manifold on how where obtain access tokens for the
>>> content that it is crawling? I see that there is a "security" tab in the
>>> job setup that allows me to specify access tokens, but this is static. My
>>> content will have different tokens associated with it per row. How do I
>>> accomplish this? Is this a place where I need to add custom code to fetch
>>> the authorization detail and apply it to content?
>>>
>>> Any guidance is greatly appreciated!
>>>
>>> Thanks,
>>>
>>> Alejandro
>>>
>>>
>>>
>>
>
Re: JDBC Repository Connector + Access Token Question
Posted by Alejandro Calbazana <ac...@gmail.com>.
Hi Karl,
Thanks for the quick reply.
(a) What authority(s) do you hope to use to secure JDBC-based document
content?
I was hoping to tie together 3 authorities in one group. One for user
level ACLs, one for roles, and another for "group" level authorization
(although its possible that group could be a variation of a role in my
context). All of these would be JDBC bound. Please let me know if I missed
your question.
(b) Do you need support for both "allow" and "deny" ACLs?
No. Allow meets my use case.
(c) Are there multiple levels of ACL required (e.g. document, folder,
share)?
No. Just documents.
(d) What is your preferred schema for access tokens in the database? a
joined table, or a delimiter-separated list?
A joined table would be preferable.
Thanks,
Alejandro
On Thu, Oct 30, 2014 at 12:55 PM, Karl Wright <da...@gmail.com> wrote:
> Hi Alejandro,
>
> The reason you are confused about document security in JDBC is because,
> heretofore, nobody has implemented document security using JDBC. We've
> been waiting for quite a while for a user who needs this functionality, so
> that we develop it properly.
>
> If you are hoping to get security tokens from each document row in the
> database, then we should talk further and I will develop a patch for you
> that works in a way that is consistent with your problem. I've created a
> ticket, CONNECTORS-1089, for tracking work on this feature.
>
> For a start, can you let me know the following:
>
> (a) What authority(s) do you hope to use to secure JDBC-based document
> content?
> (b) Do you need support for both "allow" and "deny" ACLs?
> (c) Are there multiple levels of ACL required (e.g. document, folder,
> share)?
> (d) What is your preferred schema for access tokens in the database? a
> joined table, or a delimiter-separated list?
>
> Thanks,
> Karl
>
>
> On Thu, Oct 30, 2014 at 12:37 PM, Alejandro Calbazana <
> acalbazana@gmail.com> wrote:
>
>> Hello,
>>
>> I'm getting started with ManifoldCF and I'm scratching my head a little
>> with how to get access tokens into my indexed content using a JDBC
>> repository. Here is my setup:
>>
>> - JDBC repository connector, indicating which authority group to use
>> - Solr output connector
>> - Auth connector, specifying user query and access token query
>> - Job tying the connectors together indicating the queries for content
>> IDs and content data
>>
>> Where do I instruct Manifold on how where obtain access tokens for the
>> content that it is crawling? I see that there is a "security" tab in the
>> job setup that allows me to specify access tokens, but this is static. My
>> content will have different tokens associated with it per row. How do I
>> accomplish this? Is this a place where I need to add custom code to fetch
>> the authorization detail and apply it to content?
>>
>> Any guidance is greatly appreciated!
>>
>> Thanks,
>>
>> Alejandro
>>
>>
>>
>
Re: JDBC Repository Connector + Access Token Question
Posted by Karl Wright <da...@gmail.com>.
Hi Alejandro,
The reason you are confused about document security in JDBC is because,
heretofore, nobody has implemented document security using JDBC. We've
been waiting for quite a while for a user who needs this functionality, so
that we develop it properly.
If you are hoping to get security tokens from each document row in the
database, then we should talk further and I will develop a patch for you
that works in a way that is consistent with your problem. I've created a
ticket, CONNECTORS-1089, for tracking work on this feature.
For a start, can you let me know the following:
(a) What authority(s) do you hope to use to secure JDBC-based document
content?
(b) Do you need support for both "allow" and "deny" ACLs?
(c) Are there multiple levels of ACL required (e.g. document, folder,
share)?
(d) What is your preferred schema for access tokens in the database? a
joined table, or a delimiter-separated list?
Thanks,
Karl
On Thu, Oct 30, 2014 at 12:37 PM, Alejandro Calbazana <ac...@gmail.com>
wrote:
> Hello,
>
> I'm getting started with ManifoldCF and I'm scratching my head a little
> with how to get access tokens into my indexed content using a JDBC
> repository. Here is my setup:
>
> - JDBC repository connector, indicating which authority group to use
> - Solr output connector
> - Auth connector, specifying user query and access token query
> - Job tying the connectors together indicating the queries for content IDs
> and content data
>
> Where do I instruct Manifold on how where obtain access tokens for the
> content that it is crawling? I see that there is a "security" tab in the
> job setup that allows me to specify access tokens, but this is static. My
> content will have different tokens associated with it per row. How do I
> accomplish this? Is this a place where I need to add custom code to fetch
> the authorization detail and apply it to content?
>
> Any guidance is greatly appreciated!
>
> Thanks,
>
> Alejandro
>
>
>