You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@manifoldcf.apache.org by "Jack Krupansky (JIRA)" <ji...@apache.org> on 2010/07/09 00:27:49 UTC
[jira] Created: (CONNECTORS-56) All features should be accessible
through an API
All features should be accessible through an API
------------------------------------------------
Key: CONNECTORS-56
URL: https://issues.apache.org/jira/browse/CONNECTORS-56
Project: Lucene Connector Framework
Issue Type: Improvement
Components: Framework core
Reporter: Jack Krupansky
LCF consists of a full-featured crawling engine and a full-featured user interface to access the features of that engine, but some applications are better served with a full API that lets the application control the crawling engine, including creation and editing of connections and creation, editing, and control of jobs. Put simply, everything that a user can accomplish via the LCF UI should be doable through an LCF API. All LCF objects should be queryable through the API.
A primary use case is Solr applications which currently use Aperture for crawling, but would prefer the full-featured capabilities of LCF as a crawling engine over Aperture.
I do not wish to over-specify the API in this initial description, but I think the LCF API should probably be a traditional REST API., with some of the API elements specified via the context path, some parameters via URL query parameters, and complex, detailed structures as JSON (or similar.). The precise details of the API are beyond the scope of this initial description and will be added incrementally once the high-level approach to the API becomes reasonably settled.
A job status and event reporting scheme is also needed in conjunction with the LCF API. That requirement has already been captured as CONNECTORS-41.
The intention for the API is to create, edit, access, and control all of the objects managed by LCF. The main focus is on repositories, jobs, and status, and less about document-specific crawling information, but there may be some benefit to querying crawling status for individual documents as well.
Nothing in this proposal should in any way limit or constrain the features that will be available in the LCF UI. The intent is that LCF should continue to have a full-featured UI, but in addition to a full-featured API.
Note: This issue is part of Phase 2 of the CONNECTORS-50 umbrella issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CONNECTORS-56) All features should be accessible
through an API
Posted by "Jack Krupansky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-56?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888377#action_12888377 ]
Jack Krupansky commented on CONNECTORS-56:
------------------------------------------
Some cURL and/or Perl test scripts to illustrate use of the API would be helpful.
> All features should be accessible through an API
> ------------------------------------------------
>
> Key: CONNECTORS-56
> URL: https://issues.apache.org/jira/browse/CONNECTORS-56
> Project: Lucene Connector Framework
> Issue Type: Sub-task
> Components: Framework core
> Reporter: Jack Krupansky
>
> LCF consists of a full-featured crawling engine and a full-featured user interface to access the features of that engine, but some applications are better served with a full API that lets the application control the crawling engine, including creation and editing of connections and creation, editing, and control of jobs. Put simply, everything that a user can accomplish via the LCF UI should be doable through an LCF API. All LCF objects should be queryable through the API.
> A primary use case is Solr applications which currently use Aperture for crawling, but would prefer the full-featured capabilities of LCF as a crawling engine over Aperture.
> I do not wish to over-specify the API in this initial description, but I think the LCF API should probably be a traditional REST API., with some of the API elements specified via the context path, some parameters via URL query parameters, and complex, detailed structures as JSON (or similar.). The precise details of the API are beyond the scope of this initial description and will be added incrementally once the high-level approach to the API becomes reasonably settled.
> A job status and event reporting scheme is also needed in conjunction with the LCF API. That requirement has already been captured as CONNECTORS-41.
> The intention for the API is to create, edit, access, and control all of the objects managed by LCF. The main focus is on repositories, jobs, and status, and less about document-specific crawling information, but there may be some benefit to querying crawling status for individual documents as well.
> Nothing in this proposal should in any way limit or constrain the features that will be available in the LCF UI. The intent is that LCF should continue to have a full-featured UI, but in addition to a full-featured API.
> Note: This issue is part of Phase 2 of the CONNECTORS-50 umbrella issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CONNECTORS-56) All features should be accessible
through an API
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-56?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888508#action_12888508 ]
Karl Wright commented on CONNECTORS-56:
---------------------------------------
Checked in code which I claim constitutes the 0.5 version of the API. This includes complete json-based operations for the following:
- /job/list
- /job/get
- /job/save
- /job/delete
- /jobstatus/list
- /jobstatus/start
- /jobstatus/abort
- /jobstatus/restart
- /jobstatus/pause
- /jobstatus/resume
- /outputconnection/list
- /outputconnection/get
- /outputconnection/save
- /outputconnection/delete
- /outputconnection/checkstatus
- /repositoryconnection/list
- /repositoryconnection/get
- /repositoryconnection/save
- /repositoryconnection/delete
- /repositoryconnection/checkstatus
- /authorityconnection/list
- /authorityconnection/get
- /authorityconnection/save
- /authorityconnection/delete
- /authorityconnection/checkstatus
Some minor immediate refinements are possible, specifically around the way error handling is done. Currently, errors that toss exceptions cause a 500 error (with some helpful text). It may be that it's better to get the exception text back as part of the JSON object being returned. Comments/preferences welcome.
Also not yet implemented are the following methods:
- /report/documentstatus
- /report/queuestatus
- /report/simplehistory
- /report/maximumbandwidth
- /report/maximumactivity
- /report/resultsummary
These I'd propose to hold of on right now.
I've created a separate ticket for documentation/examples. I'm still intending to create some tests to exercise at least major portions of the API before closing THIS ticket.
> All features should be accessible through an API
> ------------------------------------------------
>
> Key: CONNECTORS-56
> URL: https://issues.apache.org/jira/browse/CONNECTORS-56
> Project: Lucene Connector Framework
> Issue Type: Sub-task
> Components: Framework core
> Reporter: Jack Krupansky
>
> LCF consists of a full-featured crawling engine and a full-featured user interface to access the features of that engine, but some applications are better served with a full API that lets the application control the crawling engine, including creation and editing of connections and creation, editing, and control of jobs. Put simply, everything that a user can accomplish via the LCF UI should be doable through an LCF API. All LCF objects should be queryable through the API.
> A primary use case is Solr applications which currently use Aperture for crawling, but would prefer the full-featured capabilities of LCF as a crawling engine over Aperture.
> I do not wish to over-specify the API in this initial description, but I think the LCF API should probably be a traditional REST API., with some of the API elements specified via the context path, some parameters via URL query parameters, and complex, detailed structures as JSON (or similar.). The precise details of the API are beyond the scope of this initial description and will be added incrementally once the high-level approach to the API becomes reasonably settled.
> A job status and event reporting scheme is also needed in conjunction with the LCF API. That requirement has already been captured as CONNECTORS-41.
> The intention for the API is to create, edit, access, and control all of the objects managed by LCF. The main focus is on repositories, jobs, and status, and less about document-specific crawling information, but there may be some benefit to querying crawling status for individual documents as well.
> Nothing in this proposal should in any way limit or constrain the features that will be available in the LCF UI. The intent is that LCF should continue to have a full-featured UI, but in addition to a full-featured API.
> Note: This issue is part of Phase 2 of the CONNECTORS-50 umbrella issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CONNECTORS-56) All features should be accessible
through an API
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-56?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889251#action_12889251 ]
Karl Wright commented on CONNECTORS-56:
---------------------------------------
HTTP methods other than GET or PUT are in fact poorly supported in many HTTP clients, including Apache Commons HTTPClient. I am also unsure of whether Jetty supports the DELETE method at the servlet level. I therefore think your suggestion would potentially cause a great deal of headache for no tangible benefit.
> All features should be accessible through an API
> ------------------------------------------------
>
> Key: CONNECTORS-56
> URL: https://issues.apache.org/jira/browse/CONNECTORS-56
> Project: Lucene Connector Framework
> Issue Type: Sub-task
> Components: Framework core
> Reporter: Jack Krupansky
> Assignee: Karl Wright
>
> LCF consists of a full-featured crawling engine and a full-featured user interface to access the features of that engine, but some applications are better served with a full API that lets the application control the crawling engine, including creation and editing of connections and creation, editing, and control of jobs. Put simply, everything that a user can accomplish via the LCF UI should be doable through an LCF API. All LCF objects should be queryable through the API.
> A primary use case is Solr applications which currently use Aperture for crawling, but would prefer the full-featured capabilities of LCF as a crawling engine over Aperture.
> I do not wish to over-specify the API in this initial description, but I think the LCF API should probably be a traditional REST API., with some of the API elements specified via the context path, some parameters via URL query parameters, and complex, detailed structures as JSON (or similar.). The precise details of the API are beyond the scope of this initial description and will be added incrementally once the high-level approach to the API becomes reasonably settled.
> A job status and event reporting scheme is also needed in conjunction with the LCF API. That requirement has already been captured as CONNECTORS-41.
> The intention for the API is to create, edit, access, and control all of the objects managed by LCF. The main focus is on repositories, jobs, and status, and less about document-specific crawling information, but there may be some benefit to querying crawling status for individual documents as well.
> Nothing in this proposal should in any way limit or constrain the features that will be available in the LCF UI. The intent is that LCF should continue to have a full-featured UI, but in addition to a full-featured API.
> Note: This issue is part of Phase 2 of the CONNECTORS-50 umbrella issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CONNECTORS-56) All features should be accessible
through an API
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-56?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886744#action_12886744 ]
Karl Wright commented on CONNECTORS-56:
---------------------------------------
The general approach for an API that I'd suggest, which would be completely compatibly with the Quick Start version of LCF, would basically consist of its own web application. The web application would consist completely of a servlet, which would interpret path and argument information as commands and posted command arguments, respectively. A separate web application would permit users to control access to the API using standard application server access management mechanisms.
All commands would consist of HTTP posts that address the API servlet. The commands themselves would come in broad categories, and be specified as part of the "path", as follows:
job/find - get existing job information
job/start - start a job
job/abort - abort a job
job/restart - restart a job
job/delete - delete a job
job/save - save a job
...
report/simplehistory - generate a simple history report
report/documentstatus - generate a document status report
...
etc.
As for arguments and return values, my sense is that have a choice of either XML or JSON here. It's not clear which is better, but my preference would be JSON, because of its relative simplicity, and because otherwise people will be tempted to want us to use full SOAP, with WSDLs etc. That would add a lot of overhead to the solution, in my opinion.
The API commands would be stateless, in that there would be no explicit or implicit session created. All arguments are therefore explicit.
Connection configuration, document specification, and output specification information is the main problem. This information is easily mappable to unstructured XML, and is managed by LCF in this way. The contents of the XML is determined wholly by the involved connector code, and is therefore opaque as far as the API is concerned. Embedding such XML as a JSON field is certainly possible, or it would even be possible to convert it to embedded or nested JSON. It's really not clear to me what the best approach is here, although embedded JSON would require fewer moving parts in the client.
If this approach is to fly, somebody will need to document these opaque configuration structures, which will mean that these structures must remain backwards compatible as they evolve.
Thoughts?
> All features should be accessible through an API
> ------------------------------------------------
>
> Key: CONNECTORS-56
> URL: https://issues.apache.org/jira/browse/CONNECTORS-56
> Project: Lucene Connector Framework
> Issue Type: Improvement
> Components: Framework core
> Reporter: Jack Krupansky
>
> LCF consists of a full-featured crawling engine and a full-featured user interface to access the features of that engine, but some applications are better served with a full API that lets the application control the crawling engine, including creation and editing of connections and creation, editing, and control of jobs. Put simply, everything that a user can accomplish via the LCF UI should be doable through an LCF API. All LCF objects should be queryable through the API.
> A primary use case is Solr applications which currently use Aperture for crawling, but would prefer the full-featured capabilities of LCF as a crawling engine over Aperture.
> I do not wish to over-specify the API in this initial description, but I think the LCF API should probably be a traditional REST API., with some of the API elements specified via the context path, some parameters via URL query parameters, and complex, detailed structures as JSON (or similar.). The precise details of the API are beyond the scope of this initial description and will be added incrementally once the high-level approach to the API becomes reasonably settled.
> A job status and event reporting scheme is also needed in conjunction with the LCF API. That requirement has already been captured as CONNECTORS-41.
> The intention for the API is to create, edit, access, and control all of the objects managed by LCF. The main focus is on repositories, jobs, and status, and less about document-specific crawling information, but there may be some benefit to querying crawling status for individual documents as well.
> Nothing in this proposal should in any way limit or constrain the features that will be available in the LCF UI. The intent is that LCF should continue to have a full-featured UI, but in addition to a full-featured API.
> Note: This issue is part of Phase 2 of the CONNECTORS-50 umbrella issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CONNECTORS-56) All features should be accessible
through an API
Posted by "Jack Krupansky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-56?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889237#action_12889237 ]
Jack Krupansky commented on CONNECTORS-56:
------------------------------------------
It has come to my attention that the API would be more "pure" RESTful if the API verb was represented using the HTTP GET/PUT/DELETE verb and the input argument identifier represented in the context path.
So, GET outputconnection/get \{"connection_name":_<connection_name>_\} would be GET outputconnections/<connection_name>
and GET outputconnection/delete \{"connection_name":_<connection_name>_\} would be DELETE outputconnections/<connection_name>
and GET outputconnection/list would be GET outputconnections
and PUT outputconnection/save \{"outputconnection":_<output_connection_object>_\} would be PUT outputconnections/<connection_name> \{"outputconnection":_<output_connection_object>_\}
What we have today is certainly workable, but just not as "pure" as some might desire.
I am not going to classify this as a required issue just yet, but this would be a great time to change things before the API gets cast in concrete.
Comments?
> All features should be accessible through an API
> ------------------------------------------------
>
> Key: CONNECTORS-56
> URL: https://issues.apache.org/jira/browse/CONNECTORS-56
> Project: Lucene Connector Framework
> Issue Type: Sub-task
> Components: Framework core
> Reporter: Jack Krupansky
> Assignee: Karl Wright
>
> LCF consists of a full-featured crawling engine and a full-featured user interface to access the features of that engine, but some applications are better served with a full API that lets the application control the crawling engine, including creation and editing of connections and creation, editing, and control of jobs. Put simply, everything that a user can accomplish via the LCF UI should be doable through an LCF API. All LCF objects should be queryable through the API.
> A primary use case is Solr applications which currently use Aperture for crawling, but would prefer the full-featured capabilities of LCF as a crawling engine over Aperture.
> I do not wish to over-specify the API in this initial description, but I think the LCF API should probably be a traditional REST API., with some of the API elements specified via the context path, some parameters via URL query parameters, and complex, detailed structures as JSON (or similar.). The precise details of the API are beyond the scope of this initial description and will be added incrementally once the high-level approach to the API becomes reasonably settled.
> A job status and event reporting scheme is also needed in conjunction with the LCF API. That requirement has already been captured as CONNECTORS-41.
> The intention for the API is to create, edit, access, and control all of the objects managed by LCF. The main focus is on repositories, jobs, and status, and less about document-specific crawling information, but there may be some benefit to querying crawling status for individual documents as well.
> Nothing in this proposal should in any way limit or constrain the features that will be available in the LCF UI. The intent is that LCF should continue to have a full-featured UI, but in addition to a full-featured API.
> Note: This issue is part of Phase 2 of the CONNECTORS-50 umbrella issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CONNECTORS-56) All features should be accessible
through an API
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-56?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902940#action_12902940 ]
Mark Miller commented on CONNECTORS-56:
---------------------------------------
bq. HTTP methods other than GET or PUT are in fact poorly supported in many HTTP clients, including Apache Commons HTTPClient.
That's untrue.
bq. I am also unsure of whether Jetty supports the DELETE method at the servlet level.
Jetty has no issues with DELETE, POST, PUT, or GET. Nor does Tomcat or any other container I have seen.
bq. I therefore think your suggestion would potentially cause a great deal of headache for no tangible benefit.
Again, I don't agree - it would cause less headaches, as REST is somewhat of a standard rather than an ad hoc api. There are many advantages to having a consistent RESTful api.
> All features should be accessible through an API
> ------------------------------------------------
>
> Key: CONNECTORS-56
> URL: https://issues.apache.org/jira/browse/CONNECTORS-56
> Project: Apache Connectors Framework
> Issue Type: Sub-task
> Components: Framework core
> Reporter: Jack Krupansky
> Assignee: Karl Wright
>
> LCF consists of a full-featured crawling engine and a full-featured user interface to access the features of that engine, but some applications are better served with a full API that lets the application control the crawling engine, including creation and editing of connections and creation, editing, and control of jobs. Put simply, everything that a user can accomplish via the LCF UI should be doable through an LCF API. All LCF objects should be queryable through the API.
> A primary use case is Solr applications which currently use Aperture for crawling, but would prefer the full-featured capabilities of LCF as a crawling engine over Aperture.
> I do not wish to over-specify the API in this initial description, but I think the LCF API should probably be a traditional REST API., with some of the API elements specified via the context path, some parameters via URL query parameters, and complex, detailed structures as JSON (or similar.). The precise details of the API are beyond the scope of this initial description and will be added incrementally once the high-level approach to the API becomes reasonably settled.
> A job status and event reporting scheme is also needed in conjunction with the LCF API. That requirement has already been captured as CONNECTORS-41.
> The intention for the API is to create, edit, access, and control all of the objects managed by LCF. The main focus is on repositories, jobs, and status, and less about document-specific crawling information, but there may be some benefit to querying crawling status for individual documents as well.
> Nothing in this proposal should in any way limit or constrain the features that will be available in the LCF UI. The intent is that LCF should continue to have a full-featured UI, but in addition to a full-featured API.
> Note: This issue is part of Phase 2 of the CONNECTORS-50 umbrella issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CONNECTORS-56) All features should be accessible
through an API
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-56?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karl Wright resolved CONNECTORS-56.
-----------------------------------
Assignee: Karl Wright
Resolution: Fixed
I've created tests, and a separate ticket for API documentation and for adding report functions to the API. So I'm going to close this one, and use the other tickets to track those other issues.
> All features should be accessible through an API
> ------------------------------------------------
>
> Key: CONNECTORS-56
> URL: https://issues.apache.org/jira/browse/CONNECTORS-56
> Project: Lucene Connector Framework
> Issue Type: Sub-task
> Components: Framework core
> Reporter: Jack Krupansky
> Assignee: Karl Wright
>
> LCF consists of a full-featured crawling engine and a full-featured user interface to access the features of that engine, but some applications are better served with a full API that lets the application control the crawling engine, including creation and editing of connections and creation, editing, and control of jobs. Put simply, everything that a user can accomplish via the LCF UI should be doable through an LCF API. All LCF objects should be queryable through the API.
> A primary use case is Solr applications which currently use Aperture for crawling, but would prefer the full-featured capabilities of LCF as a crawling engine over Aperture.
> I do not wish to over-specify the API in this initial description, but I think the LCF API should probably be a traditional REST API., with some of the API elements specified via the context path, some parameters via URL query parameters, and complex, detailed structures as JSON (or similar.). The precise details of the API are beyond the scope of this initial description and will be added incrementally once the high-level approach to the API becomes reasonably settled.
> A job status and event reporting scheme is also needed in conjunction with the LCF API. That requirement has already been captured as CONNECTORS-41.
> The intention for the API is to create, edit, access, and control all of the objects managed by LCF. The main focus is on repositories, jobs, and status, and less about document-specific crawling information, but there may be some benefit to querying crawling status for individual documents as well.
> Nothing in this proposal should in any way limit or constrain the features that will be available in the LCF UI. The intent is that LCF should continue to have a full-featured UI, but in addition to a full-featured API.
> Note: This issue is part of Phase 2 of the CONNECTORS-50 umbrella issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CONNECTORS-56) All features should be accessible
through an API
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-56?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889255#action_12889255 ]
Karl Wright commented on CONNECTORS-56:
---------------------------------------
Another potential problem with using PUT is that there is no support in PUT I am aware of for large data. You need multipart POST for that. Thus, PUT would fail for large jobs (e.g. RSS jobs with lots of seeds).
> All features should be accessible through an API
> ------------------------------------------------
>
> Key: CONNECTORS-56
> URL: https://issues.apache.org/jira/browse/CONNECTORS-56
> Project: Lucene Connector Framework
> Issue Type: Sub-task
> Components: Framework core
> Reporter: Jack Krupansky
> Assignee: Karl Wright
>
> LCF consists of a full-featured crawling engine and a full-featured user interface to access the features of that engine, but some applications are better served with a full API that lets the application control the crawling engine, including creation and editing of connections and creation, editing, and control of jobs. Put simply, everything that a user can accomplish via the LCF UI should be doable through an LCF API. All LCF objects should be queryable through the API.
> A primary use case is Solr applications which currently use Aperture for crawling, but would prefer the full-featured capabilities of LCF as a crawling engine over Aperture.
> I do not wish to over-specify the API in this initial description, but I think the LCF API should probably be a traditional REST API., with some of the API elements specified via the context path, some parameters via URL query parameters, and complex, detailed structures as JSON (or similar.). The precise details of the API are beyond the scope of this initial description and will be added incrementally once the high-level approach to the API becomes reasonably settled.
> A job status and event reporting scheme is also needed in conjunction with the LCF API. That requirement has already been captured as CONNECTORS-41.
> The intention for the API is to create, edit, access, and control all of the objects managed by LCF. The main focus is on repositories, jobs, and status, and less about document-specific crawling information, but there may be some benefit to querying crawling status for individual documents as well.
> Nothing in this proposal should in any way limit or constrain the features that will be available in the LCF UI. The intent is that LCF should continue to have a full-featured UI, but in addition to a full-featured API.
> Note: This issue is part of Phase 2 of the CONNECTORS-50 umbrella issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CONNECTORS-56) All features should be accessible
through an API
Posted by "Jack Krupansky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-56?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jack Krupansky updated CONNECTORS-56:
-------------------------------------
Parent: CONNECTORS-50
Issue Type: Sub-task (was: Improvement)
> All features should be accessible through an API
> ------------------------------------------------
>
> Key: CONNECTORS-56
> URL: https://issues.apache.org/jira/browse/CONNECTORS-56
> Project: Lucene Connector Framework
> Issue Type: Sub-task
> Components: Framework core
> Reporter: Jack Krupansky
>
> LCF consists of a full-featured crawling engine and a full-featured user interface to access the features of that engine, but some applications are better served with a full API that lets the application control the crawling engine, including creation and editing of connections and creation, editing, and control of jobs. Put simply, everything that a user can accomplish via the LCF UI should be doable through an LCF API. All LCF objects should be queryable through the API.
> A primary use case is Solr applications which currently use Aperture for crawling, but would prefer the full-featured capabilities of LCF as a crawling engine over Aperture.
> I do not wish to over-specify the API in this initial description, but I think the LCF API should probably be a traditional REST API., with some of the API elements specified via the context path, some parameters via URL query parameters, and complex, detailed structures as JSON (or similar.). The precise details of the API are beyond the scope of this initial description and will be added incrementally once the high-level approach to the API becomes reasonably settled.
> A job status and event reporting scheme is also needed in conjunction with the LCF API. That requirement has already been captured as CONNECTORS-41.
> The intention for the API is to create, edit, access, and control all of the objects managed by LCF. The main focus is on repositories, jobs, and status, and less about document-specific crawling information, but there may be some benefit to querying crawling status for individual documents as well.
> Nothing in this proposal should in any way limit or constrain the features that will be available in the LCF UI. The intent is that LCF should continue to have a full-featured UI, but in addition to a full-featured API.
> Note: This issue is part of Phase 2 of the CONNECTORS-50 umbrella issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CONNECTORS-56) All features should be accessible
through an API
Posted by "Jack Krupansky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-56?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886932#action_12886932 ]
Jack Krupansky commented on CONNECTORS-56:
------------------------------------------
Karl's suggested approach seems consistent with my own thoughts.
More details and discussion to follow, but I'd be interested in more community feedback of the overall, high-level concept before we get too detailed.
Also, just to remind people that my suggestion was that a full API would not be a requirement for the initial release. Better to get QuickStart and basic capabilities in peoples' hands, but some aspects of the API, such as factoring needed to facilitate the API might well be better off being done sooner than a second release. So, maybe a portion or foundation of the API would be in the initial release.
> All features should be accessible through an API
> ------------------------------------------------
>
> Key: CONNECTORS-56
> URL: https://issues.apache.org/jira/browse/CONNECTORS-56
> Project: Lucene Connector Framework
> Issue Type: Improvement
> Components: Framework core
> Reporter: Jack Krupansky
>
> LCF consists of a full-featured crawling engine and a full-featured user interface to access the features of that engine, but some applications are better served with a full API that lets the application control the crawling engine, including creation and editing of connections and creation, editing, and control of jobs. Put simply, everything that a user can accomplish via the LCF UI should be doable through an LCF API. All LCF objects should be queryable through the API.
> A primary use case is Solr applications which currently use Aperture for crawling, but would prefer the full-featured capabilities of LCF as a crawling engine over Aperture.
> I do not wish to over-specify the API in this initial description, but I think the LCF API should probably be a traditional REST API., with some of the API elements specified via the context path, some parameters via URL query parameters, and complex, detailed structures as JSON (or similar.). The precise details of the API are beyond the scope of this initial description and will be added incrementally once the high-level approach to the API becomes reasonably settled.
> A job status and event reporting scheme is also needed in conjunction with the LCF API. That requirement has already been captured as CONNECTORS-41.
> The intention for the API is to create, edit, access, and control all of the objects managed by LCF. The main focus is on repositories, jobs, and status, and less about document-specific crawling information, but there may be some benefit to querying crawling status for individual documents as well.
> Nothing in this proposal should in any way limit or constrain the features that will be available in the LCF UI. The intent is that LCF should continue to have a full-featured UI, but in addition to a full-featured API.
> Note: This issue is part of Phase 2 of the CONNECTORS-50 umbrella issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.