You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@manifoldcf.apache.org by "Karl Wright (JIRA)" <ji...@apache.org> on 2011/09/19 14:41:08 UTC
[jira] [Created] (CONNECTORS-256) Connector for crawling Wikis
Connector for crawling Wikis
----------------------------
Key: CONNECTORS-256
URL: https://issues.apache.org/jira/browse/CONNECTORS-256
Project: ManifoldCF
Issue Type: New Feature
Affects Versions: ManifoldCF 0.4
Reporter: Karl Wright
Assignee: Karl Wright
Fix For: ManifoldCF 0.4
People have been trying to crawl wikis with ManifoldCF, but using the generic crawler is not a good way to do this. Instead, it looks like we really could use a wiki connector, which would understand the wiki API and thus crawl wiki content quickly and effectively.
Some pertinent API references follow:
I don't know if it is possible to link to a wiki document with just the pageid, but it is possible to to get the url for the referring pageid via api:
http://en.wikipedia.org/w/api.php?action=query&prop=info&pageids=27697087&inprop=url
It is possible to get the metadata of a document using the pages id (instead of title) directly:
Titel -> http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API&rvprop=timestamp|user|comment|content
PageID -> http://en.wikipedia.org/w/api.php?action=query&prop=revisions&pageids=27697087&rvprop=timestamp|user|comment|content
- There needs to be some notion of an overall list of pages:
- http://www.mediawiki.org/wiki/API:Allpages
- Example: http://en.wikipedia.org/w/api.php?action=query&list=allpages&apfrom=Kre&aplimit=5
- Metadata information (author and pub date) also needs to be separated out in some way:
- http://www.mediawiki.org/wiki/API:Properties#Revisions:_Example
- Example: http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API|Main%20Page&rvprop=timestamp|user|comment|content
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CONNECTORS-256) Connector for crawling Wikis
Posted by "Tobias Wunderlich (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127509#comment-13127509 ]
Tobias Wunderlich commented on CONNECTORS-256:
----------------------------------------------
I tried the new WikiConnector with an SolrOutputConnector and a MediaWiki (Version 1.15.3). The id (url), title and author were determined correctly. Unfortunately the content wasn't. I only got fragments of the original content. Also, information about the last_modified date was missing completely. Could the old MediaWiki version be the reason?
> Connector for crawling Wikis
> ----------------------------
>
> Key: CONNECTORS-256
> URL: https://issues.apache.org/jira/browse/CONNECTORS-256
> Project: ManifoldCF
> Issue Type: New Feature
> Components: Wiki connector
> Affects Versions: ManifoldCF 0.4
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.4
>
>
> People have been trying to crawl wikis with ManifoldCF, but using the generic crawler is not a good way to do this. Instead, it looks like we really could use a wiki connector, which would understand the wiki API and thus crawl wiki content quickly and effectively.
> Some pertinent API references follow:
> I don't know if it is possible to link to a wiki document with just the pageid, but it is possible to to get the url for the referring pageid via api:
> http://en.wikipedia.org/w/api.php?action=query&prop=info&pageids=27697087&inprop=url
> It is possible to get the metadata of a document using the pages id (instead of title) directly:
> Titel -> http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API&rvprop=timestamp|user|comment|content
> PageID -> http://en.wikipedia.org/w/api.php?action=query&prop=revisions&pageids=27697087&rvprop=timestamp|user|comment|content
> - There needs to be some notion of an overall list of pages:
> - http://www.mediawiki.org/wiki/API:Allpages
> - Example: http://en.wikipedia.org/w/api.php?action=query&list=allpages&apfrom=Kre&aplimit=5
> - Metadata information (author and pub date) also needs to be separated out in some way:
> - http://www.mediawiki.org/wiki/API:Properties#Revisions:_Example
> - Example: http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API|Main%20Page&rvprop=timestamp|user|comment|content
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CONNECTORS-256) Connector for crawling Wikis
Posted by "Tobias Wunderlich (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127555#comment-13127555 ]
Tobias Wunderlich commented on CONNECTORS-256:
----------------------------------------------
I testet all api-commands you used in WikiConnector.java, they all work.
As for an example of the content i got and what i was expecting:
content indexed:
"ff Nienhagen *> "
content expected:
"...
== Akquise ==
* Riff Nienhagen
..."
> Connector for crawling Wikis
> ----------------------------
>
> Key: CONNECTORS-256
> URL: https://issues.apache.org/jira/browse/CONNECTORS-256
> Project: ManifoldCF
> Issue Type: New Feature
> Components: Wiki connector
> Affects Versions: ManifoldCF 0.4
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.4
>
>
> People have been trying to crawl wikis with ManifoldCF, but using the generic crawler is not a good way to do this. Instead, it looks like we really could use a wiki connector, which would understand the wiki API and thus crawl wiki content quickly and effectively.
> Some pertinent API references follow:
> I don't know if it is possible to link to a wiki document with just the pageid, but it is possible to to get the url for the referring pageid via api:
> http://en.wikipedia.org/w/api.php?action=query&prop=info&pageids=27697087&inprop=url
> It is possible to get the metadata of a document using the pages id (instead of title) directly:
> Titel -> http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API&rvprop=timestamp|user|comment|content
> PageID -> http://en.wikipedia.org/w/api.php?action=query&prop=revisions&pageids=27697087&rvprop=timestamp|user|comment|content
> - There needs to be some notion of an overall list of pages:
> - http://www.mediawiki.org/wiki/API:Allpages
> - Example: http://en.wikipedia.org/w/api.php?action=query&list=allpages&apfrom=Kre&aplimit=5
> - Metadata information (author and pub date) also needs to be separated out in some way:
> - http://www.mediawiki.org/wiki/API:Properties#Revisions:_Example
> - Example: http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API|Main%20Page&rvprop=timestamp|user|comment|content
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CONNECTORS-256) Connector for crawling Wikis
Posted by "Karl Wright (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127534#comment-13127534 ]
Karl Wright commented on CONNECTORS-256:
----------------------------------------
bq. Also, information about the last_modified date was missing completely.
The last_modified date I did not include in the metadata indexed; this is why I wanted other folks to try it out. I'll open a new ticket to capture that enhancement.
> Connector for crawling Wikis
> ----------------------------
>
> Key: CONNECTORS-256
> URL: https://issues.apache.org/jira/browse/CONNECTORS-256
> Project: ManifoldCF
> Issue Type: New Feature
> Components: Wiki connector
> Affects Versions: ManifoldCF 0.4
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.4
>
>
> People have been trying to crawl wikis with ManifoldCF, but using the generic crawler is not a good way to do this. Instead, it looks like we really could use a wiki connector, which would understand the wiki API and thus crawl wiki content quickly and effectively.
> Some pertinent API references follow:
> I don't know if it is possible to link to a wiki document with just the pageid, but it is possible to to get the url for the referring pageid via api:
> http://en.wikipedia.org/w/api.php?action=query&prop=info&pageids=27697087&inprop=url
> It is possible to get the metadata of a document using the pages id (instead of title) directly:
> Titel -> http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API&rvprop=timestamp|user|comment|content
> PageID -> http://en.wikipedia.org/w/api.php?action=query&prop=revisions&pageids=27697087&rvprop=timestamp|user|comment|content
> - There needs to be some notion of an overall list of pages:
> - http://www.mediawiki.org/wiki/API:Allpages
> - Example: http://en.wikipedia.org/w/api.php?action=query&list=allpages&apfrom=Kre&aplimit=5
> - Metadata information (author and pub date) also needs to be separated out in some way:
> - http://www.mediawiki.org/wiki/API:Properties#Revisions:_Example
> - Example: http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API|Main%20Page&rvprop=timestamp|user|comment|content
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CONNECTORS-256) Connector for crawling Wikis
Posted by "Karl Wright (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127560#comment-13127560 ]
Karl Wright commented on CONNECTORS-256:
----------------------------------------
I've opened a ticket for the content corruption - CONNECTORS-274. Please attach further data to that ticket instead.
> Connector for crawling Wikis
> ----------------------------
>
> Key: CONNECTORS-256
> URL: https://issues.apache.org/jira/browse/CONNECTORS-256
> Project: ManifoldCF
> Issue Type: New Feature
> Components: Wiki connector
> Affects Versions: ManifoldCF 0.4
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.4
>
>
> People have been trying to crawl wikis with ManifoldCF, but using the generic crawler is not a good way to do this. Instead, it looks like we really could use a wiki connector, which would understand the wiki API and thus crawl wiki content quickly and effectively.
> Some pertinent API references follow:
> I don't know if it is possible to link to a wiki document with just the pageid, but it is possible to to get the url for the referring pageid via api:
> http://en.wikipedia.org/w/api.php?action=query&prop=info&pageids=27697087&inprop=url
> It is possible to get the metadata of a document using the pages id (instead of title) directly:
> Titel -> http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API&rvprop=timestamp|user|comment|content
> PageID -> http://en.wikipedia.org/w/api.php?action=query&prop=revisions&pageids=27697087&rvprop=timestamp|user|comment|content
> - There needs to be some notion of an overall list of pages:
> - http://www.mediawiki.org/wiki/API:Allpages
> - Example: http://en.wikipedia.org/w/api.php?action=query&list=allpages&apfrom=Kre&aplimit=5
> - Metadata information (author and pub date) also needs to be separated out in some way:
> - http://www.mediawiki.org/wiki/API:Properties#Revisions:_Example
> - Example: http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API|Main%20Page&rvprop=timestamp|user|comment|content
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CONNECTORS-256) Connector for crawling Wikis
Posted by "Karl Wright (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13120623#comment-13120623 ]
Karl Wright commented on CONNECTORS-256:
----------------------------------------
For the "allpages" one, aplimit must be set less than 500. This means that the enumeration must occur in chunks of 500, and we have to keep track of the last title before issuing the next query.
> Connector for crawling Wikis
> ----------------------------
>
> Key: CONNECTORS-256
> URL: https://issues.apache.org/jira/browse/CONNECTORS-256
> Project: ManifoldCF
> Issue Type: New Feature
> Components: Wiki connector
> Affects Versions: ManifoldCF 0.4
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.4
>
>
> People have been trying to crawl wikis with ManifoldCF, but using the generic crawler is not a good way to do this. Instead, it looks like we really could use a wiki connector, which would understand the wiki API and thus crawl wiki content quickly and effectively.
> Some pertinent API references follow:
> I don't know if it is possible to link to a wiki document with just the pageid, but it is possible to to get the url for the referring pageid via api:
> http://en.wikipedia.org/w/api.php?action=query&prop=info&pageids=27697087&inprop=url
> It is possible to get the metadata of a document using the pages id (instead of title) directly:
> Titel -> http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API&rvprop=timestamp|user|comment|content
> PageID -> http://en.wikipedia.org/w/api.php?action=query&prop=revisions&pageids=27697087&rvprop=timestamp|user|comment|content
> - There needs to be some notion of an overall list of pages:
> - http://www.mediawiki.org/wiki/API:Allpages
> - Example: http://en.wikipedia.org/w/api.php?action=query&list=allpages&apfrom=Kre&aplimit=5
> - Metadata information (author and pub date) also needs to be separated out in some way:
> - http://www.mediawiki.org/wiki/API:Properties#Revisions:_Example
> - Example: http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API|Main%20Page&rvprop=timestamp|user|comment|content
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-256) Connector for
crawling Wikis
Posted by "Karl Wright (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127534#comment-13127534 ]
Karl Wright edited comment on CONNECTORS-256 at 10/14/11 1:30 PM:
------------------------------------------------------------------
bq. Also, information about the last_modified date was missing completely.
The last_modified date I did not include in the metadata indexed; this is why I wanted other folks to try it out. I've opened a new ticket to capture that enhancement: CONNECTORS-273.
was (Author: kwright@metacarta.com):
bq. Also, information about the last_modified date was missing completely.
The last_modified date I did not include in the metadata indexed; this is why I wanted other folks to try it out. I'll open a new ticket to capture that enhancement.
> Connector for crawling Wikis
> ----------------------------
>
> Key: CONNECTORS-256
> URL: https://issues.apache.org/jira/browse/CONNECTORS-256
> Project: ManifoldCF
> Issue Type: New Feature
> Components: Wiki connector
> Affects Versions: ManifoldCF 0.4
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.4
>
>
> People have been trying to crawl wikis with ManifoldCF, but using the generic crawler is not a good way to do this. Instead, it looks like we really could use a wiki connector, which would understand the wiki API and thus crawl wiki content quickly and effectively.
> Some pertinent API references follow:
> I don't know if it is possible to link to a wiki document with just the pageid, but it is possible to to get the url for the referring pageid via api:
> http://en.wikipedia.org/w/api.php?action=query&prop=info&pageids=27697087&inprop=url
> It is possible to get the metadata of a document using the pages id (instead of title) directly:
> Titel -> http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API&rvprop=timestamp|user|comment|content
> PageID -> http://en.wikipedia.org/w/api.php?action=query&prop=revisions&pageids=27697087&rvprop=timestamp|user|comment|content
> - There needs to be some notion of an overall list of pages:
> - http://www.mediawiki.org/wiki/API:Allpages
> - Example: http://en.wikipedia.org/w/api.php?action=query&list=allpages&apfrom=Kre&aplimit=5
> - Metadata information (author and pub date) also needs to be separated out in some way:
> - http://www.mediawiki.org/wiki/API:Properties#Revisions:_Example
> - Example: http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API|Main%20Page&rvprop=timestamp|user|comment|content
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CONNECTORS-256) Connector for crawling Wikis
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karl Wright updated CONNECTORS-256:
-----------------------------------
Component/s: Wiki connector
> Connector for crawling Wikis
> ----------------------------
>
> Key: CONNECTORS-256
> URL: https://issues.apache.org/jira/browse/CONNECTORS-256
> Project: ManifoldCF
> Issue Type: New Feature
> Components: Wiki connector
> Affects Versions: ManifoldCF 0.4
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.4
>
>
> People have been trying to crawl wikis with ManifoldCF, but using the generic crawler is not a good way to do this. Instead, it looks like we really could use a wiki connector, which would understand the wiki API and thus crawl wiki content quickly and effectively.
> Some pertinent API references follow:
> I don't know if it is possible to link to a wiki document with just the pageid, but it is possible to to get the url for the referring pageid via api:
> http://en.wikipedia.org/w/api.php?action=query&prop=info&pageids=27697087&inprop=url
> It is possible to get the metadata of a document using the pages id (instead of title) directly:
> Titel -> http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API&rvprop=timestamp|user|comment|content
> PageID -> http://en.wikipedia.org/w/api.php?action=query&prop=revisions&pageids=27697087&rvprop=timestamp|user|comment|content
> - There needs to be some notion of an overall list of pages:
> - http://www.mediawiki.org/wiki/API:Allpages
> - Example: http://en.wikipedia.org/w/api.php?action=query&list=allpages&apfrom=Kre&aplimit=5
> - Metadata information (author and pub date) also needs to be separated out in some way:
> - http://www.mediawiki.org/wiki/API:Properties#Revisions:_Example
> - Example: http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API|Main%20Page&rvprop=timestamp|user|comment|content
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CONNECTORS-256) Connector for crawling Wikis
Posted by "Karl Wright (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127531#comment-13127531 ]
Karl Wright commented on CONNECTORS-256:
----------------------------------------
Can you be as specific as possible as to what you expected to see in the content and what you got? Also, can you send your browser to your mediawiki with a URL of the form:
http://xxxx/api.php?format=xml&action=query&prop=revisions&pageids=<your_page>&rvprop=user%7ccomment%7ccontent
The content should be included in the XML response. Can you compare what you see vs. what you EXPECT to see?
> Connector for crawling Wikis
> ----------------------------
>
> Key: CONNECTORS-256
> URL: https://issues.apache.org/jira/browse/CONNECTORS-256
> Project: ManifoldCF
> Issue Type: New Feature
> Components: Wiki connector
> Affects Versions: ManifoldCF 0.4
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.4
>
>
> People have been trying to crawl wikis with ManifoldCF, but using the generic crawler is not a good way to do this. Instead, it looks like we really could use a wiki connector, which would understand the wiki API and thus crawl wiki content quickly and effectively.
> Some pertinent API references follow:
> I don't know if it is possible to link to a wiki document with just the pageid, but it is possible to to get the url for the referring pageid via api:
> http://en.wikipedia.org/w/api.php?action=query&prop=info&pageids=27697087&inprop=url
> It is possible to get the metadata of a document using the pages id (instead of title) directly:
> Titel -> http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API&rvprop=timestamp|user|comment|content
> PageID -> http://en.wikipedia.org/w/api.php?action=query&prop=revisions&pageids=27697087&rvprop=timestamp|user|comment|content
> - There needs to be some notion of an overall list of pages:
> - http://www.mediawiki.org/wiki/API:Allpages
> - Example: http://en.wikipedia.org/w/api.php?action=query&list=allpages&apfrom=Kre&aplimit=5
> - Metadata information (author and pub date) also needs to be separated out in some way:
> - http://www.mediawiki.org/wiki/API:Properties#Revisions:_Example
> - Example: http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API|Main%20Page&rvprop=timestamp|user|comment|content
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CONNECTORS-256) Connector for crawling Wikis
Posted by "Karl Wright (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127558#comment-13127558 ]
Karl Wright commented on CONNECTORS-256:
----------------------------------------
So are you saying that the content gets only the end?
Can you email me the XML response you got from the content fetch, or attach it to this ticket? I can add it to the test suite and debug why the standard SAX XML parsing is missing data. Maybe it's CDATA or something...
> Connector for crawling Wikis
> ----------------------------
>
> Key: CONNECTORS-256
> URL: https://issues.apache.org/jira/browse/CONNECTORS-256
> Project: ManifoldCF
> Issue Type: New Feature
> Components: Wiki connector
> Affects Versions: ManifoldCF 0.4
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.4
>
>
> People have been trying to crawl wikis with ManifoldCF, but using the generic crawler is not a good way to do this. Instead, it looks like we really could use a wiki connector, which would understand the wiki API and thus crawl wiki content quickly and effectively.
> Some pertinent API references follow:
> I don't know if it is possible to link to a wiki document with just the pageid, but it is possible to to get the url for the referring pageid via api:
> http://en.wikipedia.org/w/api.php?action=query&prop=info&pageids=27697087&inprop=url
> It is possible to get the metadata of a document using the pages id (instead of title) directly:
> Titel -> http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API&rvprop=timestamp|user|comment|content
> PageID -> http://en.wikipedia.org/w/api.php?action=query&prop=revisions&pageids=27697087&rvprop=timestamp|user|comment|content
> - There needs to be some notion of an overall list of pages:
> - http://www.mediawiki.org/wiki/API:Allpages
> - Example: http://en.wikipedia.org/w/api.php?action=query&list=allpages&apfrom=Kre&aplimit=5
> - Metadata information (author and pub date) also needs to be separated out in some way:
> - http://www.mediawiki.org/wiki/API:Properties#Revisions:_Example
> - Example: http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API|Main%20Page&rvprop=timestamp|user|comment|content
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CONNECTORS-256) Connector for crawling Wikis
Posted by "Karl Wright (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karl Wright resolved CONNECTORS-256.
------------------------------------
Resolution: Fixed
r1182834.
> Connector for crawling Wikis
> ----------------------------
>
> Key: CONNECTORS-256
> URL: https://issues.apache.org/jira/browse/CONNECTORS-256
> Project: ManifoldCF
> Issue Type: New Feature
> Components: Wiki connector
> Affects Versions: ManifoldCF 0.4
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.4
>
>
> People have been trying to crawl wikis with ManifoldCF, but using the generic crawler is not a good way to do this. Instead, it looks like we really could use a wiki connector, which would understand the wiki API and thus crawl wiki content quickly and effectively.
> Some pertinent API references follow:
> I don't know if it is possible to link to a wiki document with just the pageid, but it is possible to to get the url for the referring pageid via api:
> http://en.wikipedia.org/w/api.php?action=query&prop=info&pageids=27697087&inprop=url
> It is possible to get the metadata of a document using the pages id (instead of title) directly:
> Titel -> http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API&rvprop=timestamp|user|comment|content
> PageID -> http://en.wikipedia.org/w/api.php?action=query&prop=revisions&pageids=27697087&rvprop=timestamp|user|comment|content
> - There needs to be some notion of an overall list of pages:
> - http://www.mediawiki.org/wiki/API:Allpages
> - Example: http://en.wikipedia.org/w/api.php?action=query&list=allpages&apfrom=Kre&aplimit=5
> - Metadata information (author and pub date) also needs to be separated out in some way:
> - http://www.mediawiki.org/wiki/API:Properties#Revisions:_Example
> - Example: http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API|Main%20Page&rvprop=timestamp|user|comment|content
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira