You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Grant Ingersoll (JIRA)" <ji...@apache.org> on 2010/10/28 17:53:21 UTC
[jira] Created: (LUCENE-2728) EnwikiContentSource does not properly
identify the name/id of the Wikipedia article
EnwikiContentSource does not properly identify the name/id of the Wikipedia article
-----------------------------------------------------------------------------------
Key: LUCENE-2728
URL: https://issues.apache.org/jira/browse/LUCENE-2728
Project: Lucene - Java
Issue Type: Bug
Components: contrib/benchmark
Reporter: Grant Ingersoll
Assignee: Grant Ingersoll
Priority: Minor
The EnwikiContentSource does not properly identify the id (name in benchmark parlance) of the documents. It currently produces assigns the id on the last <id> tag it sees in the document, as opposed to the id of the document. Most documents have multiple <id> tags in them. This prevents the ContentSource from being used effectively in producing documents for updating.
Example doc:
{quote}
<page>
<title>AlgeriA</title>
<id>5</id>
<revision>
<id>133452200</id>
<timestamp>2007-05-25T17:11:48Z</timestamp>
<contributor>
<username>Gurch</username>
<id>241822</id>
</contributor>
<minor />
<comment>[[WP:AES|â<86><90>]]Redirected page to [[Algeria]]</comment>
<text xml:space="preserve">#REDIRECT [[Algeria]] {{R from CamelCase}}</text>
</revision>
</page>
{quote}
In this case, the getName() return 241822 instead of 5. page/id is unique according to the schema at http://www.mediawiki.org/xml/export-0.3.xsd, so we should just get that one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2728) EnwikiContentSource does not properly
identify the name/id of the Wikipedia article
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Grant Ingersoll updated LUCENE-2728:
------------------------------------
Attachment: LUCENE-2728.patch
Patch applies from the contrib/benchmark directory.
> EnwikiContentSource does not properly identify the name/id of the Wikipedia article
> -----------------------------------------------------------------------------------
>
> Key: LUCENE-2728
> URL: https://issues.apache.org/jira/browse/LUCENE-2728
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/benchmark
> Reporter: Grant Ingersoll
> Assignee: Grant Ingersoll
> Priority: Minor
> Attachments: LUCENE-2728.patch
>
>
> The EnwikiContentSource does not properly identify the id (name in benchmark parlance) of the documents. It currently produces assigns the id on the last <id> tag it sees in the document, as opposed to the id of the document. Most documents have multiple <id> tags in them. This prevents the ContentSource from being used effectively in producing documents for updating.
> Example doc:
> {quote}
> <page>
> <title>AlgeriA</title>
> <id>5</id>
> <revision>
> <id>133452200</id>
> <timestamp>2007-05-25T17:11:48Z</timestamp>
> <contributor>
> <username>Gurch</username>
> <id>241822</id>
> </contributor>
> <minor />
> <comment>[[WP:AES|â<86><90>]]Redirected page to [[Algeria]]</comment>
> <text xml:space="preserve">#REDIRECT [[Algeria]] {{R from CamelCase}}</text>
> </revision>
> </page>
> {quote}
> In this case, the getName() return 241822 instead of 5. page/id is unique according to the schema at http://www.mediawiki.org/xml/export-0.3.xsd, so we should just get that one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-2728) EnwikiContentSource does not
properly identify the name/id of the Wikipedia article
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Grant Ingersoll resolved LUCENE-2728.
-------------------------------------
Resolution: Fixed
> EnwikiContentSource does not properly identify the name/id of the Wikipedia article
> -----------------------------------------------------------------------------------
>
> Key: LUCENE-2728
> URL: https://issues.apache.org/jira/browse/LUCENE-2728
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/benchmark
> Reporter: Grant Ingersoll
> Assignee: Grant Ingersoll
> Priority: Minor
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2728.patch
>
>
> The EnwikiContentSource does not properly identify the id (name in benchmark parlance) of the documents. It currently produces assigns the id on the last <id> tag it sees in the document, as opposed to the id of the document. Most documents have multiple <id> tags in them. This prevents the ContentSource from being used effectively in producing documents for updating.
> Example doc:
> {quote}
> <page>
> <title>AlgeriA</title>
> <id>5</id>
> <revision>
> <id>133452200</id>
> <timestamp>2007-05-25T17:11:48Z</timestamp>
> <contributor>
> <username>Gurch</username>
> <id>241822</id>
> </contributor>
> <minor />
> <comment>[[WP:AES|â<86><90>]]Redirected page to [[Algeria]]</comment>
> <text xml:space="preserve">#REDIRECT [[Algeria]] {{R from CamelCase}}</text>
> </revision>
> </page>
> {quote}
> In this case, the getName() return 241822 instead of 5. page/id is unique according to the schema at http://www.mediawiki.org/xml/export-0.3.xsd, so we should just get that one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2728) EnwikiContentSource does not
properly identify the name/id of the Wikipedia article
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925916#action_12925916 ]
Grant Ingersoll commented on LUCENE-2728:
-----------------------------------------
Yes. I'm trying at the moment, but my IntelliJ setup is screwed up which makes it harder to merge
> EnwikiContentSource does not properly identify the name/id of the Wikipedia article
> -----------------------------------------------------------------------------------
>
> Key: LUCENE-2728
> URL: https://issues.apache.org/jira/browse/LUCENE-2728
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/benchmark
> Reporter: Grant Ingersoll
> Assignee: Grant Ingersoll
> Priority: Minor
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2728.patch
>
>
> The EnwikiContentSource does not properly identify the id (name in benchmark parlance) of the documents. It currently produces assigns the id on the last <id> tag it sees in the document, as opposed to the id of the document. Most documents have multiple <id> tags in them. This prevents the ContentSource from being used effectively in producing documents for updating.
> Example doc:
> {quote}
> <page>
> <title>AlgeriA</title>
> <id>5</id>
> <revision>
> <id>133452200</id>
> <timestamp>2007-05-25T17:11:48Z</timestamp>
> <contributor>
> <username>Gurch</username>
> <id>241822</id>
> </contributor>
> <minor />
> <comment>[[WP:AES|â<86><90>]]Redirected page to [[Algeria]]</comment>
> <text xml:space="preserve">#REDIRECT [[Algeria]] {{R from CamelCase}}</text>
> </revision>
> </page>
> {quote}
> In this case, the getName() return 241822 instead of 5. page/id is unique according to the schema at http://www.mediawiki.org/xml/export-0.3.xsd, so we should just get that one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2728) EnwikiContentSource does not properly
identify the name/id of the Wikipedia article
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-2728:
---------------------------------------
Fix Version/s: 4.0
3.1
> EnwikiContentSource does not properly identify the name/id of the Wikipedia article
> -----------------------------------------------------------------------------------
>
> Key: LUCENE-2728
> URL: https://issues.apache.org/jira/browse/LUCENE-2728
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/benchmark
> Reporter: Grant Ingersoll
> Assignee: Grant Ingersoll
> Priority: Minor
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2728.patch
>
>
> The EnwikiContentSource does not properly identify the id (name in benchmark parlance) of the documents. It currently produces assigns the id on the last <id> tag it sees in the document, as opposed to the id of the document. Most documents have multiple <id> tags in them. This prevents the ContentSource from being used effectively in producing documents for updating.
> Example doc:
> {quote}
> <page>
> <title>AlgeriA</title>
> <id>5</id>
> <revision>
> <id>133452200</id>
> <timestamp>2007-05-25T17:11:48Z</timestamp>
> <contributor>
> <username>Gurch</username>
> <id>241822</id>
> </contributor>
> <minor />
> <comment>[[WP:AES|â<86><90>]]Redirected page to [[Algeria]]</comment>
> <text xml:space="preserve">#REDIRECT [[Algeria]] {{R from CamelCase}}</text>
> </revision>
> </page>
> {quote}
> In this case, the getName() return 241822 instead of 5. page/id is unique according to the schema at http://www.mediawiki.org/xml/export-0.3.xsd, so we should just get that one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2728) EnwikiContentSource does not
properly identify the name/id of the Wikipedia article
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925848#action_12925848 ]
Grant Ingersoll commented on LUCENE-2728:
-----------------------------------------
Also, the doc id is also guaranteed to be the second item. So, upon seeing the first id for a page, we can ignore all other ids.
> EnwikiContentSource does not properly identify the name/id of the Wikipedia article
> -----------------------------------------------------------------------------------
>
> Key: LUCENE-2728
> URL: https://issues.apache.org/jira/browse/LUCENE-2728
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/benchmark
> Reporter: Grant Ingersoll
> Assignee: Grant Ingersoll
> Priority: Minor
>
> The EnwikiContentSource does not properly identify the id (name in benchmark parlance) of the documents. It currently produces assigns the id on the last <id> tag it sees in the document, as opposed to the id of the document. Most documents have multiple <id> tags in them. This prevents the ContentSource from being used effectively in producing documents for updating.
> Example doc:
> {quote}
> <page>
> <title>AlgeriA</title>
> <id>5</id>
> <revision>
> <id>133452200</id>
> <timestamp>2007-05-25T17:11:48Z</timestamp>
> <contributor>
> <username>Gurch</username>
> <id>241822</id>
> </contributor>
> <minor />
> <comment>[[WP:AES|â<86><90>]]Redirected page to [[Algeria]]</comment>
> <text xml:space="preserve">#REDIRECT [[Algeria]] {{R from CamelCase}}</text>
> </revision>
> </page>
> {quote}
> In this case, the getName() return 241822 instead of 5. page/id is unique according to the schema at http://www.mediawiki.org/xml/export-0.3.xsd, so we should just get that one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2728) EnwikiContentSource does not
properly identify the name/id of the Wikipedia article
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925872#action_12925872 ]
Grant Ingersoll commented on LUCENE-2728:
-----------------------------------------
Trunk: Committed revision 1028386.
> EnwikiContentSource does not properly identify the name/id of the Wikipedia article
> -----------------------------------------------------------------------------------
>
> Key: LUCENE-2728
> URL: https://issues.apache.org/jira/browse/LUCENE-2728
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/benchmark
> Reporter: Grant Ingersoll
> Assignee: Grant Ingersoll
> Priority: Minor
> Attachments: LUCENE-2728.patch
>
>
> The EnwikiContentSource does not properly identify the id (name in benchmark parlance) of the documents. It currently produces assigns the id on the last <id> tag it sees in the document, as opposed to the id of the document. Most documents have multiple <id> tags in them. This prevents the ContentSource from being used effectively in producing documents for updating.
> Example doc:
> {quote}
> <page>
> <title>AlgeriA</title>
> <id>5</id>
> <revision>
> <id>133452200</id>
> <timestamp>2007-05-25T17:11:48Z</timestamp>
> <contributor>
> <username>Gurch</username>
> <id>241822</id>
> </contributor>
> <minor />
> <comment>[[WP:AES|â<86><90>]]Redirected page to [[Algeria]]</comment>
> <text xml:space="preserve">#REDIRECT [[Algeria]] {{R from CamelCase}}</text>
> </revision>
> </page>
> {quote}
> In this case, the getName() return 241822 instead of 5. page/id is unique according to the schema at http://www.mediawiki.org/xml/export-0.3.xsd, so we should just get that one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2728) EnwikiContentSource does not
properly identify the name/id of the Wikipedia article
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925911#action_12925911 ]
Michael McCandless commented on LUCENE-2728:
--------------------------------------------
Grant are you going to backport to 3.x?
> EnwikiContentSource does not properly identify the name/id of the Wikipedia article
> -----------------------------------------------------------------------------------
>
> Key: LUCENE-2728
> URL: https://issues.apache.org/jira/browse/LUCENE-2728
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/benchmark
> Reporter: Grant Ingersoll
> Assignee: Grant Ingersoll
> Priority: Minor
> Attachments: LUCENE-2728.patch
>
>
> The EnwikiContentSource does not properly identify the id (name in benchmark parlance) of the documents. It currently produces assigns the id on the last <id> tag it sees in the document, as opposed to the id of the document. Most documents have multiple <id> tags in them. This prevents the ContentSource from being used effectively in producing documents for updating.
> Example doc:
> {quote}
> <page>
> <title>AlgeriA</title>
> <id>5</id>
> <revision>
> <id>133452200</id>
> <timestamp>2007-05-25T17:11:48Z</timestamp>
> <contributor>
> <username>Gurch</username>
> <id>241822</id>
> </contributor>
> <minor />
> <comment>[[WP:AES|â<86><90>]]Redirected page to [[Algeria]]</comment>
> <text xml:space="preserve">#REDIRECT [[Algeria]] {{R from CamelCase}}</text>
> </revision>
> </page>
> {quote}
> In this case, the getName() return 241822 instead of 5. page/id is unique according to the schema at http://www.mediawiki.org/xml/export-0.3.xsd, so we should just get that one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org