You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Ali Syed (JIRA)" <ji...@apache.org> on 2009/05/22 02:59:45 UTC

[jira] Created: (SOLR-1180) Delete only delta import does not commit index

Delete only delta import does not commit index
----------------------------------------------

                 Key: SOLR-1180
                 URL: https://issues.apache.org/jira/browse/SOLR-1180
             Project: Solr
          Issue Type: Bug
          Components: contrib - DataImportHandler
    Affects Versions: 1.3
         Environment: OS: Windows & Linux. 
Java: 1.6
DB: MySQL & SQL Server
            Reporter: Ali Syed
            Priority: Minor


Entity is set up in db-data-config.xml as follows

I<entity dataSource="mysqlDS" name="contact" pk="id"
    query="SELECT o.* FROM Contact o WHERE (o.deleted is null OR o.deleted = 0)"
    deltaQuery="select id from Contact o where (o.deleted is null OR o.deleted = 0) AND (o.createDate &gt; '${dataimporter.last_index_time}' OR o.updateDate &gt; '${dataimporter.last_index_time}')"
    deletedPkQuery="select id FROM Contact o WHERE o.deleted = 1 AND (o.createDate &gt; '${dataimporter.last_index_time}' OR o.updateDate &gt; '${dataimporter.last_index_time}')">
    ...
</entity>

If a delta import is trigger which causes only documents to be deleted the index is not commit. 

Problem seems to be in DocBuilder.execute() method checks for deletedDocCount before commiting.
if (importStatistics.docCount.get() > 0 || importStatistics.deletedDocCount.get() > 0) {
  commit(lastIndexTimeProps);
}

doDelta() method sets the count as follows
importStatistics.deletedDocCount.addAndGet(deletedKeys.size());

but after deleteAll(Set) method removes the each key from iterator
while (...) ...
      iter.remove();

Simply remove iter.remove() line should fix the problem. I am not sure what is the point of removing the keys from the iterator.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Assigned: (SOLR-1180) Delete only delta import does not commit index

Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/SOLR-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Noble Paul reassigned SOLR-1180:
--------------------------------

    Assignee: Noble Paul

> Delete only delta import does not commit index
> ----------------------------------------------
>
>                 Key: SOLR-1180
>                 URL: https://issues.apache.org/jira/browse/SOLR-1180
>             Project: Solr
>          Issue Type: Bug
>          Components: contrib - DataImportHandler
>    Affects Versions: 1.3
>         Environment: OS: Windows & Linux. 
> Java: 1.6
> DB: MySQL & SQL Server
>            Reporter: Ali Syed
>            Assignee: Noble Paul
>            Priority: Minor
>             Fix For: 1.4
>
>
> Entity is set up in db-data-config.xml as follows
> I<entity dataSource="mysqlDS" name="contact" pk="id"
>     query="SELECT o.* FROM Contact o WHERE (o.deleted is null OR o.deleted = 0)"
>     deltaQuery="select id from Contact o where (o.deleted is null OR o.deleted = 0) AND (o.createDate &gt; '${dataimporter.last_index_time}' OR o.updateDate &gt; '${dataimporter.last_index_time}')"
>     deletedPkQuery="select id FROM Contact o WHERE o.deleted = 1 AND (o.createDate &gt; '${dataimporter.last_index_time}' OR o.updateDate &gt; '${dataimporter.last_index_time}')">
>     ...
> </entity>
> If a delta import is trigger which causes only documents to be deleted the index is not commit. 
> Problem seems to be in DocBuilder.execute() method checks for deletedDocCount before commiting.
> if (importStatistics.docCount.get() > 0 || importStatistics.deletedDocCount.get() > 0) {
>   commit(lastIndexTimeProps);
> }
> doDelta() method sets the count as follows
> importStatistics.deletedDocCount.addAndGet(deletedKeys.size());
> but after deleteAll(Set) method removes the each key from iterator
> while (...) ...
>       iter.remove();
> Simply remove iter.remove() line should fix the problem. I am not sure what is the point of removing the keys from the iterator.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (SOLR-1180) Delete only delta import does not commit index

Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/SOLR-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Noble Paul updated SOLR-1180:
-----------------------------

    Attachment: SOLR-1180.patch

> Delete only delta import does not commit index
> ----------------------------------------------
>
>                 Key: SOLR-1180
>                 URL: https://issues.apache.org/jira/browse/SOLR-1180
>             Project: Solr
>          Issue Type: Bug
>          Components: contrib - DataImportHandler
>    Affects Versions: 1.3
>         Environment: OS: Windows & Linux. 
> Java: 1.6
> DB: MySQL & SQL Server
>            Reporter: Ali Syed
>            Assignee: Noble Paul
>            Priority: Minor
>             Fix For: 1.4
>
>         Attachments: SOLR-1180.patch
>
>
> Entity is set up in db-data-config.xml as follows
> I<entity dataSource="mysqlDS" name="contact" pk="id"
>     query="SELECT o.* FROM Contact o WHERE (o.deleted is null OR o.deleted = 0)"
>     deltaQuery="select id from Contact o where (o.deleted is null OR o.deleted = 0) AND (o.createDate &gt; '${dataimporter.last_index_time}' OR o.updateDate &gt; '${dataimporter.last_index_time}')"
>     deletedPkQuery="select id FROM Contact o WHERE o.deleted = 1 AND (o.createDate &gt; '${dataimporter.last_index_time}' OR o.updateDate &gt; '${dataimporter.last_index_time}')">
>     ...
> </entity>
> If a delta import is trigger which causes only documents to be deleted the index is not commit. 
> Problem seems to be in DocBuilder.execute() method checks for deletedDocCount before commiting.
> if (importStatistics.docCount.get() > 0 || importStatistics.deletedDocCount.get() > 0) {
>   commit(lastIndexTimeProps);
> }
> doDelta() method sets the count as follows
> importStatistics.deletedDocCount.addAndGet(deletedKeys.size());
> but after deleteAll(Set) method removes the each key from iterator
> while (...) ...
>       iter.remove();
> Simply remove iter.remove() line should fix the problem. I am not sure what is the point of removing the keys from the iterator.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (SOLR-1180) Delete only delta import does not commit index

Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712035#action_12712035 ] 

Noble Paul commented on SOLR-1180:
----------------------------------

committed revision:777479
thanks Ali Syed

> Delete only delta import does not commit index
> ----------------------------------------------
>
>                 Key: SOLR-1180
>                 URL: https://issues.apache.org/jira/browse/SOLR-1180
>             Project: Solr
>          Issue Type: Bug
>          Components: contrib - DataImportHandler
>    Affects Versions: 1.3
>         Environment: OS: Windows & Linux. 
> Java: 1.6
> DB: MySQL & SQL Server
>            Reporter: Ali Syed
>            Assignee: Noble Paul
>            Priority: Minor
>             Fix For: 1.4
>
>         Attachments: SOLR-1180.patch
>
>
> Entity is set up in db-data-config.xml as follows
> I<entity dataSource="mysqlDS" name="contact" pk="id"
>     query="SELECT o.* FROM Contact o WHERE (o.deleted is null OR o.deleted = 0)"
>     deltaQuery="select id from Contact o where (o.deleted is null OR o.deleted = 0) AND (o.createDate &gt; '${dataimporter.last_index_time}' OR o.updateDate &gt; '${dataimporter.last_index_time}')"
>     deletedPkQuery="select id FROM Contact o WHERE o.deleted = 1 AND (o.createDate &gt; '${dataimporter.last_index_time}' OR o.updateDate &gt; '${dataimporter.last_index_time}')">
>     ...
> </entity>
> If a delta import is trigger which causes only documents to be deleted the index is not commit. 
> Problem seems to be in DocBuilder.execute() method checks for deletedDocCount before commiting.
> if (importStatistics.docCount.get() > 0 || importStatistics.deletedDocCount.get() > 0) {
>   commit(lastIndexTimeProps);
> }
> doDelta() method sets the count as follows
> importStatistics.deletedDocCount.addAndGet(deletedKeys.size());
> but after deleteAll(Set) method removes the each key from iterator
> while (...) ...
>       iter.remove();
> Simply remove iter.remove() line should fix the problem. I am not sure what is the point of removing the keys from the iterator.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (SOLR-1180) Delete only delta import does not commit index

Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/SOLR-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Noble Paul updated SOLR-1180:
-----------------------------

    Fix Version/s: 1.4

> Delete only delta import does not commit index
> ----------------------------------------------
>
>                 Key: SOLR-1180
>                 URL: https://issues.apache.org/jira/browse/SOLR-1180
>             Project: Solr
>          Issue Type: Bug
>          Components: contrib - DataImportHandler
>    Affects Versions: 1.3
>         Environment: OS: Windows & Linux. 
> Java: 1.6
> DB: MySQL & SQL Server
>            Reporter: Ali Syed
>            Assignee: Noble Paul
>            Priority: Minor
>             Fix For: 1.4
>
>
> Entity is set up in db-data-config.xml as follows
> I<entity dataSource="mysqlDS" name="contact" pk="id"
>     query="SELECT o.* FROM Contact o WHERE (o.deleted is null OR o.deleted = 0)"
>     deltaQuery="select id from Contact o where (o.deleted is null OR o.deleted = 0) AND (o.createDate &gt; '${dataimporter.last_index_time}' OR o.updateDate &gt; '${dataimporter.last_index_time}')"
>     deletedPkQuery="select id FROM Contact o WHERE o.deleted = 1 AND (o.createDate &gt; '${dataimporter.last_index_time}' OR o.updateDate &gt; '${dataimporter.last_index_time}')">
>     ...
> </entity>
> If a delta import is trigger which causes only documents to be deleted the index is not commit. 
> Problem seems to be in DocBuilder.execute() method checks for deletedDocCount before commiting.
> if (importStatistics.docCount.get() > 0 || importStatistics.deletedDocCount.get() > 0) {
>   commit(lastIndexTimeProps);
> }
> doDelta() method sets the count as follows
> importStatistics.deletedDocCount.addAndGet(deletedKeys.size());
> but after deleteAll(Set) method removes the each key from iterator
> while (...) ...
>       iter.remove();
> Simply remove iter.remove() line should fix the problem. I am not sure what is the point of removing the keys from the iterator.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (SOLR-1180) Delete only delta import does not commit index

Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/SOLR-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Noble Paul resolved SOLR-1180.
------------------------------

    Resolution: Fixed

> Delete only delta import does not commit index
> ----------------------------------------------
>
>                 Key: SOLR-1180
>                 URL: https://issues.apache.org/jira/browse/SOLR-1180
>             Project: Solr
>          Issue Type: Bug
>          Components: contrib - DataImportHandler
>    Affects Versions: 1.3
>         Environment: OS: Windows & Linux. 
> Java: 1.6
> DB: MySQL & SQL Server
>            Reporter: Ali Syed
>            Assignee: Noble Paul
>            Priority: Minor
>             Fix For: 1.4
>
>         Attachments: SOLR-1180.patch
>
>
> Entity is set up in db-data-config.xml as follows
> I<entity dataSource="mysqlDS" name="contact" pk="id"
>     query="SELECT o.* FROM Contact o WHERE (o.deleted is null OR o.deleted = 0)"
>     deltaQuery="select id from Contact o where (o.deleted is null OR o.deleted = 0) AND (o.createDate &gt; '${dataimporter.last_index_time}' OR o.updateDate &gt; '${dataimporter.last_index_time}')"
>     deletedPkQuery="select id FROM Contact o WHERE o.deleted = 1 AND (o.createDate &gt; '${dataimporter.last_index_time}' OR o.updateDate &gt; '${dataimporter.last_index_time}')">
>     ...
> </entity>
> If a delta import is trigger which causes only documents to be deleted the index is not commit. 
> Problem seems to be in DocBuilder.execute() method checks for deletedDocCount before commiting.
> if (importStatistics.docCount.get() > 0 || importStatistics.deletedDocCount.get() > 0) {
>   commit(lastIndexTimeProps);
> }
> doDelta() method sets the count as follows
> importStatistics.deletedDocCount.addAndGet(deletedKeys.size());
> but after deleteAll(Set) method removes the each key from iterator
> while (...) ...
>       iter.remove();
> Simply remove iter.remove() line should fix the problem. I am not sure what is the point of removing the keys from the iterator.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (SOLR-1180) Delete only delta import does not commit index

Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711943#action_12711943 ] 

Noble Paul commented on SOLR-1180:
----------------------------------

bq.I am not sure what is the point of removing the keys from the iterator.

If the no:of items in that list is huge, we want to leave the objects to GC

> Delete only delta import does not commit index
> ----------------------------------------------
>
>                 Key: SOLR-1180
>                 URL: https://issues.apache.org/jira/browse/SOLR-1180
>             Project: Solr
>          Issue Type: Bug
>          Components: contrib - DataImportHandler
>    Affects Versions: 1.3
>         Environment: OS: Windows & Linux. 
> Java: 1.6
> DB: MySQL & SQL Server
>            Reporter: Ali Syed
>            Priority: Minor
>             Fix For: 1.4
>
>
> Entity is set up in db-data-config.xml as follows
> I<entity dataSource="mysqlDS" name="contact" pk="id"
>     query="SELECT o.* FROM Contact o WHERE (o.deleted is null OR o.deleted = 0)"
>     deltaQuery="select id from Contact o where (o.deleted is null OR o.deleted = 0) AND (o.createDate &gt; '${dataimporter.last_index_time}' OR o.updateDate &gt; '${dataimporter.last_index_time}')"
>     deletedPkQuery="select id FROM Contact o WHERE o.deleted = 1 AND (o.createDate &gt; '${dataimporter.last_index_time}' OR o.updateDate &gt; '${dataimporter.last_index_time}')">
>     ...
> </entity>
> If a delta import is trigger which causes only documents to be deleted the index is not commit. 
> Problem seems to be in DocBuilder.execute() method checks for deletedDocCount before commiting.
> if (importStatistics.docCount.get() > 0 || importStatistics.deletedDocCount.get() > 0) {
>   commit(lastIndexTimeProps);
> }
> doDelta() method sets the count as follows
> importStatistics.deletedDocCount.addAndGet(deletedKeys.size());
> but after deleteAll(Set) method removes the each key from iterator
> while (...) ...
>       iter.remove();
> Simply remove iter.remove() line should fix the problem. I am not sure what is the point of removing the keys from the iterator.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.