You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by cveres <cs...@me.com> on 2013/02/21 09:57:53 UTC
DIH deleting documents
I am adding documents with data import handler from a mysql database. I
create a unique id for each document by concatenating a couple of fields in
the database. Every id is unique.
After the import, over half the documents which were imported are deleted
again, leaving me with less then half the documents in the database ending
up in the Solr index.
Is there a way to get a list of the deleted documents, so that I can start
troubleshooting what went wrong?
thanks,
Csaba
--
View this message in context: http://lucene.472066.n3.nabble.com/DIH-deleting-documents-tp4041809.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: DIH deleting documents
Posted by cveres <cs...@me.com>.
Thanks Arcadius,
Excellent suggestion about the view.I'll try to simplify things and see how
I go.
thanks,
Csaba
--
View this message in context: http://lucene.472066.n3.nabble.com/DIH-deleting-documents-tp4041811p4042663.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: DIH deleting documents
Posted by Arcadius Ahouansou <ar...@menelic.com>.
Hello Cveres.
I know you said the IDs are unique, but having a look at your config,
your id is defined as
CONCAT(CAST('${book_chapter.title}' AS CHAR),'-',CAST(chapter AS
CHAR)) as solr_id
There are many books out there with the same title and most of them
have a chapter titled "Introduction".
So, your solr_id may or may not be unique depending on your data.
It may help if you have a database view that would hide the complex
queries from solr.
You may also want to start with something simpler, a single entity and
see how it goes, then add the rest.
Thanks.
Arcadius.
On 23 February 2013 10:43, cveres <cs...@me.com> wrote:
> Thanks Gora. I do not have optimise automatically enabled, but I am new to
> Solr so I am not 100% familiar with all the steps that go on.
>
> I will try your suggestion, but I was hoping first that I could get the data
> straight from Solr.
>
> thanks, Csaba
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/DIH-deleting-documents-tp4041811p4042426.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: DIH deleting documents
Posted by cveres <cs...@me.com>.
Thanks Gora. I do not have optimise automatically enabled, but I am new to
Solr so I am not 100% familiar with all the steps that go on.
I will try your suggestion, but I was hoping first that I could get the data
straight from Solr.
thanks, Csaba
--
View this message in context: http://lucene.472066.n3.nabble.com/DIH-deleting-documents-tp4041811p4042426.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: DIH deleting documents
Posted by Gora Mohanty <go...@mimirtech.com>.
On 22 February 2013 12:11, cveres <cs...@me.com> wrote:
> I should also add that some of the books don't have chapters, so the query
> won't succeed for these books.
> But in this case I expected that the document won't be added at all ..
> rather than first added then deleted (which I am now suspecting is the
> case).
> It would be very helpful if I could see a list of deleted documents! I was
> trying to look in the terminal window (Jetty) but that did not help. I don't
> know where else Solr might put logs. I looked in /var/log.. but did not find
> anything useful looking.
The deletion probably happens during the optimize at the
end of the import. I am not sure if there is a way of getting
that logged. However, why can't you do what I suggested
earlier, i.e., run the SELECT for IDs directly against the
database, save it to a text file, and run something like uniq
on it to find out how IDs are getting duplicated.
Regards,
Gora
Re: DIH deleting documents
Posted by cveres <cs...@me.com>.
I should also add that some of the books don't have chapters, so the query
won't succeed for these books.
But in this case I expected that the document won't be added at all ..
rather than first added then deleted (which I am now suspecting is the
case).
It would be very helpful if I could see a list of deleted documents! I was
trying to look in the terminal window (Jetty) but that did not help. I don't
know where else Solr might put logs. I looked in /var/log.. but did not find
anything useful looking.
--
View this message in context: http://lucene.472066.n3.nabble.com/DIH-deleting-documents-tp4041811p4042149.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: DIH deleting documents
Posted by cveres <cs...@me.com>.
Hi Gora and Arcadius,
Thanks for your help. I'll try and answer both your questions here.
I am interested in three database tables. "Book" contains information about
books, "page" has the content of each book page by page, and "chapter"
contains the title of each chapter in every book, and the page on which the
chapter begins. It is a bit of a mess because I need the contents of each
chapter in every book, but I have to infer which pages each chapter contains
by its page number. So there is quite a complex query.
There are 8764 rows in the chapter table .. so 8764 unique chapter headings
.. and 6870 books.
When I import, I get
Num Docs:
2784
Max Doc:
9488
Deleted Docs:
6704
Here is the config file (the relevant part):
<entity name="book_chapter" PK="ID" rootEntity="false"
query="select id as b_id,title,type_id from book">
<entity name="chapter"
query="SELECT CONCAT(CAST('${book_chapter.title}' AS
CHAR),'-',CAST(chapter AS CHAR)) as solr_id, book_id,'chapter' as
entityType,GROUP_CONCAT(content_raw) from (select id as page_id, book_id,
page_no, content_raw, (select title from chapter ch where
(ch.begin_page_no < p.page_no OR ch.begin_page_no = p.page_no) and
ch.book_id = p.book_id and ch.parent_id = 0 order by begin_page_no desc
LIMIT 1) as chapter from page p where book_id = '${book_chapter.b_id}') a
group by chapter">
<field column="solr_id" name="id" />
<field column="title" name="title"/>
<field column="GROUP_CONCAT(content_raw)" name="pageText"/>
<field column="entityType" name="entityType"/>
<entity name="book-type2" query="select name,id from book_type
where id='${book_chapter.type_id}'">
<field column="name" name="contentType"/>
</entity>
</entity>
</entity>
thanks,
Csaba
--
View this message in context: http://lucene.472066.n3.nabble.com/DIH-deleting-documents-tp4041811p4041996.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: DIH deleting documents
Posted by Arcadius Ahouansou <ar...@menelic.com>.
Hi Csaba.
Would you mind posting your DIHconfig/data-config.xml and the command
you use for the import?
Thanks.
Arcadius.
On 21 February 2013 17:55, Gora Mohanty <go...@mimirtech.com> wrote:
> On 21 February 2013 19:30, cveres <cs...@me.com> wrote:
>> Thanks Gora,
>>
>> Sorry I might not have been sufficiently clear.
>>
>> I start with an empty index, then add documents.
>> 9000 are added and 6000 immediately deleted again, leaving 3000.
>> I assume this can only happen with duplicate IDs, but that should not be
>> possible! So I wanted to get a list of deleted documents so that I could try
>> and figure out why they were deleted immediately.
> [...]
>
> What do you mean by "9000 are added and 6000
> immediately deleted again"? How are you getting
> the number added, and the number deleted? How
> many documents does DIH report on the final screen
> after the full-import completes?
>
> From what you describe, it is most likely duplicate IDs.
> Could you do a SELECT from the database outside of
> Solr, create the IDs as you do with DIH, and see what is
> going wrong there?
>
> Regards,
> Gora
Re: DIH deleting documents
Posted by Gora Mohanty <go...@mimirtech.com>.
On 21 February 2013 19:30, cveres <cs...@me.com> wrote:
> Thanks Gora,
>
> Sorry I might not have been sufficiently clear.
>
> I start with an empty index, then add documents.
> 9000 are added and 6000 immediately deleted again, leaving 3000.
> I assume this can only happen with duplicate IDs, but that should not be
> possible! So I wanted to get a list of deleted documents so that I could try
> and figure out why they were deleted immediately.
[...]
What do you mean by "9000 are added and 6000
immediately deleted again"? How are you getting
the number added, and the number deleted? How
many documents does DIH report on the final screen
after the full-import completes?
>From what you describe, it is most likely duplicate IDs.
Could you do a SELECT from the database outside of
Solr, create the IDs as you do with DIH, and see what is
going wrong there?
Regards,
Gora
Re: DIH deleting documents
Posted by cveres <cs...@me.com>.
Thanks Gora,
Sorry I might not have been sufficiently clear.
I start with an empty index, then add documents.
9000 are added and 6000 immediately deleted again, leaving 3000.
I assume this can only happen with duplicate IDs, but that should not be
possible! So I wanted to get a list of deleted documents so that I could try
and figure out why they were deleted immediately.
thanks,
Csaba
--
View this message in context: http://lucene.472066.n3.nabble.com/DIH-deleting-documents-tp4041811p4041887.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: DIH deleting documents
Posted by Gora Mohanty <go...@mimirtech.com>.
On 21 February 2013 14:27, cveres <cs...@me.com> wrote:
> I am adding documents with data import handler from a mysql database. I
> create a unique id for each document by concatenating a couple of fields in
> the database. Every id is unique.
>
> After the import, over half the documents which were imported are deleted
> again, leaving me with less then half the documents in the database ending
> up in the Solr index.
[...]
By default, DIH cleans the index at the beginning, which means
that any older records that are no longer in the mysql database
will disappear from the Solr index. If you do not want that, set
clean=false as an attribute in the URL of the DIH full-import. This
will leave old records in the Solr index, update any records with
matching IDs, and add any new records with IDs that are not in
the Solr index. Please see http://wiki.apache.org/solr/DataImportHandler
Regards,
Gora