You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Jan Høydahl <ja...@cominvent.com> on 2018/01/10 23:42:13 UTC
Mixing simple and nested docs in same update?
Hi,
We index several large nested documents. We found that querying the data behaves differently depending on how the documents are indexed.
To reproduce:
solr start
solr create -c nested
# Index one plain document, “friend" and a nested one, “mother” and “daughter”, in same request:
curl localhost:8983/solr/nested/update -d ‘
<add>
<doc>
<field name="id">friend</field>
<field name="type">other</field>
</doc>
<doc>
<field name="id">mother</field>
<field name="type">parent</field>
<doc>
<field name="id">daughter</field>
<field name="type">child</field>
</doc>
</doc>
</add>'
# Query for mother’s children using either child transformer or child query parser
curl "localhost:8983/solr/a/query?q=id:mother&fl=%2A%2C%5Bchild%20parentFilter%3Dtype%3Aparent%5D”
{
"responseHeader":{
"zkConnected":true,
"status":0,
"QTime":4,
"params":{
"q":"id:mother",
"fl":"*,[child parentFilter=type:parent]"}},
"response":{"numFound":1,"start":0,"docs":[
{
"id":"mother",
"type":["parent"],
"_version_":1589249812802306048,
"type_str":["parent"],
"_childDocuments_":[
{
"id":"friend",
"type":["other"],
"_version_":1589249812729954304,
"type_str":["other"]},
{
"id":"daughter",
"type":["child"],
"_version_":1589249812802306048,
"type_str":["child"]}]}]
}}
As you can see, the “friend” got included as a child of “mother”.
If you index the exact same request, putting “friend” after “mother” in the xml,
the query works as expected.
Inspecting the index, everything looks correct, and only “daughter” and “mother” have _root_=mother.
Is there a rule that you should start a new update request for each type of parent/child relationship
that you need to index, and not mix them in the same request?
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
Re: Mixing simple and nested docs in same update?
Posted by Jan Høydahl <ja...@cominvent.com>.
Hi Mikhail,
Lacking clarity on this in the Ref Guide, I'm trying to understand all requirements for block join here.
I have noticed that if I index the blocks as one ADD request and then afterwards index the "other"
single documents in another request, then the results look ok.
But is it enough with a new ADD to divide the two or do we actually need a COMMIT in between?
I'm also worried that after some index merges that the docs may be mixed up again?
Is there some structure in the segments that prevent that from happening?
Finally, when we some day need to SPLITSHARD, is the SPLIT API aware of blocks so that it will
never split in the middle of a block?
I hope to perhaps update the RefGuide documentation to clarify all of these constraints and pitfalls.
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
> 5. feb. 2018 kl. 20:49 skrev Mikhail Khludnev <mk...@apache.org>:
>
> Jan, mixing plan docs and blocks are not supported.
>
> On Thu, Jan 11, 2018 at 2:42 AM, Jan Høydahl <ja...@cominvent.com> wrote:
>
>> Hi,
>>
>> We index several large nested documents. We found that querying the data
>> behaves differently depending on how the documents are indexed.
>>
>> To reproduce:
>>
>> solr start
>> solr create -c nested
>> # Index one plain document, “friend" and a nested one, “mother” and
>> “daughter”, in same request:
>> curl localhost:8983/solr/nested/update -d ‘
>> <add>
>> <doc>
>> <field name="id">friend</field>
>> <field name="type">other</field>
>> </doc>
>> <doc>
>> <field name="id">mother</field>
>> <field name="type">parent</field>
>> <doc>
>> <field name="id">daughter</field>
>> <field name="type">child</field>
>> </doc>
>> </doc>
>> </add>'
>>
>> # Query for mother’s children using either child transformer or child
>> query parser
>> curl "localhost:8983/solr/a/query?q=id:mother&fl=%2A%2C%5Bchild%
>> 20parentFilter%3Dtype%3Aparent%5D”
>> {
>> "responseHeader":{
>> "zkConnected":true,
>> "status":0,
>> "QTime":4,
>> "params":{
>> "q":"id:mother",
>> "fl":"*,[child parentFilter=type:parent]"}},
>> "response":{"numFound":1,"start":0,"docs":[
>> {
>> "id":"mother",
>> "type":["parent"],
>> "_version_":1589249812802306048,
>> "type_str":["parent"],
>> "_childDocuments_":[
>> {
>> "id":"friend",
>> "type":["other"],
>> "_version_":1589249812729954304,
>> "type_str":["other"]},
>> {
>> "id":"daughter",
>> "type":["child"],
>> "_version_":1589249812802306048,
>> "type_str":["child"]}]}]
>> }}
>>
>> As you can see, the “friend” got included as a child of “mother”.
>> If you index the exact same request, putting “friend” after “mother” in
>> the xml,
>> the query works as expected.
>>
>> Inspecting the index, everything looks correct, and only “daughter” and
>> “mother” have _root_=mother.
>> Is there a rule that you should start a new update request for each type
>> of parent/child relationship
>> that you need to index, and not mix them in the same request?
>>
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>>
>>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
Re: Mixing simple and nested docs in same update?
Posted by Mikhail Khludnev <mk...@apache.org>.
Jan, mixing plan docs and blocks are not supported.
On Thu, Jan 11, 2018 at 2:42 AM, Jan Høydahl <ja...@cominvent.com> wrote:
> Hi,
>
> We index several large nested documents. We found that querying the data
> behaves differently depending on how the documents are indexed.
>
> To reproduce:
>
> solr start
> solr create -c nested
> # Index one plain document, “friend" and a nested one, “mother” and
> “daughter”, in same request:
> curl localhost:8983/solr/nested/update -d ‘
> <add>
> <doc>
> <field name="id">friend</field>
> <field name="type">other</field>
> </doc>
> <doc>
> <field name="id">mother</field>
> <field name="type">parent</field>
> <doc>
> <field name="id">daughter</field>
> <field name="type">child</field>
> </doc>
> </doc>
> </add>'
>
> # Query for mother’s children using either child transformer or child
> query parser
> curl "localhost:8983/solr/a/query?q=id:mother&fl=%2A%2C%5Bchild%
> 20parentFilter%3Dtype%3Aparent%5D”
> {
> "responseHeader":{
> "zkConnected":true,
> "status":0,
> "QTime":4,
> "params":{
> "q":"id:mother",
> "fl":"*,[child parentFilter=type:parent]"}},
> "response":{"numFound":1,"start":0,"docs":[
> {
> "id":"mother",
> "type":["parent"],
> "_version_":1589249812802306048,
> "type_str":["parent"],
> "_childDocuments_":[
> {
> "id":"friend",
> "type":["other"],
> "_version_":1589249812729954304,
> "type_str":["other"]},
> {
> "id":"daughter",
> "type":["child"],
> "_version_":1589249812802306048,
> "type_str":["child"]}]}]
> }}
>
> As you can see, the “friend” got included as a child of “mother”.
> If you index the exact same request, putting “friend” after “mother” in
> the xml,
> the query works as expected.
>
> Inspecting the index, everything looks correct, and only “daughter” and
> “mother” have _root_=mother.
> Is there a rule that you should start a new update request for each type
> of parent/child relationship
> that you need to index, and not mix them in the same request?
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
>
--
Sincerely yours
Mikhail Khludnev
Re: Mixing simple and nested docs in same update?
Posted by Jan Høydahl <ja...@cominvent.com>.
Thanks for the reply.
I see that the child doctransformer (https://lucene.apache.org/solr/guide/6_6/transforming-result-documents.html#TransformingResultDocuments-_child_-ChildDocTransformerFactory) has a childFilter= option which, when used, solves the issue/bug.
But such a childFilter does not exist for the BlockJoin QParsers.
Still not sure whether it is a bug or not...
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
> 31. jan. 2018 kl. 00:30 skrev Tomas Fernandez Lobbe <tf...@apple.com>:
>
> I believe the problem is that:
> * BlockJoin queries do not know about your “types”, in the BlockJoin query world, everything that’s not a parent (matches the parentFilter) is a child.
> * All docs indexed before a parent are considered childs of that doc.
> That’s why in your first case it considers “friend” (not a parent, then a child) to be a child of the first parent it can find in the segment (mother).
> In the second case, the “friend” doc would have no parent. No parent document matches the filter after it, so it’s not considered a match.
> Maybe if you try your query with parentFilter=-type:child, this particular example works (I haven’t tried it)?
>
> Note that when you send docs with childs to Solr, Solr will make sure the childs are indexed before the parent. Also note that there are some other open bugs related to child docs, and in particular, with mixing child docs with non-child docs, depending on which features you need this may be a problem.
>
> Tomás
>
>> On Jan 30, 2018, at 5:48 AM, Jan Høydahl <ja...@cominvent.com> wrote:
>>
>> Pasting the GIST link :-) https://gist.github.com/45640fe3bad696d53ef8a0930a35d163 <https://gist.github.com/45640fe3bad696d53ef8a0930a35d163>
>> Anyone knows if this is expected behavior?
>>
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>>
>>> 15. jan. 2018 kl. 14:08 skrev Jan Høydahl <ja...@cominvent.com>:
>>>
>>> Radio silence…
>>>
>>> Here is a GIST for easy reproduction. Is this by design?
>>>
>>> --
>>> Jan Høydahl, search solution architect
>>> Cominvent AS - www.cominvent.com
>>>
>>>> 11. jan. 2018 kl. 00:42 skrev Jan Høydahl <ja...@cominvent.com>:
>>>>
>>>> Hi,
>>>>
>>>> We index several large nested documents. We found that querying the data behaves differently depending on how the documents are indexed.
>>>>
>>>> To reproduce:
>>>>
>>>> solr start
>>>> solr create -c nested
>>>> # Index one plain document, “friend" and a nested one, “mother” and “daughter”, in same request:
>>>> curl localhost:8983/solr/nested/update -d ‘
>>>> <add>
>>>> <doc>
>>>> <field name="id">friend</field>
>>>> <field name="type">other</field>
>>>> </doc>
>>>> <doc>
>>>> <field name="id">mother</field>
>>>> <field name="type">parent</field>
>>>> <doc>
>>>> <field name="id">daughter</field>
>>>> <field name="type">child</field>
>>>> </doc>
>>>> </doc>
>>>> </add>'
>>>>
>>>> # Query for mother’s children using either child transformer or child query parser
>>>> curl "localhost:8983/solr/a/query?q=id:mother&fl=%2A%2C%5Bchild%20parentFilter%3Dtype%3Aparent%5D”
>>>> {
>>>> "responseHeader":{
>>>> "zkConnected":true,
>>>> "status":0,
>>>> "QTime":4,
>>>> "params":{
>>>> "q":"id:mother",
>>>> "fl":"*,[child parentFilter=type:parent]"}},
>>>> "response":{"numFound":1,"start":0,"docs":[
>>>> {
>>>> "id":"mother",
>>>> "type":["parent"],
>>>> "_version_":1589249812802306048,
>>>> "type_str":["parent"],
>>>> "_childDocuments_":[
>>>> {
>>>> "id":"friend",
>>>> "type":["other"],
>>>> "_version_":1589249812729954304,
>>>> "type_str":["other"]},
>>>> {
>>>> "id":"daughter",
>>>> "type":["child"],
>>>> "_version_":1589249812802306048,
>>>> "type_str":["child"]}]}]
>>>> }}
>>>>
>>>> As you can see, the “friend” got included as a child of “mother”.
>>>> If you index the exact same request, putting “friend” after “mother” in the xml,
>>>> the query works as expected.
>>>>
>>>> Inspecting the index, everything looks correct, and only “daughter” and “mother” have _root_=mother.
>>>> Is there a rule that you should start a new update request for each type of parent/child relationship
>>>> that you need to index, and not mix them in the same request?
>>>>
>>>> --
>>>> Jan Høydahl, search solution architect
>>>> Cominvent AS - www.cominvent.com
>>>>
>>>
>>
>
Re: Mixing simple and nested docs in same update?
Posted by Tomas Fernandez Lobbe <tf...@apple.com>.
I believe the problem is that:
* BlockJoin queries do not know about your “types”, in the BlockJoin query world, everything that’s not a parent (matches the parentFilter) is a child.
* All docs indexed before a parent are considered childs of that doc.
That’s why in your first case it considers “friend” (not a parent, then a child) to be a child of the first parent it can find in the segment (mother).
In the second case, the “friend” doc would have no parent. No parent document matches the filter after it, so it’s not considered a match.
Maybe if you try your query with parentFilter=-type:child, this particular example works (I haven’t tried it)?
Note that when you send docs with childs to Solr, Solr will make sure the childs are indexed before the parent. Also note that there are some other open bugs related to child docs, and in particular, with mixing child docs with non-child docs, depending on which features you need this may be a problem.
Tomás
> On Jan 30, 2018, at 5:48 AM, Jan Høydahl <ja...@cominvent.com> wrote:
>
> Pasting the GIST link :-) https://gist.github.com/45640fe3bad696d53ef8a0930a35d163 <https://gist.github.com/45640fe3bad696d53ef8a0930a35d163>
> Anyone knows if this is expected behavior?
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
>> 15. jan. 2018 kl. 14:08 skrev Jan Høydahl <ja...@cominvent.com>:
>>
>> Radio silence…
>>
>> Here is a GIST for easy reproduction. Is this by design?
>>
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>>
>>> 11. jan. 2018 kl. 00:42 skrev Jan Høydahl <ja...@cominvent.com>:
>>>
>>> Hi,
>>>
>>> We index several large nested documents. We found that querying the data behaves differently depending on how the documents are indexed.
>>>
>>> To reproduce:
>>>
>>> solr start
>>> solr create -c nested
>>> # Index one plain document, “friend" and a nested one, “mother” and “daughter”, in same request:
>>> curl localhost:8983/solr/nested/update -d ‘
>>> <add>
>>> <doc>
>>> <field name="id">friend</field>
>>> <field name="type">other</field>
>>> </doc>
>>> <doc>
>>> <field name="id">mother</field>
>>> <field name="type">parent</field>
>>> <doc>
>>> <field name="id">daughter</field>
>>> <field name="type">child</field>
>>> </doc>
>>> </doc>
>>> </add>'
>>>
>>> # Query for mother’s children using either child transformer or child query parser
>>> curl "localhost:8983/solr/a/query?q=id:mother&fl=%2A%2C%5Bchild%20parentFilter%3Dtype%3Aparent%5D”
>>> {
>>> "responseHeader":{
>>> "zkConnected":true,
>>> "status":0,
>>> "QTime":4,
>>> "params":{
>>> "q":"id:mother",
>>> "fl":"*,[child parentFilter=type:parent]"}},
>>> "response":{"numFound":1,"start":0,"docs":[
>>> {
>>> "id":"mother",
>>> "type":["parent"],
>>> "_version_":1589249812802306048,
>>> "type_str":["parent"],
>>> "_childDocuments_":[
>>> {
>>> "id":"friend",
>>> "type":["other"],
>>> "_version_":1589249812729954304,
>>> "type_str":["other"]},
>>> {
>>> "id":"daughter",
>>> "type":["child"],
>>> "_version_":1589249812802306048,
>>> "type_str":["child"]}]}]
>>> }}
>>>
>>> As you can see, the “friend” got included as a child of “mother”.
>>> If you index the exact same request, putting “friend” after “mother” in the xml,
>>> the query works as expected.
>>>
>>> Inspecting the index, everything looks correct, and only “daughter” and “mother” have _root_=mother.
>>> Is there a rule that you should start a new update request for each type of parent/child relationship
>>> that you need to index, and not mix them in the same request?
>>>
>>> --
>>> Jan Høydahl, search solution architect
>>> Cominvent AS - www.cominvent.com
>>>
>>
>
Re: Mixing simple and nested docs in same update?
Posted by Jan Høydahl <ja...@cominvent.com>.
Pasting the GIST link :-) https://gist.github.com/45640fe3bad696d53ef8a0930a35d163 <https://gist.github.com/45640fe3bad696d53ef8a0930a35d163>
Anyone knows if this is expected behavior?
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
> 15. jan. 2018 kl. 14:08 skrev Jan Høydahl <ja...@cominvent.com>:
>
> Radio silence…
>
> Here is a GIST for easy reproduction. Is this by design?
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
>> 11. jan. 2018 kl. 00:42 skrev Jan Høydahl <ja...@cominvent.com>:
>>
>> Hi,
>>
>> We index several large nested documents. We found that querying the data behaves differently depending on how the documents are indexed.
>>
>> To reproduce:
>>
>> solr start
>> solr create -c nested
>> # Index one plain document, “friend" and a nested one, “mother” and “daughter”, in same request:
>> curl localhost:8983/solr/nested/update -d ‘
>> <add>
>> <doc>
>> <field name="id">friend</field>
>> <field name="type">other</field>
>> </doc>
>> <doc>
>> <field name="id">mother</field>
>> <field name="type">parent</field>
>> <doc>
>> <field name="id">daughter</field>
>> <field name="type">child</field>
>> </doc>
>> </doc>
>> </add>'
>>
>> # Query for mother’s children using either child transformer or child query parser
>> curl "localhost:8983/solr/a/query?q=id:mother&fl=%2A%2C%5Bchild%20parentFilter%3Dtype%3Aparent%5D”
>> {
>> "responseHeader":{
>> "zkConnected":true,
>> "status":0,
>> "QTime":4,
>> "params":{
>> "q":"id:mother",
>> "fl":"*,[child parentFilter=type:parent]"}},
>> "response":{"numFound":1,"start":0,"docs":[
>> {
>> "id":"mother",
>> "type":["parent"],
>> "_version_":1589249812802306048,
>> "type_str":["parent"],
>> "_childDocuments_":[
>> {
>> "id":"friend",
>> "type":["other"],
>> "_version_":1589249812729954304,
>> "type_str":["other"]},
>> {
>> "id":"daughter",
>> "type":["child"],
>> "_version_":1589249812802306048,
>> "type_str":["child"]}]}]
>> }}
>>
>> As you can see, the “friend” got included as a child of “mother”.
>> If you index the exact same request, putting “friend” after “mother” in the xml,
>> the query works as expected.
>>
>> Inspecting the index, everything looks correct, and only “daughter” and “mother” have _root_=mother.
>> Is there a rule that you should start a new update request for each type of parent/child relationship
>> that you need to index, and not mix them in the same request?
>>
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>>
>
Re: Mixing simple and nested docs in same update?
Posted by Jan Høydahl <ja...@cominvent.com>.
Radio silence…
Here is a GIST for easy reproduction. Is this by design?
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
> 11. jan. 2018 kl. 00:42 skrev Jan Høydahl <ja...@cominvent.com>:
>
> Hi,
>
> We index several large nested documents. We found that querying the data behaves differently depending on how the documents are indexed.
>
> To reproduce:
>
> solr start
> solr create -c nested
> # Index one plain document, “friend" and a nested one, “mother” and “daughter”, in same request:
> curl localhost:8983/solr/nested/update -d ‘
> <add>
> <doc>
> <field name="id">friend</field>
> <field name="type">other</field>
> </doc>
> <doc>
> <field name="id">mother</field>
> <field name="type">parent</field>
> <doc>
> <field name="id">daughter</field>
> <field name="type">child</field>
> </doc>
> </doc>
> </add>'
>
> # Query for mother’s children using either child transformer or child query parser
> curl "localhost:8983/solr/a/query?q=id:mother&fl=%2A%2C%5Bchild%20parentFilter%3Dtype%3Aparent%5D”
> {
> "responseHeader":{
> "zkConnected":true,
> "status":0,
> "QTime":4,
> "params":{
> "q":"id:mother",
> "fl":"*,[child parentFilter=type:parent]"}},
> "response":{"numFound":1,"start":0,"docs":[
> {
> "id":"mother",
> "type":["parent"],
> "_version_":1589249812802306048,
> "type_str":["parent"],
> "_childDocuments_":[
> {
> "id":"friend",
> "type":["other"],
> "_version_":1589249812729954304,
> "type_str":["other"]},
> {
> "id":"daughter",
> "type":["child"],
> "_version_":1589249812802306048,
> "type_str":["child"]}]}]
> }}
>
> As you can see, the “friend” got included as a child of “mother”.
> If you index the exact same request, putting “friend” after “mother” in the xml,
> the query works as expected.
>
> Inspecting the index, everything looks correct, and only “daughter” and “mother” have _root_=mother.
> Is there a rule that you should start a new update request for each type of parent/child relationship
> that you need to index, and not mix them in the same request?
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>