You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Scott Prentice <sp...@leximation.com> on 2018/08/29 22:02:14 UTC
Atomic updates and POST command?
Hi...
I'm trying to get atomic updates working and am seeing some strangeness.
Here's my JSON with the data to update ..
[{"id":"/unique/path/id",
"field1":{"set","newvalue1"},
"field2":{"set","newvalue2"}
}]
If I use the REST API via curl it works fine. With the following
command, the field1 and field2 fields get the new values, and all's well.
curl 'http://localhost:8983/solr/core01/update/json?commit=true'
--data-binary @test1.json -H 'Content-type:application/json'
BUT, if I use the post command ..
./bin/post -c core01 /home/xtech/solrtest/test1.json
.. the record gets updated with new fields named "field1.set" and
"field2.set", and the managed-schema file is modified to include these
new field definitions. Not at all what I'd expect or want. Is there some
setting or switch that will let the post command work "properly", or am
I misunderstanding what's correct? I can use curl, but our current
workflow uses the post command so I thought that might do the job.
Any thoughts are welcome!
Thanks,
...scott
Re: BUMP: Atomic updates and POST command?
Posted by Scott Prentice <sp...@leximation.com>.
Thanks, Shawn. That helps with the meaning of the "solr" format.
Our needs are pretty basic. We have some upstream processes that crawl
the data and generate a JSON feed that works with the default post
command. So far this works well and keeps things simple.
Thanks!
...scott
On 9/1/18 9:26 PM, Shawn Heisey wrote:
> On 8/31/2018 7:18 PM, Scott Prentice wrote:
>> Yup. That does the trick! Here's my command line ..
>>
>> $ ./bin/post -c core01 -format solr /home/xtech/solrtest/test1b.json
>>
>> I saw that "-format solr" option, but it wasn't clear what it did.
>> It's still not clear to me how that changes the endpoint to allow for
>> updates. But nice to see that it works!
>
> I think the assumption with JSON-style updates and the post tool is
> that you are sending "generic" json documents, not Solr-formatted json
> commands. So the post tool sends to the /update/json/docs handler,
> which can handle those easily. I believe that telling it that the
> format is "solr" means that the JSON input is instructions to Solr,
> not just document content. It very likely sends it to /update/json or
> /update when that's the case.
>
> I don't know if you know this, but the bin/post command calls
> something in Solr that is named SimplePostTool. It is, as that name
> suggests, a very simple tool. Although you CAN use it in production,
> a large percentage of users find that they outgrow its capabilities
> and must move to writing their own indexing system.
>
> Thanks,
> Shawn
>
>
Re: BUMP: Atomic updates and POST command?
Posted by Shawn Heisey <ap...@elyograg.org>.
On 8/31/2018 7:18 PM, Scott Prentice wrote:
> Yup. That does the trick! Here's my command line ..
>
> $ ./bin/post -c core01 -format solr /home/xtech/solrtest/test1b.json
>
> I saw that "-format solr" option, but it wasn't clear what it did.
> It's still not clear to me how that changes the endpoint to allow for
> updates. But nice to see that it works!
I think the assumption with JSON-style updates and the post tool is that
you are sending "generic" json documents, not Solr-formatted json
commands. So the post tool sends to the /update/json/docs handler,
which can handle those easily. I believe that telling it that the
format is "solr" means that the JSON input is instructions to Solr, not
just document content. It very likely sends it to /update/json or
/update when that's the case.
I don't know if you know this, but the bin/post command calls something
in Solr that is named SimplePostTool. It is, as that name suggests, a
very simple tool. Although you CAN use it in production, a large
percentage of users find that they outgrow its capabilities and must
move to writing their own indexing system.
Thanks,
Shawn
Re: BUMP: Atomic updates and POST command?
Posted by Scott Prentice <sp...@leximation.com>.
Yup. That does the trick! Here's my command line ..
$ ./bin/post -c core01 -format solr /home/xtech/solrtest/test1b.json
I saw that "-format solr" option, but it wasn't clear what it did. It's
still not clear to me how that changes the endpoint to allow for
updates. But nice to see that it works!
Thanks for your help!
...scott
On 8/31/18 6:04 PM, Alexandre Rafalovitch wrote:
> Ok,
>
> Try "-format solr" instead of "-url ...".
>
> Regards,
> Alex.
>
> On 31 August 2018 at 20:54, Scott Prentice <sp...@leximation.com> wrote:
>> Nope. That's not it. It complains about this path not being found ..
>>
>> /solr/core01/update/json/json/docs
>>
>> So, I changed the -url value to this
>> "http://localhost:8983/solr/core01/update" .. which was "successful", but
>> created the same odd index structure of "field.set".
>>
>> I'm clearly flailing. If you have any thoughts on this, do let me know.
>>
>> Thanks!
>> ...scott
>>
>>
>>
>> On 8/31/18 5:20 PM, Scott Prentice wrote:
>>> Ah .. is this done with the -url parameter? As in ..
>>>
>>> ./bin/post -url http://localhost:8983/solr/core01/update/json
>>> /home/xtech/solrtest/test1.json
>>>
>>> Will test.
>>>
>>> Thanks,
>>> ...scott
>>>
>>>
>>> On 8/31/18 5:15 PM, Scott Prentice wrote:
>>>> Hmm. That makes sense .. but where do you provide the endpoint to post?
>>>> Is that additional commands within the JSON or a parameter at the command
>>>> line?
>>>>
>>>> Thanks,
>>>> ...scott
>>>>
>>>>
>>>> On 8/31/18 4:48 PM, Alexandre Rafalovitch wrote:
>>>>> I think you are using different end points there. /update by default vs
>>>>> /update/json
>>>>>
>>>>> So i think the post gets treated as generic json parsing.
>>>>>
>>>>> Can you try the same end point?
>>>>>
>>>>> Regards,
>>>>> Alex
>>>>>
>>>>>
>>>>> On Fri, Aug 31, 2018, 7:05 PM Scott Prentice wrote:
>>>>>
>>>>>> Just bumping this post from a few days ago.
>>>>>>
>>>>>> Is anyone using atomic updates? If so, how are you passing the updates
>>>>>> to Solr? I'm seeing a significant difference between the REST API and
>>>>>> the post command .. is this to be expected? What's the recommended
>>>>>> method for doing the update?
>>>>>>
>>>>>> Thanks!
>>>>>> ...scott
>>>>>>
>>>>>>
>>>>>> On 8/29/18 3:02 PM, Scott Prentice wrote:
>>>>>>> Hi...
>>>>>>>
>>>>>>> I'm trying to get atomic updates working and am seeing some
>>>>>>> strangeness. Here's my JSON with the data to update ..
>>>>>>>
>>>>>>> [{"id":"/unique/path/id",
>>>>>>> "field1":{"set","newvalue1"},
>>>>>>> "field2":{"set","newvalue2"}
>>>>>>> }]
>>>>>>>
>>>>>>> If I use the REST API via curl it works fine. With the following
>>>>>>> command, the field1 and field2 fields get the new values, and all's
>>>>>>> well.
>>>>>>>
>>>>>>> curl 'http://localhost:8983/solr/core01/update/json?commit=true'
>>>>>>> --data-binary @test1.json -H 'Content-type:application/json'
>>>>>>>
>>>>>>> BUT, if I use the post command ..
>>>>>>>
>>>>>>> ./bin/post -c core01 /home/xtech/solrtest/test1.json
>>>>>>>
>>>>>>> .. the record gets updated with new fields named "field1.set" and
>>>>>>> "field2.set", and the managed-schema file is modified to include these
>>>>>>> new field definitions. Not at all what I'd expect or want. Is there
>>>>>>> some setting or switch that will let the post command work "properly",
>>>>>>> or am I misunderstanding what's correct? I can use curl, but our
>>>>>>> current workflow uses the post command so I thought that might do the
>>>>>>> job.
>>>>>>>
>>>>>>> Any thoughts are welcome!
>>>>>>>
>>>>>>> Thanks,
>>>>>>> ...scott
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>
>>>
Re: BUMP: Atomic updates and POST command?
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Ok,
Try "-format solr" instead of "-url ...".
Regards,
Alex.
On 31 August 2018 at 20:54, Scott Prentice <sp...@leximation.com> wrote:
> Nope. That's not it. It complains about this path not being found ..
>
> /solr/core01/update/json/json/docs
>
> So, I changed the -url value to this
> "http://localhost:8983/solr/core01/update" .. which was "successful", but
> created the same odd index structure of "field.set".
>
> I'm clearly flailing. If you have any thoughts on this, do let me know.
>
> Thanks!
> ...scott
>
>
>
> On 8/31/18 5:20 PM, Scott Prentice wrote:
>>
>> Ah .. is this done with the -url parameter? As in ..
>>
>> ./bin/post -url http://localhost:8983/solr/core01/update/json
>> /home/xtech/solrtest/test1.json
>>
>> Will test.
>>
>> Thanks,
>> ...scott
>>
>>
>> On 8/31/18 5:15 PM, Scott Prentice wrote:
>>>
>>> Hmm. That makes sense .. but where do you provide the endpoint to post?
>>> Is that additional commands within the JSON or a parameter at the command
>>> line?
>>>
>>> Thanks,
>>> ...scott
>>>
>>>
>>> On 8/31/18 4:48 PM, Alexandre Rafalovitch wrote:
>>>>
>>>> I think you are using different end points there. /update by default vs
>>>> /update/json
>>>>
>>>> So i think the post gets treated as generic json parsing.
>>>>
>>>> Can you try the same end point?
>>>>
>>>> Regards,
>>>> Alex
>>>>
>>>>
>>>> On Fri, Aug 31, 2018, 7:05 PM Scott Prentice wrote:
>>>>
>>>>> Just bumping this post from a few days ago.
>>>>>
>>>>> Is anyone using atomic updates? If so, how are you passing the updates
>>>>> to Solr? I'm seeing a significant difference between the REST API and
>>>>> the post command .. is this to be expected? What's the recommended
>>>>> method for doing the update?
>>>>>
>>>>> Thanks!
>>>>> ...scott
>>>>>
>>>>>
>>>>> On 8/29/18 3:02 PM, Scott Prentice wrote:
>>>>>>
>>>>>> Hi...
>>>>>>
>>>>>> I'm trying to get atomic updates working and am seeing some
>>>>>> strangeness. Here's my JSON with the data to update ..
>>>>>>
>>>>>> [{"id":"/unique/path/id",
>>>>>> "field1":{"set","newvalue1"},
>>>>>> "field2":{"set","newvalue2"}
>>>>>> }]
>>>>>>
>>>>>> If I use the REST API via curl it works fine. With the following
>>>>>> command, the field1 and field2 fields get the new values, and all's
>>>>>> well.
>>>>>>
>>>>>> curl 'http://localhost:8983/solr/core01/update/json?commit=true'
>>>>>> --data-binary @test1.json -H 'Content-type:application/json'
>>>>>>
>>>>>> BUT, if I use the post command ..
>>>>>>
>>>>>> ./bin/post -c core01 /home/xtech/solrtest/test1.json
>>>>>>
>>>>>> .. the record gets updated with new fields named "field1.set" and
>>>>>> "field2.set", and the managed-schema file is modified to include these
>>>>>> new field definitions. Not at all what I'd expect or want. Is there
>>>>>> some setting or switch that will let the post command work "properly",
>>>>>> or am I misunderstanding what's correct? I can use curl, but our
>>>>>> current workflow uses the post command so I thought that might do the
>>>>>> job.
>>>>>>
>>>>>> Any thoughts are welcome!
>>>>>>
>>>>>> Thanks,
>>>>>> ...scott
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>>>
>>
>>
>
Re: BUMP: Atomic updates and POST command?
Posted by Scott Prentice <sp...@leximation.com>.
Nope. That's not it. It complains about this path not being found ..
/solr/core01/update/json/json/docs
So, I changed the -url value to this
"http://localhost:8983/solr/core01/update" .. which was "successful",
but created the same odd index structure of "field.set".
I'm clearly flailing. If you have any thoughts on this, do let me know.
Thanks!
...scott
On 8/31/18 5:20 PM, Scott Prentice wrote:
> Ah .. is this done with the -url parameter? As in ..
>
> ./bin/post -url http://localhost:8983/solr/core01/update/json
> /home/xtech/solrtest/test1.json
>
> Will test.
>
> Thanks,
> ...scott
>
>
> On 8/31/18 5:15 PM, Scott Prentice wrote:
>> Hmm. That makes sense .. but where do you provide the endpoint to
>> post? Is that additional commands within the JSON or a parameter at
>> the command line?
>>
>> Thanks,
>> ...scott
>>
>>
>> On 8/31/18 4:48 PM, Alexandre Rafalovitch wrote:
>>> I think you are using different end points there. /update by default vs
>>> /update/json
>>>
>>> So i think the post gets treated as generic json parsing.
>>>
>>> Can you try the same end point?
>>>
>>> Regards,
>>> Alex
>>>
>>>
>>> On Fri, Aug 31, 2018, 7:05 PM Scott Prentice wrote:
>>>
>>>> Just bumping this post from a few days ago.
>>>>
>>>> Is anyone using atomic updates? If so, how are you passing the updates
>>>> to Solr? I'm seeing a significant difference between the REST API and
>>>> the post command .. is this to be expected? What's the recommended
>>>> method for doing the update?
>>>>
>>>> Thanks!
>>>> ...scott
>>>>
>>>>
>>>> On 8/29/18 3:02 PM, Scott Prentice wrote:
>>>>> Hi...
>>>>>
>>>>> I'm trying to get atomic updates working and am seeing some
>>>>> strangeness. Here's my JSON with the data to update ..
>>>>>
>>>>> [{"id":"/unique/path/id",
>>>>> "field1":{"set","newvalue1"},
>>>>> "field2":{"set","newvalue2"}
>>>>> }]
>>>>>
>>>>> If I use the REST API via curl it works fine. With the following
>>>>> command, the field1 and field2 fields get the new values, and
>>>>> all's well.
>>>>>
>>>>> curl 'http://localhost:8983/solr/core01/update/json?commit=true'
>>>>> --data-binary @test1.json -H 'Content-type:application/json'
>>>>>
>>>>> BUT, if I use the post command ..
>>>>>
>>>>> ./bin/post -c core01 /home/xtech/solrtest/test1.json
>>>>>
>>>>> .. the record gets updated with new fields named "field1.set" and
>>>>> "field2.set", and the managed-schema file is modified to include
>>>>> these
>>>>> new field definitions. Not at all what I'd expect or want. Is there
>>>>> some setting or switch that will let the post command work
>>>>> "properly",
>>>>> or am I misunderstanding what's correct? I can use curl, but our
>>>>> current workflow uses the post command so I thought that might do the
>>>>> job.
>>>>>
>>>>> Any thoughts are welcome!
>>>>>
>>>>> Thanks,
>>>>> ...scott
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>
>>
>
>
Re: BUMP: Atomic updates and POST command?
Posted by Scott Prentice <sp...@leximation.com>.
Ah .. is this done with the -url parameter? As in ..
./bin/post -url http://localhost:8983/solr/core01/update/json
/home/xtech/solrtest/test1.json
Will test.
Thanks,
...scott
On 8/31/18 5:15 PM, Scott Prentice wrote:
> Hmm. That makes sense .. but where do you provide the endpoint to
> post? Is that additional commands within the JSON or a parameter at
> the command line?
>
> Thanks,
> ...scott
>
>
> On 8/31/18 4:48 PM, Alexandre Rafalovitch wrote:
>> I think you are using different end points there. /update by default vs
>> /update/json
>>
>> So i think the post gets treated as generic json parsing.
>>
>> Can you try the same end point?
>>
>> Regards,
>> Alex
>>
>>
>> On Fri, Aug 31, 2018, 7:05 PM Scott Prentice wrote:
>>
>>> Just bumping this post from a few days ago.
>>>
>>> Is anyone using atomic updates? If so, how are you passing the updates
>>> to Solr? I'm seeing a significant difference between the REST API and
>>> the post command .. is this to be expected? What's the recommended
>>> method for doing the update?
>>>
>>> Thanks!
>>> ...scott
>>>
>>>
>>> On 8/29/18 3:02 PM, Scott Prentice wrote:
>>>> Hi...
>>>>
>>>> I'm trying to get atomic updates working and am seeing some
>>>> strangeness. Here's my JSON with the data to update ..
>>>>
>>>> [{"id":"/unique/path/id",
>>>> "field1":{"set","newvalue1"},
>>>> "field2":{"set","newvalue2"}
>>>> }]
>>>>
>>>> If I use the REST API via curl it works fine. With the following
>>>> command, the field1 and field2 fields get the new values, and all's
>>>> well.
>>>>
>>>> curl 'http://localhost:8983/solr/core01/update/json?commit=true'
>>>> --data-binary @test1.json -H 'Content-type:application/json'
>>>>
>>>> BUT, if I use the post command ..
>>>>
>>>> ./bin/post -c core01 /home/xtech/solrtest/test1.json
>>>>
>>>> .. the record gets updated with new fields named "field1.set" and
>>>> "field2.set", and the managed-schema file is modified to include these
>>>> new field definitions. Not at all what I'd expect or want. Is there
>>>> some setting or switch that will let the post command work "properly",
>>>> or am I misunderstanding what's correct? I can use curl, but our
>>>> current workflow uses the post command so I thought that might do the
>>>> job.
>>>>
>>>> Any thoughts are welcome!
>>>>
>>>> Thanks,
>>>> ...scott
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>
>
Re: BUMP: Atomic updates and POST command?
Posted by Scott Prentice <sp...@leximation.com>.
Hmm. That makes sense .. but where do you provide the endpoint to post?
Is that additional commands within the JSON or a parameter at the
command line?
Thanks,
...scott
On 8/31/18 4:48 PM, Alexandre Rafalovitch wrote:
> I think you are using different end points there. /update by default vs
> /update/json
>
> So i think the post gets treated as generic json parsing.
>
> Can you try the same end point?
>
> Regards,
> Alex
>
>
> On Fri, Aug 31, 2018, 7:05 PM Scott Prentice wrote:
>
>> Just bumping this post from a few days ago.
>>
>> Is anyone using atomic updates? If so, how are you passing the updates
>> to Solr? I'm seeing a significant difference between the REST API and
>> the post command .. is this to be expected? What's the recommended
>> method for doing the update?
>>
>> Thanks!
>> ...scott
>>
>>
>> On 8/29/18 3:02 PM, Scott Prentice wrote:
>>> Hi...
>>>
>>> I'm trying to get atomic updates working and am seeing some
>>> strangeness. Here's my JSON with the data to update ..
>>>
>>> [{"id":"/unique/path/id",
>>> "field1":{"set","newvalue1"},
>>> "field2":{"set","newvalue2"}
>>> }]
>>>
>>> If I use the REST API via curl it works fine. With the following
>>> command, the field1 and field2 fields get the new values, and all's well.
>>>
>>> curl 'http://localhost:8983/solr/core01/update/json?commit=true'
>>> --data-binary @test1.json -H 'Content-type:application/json'
>>>
>>> BUT, if I use the post command ..
>>>
>>> ./bin/post -c core01 /home/xtech/solrtest/test1.json
>>>
>>> .. the record gets updated with new fields named "field1.set" and
>>> "field2.set", and the managed-schema file is modified to include these
>>> new field definitions. Not at all what I'd expect or want. Is there
>>> some setting or switch that will let the post command work "properly",
>>> or am I misunderstanding what's correct? I can use curl, but our
>>> current workflow uses the post command so I thought that might do the
>>> job.
>>>
>>> Any thoughts are welcome!
>>>
>>> Thanks,
>>> ...scott
>>>
>>>
>>>
>>>
>>>
>>
Re: BUMP: Atomic updates and POST command?
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
I think you are using different end points there. /update by default vs
/update/json
So i think the post gets treated as generic json parsing.
Can you try the same end point?
Regards,
Alex
On Fri, Aug 31, 2018, 7:05 PM Scott Prentice, <sp...@leximation.com> wrote:
> Just bumping this post from a few days ago.
>
> Is anyone using atomic updates? If so, how are you passing the updates
> to Solr? I'm seeing a significant difference between the REST API and
> the post command .. is this to be expected? What's the recommended
> method for doing the update?
>
> Thanks!
> ...scott
>
>
> On 8/29/18 3:02 PM, Scott Prentice wrote:
> > Hi...
> >
> > I'm trying to get atomic updates working and am seeing some
> > strangeness. Here's my JSON with the data to update ..
> >
> > [{"id":"/unique/path/id",
> > "field1":{"set","newvalue1"},
> > "field2":{"set","newvalue2"}
> > }]
> >
> > If I use the REST API via curl it works fine. With the following
> > command, the field1 and field2 fields get the new values, and all's well.
> >
> > curl 'http://localhost:8983/solr/core01/update/json?commit=true'
> > --data-binary @test1.json -H 'Content-type:application/json'
> >
> > BUT, if I use the post command ..
> >
> > ./bin/post -c core01 /home/xtech/solrtest/test1.json
> >
> > .. the record gets updated with new fields named "field1.set" and
> > "field2.set", and the managed-schema file is modified to include these
> > new field definitions. Not at all what I'd expect or want. Is there
> > some setting or switch that will let the post command work "properly",
> > or am I misunderstanding what's correct? I can use curl, but our
> > current workflow uses the post command so I thought that might do the
> > job.
> >
> > Any thoughts are welcome!
> >
> > Thanks,
> > ...scott
> >
> >
> >
> >
> >
>
>
BUMP: Atomic updates and POST command?
Posted by Scott Prentice <sp...@leximation.com>.
Just bumping this post from a few days ago.
Is anyone using atomic updates? If so, how are you passing the updates
to Solr? I'm seeing a significant difference between the REST API and
the post command .. is this to be expected? What's the recommended
method for doing the update?
Thanks!
...scott
On 8/29/18 3:02 PM, Scott Prentice wrote:
> Hi...
>
> I'm trying to get atomic updates working and am seeing some
> strangeness. Here's my JSON with the data to update ..
>
> [{"id":"/unique/path/id",
> "field1":{"set","newvalue1"},
> "field2":{"set","newvalue2"}
> }]
>
> If I use the REST API via curl it works fine. With the following
> command, the field1 and field2 fields get the new values, and all's well.
>
> curl 'http://localhost:8983/solr/core01/update/json?commit=true'
> --data-binary @test1.json -H 'Content-type:application/json'
>
> BUT, if I use the post command ..
>
> ./bin/post -c core01 /home/xtech/solrtest/test1.json
>
> .. the record gets updated with new fields named "field1.set" and
> "field2.set", and the managed-schema file is modified to include these
> new field definitions. Not at all what I'd expect or want. Is there
> some setting or switch that will let the post command work "properly",
> or am I misunderstanding what's correct? I can use curl, but our
> current workflow uses the post command so I thought that might do the
> job.
>
> Any thoughts are welcome!
>
> Thanks,
> ...scott
>
>
>
>
>