Problems running ES 5.x on pm-test

[sebb <se...@gmail.com>]

The test VM is now running ES 5.x with a basic database.
The Apache config appears OK, Javascript is able to invoke lua scripts OK.

However a couple of the queries fail:

api/pminfo.lua:169: Backend Database returned code 400!

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"No
search type for
[scan]"}],"type":"illegal_argument_exception","reason":"No search type
for [scan]"},"status":400}

api/stats.lua:410: Backend Database returned code 400!

{"error":{"root_cause":[{"type":"parsing_exception","reason":"no
[query] registered for
[limit]","line":1,"col":405}],"type":"parsing_exception","reason":"no
[query] registered for [limit]","line":1,"col":405},"status":400}

I have tried the stats.lua query directly in curl and it fails with
the same error, so this is not a Lua issue.

It looks like the syntax for these particular queries has changed in 5.x.

However I was told that lists.a.o was running on 5.x.
If so, then I assume it must have had some fixes to the syntax which
have not been back-ported to the mainline code?

Anyone got any idea what needs fixing?

Re: Problems running ES 5.x on pm-test

[sebb <se...@gmail.com>]

On 3 January 2017 at 16:26, Daniel Gruno <hu...@apache.org> wrote:
> On 01/03/2017 05:23 PM, sebb wrote:
>> On 3 January 2017 at 14:47, Daniel Gruno <hu...@apache.org> wrote:
>>> On 01/03/2017 03:44 PM, sebb wrote:
>>>> On 3 January 2017 at 12:33, Daniel Gruno <hu...@apache.org> wrote:
>>>>> On 01/03/2017 02:12 AM, sebb wrote:
>>>>>> The test VM is now running ES 5.x with a basic database.
>>>>>> The Apache config appears OK, Javascript is able to invoke lua scripts OK.
>>>>>>
>>>>>> However a couple of the queries fail:
>>>>>>
>>>>>> api/pminfo.lua:169: Backend Database returned code 400!
>>>>>>
>>>>>> {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"No
>>>>>> search type for
>>>>>> [scan]"}],"type":"illegal_argument_exception","reason":"No search type
>>>>>> for [scan]"},"status":400}
>>>>>
>>>>> search_type changed in 5.x to mean something else, and should be omitted
>>>>> in scan function in elastic.lua, AIUI.
>>>>>
>>>>>>
>>>>>> api/stats.lua:410: Backend Database returned code 400!
>>>>>>
>>>>>> {"error":{"root_cause":[{"type":"parsing_exception","reason":"no
>>>>>> [query] registered for
>>>>>> [limit]","line":1,"col":405}],"type":"parsing_exception","reason":"no
>>>>>> [query] registered for [limit]","line":1,"col":405},"status":400}
>>>>>
>>>>> limit is a thing of the past, and the size parameter has changed as well
>>>>> (it not denotes the total docs in the next scroll, not per shard/node).
>>>>> It might be easier to adjust this for ES 2.3/5.1 and not support earlier
>>>>> versions of ElasticSearch going forward.
>>>>
>>>> +1
>>>>
>>>>> I seem to also recall some harmonization needed between hard configured
>>>>> limits to results and how the scroll API works. I'll look into that ASAP.
>>>>
>>>> So what does lists.apache.org use for the queries?
>>>> Or does it not use ES 5.x?
>>>
>>> There are minor proprietary edits, but most importantly, it doesn't use
>>> the word cloud feature, so the issue is not present there.
>>
>> Disabling word-cloud avoids the crash in stats.lua, but pminfo.lua still fails.
>> Not surprising as that does not depend on the word cloud setting.
>> I have tried with both the current and 0.9 code.
>>
>> The 0.9 JS ignores the error so it is not so obvious.
>>
>> However lists.a.o does not show the problem.
>> So there must be some other code difference in lists.a.o to allow it
>> to work with ES 5.x
>
> That bit is just removing 'search_type=scan' in the scan function in
> elastic.lua

AFAIK that's not the only change needed; the response changes as well [2]
This says:
"The response of the initial search request will not contain any
results in the hits array.
The first results will be returned by the first scroll request. "

> as that changed with 5.x (or maybe 2.x, I can't remember
> and the docs don't say).

It was deprecated in 2.1 and dropped in 5.0 [1]
The doc says ES is just as efficient, but that only applies if one
sorts by the "_doc" field which mostly we don't want to do.

> Before we do that, we should agree on only supporting 2.x an onwards.

I agree that the version support strategy needs to be decided soon.

However AFAICT it's not a blocker for fixing the code to support 5.x.


[1] https://www.elastic.co/guide/en/elasticsearch/reference/5.1/breaking_50_search_changes.html#_literal_search_type_scan_literal_removed
[2] https://www.elastic.co/guide/en/elasticsearch/reference/2.0/search-request-scroll.html#scroll-scan

> With regards,
> Daniel.
>
>>
>>>>
>>>>>>
>>>>>> I have tried the stats.lua query directly in curl and it fails with
>>>>>> the same error, so this is not a Lua issue.
>>>>>>
>>>>>> It looks like the syntax for these particular queries has changed in 5.x.
>>>>>>
>>>>>> However I was told that lists.a.o was running on 5.x.
>>>>>> If so, then I assume it must have had some fixes to the syntax which
>>>>>> have not been back-ported to the mainline code?
>>>>>>
>>>>>> Anyone got any idea what needs fixing?
>>>>>>
>>>>>
>>>
>

Re: Problems running ES 5.x on pm-test

[Daniel Gruno <hu...@apache.org>]

On 01/03/2017 05:23 PM, sebb wrote:
> On 3 January 2017 at 14:47, Daniel Gruno <hu...@apache.org> wrote:
>> On 01/03/2017 03:44 PM, sebb wrote:
>>> On 3 January 2017 at 12:33, Daniel Gruno <hu...@apache.org> wrote:
>>>> On 01/03/2017 02:12 AM, sebb wrote:
>>>>> The test VM is now running ES 5.x with a basic database.
>>>>> The Apache config appears OK, Javascript is able to invoke lua scripts OK.
>>>>>
>>>>> However a couple of the queries fail:
>>>>>
>>>>> api/pminfo.lua:169: Backend Database returned code 400!
>>>>>
>>>>> {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"No
>>>>> search type for
>>>>> [scan]"}],"type":"illegal_argument_exception","reason":"No search type
>>>>> for [scan]"},"status":400}
>>>>
>>>> search_type changed in 5.x to mean something else, and should be omitted
>>>> in scan function in elastic.lua, AIUI.
>>>>
>>>>>
>>>>> api/stats.lua:410: Backend Database returned code 400!
>>>>>
>>>>> {"error":{"root_cause":[{"type":"parsing_exception","reason":"no
>>>>> [query] registered for
>>>>> [limit]","line":1,"col":405}],"type":"parsing_exception","reason":"no
>>>>> [query] registered for [limit]","line":1,"col":405},"status":400}
>>>>
>>>> limit is a thing of the past, and the size parameter has changed as well
>>>> (it not denotes the total docs in the next scroll, not per shard/node).
>>>> It might be easier to adjust this for ES 2.3/5.1 and not support earlier
>>>> versions of ElasticSearch going forward.
>>>
>>> +1
>>>
>>>> I seem to also recall some harmonization needed between hard configured
>>>> limits to results and how the scroll API works. I'll look into that ASAP.
>>>
>>> So what does lists.apache.org use for the queries?
>>> Or does it not use ES 5.x?
>>
>> There are minor proprietary edits, but most importantly, it doesn't use
>> the word cloud feature, so the issue is not present there.
> 
> Disabling word-cloud avoids the crash in stats.lua, but pminfo.lua still fails.
> Not surprising as that does not depend on the word cloud setting.
> I have tried with both the current and 0.9 code.
> 
> The 0.9 JS ignores the error so it is not so obvious.
> 
> However lists.a.o does not show the problem.
> So there must be some other code difference in lists.a.o to allow it
> to work with ES 5.x

That bit is just removing 'search_type=scan' in the scan function in
elastic.lua, as that changed with 5.x (or maybe 2.x, I can't remember
and the docs don't say).

Before we do that, we should agree on only supporting 2.x an onwards.

With regards,
Daniel.

> 
>>>
>>>>>
>>>>> I have tried the stats.lua query directly in curl and it fails with
>>>>> the same error, so this is not a Lua issue.
>>>>>
>>>>> It looks like the syntax for these particular queries has changed in 5.x.
>>>>>
>>>>> However I was told that lists.a.o was running on 5.x.
>>>>> If so, then I assume it must have had some fixes to the syntax which
>>>>> have not been back-ported to the mainline code?
>>>>>
>>>>> Anyone got any idea what needs fixing?
>>>>>
>>>>
>>

Re: Problems running ES 5.x on pm-test

[sebb <se...@gmail.com>]

On 3 January 2017 at 14:47, Daniel Gruno <hu...@apache.org> wrote:
> On 01/03/2017 03:44 PM, sebb wrote:
>> On 3 January 2017 at 12:33, Daniel Gruno <hu...@apache.org> wrote:
>>> On 01/03/2017 02:12 AM, sebb wrote:
>>>> The test VM is now running ES 5.x with a basic database.
>>>> The Apache config appears OK, Javascript is able to invoke lua scripts OK.
>>>>
>>>> However a couple of the queries fail:
>>>>
>>>> api/pminfo.lua:169: Backend Database returned code 400!
>>>>
>>>> {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"No
>>>> search type for
>>>> [scan]"}],"type":"illegal_argument_exception","reason":"No search type
>>>> for [scan]"},"status":400}
>>>
>>> search_type changed in 5.x to mean something else, and should be omitted
>>> in scan function in elastic.lua, AIUI.
>>>
>>>>
>>>> api/stats.lua:410: Backend Database returned code 400!
>>>>
>>>> {"error":{"root_cause":[{"type":"parsing_exception","reason":"no
>>>> [query] registered for
>>>> [limit]","line":1,"col":405}],"type":"parsing_exception","reason":"no
>>>> [query] registered for [limit]","line":1,"col":405},"status":400}
>>>
>>> limit is a thing of the past, and the size parameter has changed as well
>>> (it not denotes the total docs in the next scroll, not per shard/node).
>>> It might be easier to adjust this for ES 2.3/5.1 and not support earlier
>>> versions of ElasticSearch going forward.
>>
>> +1
>>
>>> I seem to also recall some harmonization needed between hard configured
>>> limits to results and how the scroll API works. I'll look into that ASAP.
>>
>> So what does lists.apache.org use for the queries?
>> Or does it not use ES 5.x?
>
> There are minor proprietary edits, but most importantly, it doesn't use
> the word cloud feature, so the issue is not present there.

Disabling word-cloud avoids the crash in stats.lua, but pminfo.lua still fails.
Not surprising as that does not depend on the word cloud setting.
I have tried with both the current and 0.9 code.

The 0.9 JS ignores the error so it is not so obvious.

However lists.a.o does not show the problem.
So there must be some other code difference in lists.a.o to allow it
to work with ES 5.x

>>
>>>>
>>>> I have tried the stats.lua query directly in curl and it fails with
>>>> the same error, so this is not a Lua issue.
>>>>
>>>> It looks like the syntax for these particular queries has changed in 5.x.
>>>>
>>>> However I was told that lists.a.o was running on 5.x.
>>>> If so, then I assume it must have had some fixes to the syntax which
>>>> have not been back-ported to the mainline code?
>>>>
>>>> Anyone got any idea what needs fixing?
>>>>
>>>
>

Re: Problems running ES 5.x on pm-test

[Daniel Gruno <hu...@apache.org>]

On 01/03/2017 03:44 PM, sebb wrote:
> On 3 January 2017 at 12:33, Daniel Gruno <hu...@apache.org> wrote:
>> On 01/03/2017 02:12 AM, sebb wrote:
>>> The test VM is now running ES 5.x with a basic database.
>>> The Apache config appears OK, Javascript is able to invoke lua scripts OK.
>>>
>>> However a couple of the queries fail:
>>>
>>> api/pminfo.lua:169: Backend Database returned code 400!
>>>
>>> {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"No
>>> search type for
>>> [scan]"}],"type":"illegal_argument_exception","reason":"No search type
>>> for [scan]"},"status":400}
>>
>> search_type changed in 5.x to mean something else, and should be omitted
>> in scan function in elastic.lua, AIUI.
>>
>>>
>>> api/stats.lua:410: Backend Database returned code 400!
>>>
>>> {"error":{"root_cause":[{"type":"parsing_exception","reason":"no
>>> [query] registered for
>>> [limit]","line":1,"col":405}],"type":"parsing_exception","reason":"no
>>> [query] registered for [limit]","line":1,"col":405},"status":400}
>>
>> limit is a thing of the past, and the size parameter has changed as well
>> (it not denotes the total docs in the next scroll, not per shard/node).
>> It might be easier to adjust this for ES 2.3/5.1 and not support earlier
>> versions of ElasticSearch going forward.
> 
> +1
> 
>> I seem to also recall some harmonization needed between hard configured
>> limits to results and how the scroll API works. I'll look into that ASAP.
> 
> So what does lists.apache.org use for the queries?
> Or does it not use ES 5.x?

There are minor proprietary edits, but most importantly, it doesn't use
the word cloud feature, so the issue is not present there.

> 
>>>
>>> I have tried the stats.lua query directly in curl and it fails with
>>> the same error, so this is not a Lua issue.
>>>
>>> It looks like the syntax for these particular queries has changed in 5.x.
>>>
>>> However I was told that lists.a.o was running on 5.x.
>>> If so, then I assume it must have had some fixes to the syntax which
>>> have not been back-ported to the mainline code?
>>>
>>> Anyone got any idea what needs fixing?
>>>
>>

Re: Problems running ES 5.x on pm-test

[sebb <se...@gmail.com>]

On 3 January 2017 at 12:33, Daniel Gruno <hu...@apache.org> wrote:
> On 01/03/2017 02:12 AM, sebb wrote:
>> The test VM is now running ES 5.x with a basic database.
>> The Apache config appears OK, Javascript is able to invoke lua scripts OK.
>>
>> However a couple of the queries fail:
>>
>> api/pminfo.lua:169: Backend Database returned code 400!
>>
>> {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"No
>> search type for
>> [scan]"}],"type":"illegal_argument_exception","reason":"No search type
>> for [scan]"},"status":400}
>
> search_type changed in 5.x to mean something else, and should be omitted
> in scan function in elastic.lua, AIUI.
>
>>
>> api/stats.lua:410: Backend Database returned code 400!
>>
>> {"error":{"root_cause":[{"type":"parsing_exception","reason":"no
>> [query] registered for
>> [limit]","line":1,"col":405}],"type":"parsing_exception","reason":"no
>> [query] registered for [limit]","line":1,"col":405},"status":400}
>
> limit is a thing of the past, and the size parameter has changed as well
> (it not denotes the total docs in the next scroll, not per shard/node).
> It might be easier to adjust this for ES 2.3/5.1 and not support earlier
> versions of ElasticSearch going forward.

+1

> I seem to also recall some harmonization needed between hard configured
> limits to results and how the scroll API works. I'll look into that ASAP.

So what does lists.apache.org use for the queries?
Or does it not use ES 5.x?

>>
>> I have tried the stats.lua query directly in curl and it fails with
>> the same error, so this is not a Lua issue.
>>
>> It looks like the syntax for these particular queries has changed in 5.x.
>>
>> However I was told that lists.a.o was running on 5.x.
>> If so, then I assume it must have had some fixes to the syntax which
>> have not been back-ported to the mainline code?
>>
>> Anyone got any idea what needs fixing?
>>
>

Re: Problems running ES 5.x on pm-test

[Daniel Gruno <hu...@apache.org>]

On 01/03/2017 02:12 AM, sebb wrote:
> The test VM is now running ES 5.x with a basic database.
> The Apache config appears OK, Javascript is able to invoke lua scripts OK.
> 
> However a couple of the queries fail:
> 
> api/pminfo.lua:169: Backend Database returned code 400!
> 
> {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"No
> search type for
> [scan]"}],"type":"illegal_argument_exception","reason":"No search type
> for [scan]"},"status":400}

search_type changed in 5.x to mean something else, and should be omitted
in scan function in elastic.lua, AIUI.

> 
> api/stats.lua:410: Backend Database returned code 400!
> 
> {"error":{"root_cause":[{"type":"parsing_exception","reason":"no
> [query] registered for
> [limit]","line":1,"col":405}],"type":"parsing_exception","reason":"no
> [query] registered for [limit]","line":1,"col":405},"status":400}

limit is a thing of the past, and the size parameter has changed as well
(it not denotes the total docs in the next scroll, not per shard/node).
It might be easier to adjust this for ES 2.3/5.1 and not support earlier
versions of ElasticSearch going forward.

I seem to also recall some harmonization needed between hard configured
limits to results and how the scroll API works. I'll look into that ASAP.

> 
> I have tried the stats.lua query directly in curl and it fails with
> the same error, so this is not a Lua issue.
> 
> It looks like the syntax for these particular queries has changed in 5.x.
> 
> However I was told that lists.a.o was running on 5.x.
> If so, then I assume it must have had some fixes to the syntax which
> have not been back-ported to the mainline code?
> 
> Anyone got any idea what needs fixing?
>