You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@whimsical.apache.org by sebb <se...@gmail.com> on 2016/02/02 16:18:48 UTC
Empty people file
One of the reporter scripts failed.
Turns out that the public_ldap_people.json file was empty except for
the top-level dict:
{
"people": {
}
}
That should not happen.
Here is the listing of the cached files:
-rw-rw-r-- 1 www-data www-data 82234 Jan 30 00:32 public_ldap_committees.json
-rw-rw-r-- 1 www-data www-data 13254 Feb 1 19:38 public_nonldap_groups.json
-rw-rw-r-- 1 www-data www-data 267146 Feb 1 19:51 public_ldap_groups.json
-rw-rw-r-- 1 www-data www-data 423795 Feb 1 20:45 committee-info.json
-rw-rw-r-- 1 www-data www-data 22 Feb 2 11:49 public_ldap_people.json
The file was fetched as per the following log entries:
Reading https://whimsy.apache.org/public/public_ldap_people.json
File public_ldap_people.json exists and check interval is zero
https://whimsy.apache.org/public/public_ldap_people.json
{'If-Modified-Since': 'Mon, 01 Feb 2016 17:19:03 GMT'}
STATUS 200
Date: Tue, 02 Feb 2016 12:00:11 GMT
Server: Apache/2.4.7 (Ubuntu)
Strict-Transport-Security: max-age=63072000; includeSubdomains; preload
Last-Modified: Tue, 02 Feb 2016 11:49:08 GMT
ETag: "16-52ac816dbfbe2"
Accept-Ranges: bytes
Content-Length: 22
Access-Control-Allow-Origin: *
Content-Type: application/json; charset=utf-8
Via: 1.1 whimsy.apache.org
Via: 1.1 whimsy.apache.org
Connection: close
So it looks as though the file was fetched correctly.
But how was it generated with no useful content?
Unfortunately the Whimsy log files are not kept for any length of time.
Re: Empty people file
Posted by sebb <se...@gmail.com>.
On 3 February 2016 at 13:53, Sam Ruby <ru...@intertwingly.net> wrote:
> On Tue, Feb 2, 2016 at 11:25 AM, Sam Ruby <ru...@intertwingly.net> wrote:
>>
>> ... which means that the following returned an empty list:
>>
>> https://github.com/apache/whimsy/blob/master/lib/whimsy/asf/ldap.rb#L129
>>
>> And that code looks problematic. The first thing I would suggest
>> would be to remove the begin/rescue/return []/end lines, allowing
>> whatever LDAP error occurs to bubble up. It may turn out that there
>> are errors that may be worth retrying (e.g. LDAP server goes down, try
>> reconnecting). If so, a much more focused rescue clause could be
>> re-added.
>>
>>> Unfortunately the Whimsy log files are not kept for any length of time.
>>
>> There wouldn't be anything in the logs. If, however, the exception
>> weren't captured, that would be logged, and that result would trigger
>> an email.
>
> Here's the log:
>
> HTTP/1.1 400 public_json ldap-people
> ["/srv/whimsy/lib/whimsy/asf/ldap.rb:129:in `search2': Time limit
> exceeded (LDAP::ResultError)", "\tfrom
> /srv/whimsy/lib/whimsy/asf/ldap.rb:129:in `search_one'", "\tfrom
> /srv/whimsy/lib/whimsy/asf/ldap.rb:262:in `preload'", "\tfrom
> roster/public_ldap_people.rb:25:in `<main>'"]
>
> So... one LDAP server (my guess: ldaps://ldap2-us-west.apache.org:636)
> times out.
We could / should log the server along with the message.
I'll have a look at doing that.
> Unfortunately, the exception raised isn't very specific, so the only
> way to recover from it would be to look at the exception text.
Or just log and retry regardless of the actual error.
> Recovery could be trying another LDAP server.
Yes.
There should be a retry limit.
> Meanwhile, this exception should have prevented an empty people file,
> the only affect (other than the alert) would be that an update made
> during that time would not be reflected until the next time the cron
> job fired.
>
> This could also have been the cause for empty email addresses in the
> reminder script.
Agreed.
> - Sam Ruby
Re: Empty people file
Posted by Sam Ruby <ru...@intertwingly.net>.
On Tue, Feb 2, 2016 at 11:25 AM, Sam Ruby <ru...@intertwingly.net> wrote:
>
> ... which means that the following returned an empty list:
>
> https://github.com/apache/whimsy/blob/master/lib/whimsy/asf/ldap.rb#L129
>
> And that code looks problematic. The first thing I would suggest
> would be to remove the begin/rescue/return []/end lines, allowing
> whatever LDAP error occurs to bubble up. It may turn out that there
> are errors that may be worth retrying (e.g. LDAP server goes down, try
> reconnecting). If so, a much more focused rescue clause could be
> re-added.
>
>> Unfortunately the Whimsy log files are not kept for any length of time.
>
> There wouldn't be anything in the logs. If, however, the exception
> weren't captured, that would be logged, and that result would trigger
> an email.
Here's the log:
HTTP/1.1 400 public_json ldap-people
["/srv/whimsy/lib/whimsy/asf/ldap.rb:129:in `search2': Time limit
exceeded (LDAP::ResultError)", "\tfrom
/srv/whimsy/lib/whimsy/asf/ldap.rb:129:in `search_one'", "\tfrom
/srv/whimsy/lib/whimsy/asf/ldap.rb:262:in `preload'", "\tfrom
roster/public_ldap_people.rb:25:in `<main>'"]
So... one LDAP server (my guess: ldaps://ldap2-us-west.apache.org:636)
times out.
Unfortunately, the exception raised isn't very specific, so the only
way to recover from it would be to look at the exception text.
Recovery could be trying another LDAP server.
Meanwhile, this exception should have prevented an empty people file,
the only affect (other than the alert) would be that an update made
during that time would not be reflected until the next time the cron
job fired.
This could also have been the cause for empty email addresses in the
reminder script.
- Sam Ruby
Re: Empty people file
Posted by Sam Ruby <ru...@intertwingly.net>.
On Tue, Feb 2, 2016 at 10:18 AM, sebb <se...@gmail.com> wrote:
> One of the reporter scripts failed.
>
> Turns out that the public_ldap_people.json file was empty except for
> the top-level dict:
>
> {
> "people": {
> }
> }
>
> That should not happen.
>
> Here is the listing of the cached files:
>
> -rw-rw-r-- 1 www-data www-data 82234 Jan 30 00:32 public_ldap_committees.json
> -rw-rw-r-- 1 www-data www-data 13254 Feb 1 19:38 public_nonldap_groups.json
> -rw-rw-r-- 1 www-data www-data 267146 Feb 1 19:51 public_ldap_groups.json
> -rw-rw-r-- 1 www-data www-data 423795 Feb 1 20:45 committee-info.json
> -rw-rw-r-- 1 www-data www-data 22 Feb 2 11:49 public_ldap_people.json
>
> The file was fetched as per the following log entries:
>
> Reading https://whimsy.apache.org/public/public_ldap_people.json
> File public_ldap_people.json exists and check interval is zero
> https://whimsy.apache.org/public/public_ldap_people.json
> {'If-Modified-Since': 'Mon, 01 Feb 2016 17:19:03 GMT'}
> STATUS 200
> Date: Tue, 02 Feb 2016 12:00:11 GMT
> Server: Apache/2.4.7 (Ubuntu)
> Strict-Transport-Security: max-age=63072000; includeSubdomains; preload
> Last-Modified: Tue, 02 Feb 2016 11:49:08 GMT
> ETag: "16-52ac816dbfbe2"
> Accept-Ranges: bytes
> Content-Length: 22
> Access-Control-Allow-Origin: *
> Content-Type: application/json; charset=utf-8
> Via: 1.1 whimsy.apache.org
> Via: 1.1 whimsy.apache.org
> Connection: close
>
> So it looks as though the file was fetched correctly.
>
> But how was it generated with no useful content?
It looks like the following returned an empty list:
https://github.com/apache/whimsy/blob/master/www/roster/public_ldap_people.rb#L25
... which means that the following returned an empty list:
https://github.com/apache/whimsy/blob/master/lib/whimsy/asf/ldap.rb#L266
... which means that the following returned an empty list:
https://github.com/apache/whimsy/blob/master/lib/whimsy/asf/ldap.rb#L129
And that code looks problematic. The first thing I would suggest
would be to remove the begin/rescue/return []/end lines, allowing
whatever LDAP error occurs to bubble up. It may turn out that there
are errors that may be worth retrying (e.g. LDAP server goes down, try
reconnecting). If so, a much more focused rescue clause could be
re-added.
> Unfortunately the Whimsy log files are not kept for any length of time.
There wouldn't be anything in the logs. If, however, the exception
weren't captured, that would be logged, and that result would trigger
an email.
- Sam Ruby