You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Jay Doane <ja...@apache.org> on 2019/08/14 06:19:49 UTC

[DISCUSS] [PROPOSAL] Accept donation of the IBM Cloudant Weather Report diagnostic tool?

In the interest of making CouchDB 3.0 "the best CouchDB Classic possible",
I'd like to discuss whether to accept a donation from Cloudant of the
"Weather Report" diagnostic tool. This tool (and dependencies) are OTP
applications, and it is typically run from an escript which connects to a
running cluster, gathers numerous diagnostics, and emits various warning
and errors when it finds something to complain about. It was originally
ported from a fork of Riaknostic (the Automated diagnostic tools for Riak)
[1] by Mike Wallace.

The checks it makes are represented by the following modules:

weatherreport_check_custodian.erl
weatherreport_check_disk.erl
weatherreport_check_internal_replication.erl
weatherreport_check_ioq.erl
weatherreport_check_mem3_sync.erl
weatherreport_check_membership.erl
weatherreport_check_memory_use.erl
weatherreport_check_message_queues.erl
weatherreport_check_node_stats.erl
weatherreport_check_nodes_connected.erl
weatherreport_check_process_calls.erl
weatherreport_check_process_memory.erl
weatherreport_check_safe_to_rebuild.erl
weatherreport_check_search.erl
weatherreport_check_tcp_queues.erl

While some of these checks are self-contained, check_node_stats,
check_process_calls, check_process_memory, and check_message_queues all use
recon [2] under the hood. Similarly, check_custodian
and check_safe_to_rebuild use another Cloudant OTP application called
Custodian, which periodically scans the "dbs" database to track the
location of every shard of every database and can integrate with sensu [3]
to ensure that operators are aware of any shard that is under-replicated.

I have created a POC branch [4] that adds Weather Report, Custodian, and
Recon to CouchDB, and when I ran it in my dev environment (without search
running), got the following diagnostic output:

$ ./weatherreport --etc ~/proj/couchdb/dev/lib/node1/etc/ -a
['node1@127.0.0.1'] [error] Local search node at 'clouseau@127.0.0.1' not
responding: pang
['node2@127.0.0.1'] [error] Local search node at 'clouseau@127.0.0.1' not
responding: pang
['node3@127.0.0.1'] [error] Local search node at 'clouseau@127.0.0.1' not
responding: pang
['node1@127.0.0.1'] [notice] Data directory
/Users/jay/proj/couchdb/dev/lib/node1/data is not mounted with 'noatime'.
Please remount its disk with the 'noatime' flag to improve performance.
['node2@127.0.0.1'] [notice] Data directory
/Users/jay/proj/couchdb/dev/lib/node2/data is not mounted with 'noatime'.
Please remount its disk with the 'noatime' flag to improve performance.
['node3@127.0.0.1'] [notice] Data directory
/Users/jay/proj/couchdb/dev/lib/node3/data is not mounted with 'noatime'.
Please remount its disk with the 'noatime' flag to improve performance.
returned 1

There is still a little cleanup to be done before these tools would be
ready to donate, but it seems that overall they already integrate tolerably
well with CouchDB.

As far as licenses go, Riaknostic is Apache 2.0. Recon is not [5], but it
seems like it should be ok to include in CouchDB based on my possibly naive
reading. Currently Custodian has no license (just Copyright 2013 Cloudant),
but I assume it would get an Apache license, just like all other donated
code.

Would this be a welcome addition to CouchDB? Please let me know what you
think.

Thanks,
Jay

[1] https://github.com/basho/riaknostic
[2] http://ferd.github.io/recon/
[3] https://sensu.io
[4]
https://github.com/apache/couchdb/compare/master...cloudant:weatherreport?expand=1
[5] https://github.com/ferd/recon/blob/master/LICENSE

Re: [DISCUSS] [PROPOSAL] Accept donation of the IBM Cloudant Weather Report diagnostic tool?

Posted by Joan Touzet <wo...@apache.org>.
Very happy to see Paul's resistance to including recon in the default 
CouchDB distribution is gone.

Big +1, and looking forward to the PR to review and help out.

Remember that this may do weird things or be completely inoperative on 
Windows, which will require larger changes. Be prepared for me to 
heavily red-line if this happens ;) You may want to get started early on 
setting up a Windows VM so you can test this yourself.

-Joan

On 2019-08-14 10:44 a.m., Paul Davis wrote:
> +1
> 
> On Wed, Aug 14, 2019 at 9:07 AM Jan Lehnardt <ja...@apache.org> wrote:
>>
>> Agreed, and recon’s BSD 3-clause is totally fine as per https://apache.org/legal/resolved.html#category-a
>>
>> +1
>>
>> Best
>> Jan
>> —
>>
>>> On 14. Aug 2019, at 15:36, Robert Newson <rn...@apache.org> wrote:
>>>
>>> I’m for the proposal and am confident IBM will apply release custodian under the ASLv2 if the community is in favour of the the proposal.
>>>
>>> B.
>>>
>>>> On 14 Aug 2019, at 07:19, Jay Doane <ja...@apache.org> wrote:
>>>>
>>>> In the interest of making CouchDB 3.0 "the best CouchDB Classic possible",
>>>> I'd like to discuss whether to accept a donation from Cloudant of the
>>>> "Weather Report" diagnostic tool. This tool (and dependencies) are OTP
>>>> applications, and it is typically run from an escript which connects to a
>>>> running cluster, gathers numerous diagnostics, and emits various warning
>>>> and errors when it finds something to complain about. It was originally
>>>> ported from a fork of Riaknostic (the Automated diagnostic tools for Riak)
>>>> [1] by Mike Wallace.
>>>>
>>>> The checks it makes are represented by the following modules:
>>>>
>>>> weatherreport_check_custodian.erl
>>>> weatherreport_check_disk.erl
>>>> weatherreport_check_internal_replication.erl
>>>> weatherreport_check_ioq.erl
>>>> weatherreport_check_mem3_sync.erl
>>>> weatherreport_check_membership.erl
>>>> weatherreport_check_memory_use.erl
>>>> weatherreport_check_message_queues.erl
>>>> weatherreport_check_node_stats.erl
>>>> weatherreport_check_nodes_connected.erl
>>>> weatherreport_check_process_calls.erl
>>>> weatherreport_check_process_memory.erl
>>>> weatherreport_check_safe_to_rebuild.erl
>>>> weatherreport_check_search.erl
>>>> weatherreport_check_tcp_queues.erl
>>>>
>>>> While some of these checks are self-contained, check_node_stats,
>>>> check_process_calls, check_process_memory, and check_message_queues all use
>>>> recon [2] under the hood. Similarly, check_custodian
>>>> and check_safe_to_rebuild use another Cloudant OTP application called
>>>> Custodian, which periodically scans the "dbs" database to track the
>>>> location of every shard of every database and can integrate with sensu [3]
>>>> to ensure that operators are aware of any shard that is under-replicated.
>>>>
>>>> I have created a POC branch [4] that adds Weather Report, Custodian, and
>>>> Recon to CouchDB, and when I ran it in my dev environment (without search
>>>> running), got the following diagnostic output:
>>>>
>>>> $ ./weatherreport --etc ~/proj/couchdb/dev/lib/node1/etc/ -a
>>>> ['node1@127.0.0.1'] [error] Local search node at 'clouseau@127.0.0.1' not
>>>> responding: pang
>>>> ['node2@127.0.0.1'] [error] Local search node at 'clouseau@127.0.0.1' not
>>>> responding: pang
>>>> ['node3@127.0.0.1'] [error] Local search node at 'clouseau@127.0.0.1' not
>>>> responding: pang
>>>> ['node1@127.0.0.1'] [notice] Data directory
>>>> /Users/jay/proj/couchdb/dev/lib/node1/data is not mounted with 'noatime'.
>>>> Please remount its disk with the 'noatime' flag to improve performance.
>>>> ['node2@127.0.0.1'] [notice] Data directory
>>>> /Users/jay/proj/couchdb/dev/lib/node2/data is not mounted with 'noatime'.
>>>> Please remount its disk with the 'noatime' flag to improve performance.
>>>> ['node3@127.0.0.1'] [notice] Data directory
>>>> /Users/jay/proj/couchdb/dev/lib/node3/data is not mounted with 'noatime'.
>>>> Please remount its disk with the 'noatime' flag to improve performance.
>>>> returned 1
>>>>
>>>> There is still a little cleanup to be done before these tools would be
>>>> ready to donate, but it seems that overall they already integrate tolerably
>>>> well with CouchDB.
>>>>
>>>> As far as licenses go, Riaknostic is Apache 2.0. Recon is not [5], but it
>>>> seems like it should be ok to include in CouchDB based on my possibly naive
>>>> reading. Currently Custodian has no license (just Copyright 2013 Cloudant),
>>>> but I assume it would get an Apache license, just like all other donated
>>>> code.
>>>>
>>>> Would this be a welcome addition to CouchDB? Please let me know what you
>>>> think.
>>>>
>>>> Thanks,
>>>> Jay
>>>>
>>>> [1] https://github.com/basho/riaknostic
>>>> [2] http://ferd.github.io/recon/
>>>> [3] https://sensu.io
>>>> [4]
>>>> https://github.com/apache/couchdb/compare/master...cloudant:weatherreport?expand=1
>>>> [5] https://github.com/ferd/recon/blob/master/LICENSE
>>>
>>
>> --
>> Professional Support for Apache CouchDB:
>> https://neighbourhood.ie/couchdb-support/
>>

Re: [DISCUSS] [PROPOSAL] Accept donation of the IBM Cloudant Weather Report diagnostic tool?

Posted by Paul Davis <pa...@gmail.com>.
+1

On Wed, Aug 14, 2019 at 9:07 AM Jan Lehnardt <ja...@apache.org> wrote:
>
> Agreed, and recon’s BSD 3-clause is totally fine as per https://apache.org/legal/resolved.html#category-a
>
> +1
>
> Best
> Jan
> —
>
> > On 14. Aug 2019, at 15:36, Robert Newson <rn...@apache.org> wrote:
> >
> > I’m for the proposal and am confident IBM will apply release custodian under the ASLv2 if the community is in favour of the the proposal.
> >
> > B.
> >
> >> On 14 Aug 2019, at 07:19, Jay Doane <ja...@apache.org> wrote:
> >>
> >> In the interest of making CouchDB 3.0 "the best CouchDB Classic possible",
> >> I'd like to discuss whether to accept a donation from Cloudant of the
> >> "Weather Report" diagnostic tool. This tool (and dependencies) are OTP
> >> applications, and it is typically run from an escript which connects to a
> >> running cluster, gathers numerous diagnostics, and emits various warning
> >> and errors when it finds something to complain about. It was originally
> >> ported from a fork of Riaknostic (the Automated diagnostic tools for Riak)
> >> [1] by Mike Wallace.
> >>
> >> The checks it makes are represented by the following modules:
> >>
> >> weatherreport_check_custodian.erl
> >> weatherreport_check_disk.erl
> >> weatherreport_check_internal_replication.erl
> >> weatherreport_check_ioq.erl
> >> weatherreport_check_mem3_sync.erl
> >> weatherreport_check_membership.erl
> >> weatherreport_check_memory_use.erl
> >> weatherreport_check_message_queues.erl
> >> weatherreport_check_node_stats.erl
> >> weatherreport_check_nodes_connected.erl
> >> weatherreport_check_process_calls.erl
> >> weatherreport_check_process_memory.erl
> >> weatherreport_check_safe_to_rebuild.erl
> >> weatherreport_check_search.erl
> >> weatherreport_check_tcp_queues.erl
> >>
> >> While some of these checks are self-contained, check_node_stats,
> >> check_process_calls, check_process_memory, and check_message_queues all use
> >> recon [2] under the hood. Similarly, check_custodian
> >> and check_safe_to_rebuild use another Cloudant OTP application called
> >> Custodian, which periodically scans the "dbs" database to track the
> >> location of every shard of every database and can integrate with sensu [3]
> >> to ensure that operators are aware of any shard that is under-replicated.
> >>
> >> I have created a POC branch [4] that adds Weather Report, Custodian, and
> >> Recon to CouchDB, and when I ran it in my dev environment (without search
> >> running), got the following diagnostic output:
> >>
> >> $ ./weatherreport --etc ~/proj/couchdb/dev/lib/node1/etc/ -a
> >> ['node1@127.0.0.1'] [error] Local search node at 'clouseau@127.0.0.1' not
> >> responding: pang
> >> ['node2@127.0.0.1'] [error] Local search node at 'clouseau@127.0.0.1' not
> >> responding: pang
> >> ['node3@127.0.0.1'] [error] Local search node at 'clouseau@127.0.0.1' not
> >> responding: pang
> >> ['node1@127.0.0.1'] [notice] Data directory
> >> /Users/jay/proj/couchdb/dev/lib/node1/data is not mounted with 'noatime'.
> >> Please remount its disk with the 'noatime' flag to improve performance.
> >> ['node2@127.0.0.1'] [notice] Data directory
> >> /Users/jay/proj/couchdb/dev/lib/node2/data is not mounted with 'noatime'.
> >> Please remount its disk with the 'noatime' flag to improve performance.
> >> ['node3@127.0.0.1'] [notice] Data directory
> >> /Users/jay/proj/couchdb/dev/lib/node3/data is not mounted with 'noatime'.
> >> Please remount its disk with the 'noatime' flag to improve performance.
> >> returned 1
> >>
> >> There is still a little cleanup to be done before these tools would be
> >> ready to donate, but it seems that overall they already integrate tolerably
> >> well with CouchDB.
> >>
> >> As far as licenses go, Riaknostic is Apache 2.0. Recon is not [5], but it
> >> seems like it should be ok to include in CouchDB based on my possibly naive
> >> reading. Currently Custodian has no license (just Copyright 2013 Cloudant),
> >> but I assume it would get an Apache license, just like all other donated
> >> code.
> >>
> >> Would this be a welcome addition to CouchDB? Please let me know what you
> >> think.
> >>
> >> Thanks,
> >> Jay
> >>
> >> [1] https://github.com/basho/riaknostic
> >> [2] http://ferd.github.io/recon/
> >> [3] https://sensu.io
> >> [4]
> >> https://github.com/apache/couchdb/compare/master...cloudant:weatherreport?expand=1
> >> [5] https://github.com/ferd/recon/blob/master/LICENSE
> >
>
> --
> Professional Support for Apache CouchDB:
> https://neighbourhood.ie/couchdb-support/
>

Re: [DISCUSS] [PROPOSAL] Accept donation of the IBM Cloudant Weather Report diagnostic tool?

Posted by Jan Lehnardt <ja...@apache.org>.
Agreed, and recon’s BSD 3-clause is totally fine as per https://apache.org/legal/resolved.html#category-a

+1

Best
Jan
—

> On 14. Aug 2019, at 15:36, Robert Newson <rn...@apache.org> wrote:
> 
> I’m for the proposal and am confident IBM will apply release custodian under the ASLv2 if the community is in favour of the the proposal. 
> 
> B. 
> 
>> On 14 Aug 2019, at 07:19, Jay Doane <ja...@apache.org> wrote:
>> 
>> In the interest of making CouchDB 3.0 "the best CouchDB Classic possible",
>> I'd like to discuss whether to accept a donation from Cloudant of the
>> "Weather Report" diagnostic tool. This tool (and dependencies) are OTP
>> applications, and it is typically run from an escript which connects to a
>> running cluster, gathers numerous diagnostics, and emits various warning
>> and errors when it finds something to complain about. It was originally
>> ported from a fork of Riaknostic (the Automated diagnostic tools for Riak)
>> [1] by Mike Wallace.
>> 
>> The checks it makes are represented by the following modules:
>> 
>> weatherreport_check_custodian.erl
>> weatherreport_check_disk.erl
>> weatherreport_check_internal_replication.erl
>> weatherreport_check_ioq.erl
>> weatherreport_check_mem3_sync.erl
>> weatherreport_check_membership.erl
>> weatherreport_check_memory_use.erl
>> weatherreport_check_message_queues.erl
>> weatherreport_check_node_stats.erl
>> weatherreport_check_nodes_connected.erl
>> weatherreport_check_process_calls.erl
>> weatherreport_check_process_memory.erl
>> weatherreport_check_safe_to_rebuild.erl
>> weatherreport_check_search.erl
>> weatherreport_check_tcp_queues.erl
>> 
>> While some of these checks are self-contained, check_node_stats,
>> check_process_calls, check_process_memory, and check_message_queues all use
>> recon [2] under the hood. Similarly, check_custodian
>> and check_safe_to_rebuild use another Cloudant OTP application called
>> Custodian, which periodically scans the "dbs" database to track the
>> location of every shard of every database and can integrate with sensu [3]
>> to ensure that operators are aware of any shard that is under-replicated.
>> 
>> I have created a POC branch [4] that adds Weather Report, Custodian, and
>> Recon to CouchDB, and when I ran it in my dev environment (without search
>> running), got the following diagnostic output:
>> 
>> $ ./weatherreport --etc ~/proj/couchdb/dev/lib/node1/etc/ -a
>> ['node1@127.0.0.1'] [error] Local search node at 'clouseau@127.0.0.1' not
>> responding: pang
>> ['node2@127.0.0.1'] [error] Local search node at 'clouseau@127.0.0.1' not
>> responding: pang
>> ['node3@127.0.0.1'] [error] Local search node at 'clouseau@127.0.0.1' not
>> responding: pang
>> ['node1@127.0.0.1'] [notice] Data directory
>> /Users/jay/proj/couchdb/dev/lib/node1/data is not mounted with 'noatime'.
>> Please remount its disk with the 'noatime' flag to improve performance.
>> ['node2@127.0.0.1'] [notice] Data directory
>> /Users/jay/proj/couchdb/dev/lib/node2/data is not mounted with 'noatime'.
>> Please remount its disk with the 'noatime' flag to improve performance.
>> ['node3@127.0.0.1'] [notice] Data directory
>> /Users/jay/proj/couchdb/dev/lib/node3/data is not mounted with 'noatime'.
>> Please remount its disk with the 'noatime' flag to improve performance.
>> returned 1
>> 
>> There is still a little cleanup to be done before these tools would be
>> ready to donate, but it seems that overall they already integrate tolerably
>> well with CouchDB.
>> 
>> As far as licenses go, Riaknostic is Apache 2.0. Recon is not [5], but it
>> seems like it should be ok to include in CouchDB based on my possibly naive
>> reading. Currently Custodian has no license (just Copyright 2013 Cloudant),
>> but I assume it would get an Apache license, just like all other donated
>> code.
>> 
>> Would this be a welcome addition to CouchDB? Please let me know what you
>> think.
>> 
>> Thanks,
>> Jay
>> 
>> [1] https://github.com/basho/riaknostic
>> [2] http://ferd.github.io/recon/
>> [3] https://sensu.io
>> [4]
>> https://github.com/apache/couchdb/compare/master...cloudant:weatherreport?expand=1
>> [5] https://github.com/ferd/recon/blob/master/LICENSE
> 

-- 
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

Re: [DISCUSS] [PROPOSAL] Accept donation of the IBM Cloudant Weather Report diagnostic tool?

Posted by Robert Newson <rn...@apache.org>.
I’m for the proposal and am confident IBM will apply release custodian under the ASLv2 if the community is in favour of the the proposal. 

B. 

> On 14 Aug 2019, at 07:19, Jay Doane <ja...@apache.org> wrote:
> 
> In the interest of making CouchDB 3.0 "the best CouchDB Classic possible",
> I'd like to discuss whether to accept a donation from Cloudant of the
> "Weather Report" diagnostic tool. This tool (and dependencies) are OTP
> applications, and it is typically run from an escript which connects to a
> running cluster, gathers numerous diagnostics, and emits various warning
> and errors when it finds something to complain about. It was originally
> ported from a fork of Riaknostic (the Automated diagnostic tools for Riak)
> [1] by Mike Wallace.
> 
> The checks it makes are represented by the following modules:
> 
> weatherreport_check_custodian.erl
> weatherreport_check_disk.erl
> weatherreport_check_internal_replication.erl
> weatherreport_check_ioq.erl
> weatherreport_check_mem3_sync.erl
> weatherreport_check_membership.erl
> weatherreport_check_memory_use.erl
> weatherreport_check_message_queues.erl
> weatherreport_check_node_stats.erl
> weatherreport_check_nodes_connected.erl
> weatherreport_check_process_calls.erl
> weatherreport_check_process_memory.erl
> weatherreport_check_safe_to_rebuild.erl
> weatherreport_check_search.erl
> weatherreport_check_tcp_queues.erl
> 
> While some of these checks are self-contained, check_node_stats,
> check_process_calls, check_process_memory, and check_message_queues all use
> recon [2] under the hood. Similarly, check_custodian
> and check_safe_to_rebuild use another Cloudant OTP application called
> Custodian, which periodically scans the "dbs" database to track the
> location of every shard of every database and can integrate with sensu [3]
> to ensure that operators are aware of any shard that is under-replicated.
> 
> I have created a POC branch [4] that adds Weather Report, Custodian, and
> Recon to CouchDB, and when I ran it in my dev environment (without search
> running), got the following diagnostic output:
> 
> $ ./weatherreport --etc ~/proj/couchdb/dev/lib/node1/etc/ -a
> ['node1@127.0.0.1'] [error] Local search node at 'clouseau@127.0.0.1' not
> responding: pang
> ['node2@127.0.0.1'] [error] Local search node at 'clouseau@127.0.0.1' not
> responding: pang
> ['node3@127.0.0.1'] [error] Local search node at 'clouseau@127.0.0.1' not
> responding: pang
> ['node1@127.0.0.1'] [notice] Data directory
> /Users/jay/proj/couchdb/dev/lib/node1/data is not mounted with 'noatime'.
> Please remount its disk with the 'noatime' flag to improve performance.
> ['node2@127.0.0.1'] [notice] Data directory
> /Users/jay/proj/couchdb/dev/lib/node2/data is not mounted with 'noatime'.
> Please remount its disk with the 'noatime' flag to improve performance.
> ['node3@127.0.0.1'] [notice] Data directory
> /Users/jay/proj/couchdb/dev/lib/node3/data is not mounted with 'noatime'.
> Please remount its disk with the 'noatime' flag to improve performance.
> returned 1
> 
> There is still a little cleanup to be done before these tools would be
> ready to donate, but it seems that overall they already integrate tolerably
> well with CouchDB.
> 
> As far as licenses go, Riaknostic is Apache 2.0. Recon is not [5], but it
> seems like it should be ok to include in CouchDB based on my possibly naive
> reading. Currently Custodian has no license (just Copyright 2013 Cloudant),
> but I assume it would get an Apache license, just like all other donated
> code.
> 
> Would this be a welcome addition to CouchDB? Please let me know what you
> think.
> 
> Thanks,
> Jay
> 
> [1] https://github.com/basho/riaknostic
> [2] http://ferd.github.io/recon/
> [3] https://sensu.io
> [4]
> https://github.com/apache/couchdb/compare/master...cloudant:weatherreport?expand=1
> [5] https://github.com/ferd/recon/blob/master/LICENSE