You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Vitalyi Brodetskyi <vb...@hortonworks.com> on 2014/03/05 16:22:45 UTC

Review Request 18772: After configuring NNHA, nn process alerts don't work

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18772/
-----------------------------------------------------------

Review request for Ambari, Dmitro Lisnichenko and Nate Cole.


Bugs: AMBARI-4954
    https://issues.apache.org/jira/browse/AMBARI-4954


Repository: ambari


Description
-------

1) Configure NN HA
2) stack 2.0.6 (but check stack 2.1 as well)
3) Two alerts show" check_tcp: Port must be a positive integer"

NameNode process on c6401.ambari.apache.org
NameNode process on c6402.ambari.apache.org

4) Looked at /etc/nagios/objects/hadoop-services.cfg
5) Saw:

{code}
define service {
        host_name               c6401.ambari.apache.org
        use                     hadoop-service
        service_description     NAMENODE::NameNode process on c6401.ambari.apache.org
        servicegroups           HDFS
        check_command           check_tcp_wrapper!//test!-w 1 -c 1
        normal_check_interval   0.5
        retry_check_interval    0.25
        max_check_attempts      3
}
define service {
        host_name               c6402.ambari.apache.org
        use                     hadoop-service
        service_description     NAMENODE::NameNode process on c6402.ambari.apache.org
        servicegroups           HDFS
        check_command           check_tcp_wrapper!//test!-w 1 -c 1
        normal_check_interval   0.5
        retry_check_interval    0.25
        max_check_attempts      3
}
{code}

Notice in the above *//test* is the name of my nameservice.

Attaching screen shot of my config. So looks like it's grabbing port from the wrong prop.


Diffs
-----

  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/NAGIOS/package/scripts/params.py 813c626 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/NAGIOS/package/templates/hadoop-services.cfg.j2 5b8fb6d 

Diff: https://reviews.apache.org/r/18772/diff/


Testing
-------


Thanks,

Vitalyi Brodetskyi


Re: Review Request 18772: After configuring NNHA, nn process alerts don't work

Posted by Vitalyi Brodetskyi <vb...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18772/
-----------------------------------------------------------

(Updated March 5, 2014, 6:30 p.m.)


Review request for Ambari, Dmitro Lisnichenko and Nate Cole.


Bugs: AMBARI-4954
    https://issues.apache.org/jira/browse/AMBARI-4954


Repository: ambari


Description
-------

1) Configure NN HA
2) stack 2.0.6 (but check stack 2.1 as well)
3) Two alerts show" check_tcp: Port must be a positive integer"

NameNode process on c6401.ambari.apache.org
NameNode process on c6402.ambari.apache.org

4) Looked at /etc/nagios/objects/hadoop-services.cfg
5) Saw:

{code}
define service {
        host_name               c6401.ambari.apache.org
        use                     hadoop-service
        service_description     NAMENODE::NameNode process on c6401.ambari.apache.org
        servicegroups           HDFS
        check_command           check_tcp_wrapper!//test!-w 1 -c 1
        normal_check_interval   0.5
        retry_check_interval    0.25
        max_check_attempts      3
}
define service {
        host_name               c6402.ambari.apache.org
        use                     hadoop-service
        service_description     NAMENODE::NameNode process on c6402.ambari.apache.org
        servicegroups           HDFS
        check_command           check_tcp_wrapper!//test!-w 1 -c 1
        normal_check_interval   0.5
        retry_check_interval    0.25
        max_check_attempts      3
}
{code}

Notice in the above *//test* is the name of my nameservice.

Attaching screen shot of my config. So looks like it's grabbing port from the wrong prop.


Diffs (updated)
-----

  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/NAGIOS/package/scripts/params.py 813c626 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/NAGIOS/package/templates/hadoop-services.cfg.j2 6df0380 

Diff: https://reviews.apache.org/r/18772/diff/


Testing
-------


Thanks,

Vitalyi Brodetskyi


Re: Review Request 18772: After configuring NNHA, nn process alerts don't work

Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18772/#review36244
-----------------------------------------------------------

Ship it!


Ship It!

- Dmitro Lisnichenko


On March 5, 2014, 5:41 p.m., Vitalyi Brodetskyi wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18772/
> -----------------------------------------------------------
> 
> (Updated March 5, 2014, 5:41 p.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko and Nate Cole.
> 
> 
> Bugs: AMBARI-4954
>     https://issues.apache.org/jira/browse/AMBARI-4954
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> 1) Configure NN HA
> 2) stack 2.0.6 (but check stack 2.1 as well)
> 3) Two alerts show" check_tcp: Port must be a positive integer"
> 
> NameNode process on c6401.ambari.apache.org
> NameNode process on c6402.ambari.apache.org
> 
> 4) Looked at /etc/nagios/objects/hadoop-services.cfg
> 5) Saw:
> 
> {code}
> define service {
>         host_name               c6401.ambari.apache.org
>         use                     hadoop-service
>         service_description     NAMENODE::NameNode process on c6401.ambari.apache.org
>         servicegroups           HDFS
>         check_command           check_tcp_wrapper!//test!-w 1 -c 1
>         normal_check_interval   0.5
>         retry_check_interval    0.25
>         max_check_attempts      3
> }
> define service {
>         host_name               c6402.ambari.apache.org
>         use                     hadoop-service
>         service_description     NAMENODE::NameNode process on c6402.ambari.apache.org
>         servicegroups           HDFS
>         check_command           check_tcp_wrapper!//test!-w 1 -c 1
>         normal_check_interval   0.5
>         retry_check_interval    0.25
>         max_check_attempts      3
> }
> {code}
> 
> Notice in the above *//test* is the name of my nameservice.
> 
> Attaching screen shot of my config. So looks like it's grabbing port from the wrong prop.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/NAGIOS/package/scripts/params.py 813c626 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/NAGIOS/package/templates/hadoop-services.cfg.j2 6df0380 
> 
> Diff: https://reviews.apache.org/r/18772/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>


Re: Review Request 18772: After configuring NNHA, nn process alerts don't work

Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18772/#review36240
-----------------------------------------------------------

Ship it!


Ship It!

- Nate Cole


On March 5, 2014, 12:41 p.m., Vitalyi Brodetskyi wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18772/
> -----------------------------------------------------------
> 
> (Updated March 5, 2014, 12:41 p.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko and Nate Cole.
> 
> 
> Bugs: AMBARI-4954
>     https://issues.apache.org/jira/browse/AMBARI-4954
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> 1) Configure NN HA
> 2) stack 2.0.6 (but check stack 2.1 as well)
> 3) Two alerts show" check_tcp: Port must be a positive integer"
> 
> NameNode process on c6401.ambari.apache.org
> NameNode process on c6402.ambari.apache.org
> 
> 4) Looked at /etc/nagios/objects/hadoop-services.cfg
> 5) Saw:
> 
> {code}
> define service {
>         host_name               c6401.ambari.apache.org
>         use                     hadoop-service
>         service_description     NAMENODE::NameNode process on c6401.ambari.apache.org
>         servicegroups           HDFS
>         check_command           check_tcp_wrapper!//test!-w 1 -c 1
>         normal_check_interval   0.5
>         retry_check_interval    0.25
>         max_check_attempts      3
> }
> define service {
>         host_name               c6402.ambari.apache.org
>         use                     hadoop-service
>         service_description     NAMENODE::NameNode process on c6402.ambari.apache.org
>         servicegroups           HDFS
>         check_command           check_tcp_wrapper!//test!-w 1 -c 1
>         normal_check_interval   0.5
>         retry_check_interval    0.25
>         max_check_attempts      3
> }
> {code}
> 
> Notice in the above *//test* is the name of my nameservice.
> 
> Attaching screen shot of my config. So looks like it's grabbing port from the wrong prop.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/NAGIOS/package/scripts/params.py 813c626 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/NAGIOS/package/templates/hadoop-services.cfg.j2 6df0380 
> 
> Diff: https://reviews.apache.org/r/18772/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>


Re: Review Request 18772: After configuring NNHA, nn process alerts don't work

Posted by Vitalyi Brodetskyi <vb...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18772/
-----------------------------------------------------------

(Updated March 5, 2014, 5:41 p.m.)


Review request for Ambari, Dmitro Lisnichenko and Nate Cole.


Changes
-------

Nate, code changes according to your comment were done.


Bugs: AMBARI-4954
    https://issues.apache.org/jira/browse/AMBARI-4954


Repository: ambari


Description
-------

1) Configure NN HA
2) stack 2.0.6 (but check stack 2.1 as well)
3) Two alerts show" check_tcp: Port must be a positive integer"

NameNode process on c6401.ambari.apache.org
NameNode process on c6402.ambari.apache.org

4) Looked at /etc/nagios/objects/hadoop-services.cfg
5) Saw:

{code}
define service {
        host_name               c6401.ambari.apache.org
        use                     hadoop-service
        service_description     NAMENODE::NameNode process on c6401.ambari.apache.org
        servicegroups           HDFS
        check_command           check_tcp_wrapper!//test!-w 1 -c 1
        normal_check_interval   0.5
        retry_check_interval    0.25
        max_check_attempts      3
}
define service {
        host_name               c6402.ambari.apache.org
        use                     hadoop-service
        service_description     NAMENODE::NameNode process on c6402.ambari.apache.org
        servicegroups           HDFS
        check_command           check_tcp_wrapper!//test!-w 1 -c 1
        normal_check_interval   0.5
        retry_check_interval    0.25
        max_check_attempts      3
}
{code}

Notice in the above *//test* is the name of my nameservice.

Attaching screen shot of my config. So looks like it's grabbing port from the wrong prop.


Diffs (updated)
-----

  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/NAGIOS/package/scripts/params.py 813c626 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/NAGIOS/package/templates/hadoop-services.cfg.j2 6df0380 

Diff: https://reviews.apache.org/r/18772/diff/


Testing
-------


Thanks,

Vitalyi Brodetskyi


Re: Review Request 18772: After configuring NNHA, nn process alerts don't work

Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18772/#review36231
-----------------------------------------------------------

Ship it!


Ship It!

- Dmitro Lisnichenko


On March 5, 2014, 3:22 p.m., Vitalyi Brodetskyi wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18772/
> -----------------------------------------------------------
> 
> (Updated March 5, 2014, 3:22 p.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko and Nate Cole.
> 
> 
> Bugs: AMBARI-4954
>     https://issues.apache.org/jira/browse/AMBARI-4954
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> 1) Configure NN HA
> 2) stack 2.0.6 (but check stack 2.1 as well)
> 3) Two alerts show" check_tcp: Port must be a positive integer"
> 
> NameNode process on c6401.ambari.apache.org
> NameNode process on c6402.ambari.apache.org
> 
> 4) Looked at /etc/nagios/objects/hadoop-services.cfg
> 5) Saw:
> 
> {code}
> define service {
>         host_name               c6401.ambari.apache.org
>         use                     hadoop-service
>         service_description     NAMENODE::NameNode process on c6401.ambari.apache.org
>         servicegroups           HDFS
>         check_command           check_tcp_wrapper!//test!-w 1 -c 1
>         normal_check_interval   0.5
>         retry_check_interval    0.25
>         max_check_attempts      3
> }
> define service {
>         host_name               c6402.ambari.apache.org
>         use                     hadoop-service
>         service_description     NAMENODE::NameNode process on c6402.ambari.apache.org
>         servicegroups           HDFS
>         check_command           check_tcp_wrapper!//test!-w 1 -c 1
>         normal_check_interval   0.5
>         retry_check_interval    0.25
>         max_check_attempts      3
> }
> {code}
> 
> Notice in the above *//test* is the name of my nameservice.
> 
> Attaching screen shot of my config. So looks like it's grabbing port from the wrong prop.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/NAGIOS/package/scripts/params.py 813c626 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/NAGIOS/package/templates/hadoop-services.cfg.j2 5b8fb6d 
> 
> Diff: https://reviews.apache.org/r/18772/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>


Re: Review Request 18772: After configuring NNHA, nn process alerts don't work

Posted by Nate Cole <nc...@hortonworks.com>.

> On March 5, 2014, 10:29 a.m., Nate Cole wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/NAGIOS/package/templates/hadoop-services.cfg.j2, lines 388-392
> > <https://reviews.apache.org/r/18772/diff/1/?file=510839#file510839line388>
> >
> >     I realize it's sometimes convenient to put logic into template files, but it makes it harder to maintain.  Can we keep the logic in the params.py script and just "resolve the port" so to speak there, and just use a single value in the template file?
> 
> Vitalyi Brodetskyi wrote:
>     Nate, main thing, that our namenodes can have different ports(user has ability to set it on UI). That's why i've implemented in such way.

I understand that, but the port can still be determined in params.py - the template doesn't need to do a dfs_ha_enabled check when params.py is already taking that code path and finding the port.


- Nate


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18772/#review36228
-----------------------------------------------------------


On March 5, 2014, 10:22 a.m., Vitalyi Brodetskyi wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18772/
> -----------------------------------------------------------
> 
> (Updated March 5, 2014, 10:22 a.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko and Nate Cole.
> 
> 
> Bugs: AMBARI-4954
>     https://issues.apache.org/jira/browse/AMBARI-4954
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> 1) Configure NN HA
> 2) stack 2.0.6 (but check stack 2.1 as well)
> 3) Two alerts show" check_tcp: Port must be a positive integer"
> 
> NameNode process on c6401.ambari.apache.org
> NameNode process on c6402.ambari.apache.org
> 
> 4) Looked at /etc/nagios/objects/hadoop-services.cfg
> 5) Saw:
> 
> {code}
> define service {
>         host_name               c6401.ambari.apache.org
>         use                     hadoop-service
>         service_description     NAMENODE::NameNode process on c6401.ambari.apache.org
>         servicegroups           HDFS
>         check_command           check_tcp_wrapper!//test!-w 1 -c 1
>         normal_check_interval   0.5
>         retry_check_interval    0.25
>         max_check_attempts      3
> }
> define service {
>         host_name               c6402.ambari.apache.org
>         use                     hadoop-service
>         service_description     NAMENODE::NameNode process on c6402.ambari.apache.org
>         servicegroups           HDFS
>         check_command           check_tcp_wrapper!//test!-w 1 -c 1
>         normal_check_interval   0.5
>         retry_check_interval    0.25
>         max_check_attempts      3
> }
> {code}
> 
> Notice in the above *//test* is the name of my nameservice.
> 
> Attaching screen shot of my config. So looks like it's grabbing port from the wrong prop.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/NAGIOS/package/scripts/params.py 813c626 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/NAGIOS/package/templates/hadoop-services.cfg.j2 5b8fb6d 
> 
> Diff: https://reviews.apache.org/r/18772/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>


Re: Review Request 18772: After configuring NNHA, nn process alerts don't work

Posted by Vitalyi Brodetskyi <vb...@hortonworks.com>.

> On March 5, 2014, 3:29 p.m., Nate Cole wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/NAGIOS/package/templates/hadoop-services.cfg.j2, lines 388-392
> > <https://reviews.apache.org/r/18772/diff/1/?file=510839#file510839line388>
> >
> >     I realize it's sometimes convenient to put logic into template files, but it makes it harder to maintain.  Can we keep the logic in the params.py script and just "resolve the port" so to speak there, and just use a single value in the template file?

Nate, main thing, that our namenodes can have different ports(user has ability to set it on UI). That's why i've implemented in such way.


- Vitalyi


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18772/#review36228
-----------------------------------------------------------


On March 5, 2014, 3:22 p.m., Vitalyi Brodetskyi wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18772/
> -----------------------------------------------------------
> 
> (Updated March 5, 2014, 3:22 p.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko and Nate Cole.
> 
> 
> Bugs: AMBARI-4954
>     https://issues.apache.org/jira/browse/AMBARI-4954
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> 1) Configure NN HA
> 2) stack 2.0.6 (but check stack 2.1 as well)
> 3) Two alerts show" check_tcp: Port must be a positive integer"
> 
> NameNode process on c6401.ambari.apache.org
> NameNode process on c6402.ambari.apache.org
> 
> 4) Looked at /etc/nagios/objects/hadoop-services.cfg
> 5) Saw:
> 
> {code}
> define service {
>         host_name               c6401.ambari.apache.org
>         use                     hadoop-service
>         service_description     NAMENODE::NameNode process on c6401.ambari.apache.org
>         servicegroups           HDFS
>         check_command           check_tcp_wrapper!//test!-w 1 -c 1
>         normal_check_interval   0.5
>         retry_check_interval    0.25
>         max_check_attempts      3
> }
> define service {
>         host_name               c6402.ambari.apache.org
>         use                     hadoop-service
>         service_description     NAMENODE::NameNode process on c6402.ambari.apache.org
>         servicegroups           HDFS
>         check_command           check_tcp_wrapper!//test!-w 1 -c 1
>         normal_check_interval   0.5
>         retry_check_interval    0.25
>         max_check_attempts      3
> }
> {code}
> 
> Notice in the above *//test* is the name of my nameservice.
> 
> Attaching screen shot of my config. So looks like it's grabbing port from the wrong prop.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/NAGIOS/package/scripts/params.py 813c626 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/NAGIOS/package/templates/hadoop-services.cfg.j2 5b8fb6d 
> 
> Diff: https://reviews.apache.org/r/18772/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>


Re: Review Request 18772: After configuring NNHA, nn process alerts don't work

Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18772/#review36228
-----------------------------------------------------------



ambari-server/src/main/resources/stacks/HDP/2.0.6/services/NAGIOS/package/templates/hadoop-services.cfg.j2
<https://reviews.apache.org/r/18772/#comment67169>

    I realize it's sometimes convenient to put logic into template files, but it makes it harder to maintain.  Can we keep the logic in the params.py script and just "resolve the port" so to speak there, and just use a single value in the template file?


- Nate Cole


On March 5, 2014, 10:22 a.m., Vitalyi Brodetskyi wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18772/
> -----------------------------------------------------------
> 
> (Updated March 5, 2014, 10:22 a.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko and Nate Cole.
> 
> 
> Bugs: AMBARI-4954
>     https://issues.apache.org/jira/browse/AMBARI-4954
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> 1) Configure NN HA
> 2) stack 2.0.6 (but check stack 2.1 as well)
> 3) Two alerts show" check_tcp: Port must be a positive integer"
> 
> NameNode process on c6401.ambari.apache.org
> NameNode process on c6402.ambari.apache.org
> 
> 4) Looked at /etc/nagios/objects/hadoop-services.cfg
> 5) Saw:
> 
> {code}
> define service {
>         host_name               c6401.ambari.apache.org
>         use                     hadoop-service
>         service_description     NAMENODE::NameNode process on c6401.ambari.apache.org
>         servicegroups           HDFS
>         check_command           check_tcp_wrapper!//test!-w 1 -c 1
>         normal_check_interval   0.5
>         retry_check_interval    0.25
>         max_check_attempts      3
> }
> define service {
>         host_name               c6402.ambari.apache.org
>         use                     hadoop-service
>         service_description     NAMENODE::NameNode process on c6402.ambari.apache.org
>         servicegroups           HDFS
>         check_command           check_tcp_wrapper!//test!-w 1 -c 1
>         normal_check_interval   0.5
>         retry_check_interval    0.25
>         max_check_attempts      3
> }
> {code}
> 
> Notice in the above *//test* is the name of my nameservice.
> 
> Attaching screen shot of my config. So looks like it's grabbing port from the wrong prop.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/NAGIOS/package/scripts/params.py 813c626 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/NAGIOS/package/templates/hadoop-services.cfg.j2 5b8fb6d 
> 
> Diff: https://reviews.apache.org/r/18772/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>