You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Richard Ding via Review Board <no...@reviews.apache.org> on 2017/08/10 17:47:25 UTC
Review Request 61561: ATLAS-1218: Atlas says it is started but does
not accept REST requests
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61561/
-----------------------------------------------------------
Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
Bugs: ATLAS-1218
https://issues.apache.org/jira/browse/ATLAS-1218
Repository: atlas
Description
-------
This patch implements a wait_for_startup function modeled after wait_for_shutdown function.
Diffs
-----
distro/src/bin/atlas_config.py 1be9ca2b
distro/src/bin/atlas_start.py a6a3455c
Diff: https://reviews.apache.org/r/61561/diff/1/
Testing
-------
Here are the output of atlas_start and atlas_stop script:
```
starting atlas on port 21000
........................................................................................................................
Apache Atlas Server started!!!
```
and
```
stopping atlas........
Apache Atlas Server stopped!!!
```
Thanks,
Richard Ding
Re: Review Request 61561: ATLAS-1218: Atlas says it is started but
does not accept REST requests
Posted by Richard Ding via Review Board <no...@reviews.apache.org>.
> On Aug. 25, 2017, 4:31 a.m., Madhan Neethiraj wrote:
> > distro/src/bin/atlas_config.py
> > Lines 474 (patched)
> > <https://reviews.apache.org/r/61561/diff/4/?file=1798971#file1798971line474>
> >
> > Instead hardcoding "localhost", use the config value of "atlas.server.bind.address". Only when this config is not set, "localhost" should be used.
> >
> > I see that "localhost" would work currently, as Atlas server listens on all IP addresses in the host - due to the following line in EmbeddedServer.getConnector():
> > connector.setHost("0.0.0.0");
> >
> > This needes to be fixed to use the value specified in this configuration. I suggest we fix this via another JIRA.
Thanks Madhan. I opened JIRA-2087 to fix this IP address issue.
OTOH, we probally can't use ""atlas.server.bind.address" on the client side since "0.0.0.0" is a valid value for this property.
I noticed that there is another property "atlas.rest.address" defined in Atlas property file but it is not synched with other properties.
- Richard
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61561/#review183822
-----------------------------------------------------------
On Aug. 16, 2017, 11:15 p.m., Richard Ding wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61561/
> -----------------------------------------------------------
>
> (Updated Aug. 16, 2017, 11:15 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-1218
> https://issues.apache.org/jira/browse/ATLAS-1218
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch implements a wait_for_startup function modeled after wait_for_shutdown function.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_config.py 1be9ca2b
> distro/src/bin/atlas_start.py a6a3455c
>
>
> Diff: https://reviews.apache.org/r/61561/diff/4/
>
>
> Testing
> -------
>
> Here are the output of atlas_start and atlas_stop script:
> ```
> starting atlas on port 21000
> ........................................................................................................................
> Apache Atlas Server started!!!
> ```
> and
> ```
> stopping atlas........
> Apache Atlas Server stopped!!!
> ```
>
>
> Thanks,
>
> Richard Ding
>
>
Re: Review Request 61561: ATLAS-1218: Atlas says it is started but
does not accept REST requests
Posted by Madhan Neethiraj <ma...@apache.org>.
> On Aug. 25, 2017, 4:31 a.m., Madhan Neethiraj wrote:
> > distro/src/bin/atlas_config.py
> > Lines 474 (patched)
> > <https://reviews.apache.org/r/61561/diff/4/?file=1798971#file1798971line474>
> >
> > Instead hardcoding "localhost", use the config value of "atlas.server.bind.address". Only when this config is not set, "localhost" should be used.
> >
> > I see that "localhost" would work currently, as Atlas server listens on all IP addresses in the host - due to the following line in EmbeddedServer.getConnector():
> > connector.setHost("0.0.0.0");
> >
> > This needes to be fixed to use the value specified in this configuration. I suggest we fix this via another JIRA.
>
> Richard Ding wrote:
> Thanks Madhan. I opened JIRA-2087 to fix this IP address issue.
>
> OTOH, we probally can't use ""atlas.server.bind.address" on the client side since "0.0.0.0" is a valid value for this property.
>
> I noticed that there is another property "atlas.rest.address" defined in Atlas property file but it is not synched with other properties.
You are right. While "0.0.0.0" is a valid value for listener, it is not valid address to connect to. Perhaps this value can be replaced by "localhost" by this script?
Given that you filed ATLAS-2087 for the update in EmbeddedServer.getConnector(), the script changes can be included in the same JIRA. I will go head and commit this patch. Thanks!
- Madhan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61561/#review183822
-----------------------------------------------------------
On Aug. 16, 2017, 11:15 p.m., Richard Ding wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61561/
> -----------------------------------------------------------
>
> (Updated Aug. 16, 2017, 11:15 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-1218
> https://issues.apache.org/jira/browse/ATLAS-1218
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch implements a wait_for_startup function modeled after wait_for_shutdown function.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_config.py 1be9ca2b
> distro/src/bin/atlas_start.py a6a3455c
>
>
> Diff: https://reviews.apache.org/r/61561/diff/4/
>
>
> Testing
> -------
>
> Here are the output of atlas_start and atlas_stop script:
> ```
> starting atlas on port 21000
> ........................................................................................................................
> Apache Atlas Server started!!!
> ```
> and
> ```
> stopping atlas........
> Apache Atlas Server stopped!!!
> ```
>
>
> Thanks,
>
> Richard Ding
>
>
Re: Review Request 61561: ATLAS-1218: Atlas says it is started but
does not accept REST requests
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61561/#review183822
-----------------------------------------------------------
distro/src/bin/atlas_config.py
Lines 474 (patched)
<https://reviews.apache.org/r/61561/#comment259891>
Instead hardcoding "localhost", use the config value of "atlas.server.bind.address". Only when this config is not set, "localhost" should be used.
I see that "localhost" would work currently, as Atlas server listens on all IP addresses in the host - due to the following line in EmbeddedServer.getConnector():
connector.setHost("0.0.0.0");
This needes to be fixed to use the value specified in this configuration. I suggest we fix this via another JIRA.
- Madhan Neethiraj
On Aug. 16, 2017, 11:15 p.m., Richard Ding wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61561/
> -----------------------------------------------------------
>
> (Updated Aug. 16, 2017, 11:15 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-1218
> https://issues.apache.org/jira/browse/ATLAS-1218
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch implements a wait_for_startup function modeled after wait_for_shutdown function.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_config.py 1be9ca2b
> distro/src/bin/atlas_start.py a6a3455c
>
>
> Diff: https://reviews.apache.org/r/61561/diff/4/
>
>
> Testing
> -------
>
> Here are the output of atlas_start and atlas_stop script:
> ```
> starting atlas on port 21000
> ........................................................................................................................
> Apache Atlas Server started!!!
> ```
> and
> ```
> stopping atlas........
> Apache Atlas Server stopped!!!
> ```
>
>
> Thanks,
>
> Richard Ding
>
>
Re: Review Request 61561: ATLAS-1218: Atlas says it is started but
does not accept REST requests
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61561/#review183889
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On Aug. 16, 2017, 11:15 p.m., Richard Ding wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61561/
> -----------------------------------------------------------
>
> (Updated Aug. 16, 2017, 11:15 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-1218
> https://issues.apache.org/jira/browse/ATLAS-1218
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch implements a wait_for_startup function modeled after wait_for_shutdown function.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_config.py 1be9ca2b
> distro/src/bin/atlas_start.py a6a3455c
>
>
> Diff: https://reviews.apache.org/r/61561/diff/4/
>
>
> Testing
> -------
>
> Here are the output of atlas_start and atlas_stop script:
> ```
> starting atlas on port 21000
> ........................................................................................................................
> Apache Atlas Server started!!!
> ```
> and
> ```
> stopping atlas........
> Apache Atlas Server stopped!!!
> ```
>
>
> Thanks,
>
> Richard Ding
>
>
Re: Review Request 61561: ATLAS-1218: Atlas says it is started but
does not accept REST requests
Posted by Richard Ding via Review Board <no...@reviews.apache.org>.
> On Aug. 25, 2017, 9:55 a.m., Nigel Jones wrote:
> > distro/src/bin/atlas_start.py
> > Lines 134 (patched)
> > <https://reviews.apache.org/r/61561/diff/4/?file=1798972#file1798972line134>
> >
> > I notice we have a hardcoded max time of 5 minutes. I wonder if there's any scenarios where we might expect longer, and whether we need to add a -maxtime nn parameter ?
> >
> > As this is a script which the user can edit I'm not overly concerned but thought it worth mentioning
Thanks Nigel. It's hard to determine the timeout value. I've talked to Ambari developers about how Ambari handles service startup. Ambari delegates service startup functions to the service (as defined in Ambari common services). For example, Atlas service has following:
```
<commandScript>
<script>scripts/metadata_server.py</script>
<scriptType>PYTHON</scriptType>
<timeout>1200</timeout>
</commandScript>
```
Each service also provides a server_check script which Ambari runs periodically to check server status. Atlas uses following command:
```
smoke_cmd = format('curl -k -s -o /dev/null -w "%{{http_code}}" {metadata_protocol}://{atlas_host}:{metadata_port}/')
```
This is similar to what we proposed.
- Richard
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61561/#review183837
-----------------------------------------------------------
On Aug. 16, 2017, 11:15 p.m., Richard Ding wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61561/
> -----------------------------------------------------------
>
> (Updated Aug. 16, 2017, 11:15 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-1218
> https://issues.apache.org/jira/browse/ATLAS-1218
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch implements a wait_for_startup function modeled after wait_for_shutdown function.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_config.py 1be9ca2b
> distro/src/bin/atlas_start.py a6a3455c
>
>
> Diff: https://reviews.apache.org/r/61561/diff/4/
>
>
> Testing
> -------
>
> Here are the output of atlas_start and atlas_stop script:
> ```
> starting atlas on port 21000
> ........................................................................................................................
> Apache Atlas Server started!!!
> ```
> and
> ```
> stopping atlas........
> Apache Atlas Server stopped!!!
> ```
>
>
> Thanks,
>
> Richard Ding
>
>
Re: Review Request 61561: ATLAS-1218: Atlas says it is started but
does not accept REST requests
Posted by Nigel Jones <jo...@uk.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61561/#review183837
-----------------------------------------------------------
distro/src/bin/atlas_start.py
Lines 134 (patched)
<https://reviews.apache.org/r/61561/#comment259906>
I notice we have a hardcoded max time of 5 minutes. I wonder if there's any scenarios where we might expect longer, and whether we need to add a -maxtime nn parameter ?
As this is a script which the user can edit I'm not overly concerned but thought it worth mentioning
- Nigel Jones
On Aug. 17, 2017, 12:15 a.m., Richard Ding wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61561/
> -----------------------------------------------------------
>
> (Updated Aug. 17, 2017, 12:15 a.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-1218
> https://issues.apache.org/jira/browse/ATLAS-1218
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch implements a wait_for_startup function modeled after wait_for_shutdown function.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_config.py 1be9ca2b
> distro/src/bin/atlas_start.py a6a3455c
>
>
> Diff: https://reviews.apache.org/r/61561/diff/4/
>
>
> Testing
> -------
>
> Here are the output of atlas_start and atlas_stop script:
> ```
> starting atlas on port 21000
> ........................................................................................................................
> Apache Atlas Server started!!!
> ```
> and
> ```
> stopping atlas........
> Apache Atlas Server stopped!!!
> ```
>
>
> Thanks,
>
> Richard Ding
>
>
Re: Review Request 61561: ATLAS-1218: Atlas says it is started but
does not accept REST requests
Posted by Richard Ding via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61561/
-----------------------------------------------------------
(Updated Aug. 16, 2017, 11:15 p.m.)
Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
Changes
-------
This patch fixed the empty string issue.
Bugs: ATLAS-1218
https://issues.apache.org/jira/browse/ATLAS-1218
Repository: atlas
Description
-------
This patch implements a wait_for_startup function modeled after wait_for_shutdown function.
Diffs (updated)
-----
distro/src/bin/atlas_config.py 1be9ca2b
distro/src/bin/atlas_start.py a6a3455c
Diff: https://reviews.apache.org/r/61561/diff/4/
Changes: https://reviews.apache.org/r/61561/diff/3-4/
Testing
-------
Here are the output of atlas_start and atlas_stop script:
```
starting atlas on port 21000
........................................................................................................................
Apache Atlas Server started!!!
```
and
```
stopping atlas........
Apache Atlas Server stopped!!!
```
Thanks,
Richard Ding
Re: Review Request 61561: ATLAS-1218: Atlas says it is started but
does not accept REST requests
Posted by Richard Ding via Review Board <no...@reviews.apache.org>.
> On Aug. 16, 2017, 9:22 a.m., David Radley wrote:
> > distro/src/bin/atlas_start.py
> > Lines 134 (patched)
> > <https://reviews.apache.org/r/61561/diff/3/?file=1798416#file1798416line134>
> >
> > You are passing an empty string as the message. I suggest removing this parameter and hard coding the message in the function.
Done.
- Richard
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61561/#review183025
-----------------------------------------------------------
On Aug. 16, 2017, 11:15 p.m., Richard Ding wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61561/
> -----------------------------------------------------------
>
> (Updated Aug. 16, 2017, 11:15 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-1218
> https://issues.apache.org/jira/browse/ATLAS-1218
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch implements a wait_for_startup function modeled after wait_for_shutdown function.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_config.py 1be9ca2b
> distro/src/bin/atlas_start.py a6a3455c
>
>
> Diff: https://reviews.apache.org/r/61561/diff/4/
>
>
> Testing
> -------
>
> Here are the output of atlas_start and atlas_stop script:
> ```
> starting atlas on port 21000
> ........................................................................................................................
> Apache Atlas Server started!!!
> ```
> and
> ```
> stopping atlas........
> Apache Atlas Server stopped!!!
> ```
>
>
> Thanks,
>
> Richard Ding
>
>
Re: Review Request 61561: ATLAS-1218: Atlas says it is started but
does not accept REST requests
Posted by David Radley <da...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61561/#review183025
-----------------------------------------------------------
distro/src/bin/atlas_start.py
Lines 134 (patched)
<https://reviews.apache.org/r/61561/#comment259021>
You are passing an empty string as the message. I suggest removing this parameter and hard coding the message in the function.
- David Radley
On Aug. 15, 2017, 10:08 p.m., Richard Ding wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61561/
> -----------------------------------------------------------
>
> (Updated Aug. 15, 2017, 10:08 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-1218
> https://issues.apache.org/jira/browse/ATLAS-1218
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch implements a wait_for_startup function modeled after wait_for_shutdown function.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_config.py 1be9ca2b
> distro/src/bin/atlas_start.py a6a3455c
>
>
> Diff: https://reviews.apache.org/r/61561/diff/3/
>
>
> Testing
> -------
>
> Here are the output of atlas_start and atlas_stop script:
> ```
> starting atlas on port 21000
> ........................................................................................................................
> Apache Atlas Server started!!!
> ```
> and
> ```
> stopping atlas........
> Apache Atlas Server stopped!!!
> ```
>
>
> Thanks,
>
> Richard Ding
>
>
Re: Review Request 61561: ATLAS-1218: Atlas says it is started but
does not accept REST requests
Posted by Richard Ding via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61561/
-----------------------------------------------------------
(Updated Aug. 15, 2017, 10:08 p.m.)
Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
Changes
-------
The new patch simplified the code with python socket module.
Bugs: ATLAS-1218
https://issues.apache.org/jira/browse/ATLAS-1218
Repository: atlas
Description
-------
This patch implements a wait_for_startup function modeled after wait_for_shutdown function.
Diffs (updated)
-----
distro/src/bin/atlas_config.py 1be9ca2b
distro/src/bin/atlas_start.py a6a3455c
Diff: https://reviews.apache.org/r/61561/diff/3/
Changes: https://reviews.apache.org/r/61561/diff/2-3/
Testing
-------
Here are the output of atlas_start and atlas_stop script:
```
starting atlas on port 21000
........................................................................................................................
Apache Atlas Server started!!!
```
and
```
stopping atlas........
Apache Atlas Server stopped!!!
```
Thanks,
Richard Ding
Re: Review Request 61561: ATLAS-1218: Atlas says it is started but
does not accept REST requests
Posted by Richard Ding via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61561/
-----------------------------------------------------------
(Updated Aug. 14, 2017, 11:19 p.m.)
Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
Changes
-------
Update the patch based on David feedbacks.
Bugs: ATLAS-1218
https://issues.apache.org/jira/browse/ATLAS-1218
Repository: atlas
Description
-------
This patch implements a wait_for_startup function modeled after wait_for_shutdown function.
Diffs (updated)
-----
distro/src/bin/atlas_config.py 1be9ca2b
distro/src/bin/atlas_start.py a6a3455c
Diff: https://reviews.apache.org/r/61561/diff/2/
Changes: https://reviews.apache.org/r/61561/diff/1-2/
Testing
-------
Here are the output of atlas_start and atlas_stop script:
```
starting atlas on port 21000
........................................................................................................................
Apache Atlas Server started!!!
```
and
```
stopping atlas........
Apache Atlas Server stopped!!!
```
Thanks,
Richard Ding
Re: Review Request 61561: ATLAS-1218: Atlas says it is started but
does not accept REST requests
Posted by Nigel Jones <jo...@uk.ibm.com>.
> On Aug. 11, 2017, 12:11 p.m., David Radley wrote:
> > distro/src/bin/atlas_config.py
> > Lines 71 (patched)
> > <https://reviews.apache.org/r/61561/diff/1/?file=1794995#file1794995line71>
> >
> > I am uncomfortable duplicating the default port values here. I suggest moving this logic to the java class that already has the default port values
>
> Richard Ding wrote:
> The default ports are hard-coded in Java class:
> ```
> appPort = StringUtils.isEmpty(enableTLSFlag) || enableTLSFlag.equals("true") ?
> configuration.getInt(ATLAS_SERVER_HTTPS_PORT, 21443) :
> configuration.getInt(ATLAS_SERVER_HTTP_PORT, 21000);
> ```
> It's also difficult to access Java class from python code.
>
> David Radley wrote:
> hi Richard , unless you move the polling logic into Java code (in a separate thread) - so you have more control over it. I assume the process starts right away, but the delay is in web server code prior to it opening the port. all the best, David.
>
> Richard Ding wrote:
> You are right. The polling logic is in the startup python script which has no access to Java classes.
>
> The process is started by the python script (atlas_start.py) and the script has no way to know when the server initialization is complete. So inside the startup script the best we can do is to ping the port. After changing the code to use socket module, the code is simplified and it appears to work well even with SSL port.
(I know this issue is closed). I think the loose coupling, python script, and making a dummy socket connection make sense (as that is basically what the end user on a browser would do). But in terms of spreading defaults across multiple places (I'm thinking docs, scripts, java source), we could do a pre-processing step which would insert and/or update ports in docs, scripts, java etc - not sure it's worth the effort though. (so won't reopen)
- Nigel
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61561/#review182676
-----------------------------------------------------------
On Aug. 17, 2017, 12:15 a.m., Richard Ding wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61561/
> -----------------------------------------------------------
>
> (Updated Aug. 17, 2017, 12:15 a.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-1218
> https://issues.apache.org/jira/browse/ATLAS-1218
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch implements a wait_for_startup function modeled after wait_for_shutdown function.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_config.py 1be9ca2b
> distro/src/bin/atlas_start.py a6a3455c
>
>
> Diff: https://reviews.apache.org/r/61561/diff/4/
>
>
> Testing
> -------
>
> Here are the output of atlas_start and atlas_stop script:
> ```
> starting atlas on port 21000
> ........................................................................................................................
> Apache Atlas Server started!!!
> ```
> and
> ```
> stopping atlas........
> Apache Atlas Server stopped!!!
> ```
>
>
> Thanks,
>
> Richard Ding
>
>
Re: Review Request 61561: ATLAS-1218: Atlas says it is started but
does not accept REST requests
Posted by Richard Ding via Review Board <no...@reviews.apache.org>.
> On Aug. 11, 2017, 11:11 a.m., David Radley wrote:
> > distro/src/bin/atlas_config.py
> > Lines 71 (patched)
> > <https://reviews.apache.org/r/61561/diff/1/?file=1794995#file1794995line71>
> >
> > I am uncomfortable duplicating the default port values here. I suggest moving this logic to the java class that already has the default port values
The default ports are hard-coded in Java class:
```
appPort = StringUtils.isEmpty(enableTLSFlag) || enableTLSFlag.equals("true") ?
configuration.getInt(ATLAS_SERVER_HTTPS_PORT, 21443) :
configuration.getInt(ATLAS_SERVER_HTTP_PORT, 21000);
```
It's also difficult to access Java class from python code.
> On Aug. 11, 2017, 11:11 a.m., David Radley wrote:
> > distro/src/bin/atlas_config.py
> > Lines 454 (patched)
> > <https://reviews.apache.org/r/61561/diff/1/?file=1794995#file1794995line454>
> >
> > is this going to work if -port was specified on the atlas_start command?
Good catch! The new patch handles -port option.
> On Aug. 11, 2017, 11:11 a.m., David Radley wrote:
> > distro/src/bin/atlas_config.py
> > Lines 473 (patched)
> > <https://reviews.apache.org/r/61561/diff/1/?file=1794995#file1794995line473>
> >
> > I was hoping we would be able to be notified when Atlas was started or the startup time reduced, by understanding why there is a delay and minimizing it.
> >
> > Once we understand that and decide that polling is the appropriate resolution - I suggest a 1 second poll - as we can do nothing with Atlas until it is started.
Good point. Right now the atlas_start.py script just launches a process asynchronously without knowning when the Atlas server will complete the initialization (e.g. creating hbase tables, solr indexes and typedefs). This solution is to let the startup script ping the endpoint of the web server.
I'll change the ping interval to 1 sec.
> On Aug. 11, 2017, 11:11 a.m., David Radley wrote:
> > distro/src/bin/atlas_config.py
> > Lines 475 (patched)
> > <https://reviews.apache.org/r/61561/diff/1/?file=1794995#file1794995line475>
> >
> > I am not expert on all the ways Atlas can be started, does this work with OAuth and does Knox work with this?
The script just ping the Atlas server endpoint without performing the login. It should work with SSO solutions. It also assumes that the script runs on the same host as the Atlas server and is able to access directly to the server endpoint.
> On Aug. 11, 2017, 11:11 a.m., David Radley wrote:
> > distro/src/bin/atlas_config.py
> > Lines 477 (patched)
> > <https://reviews.apache.org/r/61561/diff/1/?file=1794995#file1794995line477>
> >
> > If we have failures during startup, how can we tell the polling to cease?
Just like _wait_for_shutdown_, the polling has a timeout setting (5 mins). The timeout is the only way to stop the polling. There is no callbacks from the server. Maybe we can try to tail and grep the log file to see there is any failures during startup.
- Richard
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61561/#review182676
-----------------------------------------------------------
On Aug. 10, 2017, 5:47 p.m., Richard Ding wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61561/
> -----------------------------------------------------------
>
> (Updated Aug. 10, 2017, 5:47 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-1218
> https://issues.apache.org/jira/browse/ATLAS-1218
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch implements a wait_for_startup function modeled after wait_for_shutdown function.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_config.py 1be9ca2b
> distro/src/bin/atlas_start.py a6a3455c
>
>
> Diff: https://reviews.apache.org/r/61561/diff/1/
>
>
> Testing
> -------
>
> Here are the output of atlas_start and atlas_stop script:
> ```
> starting atlas on port 21000
> ........................................................................................................................
> Apache Atlas Server started!!!
> ```
> and
> ```
> stopping atlas........
> Apache Atlas Server stopped!!!
> ```
>
>
> Thanks,
>
> Richard Ding
>
>
Re: Review Request 61561: ATLAS-1218: Atlas says it is started but
does not accept REST requests
Posted by David Radley <da...@apache.org>.
> On Aug. 11, 2017, 11:11 a.m., David Radley wrote:
> > distro/src/bin/atlas_config.py
> > Lines 71 (patched)
> > <https://reviews.apache.org/r/61561/diff/1/?file=1794995#file1794995line71>
> >
> > I am uncomfortable duplicating the default port values here. I suggest moving this logic to the java class that already has the default port values
>
> Richard Ding wrote:
> The default ports are hard-coded in Java class:
> ```
> appPort = StringUtils.isEmpty(enableTLSFlag) || enableTLSFlag.equals("true") ?
> configuration.getInt(ATLAS_SERVER_HTTPS_PORT, 21443) :
> configuration.getInt(ATLAS_SERVER_HTTP_PORT, 21000);
> ```
> It's also difficult to access Java class from python code.
hi Richard , unless you move the polling logic into Java code (in a separate thread) - so you have more control over it. I assume the process starts right away, but the delay is in web server code prior to it opening the port. all the best, David.
- David
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61561/#review182676
-----------------------------------------------------------
On Aug. 15, 2017, 10:08 p.m., Richard Ding wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61561/
> -----------------------------------------------------------
>
> (Updated Aug. 15, 2017, 10:08 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-1218
> https://issues.apache.org/jira/browse/ATLAS-1218
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch implements a wait_for_startup function modeled after wait_for_shutdown function.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_config.py 1be9ca2b
> distro/src/bin/atlas_start.py a6a3455c
>
>
> Diff: https://reviews.apache.org/r/61561/diff/3/
>
>
> Testing
> -------
>
> Here are the output of atlas_start and atlas_stop script:
> ```
> starting atlas on port 21000
> ........................................................................................................................
> Apache Atlas Server started!!!
> ```
> and
> ```
> stopping atlas........
> Apache Atlas Server stopped!!!
> ```
>
>
> Thanks,
>
> Richard Ding
>
>
Re: Review Request 61561: ATLAS-1218: Atlas says it is started but
does not accept REST requests
Posted by Richard Ding via Review Board <no...@reviews.apache.org>.
> On Aug. 11, 2017, 11:11 a.m., David Radley wrote:
> > distro/src/bin/atlas_config.py
> > Lines 71 (patched)
> > <https://reviews.apache.org/r/61561/diff/1/?file=1794995#file1794995line71>
> >
> > I am uncomfortable duplicating the default port values here. I suggest moving this logic to the java class that already has the default port values
>
> Richard Ding wrote:
> The default ports are hard-coded in Java class:
> ```
> appPort = StringUtils.isEmpty(enableTLSFlag) || enableTLSFlag.equals("true") ?
> configuration.getInt(ATLAS_SERVER_HTTPS_PORT, 21443) :
> configuration.getInt(ATLAS_SERVER_HTTP_PORT, 21000);
> ```
> It's also difficult to access Java class from python code.
>
> David Radley wrote:
> hi Richard , unless you move the polling logic into Java code (in a separate thread) - so you have more control over it. I assume the process starts right away, but the delay is in web server code prior to it opening the port. all the best, David.
You are right. The polling logic is in the startup python script which has no access to Java classes.
The process is started by the python script (atlas_start.py) and the script has no way to know when the server initialization is complete. So inside the startup script the best we can do is to ping the port. After changing the code to use socket module, the code is simplified and it appears to work well even with SSL port.
- Richard
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61561/#review182676
-----------------------------------------------------------
On Aug. 15, 2017, 10:08 p.m., Richard Ding wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61561/
> -----------------------------------------------------------
>
> (Updated Aug. 15, 2017, 10:08 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-1218
> https://issues.apache.org/jira/browse/ATLAS-1218
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch implements a wait_for_startup function modeled after wait_for_shutdown function.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_config.py 1be9ca2b
> distro/src/bin/atlas_start.py a6a3455c
>
>
> Diff: https://reviews.apache.org/r/61561/diff/3/
>
>
> Testing
> -------
>
> Here are the output of atlas_start and atlas_stop script:
> ```
> starting atlas on port 21000
> ........................................................................................................................
> Apache Atlas Server started!!!
> ```
> and
> ```
> stopping atlas........
> Apache Atlas Server stopped!!!
> ```
>
>
> Thanks,
>
> Richard Ding
>
>
Re: Review Request 61561: ATLAS-1218: Atlas says it is started but
does not accept REST requests
Posted by David Radley <da...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61561/#review182676
-----------------------------------------------------------
distro/src/bin/atlas_config.py
Lines 71 (patched)
<https://reviews.apache.org/r/61561/#comment258666>
I am uncomfortable duplicating the default port values here. I suggest moving this logic to the java class that already has the default port values
distro/src/bin/atlas_config.py
Lines 454 (patched)
<https://reviews.apache.org/r/61561/#comment258667>
is this going to work if -port was specified on the atlas_start command?
distro/src/bin/atlas_config.py
Lines 473 (patched)
<https://reviews.apache.org/r/61561/#comment258663>
I was hoping we would be able to be notified when Atlas was started or the startup time reduced, by understanding why there is a delay and minimizing it.
Once we understand that and decide that polling is the appropriate resolution - I suggest a 1 second poll - as we can do nothing with Atlas until it is started.
distro/src/bin/atlas_config.py
Lines 475 (patched)
<https://reviews.apache.org/r/61561/#comment258664>
I am not expert on all the ways Atlas can be started, does this work with OAuth and does Knox work with this?
distro/src/bin/atlas_config.py
Lines 477 (patched)
<https://reviews.apache.org/r/61561/#comment258665>
If we have failures during startup, how can we tell the polling to cease?
- David Radley
On Aug. 10, 2017, 5:47 p.m., Richard Ding wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61561/
> -----------------------------------------------------------
>
> (Updated Aug. 10, 2017, 5:47 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-1218
> https://issues.apache.org/jira/browse/ATLAS-1218
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch implements a wait_for_startup function modeled after wait_for_shutdown function.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_config.py 1be9ca2b
> distro/src/bin/atlas_start.py a6a3455c
>
>
> Diff: https://reviews.apache.org/r/61561/diff/1/
>
>
> Testing
> -------
>
> Here are the output of atlas_start and atlas_stop script:
> ```
> starting atlas on port 21000
> ........................................................................................................................
> Apache Atlas Server started!!!
> ```
> and
> ```
> stopping atlas........
> Apache Atlas Server stopped!!!
> ```
>
>
> Thanks,
>
> Richard Ding
>
>