You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by bhuvnesh chaudhary <bc...@pivotal.io> on 2016/03/11 02:22:15 UTC
Review Request 44687: Update RCO : PXF should start after HDFS
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44687/
-----------------------------------------------------------
Review request for Ambari, Alejandro Fernandez, jun aoki, Jayush Luniya, Matt, and Oleksandr Diachenko.
Bugs: AMBARI-15381
https://issues.apache.org/jira/browse/AMBARI-15381
Repository: ambari
Description
-------
Update RCO to ensure that PXF is started after HDFS.
PXF connects to namenode during startup and retries couple of times until namenode has come up.
Diffs
-----
ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json b28f2a9
Diff: https://reviews.apache.org/r/44687/diff/
Testing
-------
yes. manually.
Thanks,
bhuvnesh chaudhary
Re: Review Request 44687: Update RCO : PXF should start after HDFS
Posted by bhuvnesh chaudhary <bc...@pivotal.io>.
> On March 11, 2016, 9:06 a.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json, line 14
> > <https://reviews.apache.org/r/44687/diff/1/?file=1295169#file1295169line14>
> >
> > RCO can be defined at service level (https://issues.apache.org/jira/browse/AMBARI-9363). Instead of updating RCO of HDP stack, can you move the HAWQ and PXF specific ordering into the service?
>
> bhuvnesh chaudhary wrote:
> Thanks Jayush for the suggestion, let me try that.
> However, RCO at service level will be applicable only for trunk, for 2.2 branch we still need to update HDP rco. correct ?
> If so i will remove trunk tag from the list of branches, and keep this pathc for 2.2.0 only.
Tried putting role_command_order.json under /var/lib/ambari-server/resources/common-services/HAWQ/2.0.0/role_command_order.json. However, it appears that its not loaded due to which HAWQMASTER was started before NAMENODE and it failed. Same for PXF.
Here are the contents of RCO:
[root@c6401 ~]# cat /var/lib/ambari-server/resources/common-services/HAWQ/2.0.0/role_command_order.json
{
"_comment" : "Record format:",
"_comment" : "blockedRole-blockedCommand: [blockerRole1-blockerCommand1, blockerRole2-blockerCommand2, ...]",
"general_deps" : {
"_comment" : "dependencies for all cases",
"HAWQMASTER-START" : ["NAMENODE-START", "DATANODE-START", "NODEMANAGER-START"],
"HAWQSTANDBY-START" : ["HAWQMASTER-START"],
"HAWQSEGMENT-START" : ["HAWQMASTER-START", "HAWQSTANDBY-START"],
"HAWQ_SERVICE_CHECK-SERVICE_CHECK" : ["HAWQSEGMENT-START", "HDFS_SERVICE_CHECK-SERVICE_CHECK", "YARN_SERVICE_CHECK-SERVICE_CHECK", "PXF_SERVICE_CHECK-SERVICE_CHECK"]
}
}
Checked ambari-server.log, it does not show that this RCO is loaded but shows the others under /var/lib/ambari-server/resources/stacks which are loaded, will need to debug further for the root cause.
Adding Tim Thorpe as well to the reviewers list.
- bhuvnesh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44687/#review123118
-----------------------------------------------------------
On March 11, 2016, 1:22 a.m., bhuvnesh chaudhary wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44687/
> -----------------------------------------------------------
>
> (Updated March 11, 2016, 1:22 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, jun aoki, Jayush Luniya, Matt, and Oleksandr Diachenko.
>
>
> Bugs: AMBARI-15381
> https://issues.apache.org/jira/browse/AMBARI-15381
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Update RCO to ensure that PXF is started after HDFS.
> PXF connects to namenode during startup and retries couple of times until namenode has come up.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json b28f2a9
>
> Diff: https://reviews.apache.org/r/44687/diff/
>
>
> Testing
> -------
>
> yes. manually.
>
>
> Thanks,
>
> bhuvnesh chaudhary
>
>
Re: Review Request 44687: Update RCO : PXF should start after HDFS
Posted by bhuvnesh chaudhary <bc...@pivotal.io>.
> On March 11, 2016, 9:06 a.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json, line 14
> > <https://reviews.apache.org/r/44687/diff/1/?file=1295169#file1295169line14>
> >
> > RCO can be defined at service level (https://issues.apache.org/jira/browse/AMBARI-9363). Instead of updating RCO of HDP stack, can you move the HAWQ and PXF specific ordering into the service?
Thanks Jayush for the suggestion, let me try that.
However, RCO at service level will be applicable only for trunk, for 2.2 branch we still need to update HDP rco. correct ?
If so i will remove trunk tag from the list of branches, and keep this pathc for 2.2.0 only.
- bhuvnesh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44687/#review123118
-----------------------------------------------------------
On March 11, 2016, 1:22 a.m., bhuvnesh chaudhary wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44687/
> -----------------------------------------------------------
>
> (Updated March 11, 2016, 1:22 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, jun aoki, Jayush Luniya, Matt, and Oleksandr Diachenko.
>
>
> Bugs: AMBARI-15381
> https://issues.apache.org/jira/browse/AMBARI-15381
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Update RCO to ensure that PXF is started after HDFS.
> PXF connects to namenode during startup and retries couple of times until namenode has come up.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json b28f2a9
>
> Diff: https://reviews.apache.org/r/44687/diff/
>
>
> Testing
> -------
>
> yes. manually.
>
>
> Thanks,
>
> bhuvnesh chaudhary
>
>
Re: Review Request 44687: Update RCO : PXF should start after HDFS
Posted by Tim Thorpe <tt...@ca.ibm.com>.
> On March 11, 2016, 9:06 a.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json, line 14
> > <https://reviews.apache.org/r/44687/diff/1/?file=1295169#file1295169line14>
> >
> > RCO can be defined at service level (https://issues.apache.org/jira/browse/AMBARI-9363). Instead of updating RCO of HDP stack, can you move the HAWQ and PXF specific ordering into the service?
>
> bhuvnesh chaudhary wrote:
> Thanks Jayush for the suggestion, let me try that.
> However, RCO at service level will be applicable only for trunk, for 2.2 branch we still need to update HDP rco. correct ?
> If so i will remove trunk tag from the list of branches, and keep this pathc for 2.2.0 only.
>
> bhuvnesh chaudhary wrote:
> Tried putting role_command_order.json under /var/lib/ambari-server/resources/common-services/HAWQ/2.0.0/role_command_order.json. However, it appears that its not loaded due to which HAWQMASTER was started before NAMENODE and it failed. Same for PXF.
> Here are the contents of RCO:
> [root@c6401 ~]# cat /var/lib/ambari-server/resources/common-services/HAWQ/2.0.0/role_command_order.json
> {
> "_comment" : "Record format:",
> "_comment" : "blockedRole-blockedCommand: [blockerRole1-blockerCommand1, blockerRole2-blockerCommand2, ...]",
> "general_deps" : {
> "_comment" : "dependencies for all cases",
> "HAWQMASTER-START" : ["NAMENODE-START", "DATANODE-START", "NODEMANAGER-START"],
> "HAWQSTANDBY-START" : ["HAWQMASTER-START"],
> "HAWQSEGMENT-START" : ["HAWQMASTER-START", "HAWQSTANDBY-START"],
> "HAWQ_SERVICE_CHECK-SERVICE_CHECK" : ["HAWQSEGMENT-START", "HDFS_SERVICE_CHECK-SERVICE_CHECK", "YARN_SERVICE_CHECK-SERVICE_CHECK", "PXF_SERVICE_CHECK-SERVICE_CHECK"]
> }
> }
>
> Checked ambari-server.log, it does not show that this RCO is loaded but shows the others under /var/lib/ambari-server/resources/stacks which are loaded, will need to debug further for the root cause.
> Adding Tim Thorpe as well to the reviewers list.
>
> Jayush Luniya wrote:
> @Bhuvnesh
> Go ahead with your changes without the refactoring in trunk and 2.2 branch. I dont want to block you on it. Lets handle the refactoring in a separate JIRA. @Tim Thorpe, can you check if anything is wrong with the service-level RCO?
>
> bhuvnesh chaudhary wrote:
> Thank you Jayush, will go ahead with the change.
When I added the RCO change for putting it at the service level, I only tested with services defined in the stack not having inherited the RCO from common-services. I'll look into it some more and see if I can figure out what the problem might be. Sorry was on vacation for a week.
- Tim
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44687/#review123118
-----------------------------------------------------------
On March 11, 2016, 9:50 p.m., bhuvnesh chaudhary wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44687/
> -----------------------------------------------------------
>
> (Updated March 11, 2016, 9:50 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, jun aoki, Jayush Luniya, Matt, Oleksandr Diachenko, and Tim Thorpe.
>
>
> Bugs: AMBARI-15381
> https://issues.apache.org/jira/browse/AMBARI-15381
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Update RCO to ensure that PXF is started after HDFS.
> PXF connects to namenode during startup and retries couple of times until namenode has come up.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json b28f2a9
>
> Diff: https://reviews.apache.org/r/44687/diff/
>
>
> Testing
> -------
>
> yes. manually.
>
>
> Thanks,
>
> bhuvnesh chaudhary
>
>
Re: Review Request 44687: Update RCO : PXF should start after HDFS
Posted by Jayush Luniya <jl...@hortonworks.com>.
> On March 11, 2016, 9:06 a.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json, line 14
> > <https://reviews.apache.org/r/44687/diff/1/?file=1295169#file1295169line14>
> >
> > RCO can be defined at service level (https://issues.apache.org/jira/browse/AMBARI-9363). Instead of updating RCO of HDP stack, can you move the HAWQ and PXF specific ordering into the service?
>
> bhuvnesh chaudhary wrote:
> Thanks Jayush for the suggestion, let me try that.
> However, RCO at service level will be applicable only for trunk, for 2.2 branch we still need to update HDP rco. correct ?
> If so i will remove trunk tag from the list of branches, and keep this pathc for 2.2.0 only.
>
> bhuvnesh chaudhary wrote:
> Tried putting role_command_order.json under /var/lib/ambari-server/resources/common-services/HAWQ/2.0.0/role_command_order.json. However, it appears that its not loaded due to which HAWQMASTER was started before NAMENODE and it failed. Same for PXF.
> Here are the contents of RCO:
> [root@c6401 ~]# cat /var/lib/ambari-server/resources/common-services/HAWQ/2.0.0/role_command_order.json
> {
> "_comment" : "Record format:",
> "_comment" : "blockedRole-blockedCommand: [blockerRole1-blockerCommand1, blockerRole2-blockerCommand2, ...]",
> "general_deps" : {
> "_comment" : "dependencies for all cases",
> "HAWQMASTER-START" : ["NAMENODE-START", "DATANODE-START", "NODEMANAGER-START"],
> "HAWQSTANDBY-START" : ["HAWQMASTER-START"],
> "HAWQSEGMENT-START" : ["HAWQMASTER-START", "HAWQSTANDBY-START"],
> "HAWQ_SERVICE_CHECK-SERVICE_CHECK" : ["HAWQSEGMENT-START", "HDFS_SERVICE_CHECK-SERVICE_CHECK", "YARN_SERVICE_CHECK-SERVICE_CHECK", "PXF_SERVICE_CHECK-SERVICE_CHECK"]
> }
> }
>
> Checked ambari-server.log, it does not show that this RCO is loaded but shows the others under /var/lib/ambari-server/resources/stacks which are loaded, will need to debug further for the root cause.
> Adding Tim Thorpe as well to the reviewers list.
@Bhuvnesh
Go ahead with your changes without the refactoring in trunk and 2.2 branch. I dont want to block you on it. Lets handle the refactoring in a separate JIRA. @Tim Thorpe, can you check if anything is wrong with the service-level RCO?
- Jayush
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44687/#review123118
-----------------------------------------------------------
On March 11, 2016, 9:50 p.m., bhuvnesh chaudhary wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44687/
> -----------------------------------------------------------
>
> (Updated March 11, 2016, 9:50 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, jun aoki, Jayush Luniya, Matt, Oleksandr Diachenko, and Tim Thorpe.
>
>
> Bugs: AMBARI-15381
> https://issues.apache.org/jira/browse/AMBARI-15381
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Update RCO to ensure that PXF is started after HDFS.
> PXF connects to namenode during startup and retries couple of times until namenode has come up.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json b28f2a9
>
> Diff: https://reviews.apache.org/r/44687/diff/
>
>
> Testing
> -------
>
> yes. manually.
>
>
> Thanks,
>
> bhuvnesh chaudhary
>
>
Re: Review Request 44687: Update RCO : PXF should start after HDFS
Posted by bhuvnesh chaudhary <bc...@pivotal.io>.
> On March 11, 2016, 9:06 a.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json, line 14
> > <https://reviews.apache.org/r/44687/diff/1/?file=1295169#file1295169line14>
> >
> > RCO can be defined at service level (https://issues.apache.org/jira/browse/AMBARI-9363). Instead of updating RCO of HDP stack, can you move the HAWQ and PXF specific ordering into the service?
>
> bhuvnesh chaudhary wrote:
> Thanks Jayush for the suggestion, let me try that.
> However, RCO at service level will be applicable only for trunk, for 2.2 branch we still need to update HDP rco. correct ?
> If so i will remove trunk tag from the list of branches, and keep this pathc for 2.2.0 only.
>
> bhuvnesh chaudhary wrote:
> Tried putting role_command_order.json under /var/lib/ambari-server/resources/common-services/HAWQ/2.0.0/role_command_order.json. However, it appears that its not loaded due to which HAWQMASTER was started before NAMENODE and it failed. Same for PXF.
> Here are the contents of RCO:
> [root@c6401 ~]# cat /var/lib/ambari-server/resources/common-services/HAWQ/2.0.0/role_command_order.json
> {
> "_comment" : "Record format:",
> "_comment" : "blockedRole-blockedCommand: [blockerRole1-blockerCommand1, blockerRole2-blockerCommand2, ...]",
> "general_deps" : {
> "_comment" : "dependencies for all cases",
> "HAWQMASTER-START" : ["NAMENODE-START", "DATANODE-START", "NODEMANAGER-START"],
> "HAWQSTANDBY-START" : ["HAWQMASTER-START"],
> "HAWQSEGMENT-START" : ["HAWQMASTER-START", "HAWQSTANDBY-START"],
> "HAWQ_SERVICE_CHECK-SERVICE_CHECK" : ["HAWQSEGMENT-START", "HDFS_SERVICE_CHECK-SERVICE_CHECK", "YARN_SERVICE_CHECK-SERVICE_CHECK", "PXF_SERVICE_CHECK-SERVICE_CHECK"]
> }
> }
>
> Checked ambari-server.log, it does not show that this RCO is loaded but shows the others under /var/lib/ambari-server/resources/stacks which are loaded, will need to debug further for the root cause.
> Adding Tim Thorpe as well to the reviewers list.
>
> Jayush Luniya wrote:
> @Bhuvnesh
> Go ahead with your changes without the refactoring in trunk and 2.2 branch. I dont want to block you on it. Lets handle the refactoring in a separate JIRA. @Tim Thorpe, can you check if anything is wrong with the service-level RCO?
Thank you Jayush, will go ahead with the change.
- bhuvnesh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44687/#review123118
-----------------------------------------------------------
On March 11, 2016, 9:50 p.m., bhuvnesh chaudhary wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44687/
> -----------------------------------------------------------
>
> (Updated March 11, 2016, 9:50 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, jun aoki, Jayush Luniya, Matt, Oleksandr Diachenko, and Tim Thorpe.
>
>
> Bugs: AMBARI-15381
> https://issues.apache.org/jira/browse/AMBARI-15381
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Update RCO to ensure that PXF is started after HDFS.
> PXF connects to namenode during startup and retries couple of times until namenode has come up.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json b28f2a9
>
> Diff: https://reviews.apache.org/r/44687/diff/
>
>
> Testing
> -------
>
> yes. manually.
>
>
> Thanks,
>
> bhuvnesh chaudhary
>
>
Re: Review Request 44687: Update RCO : PXF should start after HDFS
Posted by Jayush Luniya <jl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44687/#review123118
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json (line 14)
<https://reviews.apache.org/r/44687/#comment185289>
RCO can be defined at service level (https://issues.apache.org/jira/browse/AMBARI-9363). Instead of updating RCO of HDP stack, can you move the HAWQ and PXF specific ordering into the service?
- Jayush Luniya
On March 11, 2016, 1:22 a.m., bhuvnesh chaudhary wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44687/
> -----------------------------------------------------------
>
> (Updated March 11, 2016, 1:22 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, jun aoki, Jayush Luniya, Matt, and Oleksandr Diachenko.
>
>
> Bugs: AMBARI-15381
> https://issues.apache.org/jira/browse/AMBARI-15381
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Update RCO to ensure that PXF is started after HDFS.
> PXF connects to namenode during startup and retries couple of times until namenode has come up.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json b28f2a9
>
> Diff: https://reviews.apache.org/r/44687/diff/
>
>
> Testing
> -------
>
> yes. manually.
>
>
> Thanks,
>
> bhuvnesh chaudhary
>
>