You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hawq.apache.org by linwen <gi...@git.apache.org> on 2017/05/09 03:42:10 UTC
[GitHub] incubator-hawq pull request #1234: HAWQ-1436. Implement ranger pulgin servic...
GitHub user linwen opened a pull request:
https://github.com/apache/incubator-hawq/pull/1234
HAWQ-1436. Implement ranger pulgin service High Availability.
1. master will the connect to standby RPS for policy search if RPS on master failed;
2. if master has been talking to standby RPS for a period(controlled by GUC hawq_rps_check_local_interval, 5 minutes by default), it will try to connect local RPS again.
Please review, thanks!
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/linwen/incubator-hawq hawq_1436
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hawq/pull/1234.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1234
----
commit baf088d7187de4c757c7d44e7b174615c84303b1
Author: Wen Lin <wl...@pivotal.io>
Date: 2017-05-09T03:36:13Z
HAWQ-1436. Implement ranger pulgin service High Availability.
1. master will the connect to standby RPS for policy search if RPS on master failed;
2. if master has been talking to standby RPS for a period(controlled by GUC hawq_rps_check_local_interval, 5 minutes by default), it will try to connect local RPS again.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #1234: HAWQ-1436. Implement ranger pulgin service High ...
Posted by interma <gi...@git.apache.org>.
Github user interma commented on the issue:
https://github.com/apache/incubator-hawq/pull/1234
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1234: HAWQ-1436. Implement ranger pulgin servic...
Posted by stanlyxiang <gi...@git.apache.org>.
Github user stanlyxiang commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1234#discussion_r115410444
--- Diff: src/backend/libpq/rangerrest.c ---
@@ -383,58 +383,100 @@ static size_t write_callback(char *contents, size_t size, size_t nitems,
static int call_ranger_rest(CURL_HANDLE curl_handle, const char* request)
{
int ret = -1;
+ int retry = 2;
CURLcode res;
Assert(request != NULL);
/*
- * Re-initializes all options previously set on a specified CURL handle
- * to the default values. This puts back the handle to the same state as
- * it was in when it was just created with curl_easy_init.It does not
- * change the following information kept in the handle: live connections,
- * the Session ID cache, the DNS cache, the cookies and shares.
+ * If master is talking with standby RPS, for every predefined interval
+ * (controlled by a GUC hawq_rps_check_local_interval) it will check if local RPS works now.
*/
- curl_easy_reset(curl_handle->curl_handle);
- /* timeout: hard-coded temporarily and maybe should be a guc in future */
- curl_easy_setopt(curl_handle->curl_handle, CURLOPT_TIMEOUT, 30L);
-
- /* specify URL to get */
- StringInfoData tname;
- initStringInfo(&tname);
- appendStringInfo(&tname, "http://");
- appendStringInfo(&tname, "%s", master_addr_host);
- appendStringInfo(&tname, ":");
- appendStringInfo(&tname, "%d", rps_addr_port);
- appendStringInfo(&tname, "/");
- appendStringInfo(&tname, "%s", "rps");
- curl_easy_setopt(curl_handle->curl_handle, CURLOPT_URL, tname.data);
- pfree(tname.data);
-
- struct curl_slist *headers = NULL;
- headers = curl_slist_append(headers, "Content-Type:application/json");
- curl_easy_setopt(curl_handle->curl_handle, CURLOPT_HTTPHEADER, headers);
-
- curl_easy_setopt(curl_handle->curl_handle, CURLOPT_POSTFIELDS,request);
- /* send all data to this function */
- curl_easy_setopt(curl_handle->curl_handle, CURLOPT_WRITEFUNCTION, write_callback);
- curl_easy_setopt(curl_handle->curl_handle, CURLOPT_WRITEDATA, (void *)curl_handle);
-
- res = curl_easy_perform(curl_handle->curl_handle);
- if(request_id == INT_MAX)
+ if (curl_handle->talkingWithStandby)
{
- request_id = 0;
- }
- request_id++;
- /* check for errors */
- if(res != CURLE_OK)
- {
- elog(ERROR, "ranger plugin service from http://%s:%d/rps is unavailable : %s.\n",
- master_addr_host, rps_addr_port, curl_easy_strerror(res));
+ uint64_t current_time = gettime_microsec();
+ if ((current_time - curl_handle->lastCheckTimestamp) > 1000000LL * rps_check_local_interval)
+ {
+ curl_handle->talkingWithStandby = false;
+ curl_handle->lastCheckTimestamp = 0;
+ elog(RANGER_LOG,
+ "master has been talking to standby RPS for a predefined period, try switching to master RPS");
+ }
}
- else
+
+ /*
+ * try to connect standby's RPS if fail in connecting master's RPS
+ */
+ while(retry > 0 && ret != 0)
{
- ret = 0;
- elog(RANGER_LOG, "retrieved %d bytes data from ranger restful response.",
- curl_handle->response.response_size);
+ /*
+ * Re-initializes all options previously set on a specified CURL handle
+ * to the default values. This puts back the handle to the same state as
+ * it was in when it was just created with curl_easy_init.It does not
+ * change the following information kept in the handle: live connections,
+ * the Session ID cache, the DNS cache, the cookies and shares.
+ */
+ curl_easy_reset(curl_handle->curl_handle);
+ /* timeout: hard-coded temporarily and maybe should be a guc in future */
+ curl_easy_setopt(curl_handle->curl_handle, CURLOPT_TIMEOUT, 30L);
+
+ /* specify URL to get */
+ StringInfoData tname;
+ initStringInfo(&tname);
+ appendStringInfo(&tname, "http://");
+ appendStringInfo(&tname, "%s", curl_handle->talkingWithStandby?standby_addr_host:master_addr_host);
+ appendStringInfo(&tname, ":");
+ appendStringInfo(&tname, "%d", rps_addr_port);
+ appendStringInfo(&tname, "/");
+ appendStringInfo(&tname, "%s", "rps");
--- End diff --
These 2 lines can squash to 1 line . "appendStringInfo(&tname, "/rps");"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1234: HAWQ-1436. Implement ranger pulgin servic...
Posted by stanlyxiang <gi...@git.apache.org>.
Github user stanlyxiang commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1234#discussion_r115410368
--- Diff: src/backend/libpq/rangerrest.c ---
@@ -383,58 +383,100 @@ static size_t write_callback(char *contents, size_t size, size_t nitems,
static int call_ranger_rest(CURL_HANDLE curl_handle, const char* request)
{
int ret = -1;
+ int retry = 2;
CURLcode res;
Assert(request != NULL);
/*
- * Re-initializes all options previously set on a specified CURL handle
- * to the default values. This puts back the handle to the same state as
- * it was in when it was just created with curl_easy_init.It does not
- * change the following information kept in the handle: live connections,
- * the Session ID cache, the DNS cache, the cookies and shares.
+ * If master is talking with standby RPS, for every predefined interval
+ * (controlled by a GUC hawq_rps_check_local_interval) it will check if local RPS works now.
*/
- curl_easy_reset(curl_handle->curl_handle);
- /* timeout: hard-coded temporarily and maybe should be a guc in future */
- curl_easy_setopt(curl_handle->curl_handle, CURLOPT_TIMEOUT, 30L);
-
- /* specify URL to get */
- StringInfoData tname;
- initStringInfo(&tname);
- appendStringInfo(&tname, "http://");
- appendStringInfo(&tname, "%s", master_addr_host);
- appendStringInfo(&tname, ":");
- appendStringInfo(&tname, "%d", rps_addr_port);
- appendStringInfo(&tname, "/");
- appendStringInfo(&tname, "%s", "rps");
- curl_easy_setopt(curl_handle->curl_handle, CURLOPT_URL, tname.data);
- pfree(tname.data);
-
- struct curl_slist *headers = NULL;
- headers = curl_slist_append(headers, "Content-Type:application/json");
- curl_easy_setopt(curl_handle->curl_handle, CURLOPT_HTTPHEADER, headers);
-
- curl_easy_setopt(curl_handle->curl_handle, CURLOPT_POSTFIELDS,request);
- /* send all data to this function */
- curl_easy_setopt(curl_handle->curl_handle, CURLOPT_WRITEFUNCTION, write_callback);
- curl_easy_setopt(curl_handle->curl_handle, CURLOPT_WRITEDATA, (void *)curl_handle);
-
- res = curl_easy_perform(curl_handle->curl_handle);
- if(request_id == INT_MAX)
+ if (curl_handle->talkingWithStandby)
{
- request_id = 0;
- }
- request_id++;
- /* check for errors */
- if(res != CURLE_OK)
- {
- elog(ERROR, "ranger plugin service from http://%s:%d/rps is unavailable : %s.\n",
- master_addr_host, rps_addr_port, curl_easy_strerror(res));
+ uint64_t current_time = gettime_microsec();
+ if ((current_time - curl_handle->lastCheckTimestamp) > 1000000LL * rps_check_local_interval)
+ {
+ curl_handle->talkingWithStandby = false;
+ curl_handle->lastCheckTimestamp = 0;
+ elog(RANGER_LOG,
+ "master has been talking to standby RPS for a predefined period, try switching to master RPS");
--- End diff --
we could print the rps_check_local_interval time instead of "predefined period".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #1234: HAWQ-1436. Implement ranger pulgin service High ...
Posted by linwen <gi...@git.apache.org>.
Github user linwen commented on the issue:
https://github.com/apache/incubator-hawq/pull/1234
Fix and merge into master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1234: HAWQ-1436. Implement ranger pulgin servic...
Posted by linwen <gi...@git.apache.org>.
Github user linwen closed the pull request at:
https://github.com/apache/incubator-hawq/pull/1234
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---