You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2020/05/05 19:04:07 UTC
[GitHub] [hive] aasha opened a new pull request #1004: Add Ranger Replication
aasha opened a new pull request #1004:
URL: https://github.com/apache/hive/pull/1004
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aasha commented on a change in pull request #1004: HIVE-23351 : Ranger Replication Scheduling
Posted by GitBox <gi...@apache.org>.
aasha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r420743627
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplAck.java
##########
@@ -23,7 +23,9 @@
public enum ReplAck {
DUMP_ACKNOWLEDGEMENT("_finished_dump"),
EVENTS_DUMP("_events_dump"),
- LOAD_ACKNOWLEDGEMENT("_finished_load");
+ LOAD_ACKNOWLEDGEMENT("_finished_load"),
+ RANGER_DUMP_ACKNOWLEDGEMENT("_finished_ranger_dump"),
Review comment:
As discussed we will remove the Acks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aasha commented on a change in pull request #1004: HIVE-23351 : Ranger Replication Scheduling
Posted by GitBox <gi...@apache.org>.
aasha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r420744102
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##########
@@ -170,6 +175,31 @@ public int execute() {
return 0;
}
+ private void initiateAuthorizationDumpTask(Path currentDumpPath) throws SemanticException, IOException {
+ if (RANGER_AUTHORIZER.equalsIgnoreCase(conf.getVar(HiveConf.ConfVars.REPL_AUTHORIZATION_PROVIDER_SERVICE))) {
+ Path rangerDumpRoot = new Path(currentDumpPath, ReplUtils.REPL_RANGER_BASE_DIR);
+ FileSystem fs = rangerDumpRoot.getFileSystem(conf);
+ if (fs.exists(new Path(rangerDumpRoot, ReplAck.RANGER_DUMP_ACKNOWLEDGEMENT.toString()))) {
+ LOG.info("Ranger Authorization Metadata is already exported at {} ", rangerDumpRoot);
+ } else {
+ LOG.info("Exporting Authorization Metadata at {} ", rangerDumpRoot);
+ RangerDumpWork rangerDumpWork = new RangerDumpWork(rangerDumpRoot, work.dbNameOrPattern);
+ Task<RangerDumpWork> rangerDumpTask = TaskFactory.get(rangerDumpWork, conf);
+ if (childTasks == null) {
+ childTasks = new ArrayList<>();
+ }
+ childTasks.add(rangerDumpTask);
Review comment:
Its getting added before the data copy tasks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aasha commented on a change in pull request #1004: HIVE-23351 : Ranger Replication Scheduling
Posted by GitBox <gi...@apache.org>.
aasha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r422170394
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##########
@@ -103,13 +105,43 @@ public int execute() {
}
work.setRootTask(this);
this.parentTasks = null;
+ if (shouldLoadAuthorizationMetadata()) {
+ LOG.info("Loading authorization data");
+ try {
+ initiateAuthorizationLoadTask(work.dumpDirectory);
+ } catch (Exception e) {
+ LOG.error("failed", e);
+ setException(e);
+ return ErrorMsg.getErrorMsg(e.getMessage()).getErrorCode();
+ }
+ }
if (work.isIncrementalLoad()) {
return executeIncrementalLoad();
} else {
return executeBootStrapLoad();
}
}
+ private boolean shouldLoadAuthorizationMetadata() {
+ return conf.getBoolVar(HiveConf.ConfVars.REPL_INCLUDE_AUTHORIZATION_METADATA);
+ }
+
+ private void initiateAuthorizationLoadTask(String hiveDumpDirectory) throws SemanticException {
+ if (RANGER_AUTHORIZER.equalsIgnoreCase(conf.getVar(HiveConf.ConfVars.REPL_AUTHORIZATION_PROVIDER_SERVICE))) {
+ Path rangerLoadRoot = new Path(new Path(hiveDumpDirectory).getParent(), ReplUtils.REPL_RANGER_BASE_DIR);
+ LOG.info("Importing Authorization Metadata from {} ", rangerLoadRoot);
+ RangerLoadWork rangerLoadWork = new RangerLoadWork(rangerLoadRoot, work.getSourceDbName(), work.dbNameToLoadIn);
+ Task<RangerLoadWork> rangerLoadTask = TaskFactory.get(rangerLoadWork, conf);
+ if (childTasks == null) {
+ childTasks = new ArrayList<>();
+ }
+ childTasks.add(rangerLoadTask);
Review comment:
No, load tasks are added after this ranger task. Even metadata is loaded through tasks in Repl Load. No?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aasha commented on a change in pull request #1004: HIVE-23351 : Ranger Replication Scheduling
Posted by GitBox <gi...@apache.org>.
aasha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r422160855
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ranger/RangerPolicy.java
##########
@@ -0,0 +1,1513 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl.ranger;
+
+import org.codehaus.jackson.annotate.JsonAutoDetect;
+import org.codehaus.jackson.annotate.JsonAutoDetect.Visibility;
+import org.codehaus.jackson.annotate.JsonIgnoreProperties;
+import org.codehaus.jackson.map.annotate.JsonSerialize;
+
+import javax.xml.bind.annotation.XmlAccessType;
+import javax.xml.bind.annotation.XmlAccessorType;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * RangerPolicy class to contain Ranger Policy details.
+ */
+@JsonAutoDetect(fieldVisibility = Visibility.ANY)
+@JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+@JsonIgnoreProperties(ignoreUnknown = true)
+@XmlRootElement
+@XmlAccessorType(XmlAccessType.FIELD)
+public class RangerPolicy extends RangerBaseModelObject implements java.io.Serializable {
+ public static final int POLICY_TYPE_ACCESS = 0;
+ public static final int POLICY_TYPE_DATAMASK = 1;
+ public static final int POLICY_TYPE_ROWFILTER = 2;
+
+ public static final int[] POLICY_TYPES = new int[]{
+ POLICY_TYPE_ACCESS,
+ POLICY_TYPE_DATAMASK,
+ POLICY_TYPE_ROWFILTER,
+ };
+
+ public static final String MASK_TYPE_NULL = "MASK_NULL";
+ public static final String MASK_TYPE_NONE = "MASK_NONE";
+ public static final String MASK_TYPE_CUSTOM = "CUSTOM";
+
+ private static final long serialVersionUID = 1L;
+
+ private String service;
+ private String name;
+ private Integer policyType;
+ private String description;
+ private String resourceSignature;
+ private Boolean isAuditEnabled;
+ private Map<String, RangerPolicyResource> resources;
+ private List<RangerPolicyItem> policyItems;
+ private List<RangerPolicyItem> denyPolicyItems;
+ private List<RangerPolicyItem> allowExceptions;
+ private List<RangerPolicyItem> denyExceptions;
+ private List<RangerDataMaskPolicyItem> dataMaskPolicyItems;
+ private List<RangerRowFilterPolicyItem> rowFilterPolicyItems;
+
+
+ /**
+ * Ranger Policy default constructor.
+ */
+ public RangerPolicy() {
+ this(null, null, null, null, null, null, null);
+ }
+
+ /**
+ * @param service
+ * @param name
+ * @param policyType
+ * @param description
+ * @param resources
+ * @param policyItems
+ * @param resourceSignature TODO
+ */
+ public RangerPolicy(String service, String name, Integer policyType, String description,
+ Map<String, RangerPolicyResource> resources, List<RangerPolicyItem> policyItems,
+ String resourceSignature) {
+ super();
+ setService(service);
+ setName(name);
+ setPolicyType(policyType);
+ setDescription(description);
+ setResourceSignature(resourceSignature);
+ setIsAuditEnabled(null);
+ setResources(resources);
+ setPolicyItems(policyItems);
+ setDenyPolicyItems(null);
+ setAllowExceptions(null);
+ setDenyExceptions(null);
+ setDataMaskPolicyItems(null);
+ setRowFilterPolicyItems(null);
+ }
+
+ /**
+ * @param other
+ */
+ public void updateFrom(RangerPolicy other) {
+ super.updateFrom(other);
+
+ setService(other.getService());
+ setName(other.getName());
+ setPolicyType(other.getPolicyType());
+ setDescription(other.getDescription());
+ setResourceSignature(other.getResourceSignature());
+ setIsAuditEnabled(other.getIsAuditEnabled());
+ setResources(other.getResources());
+ setPolicyItems(other.getPolicyItems());
+ setDenyPolicyItems(other.getDenyPolicyItems());
+ setAllowExceptions(other.getAllowExceptions());
+ setDenyExceptions(other.getDenyExceptions());
+ setDataMaskPolicyItems(other.getDataMaskPolicyItems());
+ setRowFilterPolicyItems(other.getRowFilterPolicyItems());
+ }
+
+ /**
+ * @return the type
+ */
+ public String getService() {
+ return service;
+ }
+
+ /**
+ * @param service the type to set
+ */
+ public void setService(String service) {
+ this.service = service;
+ }
+
+ /**
+ * @return the name
+ */
+ public String getName() {
+ return name;
+ }
+
+ /**
+ * @param name the name to set
+ */
+ public void setName(String name) {
+ this.name = name;
+ }
+
+ /**
+ * @return the policyType
+ */
+ public Integer getPolicyType() {
+ return policyType;
+ }
+
+ /**
+ * @param policyType the policyType to set
+ */
+ public void setPolicyType(Integer policyType) {
+ this.policyType = policyType;
+ }
+
+ /**
+ * @return the description
+ */
+ public String getDescription() {
+ return description;
+ }
+
+ /**
+ * @param description the description to set
+ */
+ public void setDescription(String description) {
+ this.description = description;
+ }
+
+ /**
+ * @return the resourceSignature
+ */
+ public String getResourceSignature() {
+ return resourceSignature;
+ }
+
+ /**
+ * @param resourceSignature the resourceSignature to set
+ */
+ public void setResourceSignature(String resourceSignature) {
+ this.resourceSignature = resourceSignature;
+ }
+
+ /**
+ * @return the isAuditEnabled
+ */
+ public Boolean getIsAuditEnabled() {
+ return isAuditEnabled;
+ }
+
+ /**
+ * @param isAuditEnabled the isEnabled to set
+ */
+ public void setIsAuditEnabled(Boolean isAuditEnabled) {
+ this.isAuditEnabled = isAuditEnabled == null ? Boolean.TRUE : isAuditEnabled;
+ }
+
+ /**
+ * @return the resources
+ */
+ public Map<String, RangerPolicyResource> getResources() {
+ return resources;
+ }
+
+ /**
+ * @param resources the resources to set
+ */
+ public void setResources(Map<String, RangerPolicyResource> resources) {
+ if (this.resources == null) {
+ this.resources = new HashMap<>();
+ }
+
+ if (this.resources == resources) {
+ return;
+ }
+
+ this.resources.clear();
+
+ if (resources != null) {
+ for (Map.Entry<String, RangerPolicyResource> e : resources.entrySet()) {
+ this.resources.put(e.getKey(), e.getValue());
+ }
+ }
+ }
+
+ /**
+ * @return the policyItems
+ */
+ public List<RangerPolicyItem> getPolicyItems() {
+ return policyItems;
+ }
+
+ /**
+ * @param policyItems the policyItems to set
+ */
+ public void setPolicyItems(List<RangerPolicyItem> policyItems) {
+ if (this.policyItems == null) {
+ this.policyItems = new ArrayList<>();
+ }
+
+ if (this.policyItems == policyItems) {
+ return;
+ }
+
+ this.policyItems.clear();
+
+ if (policyItems != null) {
+ this.policyItems.addAll(policyItems);
+ }
+ }
+
+ /**
+ * @return the denyPolicyItems
+ */
+ public List<RangerPolicyItem> getDenyPolicyItems() {
+ return denyPolicyItems;
+ }
+
+ /**
+ * @param denyPolicyItems the denyPolicyItems to set
+ */
+ public void setDenyPolicyItems(List<RangerPolicyItem> denyPolicyItems) {
+ if (this.denyPolicyItems == null) {
+ this.denyPolicyItems = new ArrayList<>();
+ }
+
+ if (this.denyPolicyItems == denyPolicyItems) {
+ return;
+ }
+
+ this.denyPolicyItems.clear();
+
+ if (denyPolicyItems != null) {
+ this.denyPolicyItems.addAll(denyPolicyItems);
+ }
+ }
+
+ /**
+ * @return the allowExceptions
+ */
+ public List<RangerPolicyItem> getAllowExceptions() {
+ return allowExceptions;
+ }
+
+ /**
+ * @param allowExceptions the allowExceptions to set
+ */
+ public void setAllowExceptions(List<RangerPolicyItem> allowExceptions) {
+ if (this.allowExceptions == null) {
+ this.allowExceptions = new ArrayList<>();
+ }
+
+ if (this.allowExceptions == allowExceptions) {
+ return;
+ }
+
+ this.allowExceptions.clear();
+
+ if (allowExceptions != null) {
+ this.allowExceptions.addAll(allowExceptions);
+ }
+ }
+
+ /**
+ * @return the denyExceptions
+ */
+ public List<RangerPolicyItem> getDenyExceptions() {
+ return denyExceptions;
+ }
+
+ /**
+ * @param denyExceptions the denyExceptions to set
+ */
+ public void setDenyExceptions(List<RangerPolicyItem> denyExceptions) {
+ if (this.denyExceptions == null) {
+ this.denyExceptions = new ArrayList<>();
+ }
+
+ if (this.denyExceptions == denyExceptions) {
+ return;
+ }
+
+ this.denyExceptions.clear();
+
+ if (denyExceptions != null) {
+ this.denyExceptions.addAll(denyExceptions);
+ }
+ }
+
+ public List<RangerDataMaskPolicyItem> getDataMaskPolicyItems() {
+ return dataMaskPolicyItems;
+ }
+
+ public void setDataMaskPolicyItems(List<RangerDataMaskPolicyItem> dataMaskPolicyItems) {
+ if (this.dataMaskPolicyItems == null) {
+ this.dataMaskPolicyItems = new ArrayList<>();
+ }
+
+ if (this.dataMaskPolicyItems == dataMaskPolicyItems) {
+ return;
+ }
+
+ this.dataMaskPolicyItems.clear();
+
+ if (dataMaskPolicyItems != null) {
+ this.dataMaskPolicyItems.addAll(dataMaskPolicyItems);
+ }
+ }
+
+ public List<RangerRowFilterPolicyItem> getRowFilterPolicyItems() {
+ return rowFilterPolicyItems;
+ }
+
+ public void setRowFilterPolicyItems(List<RangerRowFilterPolicyItem> rowFilterPolicyItems) {
+ if (this.rowFilterPolicyItems == null) {
+ this.rowFilterPolicyItems = new ArrayList<>();
+ }
+
+ if (this.rowFilterPolicyItems == rowFilterPolicyItems) {
+ return;
+ }
+
+ this.rowFilterPolicyItems.clear();
+
+ if (rowFilterPolicyItems != null) {
+ this.rowFilterPolicyItems.addAll(rowFilterPolicyItems);
+ }
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ toString(sb);
+ return sb.toString();
+ }
+
+ public StringBuilder toString(StringBuilder sb) {
+ sb.append("RangerPolicy={");
+
+ super.toString(sb);
+
+ sb.append("service={").append(service).append("} ");
+ sb.append("name={").append(name).append("} ");
+ sb.append("policyType={").append(policyType).append("} ");
+ sb.append("description={").append(description).append("} ");
+ sb.append("resourceSignature={").append(resourceSignature).append("} ");
+ sb.append("isAuditEnabled={").append(isAuditEnabled).append("} ");
+
+ sb.append("resources={");
+ if (resources != null) {
+ for (Map.Entry<String, RangerPolicyResource> e : resources.entrySet()) {
+ sb.append(e.getKey()).append("={");
+ e.getValue().toString(sb);
+ sb.append("} ");
+ }
+ }
+ sb.append("} ");
+
+ sb.append("policyItems={");
+ if (policyItems != null) {
+ for (RangerPolicyItem policyItem : policyItems) {
+ if (policyItem != null) {
+ policyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("denyPolicyItems={");
+ if (denyPolicyItems != null) {
+ for (RangerPolicyItem policyItem : denyPolicyItems) {
+ if (policyItem != null) {
+ policyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("allowExceptions={");
+ if (allowExceptions != null) {
+ for (RangerPolicyItem policyItem : allowExceptions) {
+ if (policyItem != null) {
+ policyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("denyExceptions={");
+ if (denyExceptions != null) {
+ for (RangerPolicyItem policyItem : denyExceptions) {
+ if (policyItem != null) {
+ policyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("dataMaskPolicyItems={");
+ if (dataMaskPolicyItems != null) {
+ for (RangerDataMaskPolicyItem dataMaskPolicyItem : dataMaskPolicyItems) {
+ if (dataMaskPolicyItem != null) {
+ dataMaskPolicyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("rowFilterPolicyItems={");
+ if (rowFilterPolicyItems != null) {
+ for (RangerRowFilterPolicyItem rowFilterPolicyItem : rowFilterPolicyItems) {
+ if (rowFilterPolicyItem != null) {
+ rowFilterPolicyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("}");
+
+ return sb;
+ }
+
+ /**
+ * RangerPolicyResource class to store the resource path values.
+ */
+ @JsonAutoDetect(fieldVisibility = Visibility.ANY)
+ @JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+ @JsonIgnoreProperties(ignoreUnknown = true)
+ @XmlRootElement
+ @XmlAccessorType(XmlAccessType.FIELD)
+ public static class RangerPolicyResource implements java.io.Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private List<String> values;
+ private Boolean isExcludes;
+ private Boolean isRecursive;
+
+ public RangerPolicyResource() {
+ this((List<String>) null, null, null);
+ }
+
+ public RangerPolicyResource(String value) {
+ setValue(value);
+ setIsExcludes(null);
+ setIsRecursive(null);
+ }
+
+ public RangerPolicyResource(String value, Boolean isExcludes, Boolean isRecursive) {
+ setValue(value);
+ setIsExcludes(isExcludes);
+ setIsRecursive(isRecursive);
+ }
+
+ public RangerPolicyResource(List<String> values, Boolean isExcludes, Boolean isRecursive) {
+ setValues(values);
+ setIsExcludes(isExcludes);
+ setIsRecursive(isRecursive);
+ }
+
+ /**
+ * @return the values
+ */
+ public List<String> getValues() {
+ return values;
+ }
+
+ /**
+ * @param values the values to set
+ */
+ public void setValues(List<String> values) {
+ if (this.values == null) {
+ this.values = new ArrayList<>();
+ }
+ if (this.values == values) {
+ return;
+ }
+ this.values.clear();
+ if (values != null) {
+ this.values.addAll(values);
+ }
+ }
+
+ /**
+ * @param value the value to set
+ */
+ public void setValue(String value) {
+ if (this.values == null) {
+ this.values = new ArrayList<>();
+ }
+ this.values.clear();
+ this.values.add(value);
+ }
+
+ /**
+ * @return the isExcludes
+ */
+ public Boolean getIsExcludes() {
+ return isExcludes;
+ }
+
+ /**
+ * @param isExcludes the isExcludes to set
+ */
+ public void setIsExcludes(Boolean isExcludes) {
+ this.isExcludes = isExcludes == null ? Boolean.FALSE : isExcludes;
+ }
+
+ /**
+ * @return the isRecursive
+ */
+ public Boolean getIsRecursive() {
+ return isRecursive;
+ }
+
+ /**
+ * @param isRecursive the isRecursive to set
+ */
+ public void setIsRecursive(Boolean isRecursive) {
+ this.isRecursive = isRecursive == null ? Boolean.FALSE : isRecursive;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ toString(sb);
+ return sb.toString();
+ }
+
+ public StringBuilder toString(StringBuilder sb) {
+ sb.append("RangerPolicyResource={");
+ sb.append("values={");
+ if (values != null) {
+ for (String value : values) {
+ sb.append(value).append(" ");
+ }
+ }
+ sb.append("} ");
+ sb.append("isExcludes={").append(isExcludes).append("} ");
+ sb.append("isRecursive={").append(isRecursive).append("} ");
+ sb.append("}");
+
+ return sb;
+ }
+
+ @Override
+ public int hashCode() {
+ final int prime = 31;
+ int result = 1;
+ result = prime * result
+ + ((isExcludes == null) ? 0 : isExcludes.hashCode());
+ result = prime * result
+ + ((isRecursive == null) ? 0 : isRecursive.hashCode());
+ result = prime * result
+ + ((values == null) ? 0 : values.hashCode());
+ return result;
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (this == obj) {
+ return true;
+ }
+ if (obj == null) {
+ return false;
+ }
+ if (getClass() != obj.getClass()) {
+ return false;
+ }
+ RangerPolicyResource other = (RangerPolicyResource) obj;
+ if (isExcludes == null) {
+ if (other.isExcludes != null) {
+ return false;
+ }
+ } else if (!isExcludes.equals(other.isExcludes)) {
+ return false;
+ }
+ if (isRecursive == null) {
+ if (other.isRecursive != null) {
+ return false;
+ }
+ } else if (!isRecursive.equals(other.isRecursive)) {
+ return false;
+ }
+ if (values == null) {
+ if (other.values != null) {
+ return false;
+ }
+ } else if (!values.equals(other.values)) {
+ return false;
+ }
+ return true;
+ }
+ }
+
+ /**
+ * RangerPolicyItem class contains ranger policy items like access and permissions.
+ */
+ @JsonAutoDetect(fieldVisibility = Visibility.ANY)
+ @JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+ @JsonIgnoreProperties(ignoreUnknown = true)
+ @XmlRootElement
+ @XmlAccessorType(XmlAccessType.FIELD)
+ public static class RangerPolicyItem implements java.io.Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private List<RangerPolicyItemAccess> accesses;
+ private List<String> users;
+ private List<String> groups;
+ private List<RangerPolicyItemCondition> conditions;
+ private Boolean delegateAdmin;
+
+ public RangerPolicyItem() {
Review comment:
It will be needed to deserialize it
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ranger/RangerPolicy.java
##########
@@ -0,0 +1,1513 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl.ranger;
+
+import org.codehaus.jackson.annotate.JsonAutoDetect;
+import org.codehaus.jackson.annotate.JsonAutoDetect.Visibility;
+import org.codehaus.jackson.annotate.JsonIgnoreProperties;
+import org.codehaus.jackson.map.annotate.JsonSerialize;
+
+import javax.xml.bind.annotation.XmlAccessType;
+import javax.xml.bind.annotation.XmlAccessorType;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * RangerPolicy class to contain Ranger Policy details.
+ */
+@JsonAutoDetect(fieldVisibility = Visibility.ANY)
+@JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+@JsonIgnoreProperties(ignoreUnknown = true)
+@XmlRootElement
+@XmlAccessorType(XmlAccessType.FIELD)
+public class RangerPolicy extends RangerBaseModelObject implements java.io.Serializable {
+ public static final int POLICY_TYPE_ACCESS = 0;
+ public static final int POLICY_TYPE_DATAMASK = 1;
+ public static final int POLICY_TYPE_ROWFILTER = 2;
+
+ public static final int[] POLICY_TYPES = new int[]{
+ POLICY_TYPE_ACCESS,
+ POLICY_TYPE_DATAMASK,
+ POLICY_TYPE_ROWFILTER,
+ };
+
+ public static final String MASK_TYPE_NULL = "MASK_NULL";
+ public static final String MASK_TYPE_NONE = "MASK_NONE";
+ public static final String MASK_TYPE_CUSTOM = "CUSTOM";
+
+ private static final long serialVersionUID = 1L;
+
+ private String service;
+ private String name;
+ private Integer policyType;
+ private String description;
+ private String resourceSignature;
+ private Boolean isAuditEnabled;
+ private Map<String, RangerPolicyResource> resources;
+ private List<RangerPolicyItem> policyItems;
+ private List<RangerPolicyItem> denyPolicyItems;
+ private List<RangerPolicyItem> allowExceptions;
+ private List<RangerPolicyItem> denyExceptions;
+ private List<RangerDataMaskPolicyItem> dataMaskPolicyItems;
+ private List<RangerRowFilterPolicyItem> rowFilterPolicyItems;
+
+
+ /**
+ * Ranger Policy default constructor.
+ */
+ public RangerPolicy() {
+ this(null, null, null, null, null, null, null);
+ }
+
+ /**
+ * @param service
+ * @param name
+ * @param policyType
+ * @param description
+ * @param resources
+ * @param policyItems
+ * @param resourceSignature TODO
+ */
+ public RangerPolicy(String service, String name, Integer policyType, String description,
+ Map<String, RangerPolicyResource> resources, List<RangerPolicyItem> policyItems,
+ String resourceSignature) {
+ super();
+ setService(service);
+ setName(name);
+ setPolicyType(policyType);
+ setDescription(description);
+ setResourceSignature(resourceSignature);
+ setIsAuditEnabled(null);
+ setResources(resources);
+ setPolicyItems(policyItems);
+ setDenyPolicyItems(null);
+ setAllowExceptions(null);
+ setDenyExceptions(null);
+ setDataMaskPolicyItems(null);
+ setRowFilterPolicyItems(null);
+ }
+
+ /**
+ * @param other
+ */
+ public void updateFrom(RangerPolicy other) {
+ super.updateFrom(other);
+
+ setService(other.getService());
+ setName(other.getName());
+ setPolicyType(other.getPolicyType());
+ setDescription(other.getDescription());
+ setResourceSignature(other.getResourceSignature());
+ setIsAuditEnabled(other.getIsAuditEnabled());
+ setResources(other.getResources());
+ setPolicyItems(other.getPolicyItems());
+ setDenyPolicyItems(other.getDenyPolicyItems());
+ setAllowExceptions(other.getAllowExceptions());
+ setDenyExceptions(other.getDenyExceptions());
+ setDataMaskPolicyItems(other.getDataMaskPolicyItems());
+ setRowFilterPolicyItems(other.getRowFilterPolicyItems());
+ }
+
+ /**
+ * @return the type
+ */
+ public String getService() {
+ return service;
+ }
+
+ /**
+ * @param service the type to set
+ */
+ public void setService(String service) {
+ this.service = service;
+ }
+
+ /**
+ * @return the name
+ */
+ public String getName() {
+ return name;
+ }
+
+ /**
+ * @param name the name to set
+ */
+ public void setName(String name) {
+ this.name = name;
+ }
+
+ /**
+ * @return the policyType
+ */
+ public Integer getPolicyType() {
+ return policyType;
+ }
+
+ /**
+ * @param policyType the policyType to set
+ */
+ public void setPolicyType(Integer policyType) {
+ this.policyType = policyType;
+ }
+
+ /**
+ * @return the description
+ */
+ public String getDescription() {
+ return description;
+ }
+
+ /**
+ * @param description the description to set
+ */
+ public void setDescription(String description) {
+ this.description = description;
+ }
+
+ /**
+ * @return the resourceSignature
+ */
+ public String getResourceSignature() {
+ return resourceSignature;
+ }
+
+ /**
+ * @param resourceSignature the resourceSignature to set
+ */
+ public void setResourceSignature(String resourceSignature) {
+ this.resourceSignature = resourceSignature;
+ }
+
+ /**
+ * @return the isAuditEnabled
+ */
+ public Boolean getIsAuditEnabled() {
+ return isAuditEnabled;
+ }
+
+ /**
+ * @param isAuditEnabled the isEnabled to set
+ */
+ public void setIsAuditEnabled(Boolean isAuditEnabled) {
+ this.isAuditEnabled = isAuditEnabled == null ? Boolean.TRUE : isAuditEnabled;
+ }
+
+ /**
+ * @return the resources
+ */
+ public Map<String, RangerPolicyResource> getResources() {
+ return resources;
+ }
+
+ /**
+ * @param resources the resources to set
+ */
+ public void setResources(Map<String, RangerPolicyResource> resources) {
+ if (this.resources == null) {
+ this.resources = new HashMap<>();
+ }
+
+ if (this.resources == resources) {
+ return;
+ }
+
+ this.resources.clear();
+
+ if (resources != null) {
+ for (Map.Entry<String, RangerPolicyResource> e : resources.entrySet()) {
+ this.resources.put(e.getKey(), e.getValue());
+ }
+ }
+ }
+
+ /**
+ * @return the policyItems
+ */
+ public List<RangerPolicyItem> getPolicyItems() {
+ return policyItems;
+ }
+
+ /**
+ * @param policyItems the policyItems to set
+ */
+ public void setPolicyItems(List<RangerPolicyItem> policyItems) {
+ if (this.policyItems == null) {
+ this.policyItems = new ArrayList<>();
+ }
+
+ if (this.policyItems == policyItems) {
+ return;
+ }
+
+ this.policyItems.clear();
+
+ if (policyItems != null) {
+ this.policyItems.addAll(policyItems);
+ }
+ }
+
+ /**
+ * @return the denyPolicyItems
+ */
+ public List<RangerPolicyItem> getDenyPolicyItems() {
+ return denyPolicyItems;
+ }
+
+ /**
+ * @param denyPolicyItems the denyPolicyItems to set
+ */
+ public void setDenyPolicyItems(List<RangerPolicyItem> denyPolicyItems) {
+ if (this.denyPolicyItems == null) {
+ this.denyPolicyItems = new ArrayList<>();
+ }
+
+ if (this.denyPolicyItems == denyPolicyItems) {
+ return;
+ }
+
+ this.denyPolicyItems.clear();
+
+ if (denyPolicyItems != null) {
+ this.denyPolicyItems.addAll(denyPolicyItems);
+ }
+ }
+
+ /**
+ * @return the allowExceptions
+ */
+ public List<RangerPolicyItem> getAllowExceptions() {
+ return allowExceptions;
+ }
+
+ /**
+ * @param allowExceptions the allowExceptions to set
+ */
+ public void setAllowExceptions(List<RangerPolicyItem> allowExceptions) {
+ if (this.allowExceptions == null) {
+ this.allowExceptions = new ArrayList<>();
+ }
+
+ if (this.allowExceptions == allowExceptions) {
+ return;
+ }
+
+ this.allowExceptions.clear();
+
+ if (allowExceptions != null) {
+ this.allowExceptions.addAll(allowExceptions);
+ }
+ }
+
+ /**
+ * @return the denyExceptions
+ */
+ public List<RangerPolicyItem> getDenyExceptions() {
+ return denyExceptions;
+ }
+
+ /**
+ * @param denyExceptions the denyExceptions to set
+ */
+ public void setDenyExceptions(List<RangerPolicyItem> denyExceptions) {
+ if (this.denyExceptions == null) {
+ this.denyExceptions = new ArrayList<>();
+ }
+
+ if (this.denyExceptions == denyExceptions) {
+ return;
+ }
+
+ this.denyExceptions.clear();
+
+ if (denyExceptions != null) {
+ this.denyExceptions.addAll(denyExceptions);
+ }
+ }
+
+ public List<RangerDataMaskPolicyItem> getDataMaskPolicyItems() {
+ return dataMaskPolicyItems;
+ }
+
+ public void setDataMaskPolicyItems(List<RangerDataMaskPolicyItem> dataMaskPolicyItems) {
+ if (this.dataMaskPolicyItems == null) {
+ this.dataMaskPolicyItems = new ArrayList<>();
+ }
+
+ if (this.dataMaskPolicyItems == dataMaskPolicyItems) {
+ return;
+ }
+
+ this.dataMaskPolicyItems.clear();
+
+ if (dataMaskPolicyItems != null) {
+ this.dataMaskPolicyItems.addAll(dataMaskPolicyItems);
+ }
+ }
+
+ public List<RangerRowFilterPolicyItem> getRowFilterPolicyItems() {
+ return rowFilterPolicyItems;
+ }
+
+ public void setRowFilterPolicyItems(List<RangerRowFilterPolicyItem> rowFilterPolicyItems) {
+ if (this.rowFilterPolicyItems == null) {
+ this.rowFilterPolicyItems = new ArrayList<>();
+ }
+
+ if (this.rowFilterPolicyItems == rowFilterPolicyItems) {
+ return;
+ }
+
+ this.rowFilterPolicyItems.clear();
+
+ if (rowFilterPolicyItems != null) {
+ this.rowFilterPolicyItems.addAll(rowFilterPolicyItems);
+ }
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ toString(sb);
+ return sb.toString();
+ }
+
+ public StringBuilder toString(StringBuilder sb) {
+ sb.append("RangerPolicy={");
+
+ super.toString(sb);
+
+ sb.append("service={").append(service).append("} ");
+ sb.append("name={").append(name).append("} ");
+ sb.append("policyType={").append(policyType).append("} ");
+ sb.append("description={").append(description).append("} ");
+ sb.append("resourceSignature={").append(resourceSignature).append("} ");
+ sb.append("isAuditEnabled={").append(isAuditEnabled).append("} ");
+
+ sb.append("resources={");
+ if (resources != null) {
+ for (Map.Entry<String, RangerPolicyResource> e : resources.entrySet()) {
+ sb.append(e.getKey()).append("={");
+ e.getValue().toString(sb);
+ sb.append("} ");
+ }
+ }
+ sb.append("} ");
+
+ sb.append("policyItems={");
+ if (policyItems != null) {
+ for (RangerPolicyItem policyItem : policyItems) {
+ if (policyItem != null) {
+ policyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("denyPolicyItems={");
+ if (denyPolicyItems != null) {
+ for (RangerPolicyItem policyItem : denyPolicyItems) {
+ if (policyItem != null) {
+ policyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("allowExceptions={");
+ if (allowExceptions != null) {
+ for (RangerPolicyItem policyItem : allowExceptions) {
+ if (policyItem != null) {
+ policyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("denyExceptions={");
+ if (denyExceptions != null) {
+ for (RangerPolicyItem policyItem : denyExceptions) {
+ if (policyItem != null) {
+ policyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("dataMaskPolicyItems={");
+ if (dataMaskPolicyItems != null) {
+ for (RangerDataMaskPolicyItem dataMaskPolicyItem : dataMaskPolicyItems) {
+ if (dataMaskPolicyItem != null) {
+ dataMaskPolicyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("rowFilterPolicyItems={");
+ if (rowFilterPolicyItems != null) {
+ for (RangerRowFilterPolicyItem rowFilterPolicyItem : rowFilterPolicyItems) {
+ if (rowFilterPolicyItem != null) {
+ rowFilterPolicyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("}");
+
+ return sb;
+ }
+
+ /**
+ * RangerPolicyResource class to store the resource path values.
+ */
+ @JsonAutoDetect(fieldVisibility = Visibility.ANY)
+ @JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+ @JsonIgnoreProperties(ignoreUnknown = true)
+ @XmlRootElement
+ @XmlAccessorType(XmlAccessType.FIELD)
+ public static class RangerPolicyResource implements java.io.Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private List<String> values;
+ private Boolean isExcludes;
+ private Boolean isRecursive;
+
+ public RangerPolicyResource() {
+ this((List<String>) null, null, null);
+ }
+
+ public RangerPolicyResource(String value) {
+ setValue(value);
+ setIsExcludes(null);
+ setIsRecursive(null);
+ }
+
+ public RangerPolicyResource(String value, Boolean isExcludes, Boolean isRecursive) {
+ setValue(value);
+ setIsExcludes(isExcludes);
+ setIsRecursive(isRecursive);
+ }
+
+ public RangerPolicyResource(List<String> values, Boolean isExcludes, Boolean isRecursive) {
+ setValues(values);
+ setIsExcludes(isExcludes);
+ setIsRecursive(isRecursive);
+ }
+
+ /**
+ * @return the values
+ */
+ public List<String> getValues() {
+ return values;
+ }
+
+ /**
+ * @param values the values to set
+ */
+ public void setValues(List<String> values) {
+ if (this.values == null) {
+ this.values = new ArrayList<>();
+ }
+ if (this.values == values) {
+ return;
+ }
+ this.values.clear();
+ if (values != null) {
+ this.values.addAll(values);
+ }
+ }
+
+ /**
+ * @param value the value to set
+ */
+ public void setValue(String value) {
+ if (this.values == null) {
+ this.values = new ArrayList<>();
+ }
+ this.values.clear();
+ this.values.add(value);
+ }
+
+ /**
+ * @return the isExcludes
+ */
+ public Boolean getIsExcludes() {
+ return isExcludes;
+ }
+
+ /**
+ * @param isExcludes the isExcludes to set
+ */
+ public void setIsExcludes(Boolean isExcludes) {
+ this.isExcludes = isExcludes == null ? Boolean.FALSE : isExcludes;
+ }
+
+ /**
+ * @return the isRecursive
+ */
+ public Boolean getIsRecursive() {
+ return isRecursive;
+ }
+
+ /**
+ * @param isRecursive the isRecursive to set
+ */
+ public void setIsRecursive(Boolean isRecursive) {
+ this.isRecursive = isRecursive == null ? Boolean.FALSE : isRecursive;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ toString(sb);
+ return sb.toString();
+ }
+
+ public StringBuilder toString(StringBuilder sb) {
+ sb.append("RangerPolicyResource={");
+ sb.append("values={");
+ if (values != null) {
+ for (String value : values) {
+ sb.append(value).append(" ");
+ }
+ }
+ sb.append("} ");
+ sb.append("isExcludes={").append(isExcludes).append("} ");
+ sb.append("isRecursive={").append(isRecursive).append("} ");
+ sb.append("}");
+
+ return sb;
+ }
+
+ @Override
+ public int hashCode() {
+ final int prime = 31;
+ int result = 1;
+ result = prime * result
+ + ((isExcludes == null) ? 0 : isExcludes.hashCode());
+ result = prime * result
+ + ((isRecursive == null) ? 0 : isRecursive.hashCode());
+ result = prime * result
+ + ((values == null) ? 0 : values.hashCode());
+ return result;
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (this == obj) {
+ return true;
+ }
+ if (obj == null) {
+ return false;
+ }
+ if (getClass() != obj.getClass()) {
+ return false;
+ }
+ RangerPolicyResource other = (RangerPolicyResource) obj;
+ if (isExcludes == null) {
+ if (other.isExcludes != null) {
+ return false;
+ }
+ } else if (!isExcludes.equals(other.isExcludes)) {
+ return false;
+ }
+ if (isRecursive == null) {
+ if (other.isRecursive != null) {
+ return false;
+ }
+ } else if (!isRecursive.equals(other.isRecursive)) {
+ return false;
+ }
+ if (values == null) {
+ if (other.values != null) {
+ return false;
+ }
+ } else if (!values.equals(other.values)) {
+ return false;
+ }
+ return true;
+ }
+ }
+
+ /**
+ * RangerPolicyItem class contains ranger policy items like access and permissions.
+ */
+ @JsonAutoDetect(fieldVisibility = Visibility.ANY)
+ @JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+ @JsonIgnoreProperties(ignoreUnknown = true)
+ @XmlRootElement
+ @XmlAccessorType(XmlAccessType.FIELD)
+ public static class RangerPolicyItem implements java.io.Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private List<RangerPolicyItemAccess> accesses;
+ private List<String> users;
+ private List<String> groups;
+ private List<RangerPolicyItemCondition> conditions;
+ private Boolean delegateAdmin;
+
+ public RangerPolicyItem() {
Review comment:
It will be needed to deserialize it, the default constructor
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pkumarsinha commented on a change in pull request #1004: HIVE-23351 : Ranger Replication Scheduling
Posted by GitBox <gi...@apache.org>.
pkumarsinha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r422688212
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##########
@@ -144,6 +145,11 @@ public int execute() {
if (shouldDump(previousValidHiveDumpPath)) {
Path currentDumpPath = getCurrentDumpPath(dumpRoot, isBootstrap);
Path hiveDumpRoot = new Path(currentDumpPath, ReplUtils.REPL_HIVE_BASE_DIR);
+ work.setCurrentDumpPath(currentDumpPath);
+ if (shouldDumpAuthorizationMetadata()) {
+ LOG.info("Initiate authorization metadata dump provided by {} ", RANGER_AUTHORIZER);
+ initiateAuthorizationDumpTask(currentDumpPath);
Review comment:
No need to pass this, use work.getCurrentDumpPath().
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##########
@@ -144,6 +145,11 @@ public int execute() {
if (shouldDump(previousValidHiveDumpPath)) {
Path currentDumpPath = getCurrentDumpPath(dumpRoot, isBootstrap);
Path hiveDumpRoot = new Path(currentDumpPath, ReplUtils.REPL_HIVE_BASE_DIR);
+ work.setCurrentDumpPath(currentDumpPath);
+ if (shouldDumpAuthorizationMetadata()) {
+ LOG.info("Initiate authorization metadata dump provided by {} ", RANGER_AUTHORIZER);
Review comment:
This log and the log message inside the method "initiateAuthorizationDumpTask" looks repetitive even though they print two info. It would be better if you remove this one and club the "RANGER_AUTHORIZER" info in the the log message inside "initiateAuthorizationDumpTask".
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##########
@@ -97,214 +100,257 @@ public StageType getType() {
@Override
public int execute() {
- Task<?> rootTask = work.getRootTask();
- if (rootTask != null) {
- rootTask.setChildTasks(null);
+ try {
+ Task<?> rootTask = work.getRootTask();
+ if (rootTask != null) {
+ rootTask.setChildTasks(null);
+ }
+ work.setRootTask(this);
+ this.parentTasks = null;
+ if (shouldLoadAuthorizationMetadata()) {
+ LOG.info("Loading authorization data provided by service {} ", RANGER_AUTHORIZER);
+ initiateAuthorizationLoadTask(work.dumpDirectory);
Review comment:
Don't need to pass this work.dumpDirectory, it is available there as well
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##########
@@ -97,214 +100,257 @@ public StageType getType() {
@Override
public int execute() {
- Task<?> rootTask = work.getRootTask();
- if (rootTask != null) {
- rootTask.setChildTasks(null);
+ try {
+ Task<?> rootTask = work.getRootTask();
+ if (rootTask != null) {
+ rootTask.setChildTasks(null);
+ }
+ work.setRootTask(this);
+ this.parentTasks = null;
+ if (shouldLoadAuthorizationMetadata()) {
+ LOG.info("Loading authorization data provided by service {} ", RANGER_AUTHORIZER);
Review comment:
Remove log and club it with the log info for RANGER_AUTHORIZER in log message inside the initiateAuthorizationLoadTask
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##########
@@ -340,30 +387,30 @@ private void dropTablesExcludedInReplScope(ReplScope replScope) throws HiveExcep
// List all the tables that are excluded in the current repl scope.
Iterable<String> tableNames = Collections2.filter(db.getAllTables(dbName),
tableName -> {
- assert(tableName != null);
+ assert (tableName != null);
return !tableName.toLowerCase().startsWith(
- SemanticAnalyzer.VALUES_TMP_TABLE_NAME_PREFIX.toLowerCase())
- && !replScope.tableIncludedInReplScope(tableName);
+ SemanticAnalyzer.VALUES_TMP_TABLE_NAME_PREFIX.toLowerCase())
+ && !replScope.tableIncludedInReplScope(tableName);
});
for (String table : tableNames) {
db.dropTable(dbName + "." + table, true);
}
LOG.info("Tables in the Database: {} that are excluded in the replication scope are dropped.",
- dbName);
+ dbName);
}
private void createReplLoadCompleteAckTask() {
if ((work.isIncrementalLoad() && !work.incrementalLoadTasksBuilder().hasMoreWork() && !work.hasBootstrapLoadTasks())
- || (!work.isIncrementalLoad() && !work.hasBootstrapLoadTasks())) {
+ || (!work.isIncrementalLoad() && !work.hasBootstrapLoadTasks())) {
//All repl load tasks are executed and status is 0, create the task to add the acknowledgement
AckWork replLoadAckWork = new AckWork(
- new Path(work.dumpDirectory, LOAD_ACKNOWLEDGEMENT.toString()));
+ new Path(work.dumpDirectory, LOAD_ACKNOWLEDGEMENT.toString()));
Task<AckWork> loadAckWorkTask = TaskFactory.get(replLoadAckWork, conf);
- if (this.childTasks.isEmpty()) {
+ if (childTasks.isEmpty()) {
this.childTasks.add(loadAckWorkTask);
Review comment:
Remove this from here and from next line as well.
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##########
@@ -392,7 +439,7 @@ private void createEndReplLogTask(Context context, Scope scope,
/**
* There was a database update done before and we want to make sure we update the last repl
* id on this database as we are now going to switch to processing a new database.
- *
+ * <p>
Review comment:
Remove this.
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##########
@@ -340,30 +387,30 @@ private void dropTablesExcludedInReplScope(ReplScope replScope) throws HiveExcep
// List all the tables that are excluded in the current repl scope.
Iterable<String> tableNames = Collections2.filter(db.getAllTables(dbName),
tableName -> {
- assert(tableName != null);
+ assert (tableName != null);
return !tableName.toLowerCase().startsWith(
- SemanticAnalyzer.VALUES_TMP_TABLE_NAME_PREFIX.toLowerCase())
- && !replScope.tableIncludedInReplScope(tableName);
+ SemanticAnalyzer.VALUES_TMP_TABLE_NAME_PREFIX.toLowerCase())
+ && !replScope.tableIncludedInReplScope(tableName);
});
for (String table : tableNames) {
db.dropTable(dbName + "." + table, true);
}
LOG.info("Tables in the Database: {} that are excluded in the replication scope are dropped.",
- dbName);
+ dbName);
}
private void createReplLoadCompleteAckTask() {
if ((work.isIncrementalLoad() && !work.incrementalLoadTasksBuilder().hasMoreWork() && !work.hasBootstrapLoadTasks())
- || (!work.isIncrementalLoad() && !work.hasBootstrapLoadTasks())) {
+ || (!work.isIncrementalLoad() && !work.hasBootstrapLoadTasks())) {
//All repl load tasks are executed and status is 0, create the task to add the acknowledgement
AckWork replLoadAckWork = new AckWork(
- new Path(work.dumpDirectory, LOAD_ACKNOWLEDGEMENT.toString()));
+ new Path(work.dumpDirectory, LOAD_ACKNOWLEDGEMENT.toString()));
Task<AckWork> loadAckWorkTask = TaskFactory.get(replLoadAckWork, conf);
- if (this.childTasks.isEmpty()) {
+ if (childTasks.isEmpty()) {
this.childTasks.add(loadAckWorkTask);
} else {
DAGTraversal.traverse(this.childTasks,
- new AddDependencyToLeaves(Collections.singletonList(loadAckWorkTask)));
+ new AddDependencyToLeaves(Collections.singletonList(loadAckWorkTask)));
Review comment:
You can rather shift this to single line
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##########
@@ -452,78 +499,65 @@ private void createBuilderTask(List<Task<?>> rootTasks) {
DAGTraversal.traverse(rootTasks, new AddDependencyToLeaves(loadTask));
}
- private int executeIncrementalLoad() {
- try {
-
- // If replication policy is changed between previous and current repl load, then drop the tables
- // that are excluded in the new replication policy.
- dropTablesExcludedInReplScope(work.currentReplScope);
-
- IncrementalLoadTasksBuilder builder = work.incrementalLoadTasksBuilder();
-
- // If incremental events are already applied, then check and perform if need to bootstrap any tables.
- if (!builder.hasMoreWork() && work.isLastReplIDUpdated()) {
- if (work.hasBootstrapLoadTasks()) {
- LOG.debug("Current incremental dump have tables to be bootstrapped. Switching to bootstrap "
- + "mode after applying all events.");
- return executeBootStrapLoad();
- }
+ private int executeIncrementalLoad() throws Exception {
+ // If replication policy is changed between previous and current repl load, then drop the tables
+ // that are excluded in the new replication policy.
+ dropTablesExcludedInReplScope(work.currentReplScope);
+ IncrementalLoadTasksBuilder builder = work.incrementalLoadTasksBuilder();
+ // If incremental events are already applied, then check and perform if need to bootstrap any tables.
+ if (!builder.hasMoreWork() && work.isLastReplIDUpdated()) {
+ if (work.hasBootstrapLoadTasks()) {
+ LOG.debug("Current incremental dump have tables to be bootstrapped. Switching to bootstrap "
+ + "mode after applying all events.");
+ return executeBootStrapLoad();
}
-
- List<Task<?>> childTasks = new ArrayList<>();
- int maxTasks = conf.getIntVar(HiveConf.ConfVars.REPL_APPROX_MAX_LOAD_TASKS);
-
- TaskTracker tracker = new TaskTracker(maxTasks);
- childTasks.add(builder.build(context, getHive(), LOG, tracker));
-
- // If there are no more events to be applied, add a task to update the last.repl.id of the
- // target database to the event id of the last event considered by the dump. Next
- // incremental cycle won't consider the events in this dump again if it starts from this id.
- if (!builder.hasMoreWork()) {
- // The name of the database to be loaded into is either specified directly in REPL LOAD
- // command i.e. when dbNameToLoadIn has a valid dbname or is available through dump
- // metadata during table level replication.
- String dbName = work.dbNameToLoadIn;
- if (dbName == null || StringUtils.isBlank(dbName)) {
- if (work.currentReplScope != null) {
- String replScopeDbName = work.currentReplScope.getDbName();
- if (replScopeDbName != null && !"*".equals(replScopeDbName)) {
- dbName = replScopeDbName;
- }
+ }
+ List<Task<?>> childTasks = new ArrayList<>();
+ int maxTasks = conf.getIntVar(HiveConf.ConfVars.REPL_APPROX_MAX_LOAD_TASKS);
+ TaskTracker tracker = new TaskTracker(maxTasks);
+ childTasks.add(builder.build(context, getHive(), LOG, tracker));
+ // If there are no more events to be applied, add a task to update the last.repl.id of the
+ // target database to the event id of the last event considered by the dump. Next
+ // incremental cycle won't consider the events in this dump again if it starts from this id.
+ if (!builder.hasMoreWork()) {
+ // The name of the database to be loaded into is either specified directly in REPL LOAD
+ // command i.e. when dbNameToLoadIn has a valid dbname or is available through dump
+ // metadata during table level replication.
+ String dbName = work.dbNameToLoadIn;
+ if (dbName == null || StringUtils.isBlank(dbName)) {
+ if (work.currentReplScope != null) {
+ String replScopeDbName = work.currentReplScope.getDbName();
+ if (replScopeDbName != null && !"*".equals(replScopeDbName)) {
+ dbName = replScopeDbName;
}
}
-
- // If we are replicating to multiple databases at a time, it's not
- // possible to know which all databases we are replicating into and hence we can not
- // update repl id in all those databases.
- if (StringUtils.isNotBlank(dbName)) {
- String lastEventid = builder.eventTo().toString();
- Map<String, String> mapProp = new HashMap<>();
- mapProp.put(ReplicationSpec.KEY.CURR_STATE_ID.toString(), lastEventid);
-
- AlterDatabaseSetPropertiesDesc alterDbDesc =
- new AlterDatabaseSetPropertiesDesc(dbName, mapProp,
- new ReplicationSpec(lastEventid, lastEventid));
- Task<?> updateReplIdTask =
- TaskFactory.get(new DDLWork(new HashSet<>(), new HashSet<>(), alterDbDesc), conf);
-
- DAGTraversal.traverse(childTasks, new AddDependencyToLeaves(updateReplIdTask));
- work.setLastReplIDUpdated(true);
- LOG.debug("Added task to set last repl id of db " + dbName + " to " + lastEventid);
- }
}
-
- // Once all the incremental events are applied, enable bootstrap of tables if exist.
- if (builder.hasMoreWork() || work.hasBootstrapLoadTasks()) {
- DAGTraversal.traverse(childTasks, new AddDependencyToLeaves(TaskFactory.get(work, conf)));
+ // If we are replicating to multiple databases at a time, it's not
+ // possible to know which all databases we are replicating into and hence we can not
+ // update repl id in all those databases.
+ if (StringUtils.isNotBlank(dbName)) {
+ String lastEventid = builder.eventTo().toString();
+ Map<String, String> mapProp = new HashMap<>();
+ mapProp.put(ReplicationSpec.KEY.CURR_STATE_ID.toString(), lastEventid);
+ AlterDatabaseSetPropertiesDesc alterDbDesc =
+ new AlterDatabaseSetPropertiesDesc(dbName, mapProp,
+ new ReplicationSpec(lastEventid, lastEventid));
+ Task<?> updateReplIdTask =
+ TaskFactory.get(new DDLWork(new HashSet<>(), new HashSet<>(), alterDbDesc), conf);
+ DAGTraversal.traverse(childTasks, new AddDependencyToLeaves(updateReplIdTask));
+ work.setLastReplIDUpdated(true);
+ LOG.debug("Added task to set last repl id of db " + dbName + " to " + lastEventid);
}
- this.childTasks = childTasks;
- createReplLoadCompleteAckTask();
- return 0;
- } catch (Exception e) {
- LOG.error("failed replication", e);
- setException(e);
- return 1;
}
+ // Once all the incremental events are applied, enable bootstrap of tables if exist.
+ if (builder.hasMoreWork() || work.hasBootstrapLoadTasks()) {
+ DAGTraversal.traverse(childTasks, new AddDependencyToLeaves(TaskFactory.get(work, conf)));
+ }
+ if (this.childTasks == null) {
Review comment:
remove this.
##########
File path: ql/src/test/org/apache/hadoop/hive/ql/exec/repl/TestRangerLoadTask.java
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.exec.repl;
+
+import com.google.gson.Gson;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.exec.repl.ranger.RangerExportPolicyList;
+import org.apache.hadoop.hive.ql.exec.repl.ranger.RangerRestClientImpl;
+import org.apache.hadoop.hive.ql.exec.repl.util.ReplUtils;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.Mock;
+import org.mockito.Mockito;
+import org.mockito.junit.MockitoJUnitRunner;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+import static org.apache.hadoop.hive.conf.HiveConf.ConfVars.REPL_AUTHORIZATION_PROVIDER_SERVICE_ENDPOINT;
+
+/**
+ * Unit test class for testing Ranger Dump.
+ */
+@RunWith(MockitoJUnitRunner.class)
+public class TestRangerLoadTask {
+
+ protected static final Logger LOG = LoggerFactory.getLogger(TestRangerLoadTask.class);
+ private RangerLoadTask task;
+
+ @Mock
+ private RangerRestClientImpl mockClient;
+
+ @Mock
+ private HiveConf conf;
+
+ @Mock
+ private RangerLoadWork work;
+
+ @Before
+ public void setup() throws Exception {
+ task = new RangerLoadTask(mockClient, conf, work);
+ Mockito.when(mockClient.changeDataSet(Mockito.anyList(), Mockito.anyString(), Mockito.anyString()))
+ .thenCallRealMethod();
+ Mockito.when(mockClient.checkConnection(Mockito.anyString())).thenReturn(true);
+ }
+
+ @Test
+ public void testFailureInvalidAuthProviderEndpoint() {
+ Mockito.when(conf.getVar(REPL_AUTHORIZATION_PROVIDER_SERVICE_ENDPOINT)).thenReturn(null);
+ int status = task.execute();
+ Assert.assertEquals(40000, status);
+ }
+
+ @Test
+ public void testSuccessValidAuthProviderEndpoint() {
+ Mockito.when(conf.getVar(REPL_AUTHORIZATION_PROVIDER_SERVICE_ENDPOINT)).thenReturn("rangerEndpoint");
+ Mockito.when(work.getSourceDbName()).thenReturn("srcdb");
+ Mockito.when(work.getTargetDbName()).thenReturn("tgtdb");
+ int status = task.execute();
+ Assert.assertEquals(0, status);
+ }
+
+ @Test
+ public void testSuccessNonEmptyRangerPolicies() throws Exception {
+ String rangerResponse = "{\"metaDataInfo\":{\"Host name\":\"org.apache.ranger.com\","
Review comment:
ranger.apache.org?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aasha commented on a change in pull request #1004: HIVE-23351 : Ranger Replication Scheduling
Posted by GitBox <gi...@apache.org>.
aasha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r422161641
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ranger/RangerPolicy.java
##########
@@ -0,0 +1,1513 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl.ranger;
+
+import org.codehaus.jackson.annotate.JsonAutoDetect;
+import org.codehaus.jackson.annotate.JsonAutoDetect.Visibility;
+import org.codehaus.jackson.annotate.JsonIgnoreProperties;
+import org.codehaus.jackson.map.annotate.JsonSerialize;
+
+import javax.xml.bind.annotation.XmlAccessType;
+import javax.xml.bind.annotation.XmlAccessorType;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * RangerPolicy class to contain Ranger Policy details.
+ */
+@JsonAutoDetect(fieldVisibility = Visibility.ANY)
+@JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+@JsonIgnoreProperties(ignoreUnknown = true)
+@XmlRootElement
+@XmlAccessorType(XmlAccessType.FIELD)
+public class RangerPolicy extends RangerBaseModelObject implements java.io.Serializable {
+ public static final int POLICY_TYPE_ACCESS = 0;
+ public static final int POLICY_TYPE_DATAMASK = 1;
+ public static final int POLICY_TYPE_ROWFILTER = 2;
+
+ public static final int[] POLICY_TYPES = new int[]{
+ POLICY_TYPE_ACCESS,
+ POLICY_TYPE_DATAMASK,
+ POLICY_TYPE_ROWFILTER,
+ };
+
+ public static final String MASK_TYPE_NULL = "MASK_NULL";
+ public static final String MASK_TYPE_NONE = "MASK_NONE";
+ public static final String MASK_TYPE_CUSTOM = "CUSTOM";
+
+ private static final long serialVersionUID = 1L;
+
+ private String service;
+ private String name;
+ private Integer policyType;
+ private String description;
+ private String resourceSignature;
+ private Boolean isAuditEnabled;
+ private Map<String, RangerPolicyResource> resources;
+ private List<RangerPolicyItem> policyItems;
+ private List<RangerPolicyItem> denyPolicyItems;
+ private List<RangerPolicyItem> allowExceptions;
+ private List<RangerPolicyItem> denyExceptions;
+ private List<RangerDataMaskPolicyItem> dataMaskPolicyItems;
+ private List<RangerRowFilterPolicyItem> rowFilterPolicyItems;
+
+
+ /**
+ * Ranger Policy default constructor.
+ */
+ public RangerPolicy() {
+ this(null, null, null, null, null, null, null);
+ }
+
+ /**
+ * @param service
+ * @param name
+ * @param policyType
+ * @param description
+ * @param resources
+ * @param policyItems
+ * @param resourceSignature TODO
+ */
+ public RangerPolicy(String service, String name, Integer policyType, String description,
+ Map<String, RangerPolicyResource> resources, List<RangerPolicyItem> policyItems,
+ String resourceSignature) {
+ super();
+ setService(service);
+ setName(name);
+ setPolicyType(policyType);
+ setDescription(description);
+ setResourceSignature(resourceSignature);
+ setIsAuditEnabled(null);
+ setResources(resources);
+ setPolicyItems(policyItems);
+ setDenyPolicyItems(null);
+ setAllowExceptions(null);
+ setDenyExceptions(null);
+ setDataMaskPolicyItems(null);
+ setRowFilterPolicyItems(null);
+ }
+
+ /**
+ * @param other
+ */
+ public void updateFrom(RangerPolicy other) {
+ super.updateFrom(other);
+
+ setService(other.getService());
+ setName(other.getName());
+ setPolicyType(other.getPolicyType());
+ setDescription(other.getDescription());
+ setResourceSignature(other.getResourceSignature());
+ setIsAuditEnabled(other.getIsAuditEnabled());
+ setResources(other.getResources());
+ setPolicyItems(other.getPolicyItems());
+ setDenyPolicyItems(other.getDenyPolicyItems());
+ setAllowExceptions(other.getAllowExceptions());
+ setDenyExceptions(other.getDenyExceptions());
+ setDataMaskPolicyItems(other.getDataMaskPolicyItems());
+ setRowFilterPolicyItems(other.getRowFilterPolicyItems());
+ }
+
+ /**
+ * @return the type
+ */
+ public String getService() {
+ return service;
+ }
+
+ /**
+ * @param service the type to set
+ */
+ public void setService(String service) {
+ this.service = service;
+ }
+
+ /**
+ * @return the name
+ */
+ public String getName() {
+ return name;
+ }
+
+ /**
+ * @param name the name to set
+ */
+ public void setName(String name) {
+ this.name = name;
+ }
+
+ /**
+ * @return the policyType
+ */
+ public Integer getPolicyType() {
+ return policyType;
+ }
+
+ /**
+ * @param policyType the policyType to set
+ */
+ public void setPolicyType(Integer policyType) {
+ this.policyType = policyType;
+ }
+
+ /**
+ * @return the description
+ */
+ public String getDescription() {
+ return description;
+ }
+
+ /**
+ * @param description the description to set
+ */
+ public void setDescription(String description) {
+ this.description = description;
+ }
+
+ /**
+ * @return the resourceSignature
+ */
+ public String getResourceSignature() {
+ return resourceSignature;
+ }
+
+ /**
+ * @param resourceSignature the resourceSignature to set
+ */
+ public void setResourceSignature(String resourceSignature) {
+ this.resourceSignature = resourceSignature;
+ }
+
+ /**
+ * @return the isAuditEnabled
+ */
+ public Boolean getIsAuditEnabled() {
+ return isAuditEnabled;
+ }
+
+ /**
+ * @param isAuditEnabled the isEnabled to set
+ */
+ public void setIsAuditEnabled(Boolean isAuditEnabled) {
+ this.isAuditEnabled = isAuditEnabled == null ? Boolean.TRUE : isAuditEnabled;
+ }
+
+ /**
+ * @return the resources
+ */
+ public Map<String, RangerPolicyResource> getResources() {
+ return resources;
+ }
+
+ /**
+ * @param resources the resources to set
+ */
+ public void setResources(Map<String, RangerPolicyResource> resources) {
+ if (this.resources == null) {
+ this.resources = new HashMap<>();
+ }
+
+ if (this.resources == resources) {
+ return;
+ }
+
+ this.resources.clear();
+
+ if (resources != null) {
+ for (Map.Entry<String, RangerPolicyResource> e : resources.entrySet()) {
+ this.resources.put(e.getKey(), e.getValue());
+ }
+ }
+ }
+
+ /**
+ * @return the policyItems
+ */
+ public List<RangerPolicyItem> getPolicyItems() {
+ return policyItems;
+ }
+
+ /**
+ * @param policyItems the policyItems to set
+ */
+ public void setPolicyItems(List<RangerPolicyItem> policyItems) {
+ if (this.policyItems == null) {
+ this.policyItems = new ArrayList<>();
+ }
+
+ if (this.policyItems == policyItems) {
+ return;
+ }
+
+ this.policyItems.clear();
+
+ if (policyItems != null) {
+ this.policyItems.addAll(policyItems);
+ }
+ }
+
+ /**
+ * @return the denyPolicyItems
+ */
+ public List<RangerPolicyItem> getDenyPolicyItems() {
+ return denyPolicyItems;
+ }
+
+ /**
+ * @param denyPolicyItems the denyPolicyItems to set
+ */
+ public void setDenyPolicyItems(List<RangerPolicyItem> denyPolicyItems) {
+ if (this.denyPolicyItems == null) {
+ this.denyPolicyItems = new ArrayList<>();
+ }
+
+ if (this.denyPolicyItems == denyPolicyItems) {
+ return;
+ }
+
+ this.denyPolicyItems.clear();
+
+ if (denyPolicyItems != null) {
+ this.denyPolicyItems.addAll(denyPolicyItems);
+ }
+ }
+
+ /**
+ * @return the allowExceptions
+ */
+ public List<RangerPolicyItem> getAllowExceptions() {
+ return allowExceptions;
+ }
+
+ /**
+ * @param allowExceptions the allowExceptions to set
+ */
+ public void setAllowExceptions(List<RangerPolicyItem> allowExceptions) {
+ if (this.allowExceptions == null) {
+ this.allowExceptions = new ArrayList<>();
+ }
+
+ if (this.allowExceptions == allowExceptions) {
+ return;
+ }
+
+ this.allowExceptions.clear();
+
+ if (allowExceptions != null) {
+ this.allowExceptions.addAll(allowExceptions);
+ }
+ }
+
+ /**
+ * @return the denyExceptions
+ */
+ public List<RangerPolicyItem> getDenyExceptions() {
+ return denyExceptions;
+ }
+
+ /**
+ * @param denyExceptions the denyExceptions to set
+ */
+ public void setDenyExceptions(List<RangerPolicyItem> denyExceptions) {
+ if (this.denyExceptions == null) {
+ this.denyExceptions = new ArrayList<>();
+ }
+
+ if (this.denyExceptions == denyExceptions) {
+ return;
+ }
+
+ this.denyExceptions.clear();
+
+ if (denyExceptions != null) {
+ this.denyExceptions.addAll(denyExceptions);
+ }
+ }
+
+ public List<RangerDataMaskPolicyItem> getDataMaskPolicyItems() {
+ return dataMaskPolicyItems;
+ }
+
+ public void setDataMaskPolicyItems(List<RangerDataMaskPolicyItem> dataMaskPolicyItems) {
+ if (this.dataMaskPolicyItems == null) {
+ this.dataMaskPolicyItems = new ArrayList<>();
+ }
+
+ if (this.dataMaskPolicyItems == dataMaskPolicyItems) {
+ return;
+ }
+
+ this.dataMaskPolicyItems.clear();
+
+ if (dataMaskPolicyItems != null) {
+ this.dataMaskPolicyItems.addAll(dataMaskPolicyItems);
+ }
+ }
+
+ public List<RangerRowFilterPolicyItem> getRowFilterPolicyItems() {
+ return rowFilterPolicyItems;
+ }
+
+ public void setRowFilterPolicyItems(List<RangerRowFilterPolicyItem> rowFilterPolicyItems) {
+ if (this.rowFilterPolicyItems == null) {
+ this.rowFilterPolicyItems = new ArrayList<>();
+ }
+
+ if (this.rowFilterPolicyItems == rowFilterPolicyItems) {
+ return;
+ }
+
+ this.rowFilterPolicyItems.clear();
+
+ if (rowFilterPolicyItems != null) {
+ this.rowFilterPolicyItems.addAll(rowFilterPolicyItems);
+ }
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ toString(sb);
+ return sb.toString();
+ }
+
+ public StringBuilder toString(StringBuilder sb) {
+ sb.append("RangerPolicy={");
+
+ super.toString(sb);
+
+ sb.append("service={").append(service).append("} ");
+ sb.append("name={").append(name).append("} ");
+ sb.append("policyType={").append(policyType).append("} ");
+ sb.append("description={").append(description).append("} ");
+ sb.append("resourceSignature={").append(resourceSignature).append("} ");
+ sb.append("isAuditEnabled={").append(isAuditEnabled).append("} ");
+
+ sb.append("resources={");
+ if (resources != null) {
+ for (Map.Entry<String, RangerPolicyResource> e : resources.entrySet()) {
+ sb.append(e.getKey()).append("={");
+ e.getValue().toString(sb);
+ sb.append("} ");
+ }
+ }
+ sb.append("} ");
+
+ sb.append("policyItems={");
+ if (policyItems != null) {
+ for (RangerPolicyItem policyItem : policyItems) {
+ if (policyItem != null) {
+ policyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("denyPolicyItems={");
+ if (denyPolicyItems != null) {
+ for (RangerPolicyItem policyItem : denyPolicyItems) {
+ if (policyItem != null) {
+ policyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("allowExceptions={");
+ if (allowExceptions != null) {
+ for (RangerPolicyItem policyItem : allowExceptions) {
+ if (policyItem != null) {
+ policyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("denyExceptions={");
+ if (denyExceptions != null) {
+ for (RangerPolicyItem policyItem : denyExceptions) {
+ if (policyItem != null) {
+ policyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("dataMaskPolicyItems={");
+ if (dataMaskPolicyItems != null) {
+ for (RangerDataMaskPolicyItem dataMaskPolicyItem : dataMaskPolicyItems) {
+ if (dataMaskPolicyItem != null) {
+ dataMaskPolicyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("rowFilterPolicyItems={");
+ if (rowFilterPolicyItems != null) {
+ for (RangerRowFilterPolicyItem rowFilterPolicyItem : rowFilterPolicyItems) {
+ if (rowFilterPolicyItem != null) {
+ rowFilterPolicyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("}");
+
+ return sb;
+ }
+
+ /**
+ * RangerPolicyResource class to store the resource path values.
+ */
+ @JsonAutoDetect(fieldVisibility = Visibility.ANY)
+ @JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+ @JsonIgnoreProperties(ignoreUnknown = true)
+ @XmlRootElement
+ @XmlAccessorType(XmlAccessType.FIELD)
+ public static class RangerPolicyResource implements java.io.Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private List<String> values;
+ private Boolean isExcludes;
+ private Boolean isRecursive;
+
+ public RangerPolicyResource() {
+ this((List<String>) null, null, null);
+ }
+
+ public RangerPolicyResource(String value) {
+ setValue(value);
+ setIsExcludes(null);
+ setIsRecursive(null);
+ }
+
+ public RangerPolicyResource(String value, Boolean isExcludes, Boolean isRecursive) {
+ setValue(value);
+ setIsExcludes(isExcludes);
+ setIsRecursive(isRecursive);
+ }
+
+ public RangerPolicyResource(List<String> values, Boolean isExcludes, Boolean isRecursive) {
+ setValues(values);
+ setIsExcludes(isExcludes);
+ setIsRecursive(isRecursive);
+ }
+
+ /**
+ * @return the values
+ */
+ public List<String> getValues() {
+ return values;
+ }
+
+ /**
+ * @param values the values to set
+ */
+ public void setValues(List<String> values) {
+ if (this.values == null) {
+ this.values = new ArrayList<>();
+ }
+ if (this.values == values) {
+ return;
+ }
+ this.values.clear();
+ if (values != null) {
+ this.values.addAll(values);
+ }
+ }
+
+ /**
+ * @param value the value to set
+ */
+ public void setValue(String value) {
+ if (this.values == null) {
+ this.values = new ArrayList<>();
+ }
+ this.values.clear();
+ this.values.add(value);
+ }
+
+ /**
+ * @return the isExcludes
+ */
+ public Boolean getIsExcludes() {
+ return isExcludes;
+ }
+
+ /**
+ * @param isExcludes the isExcludes to set
+ */
+ public void setIsExcludes(Boolean isExcludes) {
+ this.isExcludes = isExcludes == null ? Boolean.FALSE : isExcludes;
+ }
+
+ /**
+ * @return the isRecursive
+ */
+ public Boolean getIsRecursive() {
+ return isRecursive;
+ }
+
+ /**
+ * @param isRecursive the isRecursive to set
+ */
+ public void setIsRecursive(Boolean isRecursive) {
+ this.isRecursive = isRecursive == null ? Boolean.FALSE : isRecursive;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ toString(sb);
+ return sb.toString();
+ }
+
+ public StringBuilder toString(StringBuilder sb) {
+ sb.append("RangerPolicyResource={");
+ sb.append("values={");
+ if (values != null) {
+ for (String value : values) {
+ sb.append(value).append(" ");
+ }
+ }
+ sb.append("} ");
+ sb.append("isExcludes={").append(isExcludes).append("} ");
+ sb.append("isRecursive={").append(isRecursive).append("} ");
+ sb.append("}");
+
+ return sb;
+ }
+
+ @Override
+ public int hashCode() {
+ final int prime = 31;
+ int result = 1;
+ result = prime * result
+ + ((isExcludes == null) ? 0 : isExcludes.hashCode());
+ result = prime * result
+ + ((isRecursive == null) ? 0 : isRecursive.hashCode());
+ result = prime * result
+ + ((values == null) ? 0 : values.hashCode());
+ return result;
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (this == obj) {
+ return true;
+ }
+ if (obj == null) {
+ return false;
+ }
+ if (getClass() != obj.getClass()) {
+ return false;
+ }
+ RangerPolicyResource other = (RangerPolicyResource) obj;
+ if (isExcludes == null) {
+ if (other.isExcludes != null) {
+ return false;
+ }
+ } else if (!isExcludes.equals(other.isExcludes)) {
+ return false;
+ }
+ if (isRecursive == null) {
+ if (other.isRecursive != null) {
+ return false;
+ }
+ } else if (!isRecursive.equals(other.isRecursive)) {
+ return false;
+ }
+ if (values == null) {
+ if (other.values != null) {
+ return false;
+ }
+ } else if (!values.equals(other.values)) {
+ return false;
+ }
+ return true;
+ }
+ }
+
+ /**
+ * RangerPolicyItem class contains ranger policy items like access and permissions.
+ */
+ @JsonAutoDetect(fieldVisibility = Visibility.ANY)
+ @JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+ @JsonIgnoreProperties(ignoreUnknown = true)
+ @XmlRootElement
+ @XmlAccessorType(XmlAccessType.FIELD)
+ public static class RangerPolicyItem implements java.io.Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private List<RangerPolicyItemAccess> accesses;
+ private List<String> users;
+ private List<String> groups;
+ private List<RangerPolicyItemCondition> conditions;
+ private Boolean delegateAdmin;
+
+ public RangerPolicyItem() {
+ this(null, null, null, null, null);
+ }
+
+ public RangerPolicyItem(List<RangerPolicyItemAccess> accessTypes, List<String> users, List<String> groups,
+ List<RangerPolicyItemCondition> conditions, Boolean delegateAdmin) {
+ setAccesses(accessTypes);
+ setUsers(users);
+ setGroups(groups);
+ setConditions(conditions);
+ setDelegateAdmin(delegateAdmin);
+ }
+
+ /**
+ * @return the accesses
+ */
+ public List<RangerPolicyItemAccess> getAccesses() {
+ return accesses;
+ }
+
+ /**
+ * @param accesses the accesses to set
+ */
+ public void setAccesses(List<RangerPolicyItemAccess> accesses) {
+ if (this.accesses == null) {
+ this.accesses = new ArrayList<>();
+ }
+
+ if (this.accesses == accesses) {
+ return;
+ }
+
+ this.accesses.clear();
+
+ if (accesses != null) {
+ this.accesses.addAll(accesses);
+ }
+ }
+
+ /**
+ * @return the users
+ */
+ public List<String> getUsers() {
+ return users;
+ }
+
+ /**
+ * @param users the users to set
+ */
+ public void setUsers(List<String> users) {
+ if (this.users == null) {
+ this.users = new ArrayList<>();
+ }
+
+ if (this.users == users) {
+ return;
+ }
+
+ this.users.clear();
+
+ if (users != null) {
+ this.users.addAll(users);
+ }
+ }
+
+ /**
+ * @return the groups
+ */
+ public List<String> getGroups() {
+ return groups;
+ }
+
+ /**
+ * @param groups the groups to set
+ */
+ public void setGroups(List<String> groups) {
+ if (this.groups == null) {
+ this.groups = new ArrayList<>();
+ }
+ if (this.groups == groups) {
+ return;
+ }
+ this.groups.clear();
+ if (groups != null) {
+ this.groups.addAll(groups);
+ }
+ }
+
+ /**
+ * @return the conditions
+ */
+ public List<RangerPolicyItemCondition> getConditions() {
+ return conditions;
+ }
+
+ /**
+ * @param conditions the conditions to set
+ */
+ public void setConditions(List<RangerPolicyItemCondition> conditions) {
+ if (this.conditions == null) {
+ this.conditions = new ArrayList<>();
+ }
+ if (this.conditions == conditions) {
+ return;
+ }
+ this.conditions.clear();
+ if (conditions != null) {
+ this.conditions.addAll(conditions);
+ }
+ }
+
+ /**
+ * @return the delegateAdmin
+ */
+ public Boolean getDelegateAdmin() {
+ return delegateAdmin;
+ }
+
+ /**
+ * @param delegateAdmin the delegateAdmin to set
+ */
+ public void setDelegateAdmin(Boolean delegateAdmin) {
+ this.delegateAdmin = delegateAdmin == null ? Boolean.FALSE : delegateAdmin;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ toString(sb);
+ return sb.toString();
+ }
+
+ public StringBuilder toString(StringBuilder sb) {
+ sb.append("RangerPolicyItem={");
+ sb.append("accessTypes={");
+ if (accesses != null) {
+ for (RangerPolicyItemAccess access : accesses) {
+ if (access != null) {
+ access.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+ sb.append("users={");
+ if (users != null) {
+ for (String user : users) {
+ if (user != null) {
+ sb.append(user).append(" ");
+ }
+ }
+ }
+ sb.append("} ");
+ sb.append("groups={");
+ if (groups != null) {
+ for (String group : groups) {
+ if (group != null) {
+ sb.append(group).append(" ");
+ }
+ }
+ }
+ sb.append("} ");
+ sb.append("conditions={");
+ if (conditions != null) {
+ for (RangerPolicyItemCondition condition : conditions) {
+ if (condition != null) {
+ condition.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+ sb.append("delegateAdmin={").append(delegateAdmin).append("} ");
+ sb.append("}");
+
+ return sb;
+ }
+
+ @Override
+ public int hashCode() {
+ final int prime = 31;
+ int result = 1;
+ result = prime * result
+ + ((accesses == null) ? 0 : accesses.hashCode());
+ result = prime * result
+ + ((conditions == null) ? 0 : conditions.hashCode());
+ result = prime * result
+ + ((delegateAdmin == null) ? 0 : delegateAdmin.hashCode());
+ result = prime * result
+ + ((groups == null) ? 0 : groups.hashCode());
+ result = prime * result + ((users == null) ? 0 : users.hashCode());
+ return result;
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (this == obj) {
+ return true;
+ }
+ if (obj == null) {
+ return false;
+ }
+ if (getClass() != obj.getClass()) {
+ return false;
+ }
+ RangerPolicyItem other = (RangerPolicyItem) obj;
+ if (accesses == null) {
+ if (other.accesses != null) {
+ return false;
+ }
+ } else if (!accesses.equals(other.accesses)) {
+ return false;
+ }
+ if (conditions == null) {
+ if (other.conditions != null) {
+ return false;
+ }
+ } else if (!conditions.equals(other.conditions)) {
+ return false;
+ }
+ if (delegateAdmin == null) {
+ if (other.delegateAdmin != null) {
+ return false;
+ }
+ } else if (!delegateAdmin.equals(other.delegateAdmin)) {
+ return false;
+ }
+ if (groups == null) {
+ if (other.groups != null) {
+ return false;
+ }
+ } else if (!groups.equals(other.groups)) {
+ return false;
+ }
+ if (users == null) {
+ if (other.users != null) {
+ return false;
+ }
+ } else if (!users.equals(other.users)) {
+ return false;
+ }
+ return true;
+
+ }
+ }
+
+ /**
+ * RangerDataMaskPolicyItem class.
+ */
+ @JsonAutoDetect(fieldVisibility = Visibility.ANY)
+ @JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+ @JsonIgnoreProperties(ignoreUnknown = true)
+ @XmlRootElement
+ @XmlAccessorType(XmlAccessType.FIELD)
+ public static class RangerDataMaskPolicyItem extends RangerPolicyItem implements java.io.Serializable {
+ private static final long serialVersionUID = 1L;
+ private RangerPolicyItemDataMaskInfo dataMaskInfo;
+
+ public RangerDataMaskPolicyItem() {
+ this(null, null, null, null, null, null);
+ }
+
+ public RangerDataMaskPolicyItem(List<RangerPolicyItemAccess> accesses,
+ RangerPolicyItemDataMaskInfo dataMaskDetail, List<String> users,
+ List<String> groups,
+ List<RangerPolicyItemCondition> conditions, Boolean delegateAdmin) {
+ super(accesses, users, groups, conditions, delegateAdmin);
+ setDataMaskInfo(dataMaskDetail);
+ }
+
+ /**
+ * @return the dataMaskInfo
+ */
+ public RangerPolicyItemDataMaskInfo getDataMaskInfo() {
+ return dataMaskInfo;
+ }
+
+ /**
+ * @param dataMaskInfo the dataMaskInfo to set
+ */
+ public void setDataMaskInfo(RangerPolicyItemDataMaskInfo dataMaskInfo) {
+ this.dataMaskInfo = dataMaskInfo == null ? new RangerPolicyItemDataMaskInfo() : dataMaskInfo;
+ }
+
+ @Override
+ public int hashCode() {
+ final int prime = 31;
+ int result = super.hashCode();
+ result = prime * result + ((dataMaskInfo == null) ? 0 : dataMaskInfo.hashCode());
+ return result;
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (!super.equals(obj)) {
+ return false;
+ }
+ if (this == obj) {
+ return true;
+ }
+ if (obj == null) {
+ return false;
+ }
+ if (getClass() != obj.getClass()) {
+ return false;
+ }
+ RangerDataMaskPolicyItem other = (RangerDataMaskPolicyItem) obj;
+ if (dataMaskInfo == null) {
+ if (other.dataMaskInfo != null) {
+ return false;
+ }
+ } else if (!dataMaskInfo.equals(other.dataMaskInfo)) {
+ return false;
+ }
+ return true;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ toString(sb);
+ return sb.toString();
+ }
+
+ public StringBuilder toString(StringBuilder sb) {
+ sb.append("RangerDataMaskPolicyItem={");
+ super.toString(sb);
+ sb.append("dataMaskInfo={");
+ if (dataMaskInfo != null) {
+ dataMaskInfo.toString(sb);
+ }
+ sb.append("} ");
+ sb.append("}");
+ return sb;
+ }
+ }
+
+ /**
+ * RangerRowFilterPolicyItem class.
+ */
+ @JsonAutoDetect(fieldVisibility = Visibility.ANY)
+ @JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+ @JsonIgnoreProperties(ignoreUnknown = true)
+ @XmlRootElement
+ @XmlAccessorType(XmlAccessType.FIELD)
+ public static class RangerRowFilterPolicyItem extends RangerPolicyItem implements java.io.Serializable {
+ private static final long serialVersionUID = 1L;
+ private RangerPolicyItemRowFilterInfo rowFilterInfo;
+
+ public RangerRowFilterPolicyItem() {
Review comment:
Needed for deserialize, the default constructor
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aasha commented on a change in pull request #1004: HIVE-23351 : Ranger Replication Scheduling
Posted by GitBox <gi...@apache.org>.
aasha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r422283834
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ranger/RangerRestClientImpl.java
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl.ranger;
+
+import com.google.gson.Gson;
+import com.google.gson.GsonBuilder;
+import com.sun.jersey.api.client.Client;
+import com.sun.jersey.api.client.ClientResponse;
+import com.sun.jersey.api.client.WebResource;
+import com.sun.jersey.api.client.config.ClientConfig;
+import com.sun.jersey.api.client.config.DefaultClientConfig;
+import com.sun.jersey.multipart.FormDataMultiPart;
+import com.sun.jersey.multipart.MultiPart;
+import com.sun.jersey.multipart.file.StreamDataBodyPart;
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.eclipse.jetty.util.MultiPartWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.servlet.http.HttpServletResponse;
+import javax.ws.rs.core.MediaType;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.File;
+import java.io.Reader;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.ArrayList;
+
+/**
+ * RangerRestClientImpl to connect to Ranger and export policies.
+ */
+public class RangerRestClientImpl implements RangerRestClient {
+ private static final Logger LOG = LoggerFactory.getLogger(RangerRestClientImpl.class);
+ private static final String RANGER_REST_URL_EXPORTJSONFILE = "/service/plugins/policies/exportJson";
+ private static final String RANGER_REST_URL_IMPORTJSONFILE =
+ "/service/plugins/policies/importPoliciesFromFile?updateIfExists=true";
+
+ public RangerExportPolicyList exportRangerPolicies(String sourceRangerEndpoint,
+ String dbName, String rangerHiveServiceName) throws Exception {
+ LOG.info("Ranger endpoint for cluster " + sourceRangerEndpoint);
+ ClientResponse clientResp = null;
+ String uri = null;
+ if (!StringUtils.isEmpty(rangerHiveServiceName)) {
+ uri = RANGER_REST_URL_EXPORTJSONFILE + "?serviceName=" + rangerHiveServiceName + "&polResource="
+ + dbName + "&resource:database=" + dbName
+ + "&serviceType=hive&resourceMatchScope=self_or_ancestor&resourceMatch=full";
+ }
+ if (sourceRangerEndpoint.endsWith("/")) {
+ sourceRangerEndpoint = StringUtils.removePattern(sourceRangerEndpoint, "/+$");
+ }
+ String url = sourceRangerEndpoint + (uri != null ? (uri.startsWith("/") ? uri : ("/" + uri)) : "");
+ LOG.debug("URL to export policies from source Ranger: {}", url);
+ RangerExportPolicyList rangerExportPolicyList = new RangerExportPolicyList();
+ WebResource.Builder builder = getRangerResourceBuilder(url);
+ clientResp = builder.get(ClientResponse.class);
+
+ String response = null;
+ if (clientResp != null) {
+ if (clientResp.getStatus() == HttpServletResponse.SC_OK) {
+ Gson gson = new GsonBuilder().create();
+ response = clientResp.getEntity(String.class);
+ LOG.debug("Response received for ranger export {} ", response);
+ if (StringUtils.isNotEmpty(response)) {
+ rangerExportPolicyList = gson.fromJson(response, RangerExportPolicyList.class);
+ return rangerExportPolicyList;
+ }
+ } else if (clientResp.getStatus() == HttpServletResponse.SC_NO_CONTENT) {
+ LOG.debug("Ranger policy export request returned empty list");
+ return rangerExportPolicyList;
+ } else if (clientResp.getStatus() == HttpServletResponse.SC_UNAUTHORIZED) {
Review comment:
Using SemanticException
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pkumarsinha commented on a change in pull request #1004: HIVE-23351 : Ranger Replication Scheduling
Posted by GitBox <gi...@apache.org>.
pkumarsinha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r421996357
##########
File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
##########
@@ -526,6 +526,22 @@ private static void populateLlapDaemonVarsSet(Set<String> llapDaemonVarsSetLocal
"This is the base directory on the target/replica warehouse under which data for "
+ "external tables is stored. This is relative base path and hence prefixed to the source "
+ "external table path on target cluster."),
+ REPL_INCLUDE_AUTHORIZATION_METADATA("hive.repl.include.authorization.metadata", false,
+ "This configuration will enable security and authorization related metadata along "
+ + "with the hive data and metadata replication. "),
+ REPL_AUTHORIZATION_PROVIDER_SERVICE("hive.repl.authorization.provider.service", "ranger",
+ "This configuration will define which service will provide the security and authorization "
+ + "related metadata that needs to be replicated along "
+ + "with the hive data and metadata replication. Set the configuration "
+ + "hive.repl.include.authorization.metadata to false to disable "
+ + "security policies being replicated "),
+ REPL_AUTHORIZATION_PROVIDER_SERVICE_ENDPOINT("hive.repl.authorization.provider.service.endpoint",
+ "",
+ "This configuration will define the authorization service endpoint"),
+ REPL_RANGER_SERVICE_NAME("hive.repl.ranger.service.name",
+ "cm_hive",
Review comment:
How is the default derived here?
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##########
@@ -144,6 +145,11 @@ public int execute() {
if (shouldDump(previousValidHiveDumpPath)) {
Path currentDumpPath = getCurrentDumpPath(dumpRoot, isBootstrap);
Path hiveDumpRoot = new Path(currentDumpPath, ReplUtils.REPL_HIVE_BASE_DIR);
+ work.setCurrentDumpPath(currentDumpPath);
+ if (shouldDumpAuthorizationMetadata()) {
+ LOG.info("Dumping authorization data");
Review comment:
Log message may be misleading as at this point we are not dumping the data.
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##########
@@ -170,6 +175,26 @@ public int execute() {
return 0;
}
+ private void initiateAuthorizationDumpTask(Path currentDumpPath) throws SemanticException {
+ if (RANGER_AUTHORIZER.equalsIgnoreCase(conf.getVar(HiveConf.ConfVars.REPL_AUTHORIZATION_PROVIDER_SERVICE))) {
+ Path rangerDumpRoot = new Path(currentDumpPath, ReplUtils.REPL_RANGER_BASE_DIR);
+ LOG.info("Exporting Authorization Metadata at {} ", rangerDumpRoot);
+ RangerDumpWork rangerDumpWork = new RangerDumpWork(rangerDumpRoot, work.dbNameOrPattern);
+ Task<RangerDumpWork> rangerDumpTask = TaskFactory.get(rangerDumpWork, conf);
+ if (childTasks == null) {
Review comment:
Shouldn't the child task must be null when we are trying to add ranger authorization task? And we should throw error if it is not?
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##########
@@ -170,6 +175,26 @@ public int execute() {
return 0;
}
+ private void initiateAuthorizationDumpTask(Path currentDumpPath) throws SemanticException {
+ if (RANGER_AUTHORIZER.equalsIgnoreCase(conf.getVar(HiveConf.ConfVars.REPL_AUTHORIZATION_PROVIDER_SERVICE))) {
+ Path rangerDumpRoot = new Path(currentDumpPath, ReplUtils.REPL_RANGER_BASE_DIR);
+ LOG.info("Exporting Authorization Metadata at {} ", rangerDumpRoot);
+ RangerDumpWork rangerDumpWork = new RangerDumpWork(rangerDumpRoot, work.dbNameOrPattern);
+ Task<RangerDumpWork> rangerDumpTask = TaskFactory.get(rangerDumpWork, conf);
+ if (childTasks == null) {
+ childTasks = new ArrayList<>();
+ }
+ childTasks.add(rangerDumpTask);
+ } else {
+ throw new SemanticException("Authorizer " + conf.getVar(HiveConf.ConfVars.REPL_AUTHORIZATION_PROVIDER_SERVICE)
Review comment:
Add a test for this.
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##########
@@ -198,7 +223,12 @@ private void initiateDataCopyTasks() throws SemanticException {
finishRemainingTasks();
} else {
DAGTraversal.traverse(childTasks, new AddDependencyToLeaves(TaskFactory.get(work, conf)));
- this.childTasks = childTasks;
+ if (this.childTasks.isEmpty()) {
Review comment:
Isn't this.childTask null by defualt? If so, then when ranger repl is not configured, this might lead to NPE?
##########
File path: ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/StageType.java
##########
@@ -26,7 +26,9 @@
REPL_TXN(15),
REPL_INCREMENTAL_LOAD(16),
SCHEDULED_QUERY_MAINT(17),
- ACK(18);
+ ACK(18),
+ RANGER_DUMP(19),
+ RANGER_LOAD(20);
Review comment:
Don't we require to fix findByValue() as well?
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##########
@@ -103,13 +105,43 @@ public int execute() {
}
work.setRootTask(this);
this.parentTasks = null;
+ if (shouldLoadAuthorizationMetadata()) {
+ LOG.info("Loading authorization data");
+ try {
Review comment:
Either move the exception handling during executeIncrementalLoad() and executeBootStrapLoad() here and have one try/catch block for all three of the method execution or handle the initiateAuthorizationLoadTask() related exception inside that. It might look more aligned. I would prefer first one if possible.
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##########
@@ -198,7 +223,12 @@ private void initiateDataCopyTasks() throws SemanticException {
finishRemainingTasks();
} else {
DAGTraversal.traverse(childTasks, new AddDependencyToLeaves(TaskFactory.get(work, conf)));
- this.childTasks = childTasks;
Review comment:
finishRemainingTasks() also creates ack file. I see a possibility that ack file will be created before ranger dump and in case ranger dump fails, the dump will be in inconsistent state but will still be picked by load as the ack exists. the side effect will further escalate.
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##########
@@ -103,13 +105,43 @@ public int execute() {
}
work.setRootTask(this);
this.parentTasks = null;
+ if (shouldLoadAuthorizationMetadata()) {
+ LOG.info("Loading authorization data");
Review comment:
We may change the log to something like "Load authorization data is enabled" or similar
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##########
@@ -103,13 +105,43 @@ public int execute() {
}
work.setRootTask(this);
this.parentTasks = null;
+ if (shouldLoadAuthorizationMetadata()) {
+ LOG.info("Loading authorization data");
+ try {
+ initiateAuthorizationLoadTask(work.dumpDirectory);
+ } catch (Exception e) {
+ LOG.error("failed", e);
+ setException(e);
+ return ErrorMsg.getErrorMsg(e.getMessage()).getErrorCode();
+ }
+ }
if (work.isIncrementalLoad()) {
return executeIncrementalLoad();
} else {
return executeBootStrapLoad();
}
}
+ private boolean shouldLoadAuthorizationMetadata() {
+ return conf.getBoolVar(HiveConf.ConfVars.REPL_INCLUDE_AUTHORIZATION_METADATA);
+ }
+
+ private void initiateAuthorizationLoadTask(String hiveDumpDirectory) throws SemanticException {
+ if (RANGER_AUTHORIZER.equalsIgnoreCase(conf.getVar(HiveConf.ConfVars.REPL_AUTHORIZATION_PROVIDER_SERVICE))) {
+ Path rangerLoadRoot = new Path(new Path(hiveDumpDirectory).getParent(), ReplUtils.REPL_RANGER_BASE_DIR);
+ LOG.info("Importing Authorization Metadata from {} ", rangerLoadRoot);
Review comment:
Change the log message to align with current state.
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##########
@@ -103,13 +105,43 @@ public int execute() {
}
work.setRootTask(this);
this.parentTasks = null;
+ if (shouldLoadAuthorizationMetadata()) {
+ LOG.info("Loading authorization data");
+ try {
+ initiateAuthorizationLoadTask(work.dumpDirectory);
+ } catch (Exception e) {
+ LOG.error("failed", e);
+ setException(e);
+ return ErrorMsg.getErrorMsg(e.getMessage()).getErrorCode();
+ }
+ }
if (work.isIncrementalLoad()) {
return executeIncrementalLoad();
} else {
return executeBootStrapLoad();
}
}
+ private boolean shouldLoadAuthorizationMetadata() {
+ return conf.getBoolVar(HiveConf.ConfVars.REPL_INCLUDE_AUTHORIZATION_METADATA);
+ }
+
+ private void initiateAuthorizationLoadTask(String hiveDumpDirectory) throws SemanticException {
+ if (RANGER_AUTHORIZER.equalsIgnoreCase(conf.getVar(HiveConf.ConfVars.REPL_AUTHORIZATION_PROVIDER_SERVICE))) {
+ Path rangerLoadRoot = new Path(new Path(hiveDumpDirectory).getParent(), ReplUtils.REPL_RANGER_BASE_DIR);
+ LOG.info("Importing Authorization Metadata from {} ", rangerLoadRoot);
+ RangerLoadWork rangerLoadWork = new RangerLoadWork(rangerLoadRoot, work.getSourceDbName(), work.dbNameToLoadIn);
+ Task<RangerLoadWork> rangerLoadTask = TaskFactory.get(rangerLoadWork, conf);
+ if (childTasks == null) {
+ childTasks = new ArrayList<>();
+ }
+ childTasks.add(rangerLoadTask);
Review comment:
This will run ranger load after hive db/table meta data. Ideally the ranger/atlas should run prior to it.
##########
File path: ql/src/test/org/apache/hadoop/hive/ql/exec/repl/TestRangerDumpTask.java
##########
@@ -0,0 +1,123 @@
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.exec.repl;
+
+import com.google.gson.Gson;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.exec.repl.ranger.RangerExportPolicyList;
+import org.apache.hadoop.hive.ql.exec.repl.ranger.RangerRestClientImpl;
+import org.apache.hadoop.hive.ql.exec.repl.ranger.RangerPolicy;
+import org.apache.hadoop.hive.ql.exec.repl.util.ReplUtils;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.Mock;
+import org.mockito.Mockito;
+import org.mockito.junit.MockitoJUnitRunner;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+
+import static org.apache.hadoop.hive.conf.HiveConf.ConfVars.REPL_AUTHORIZATION_PROVIDER_SERVICE_ENDPOINT;
+import static org.apache.hadoop.hive.conf.HiveConf.ConfVars.REPL_RANGER_SERVICE_NAME;
+
+ @RunWith(MockitoJUnitRunner.class)
+public class TestRangerDumpTask {
+
+ protected static final Logger LOG = LoggerFactory.getLogger(TestRangerDumpTask.class);
+ private RangerDumpTask task;
+
+ @Mock
+ private RangerRestClientImpl mockClient;
+
+ @Mock
+ private HiveConf conf;
+
+ @Mock
+ private RangerDumpWork work;
+
+ @Before
+ public void setup() throws Exception {
+ task = new RangerDumpTask(mockClient, conf, work);
+ Mockito.when(mockClient.removeMultiResourcePolicies(Mockito.anyList())).thenCallRealMethod();
+ Mockito.when(mockClient.saveRangerPoliciesToFile(Mockito.any(), Mockito.any(), Mockito.anyString()))
+ .thenCallRealMethod();
+ Mockito.when(mockClient.readRangerPoliciesFromJsonFile(Mockito.any())).thenCallRealMethod();
+ Mockito.when(mockClient.checkConnection(Mockito.anyString())).thenReturn(true);
+ }
+
+ @Test
+ public void testFailureInvalidAuthProviderEndpoint() throws Exception {
+ Mockito.when(conf.getVar(REPL_AUTHORIZATION_PROVIDER_SERVICE_ENDPOINT)).thenReturn(null);
+ int status = task.execute();
+ Assert.assertEquals(40000, status);
+ }
+
+ @Test
+ public void testSuccessValidAuthProviderEndpoint() throws Exception {
+ RangerExportPolicyList rangerPolicyList = new RangerExportPolicyList();
+ rangerPolicyList.setPolicies(new ArrayList<RangerPolicy>());
+ Mockito.when(mockClient.exportRangerPolicies(Mockito.anyString(), Mockito.anyString(), Mockito.anyString()))
+ .thenReturn(rangerPolicyList);
+ Mockito.when(conf.getVar(REPL_AUTHORIZATION_PROVIDER_SERVICE_ENDPOINT)).thenReturn("rangerEndpoint");
+ Mockito.when(conf.getVar(REPL_RANGER_SERVICE_NAME)).thenReturn("cm_hive");
+ Mockito.when(work.getDbName()).thenReturn("testdb");
+ int status = task.execute();
+ Assert.assertEquals(0, status);
+ }
+
+ @Test
+ public void testSuccessNonEmptyRangerPolicies() throws Exception {
+ String rangerResponse = "{\"metaDataInfo\":{\"Host name\":\"quasar-nnwpyd-1.quasar-nnwpyd.root.hwx.site\","
+ + "\"Exported by\":\"hive\",\"Export time\":\"May 5, 2020, 8:55:03 AM\",\"Ranger apache version\""
+ + ":\"2.0.0.7.2.0.0-61\"},\"policies\":[{\"service\":\"cm_hive\",\"name\":\"db-level\",\"policyType\":0,"
+ + "\"description\":\"\",\"isAuditEnabled\":true,\"resources\":{\"database\":{\"values\":[\"aa\"],"
+ + "\"isExcludes\":false,\"isRecursive\":false},\"column\":{\"values\":[\"id\"],\"isExcludes\":false,"
+ + "\"isRecursive\":false},\"table\":{\"values\":[\"*\"],\"isExcludes\":false,\"isRecursive\":false}},"
+ + "\"policyItems\":[{\"accesses\":[{\"type\":\"select\",\"isAllowed\":true},{\"type\":\"update\","
+ + "\"isAllowed\":true}],\"users\":[\"admin\"],\"groups\":[\"public\"],\"conditions\":[],"
+ + "\"delegateAdmin\":false}],\"denyPolicyItems\":[],\"allowExceptions\":[],\"denyExceptions\":[],"
+ + "\"dataMaskPolicyItems\":[],\"rowFilterPolicyItems\":[],\"id\":40,\"guid\":"
+ + "\"4e2b3406-7b9a-4004-8cdf-7a239c8e2cae\",\"isEnabled\":true,\"version\":1}]}";
+ RangerExportPolicyList rangerPolicyList = new Gson().fromJson(rangerResponse, RangerExportPolicyList.class);
+ Mockito.when(mockClient.exportRangerPolicies(Mockito.anyString(), Mockito.anyString(), Mockito.anyString()))
+ .thenReturn(rangerPolicyList);
+ Mockito.when(conf.getVar(REPL_AUTHORIZATION_PROVIDER_SERVICE_ENDPOINT)).thenReturn("rangerEndpoint");
+ Mockito.when(conf.getVar(REPL_RANGER_SERVICE_NAME)).thenReturn("cm_hive");
+ Mockito.when(work.getDbName()).thenReturn("testdb");
+ Path rangerDumpPath = new Path("/tmp");
+ Mockito.when(work.getCurrentDumpPath()).thenReturn(rangerDumpPath);
+ int status = task.execute();
+ Assert.assertEquals(0, status);
+ Path policyFile = new Path(rangerDumpPath, ReplUtils.HIVE_RANGER_POLICIES_FILE_NAME + ".json");
+ Assert.assertTrue(rangerDumpPath.getFileSystem(new Configuration()).exists(policyFile));
+ RangerExportPolicyList actualRangerExportPolicyList = mockClient.readRangerPoliciesFromJsonFile(policyFile);
+ Assert.assertEquals(1, actualRangerExportPolicyList.getPolicies().size());
+ Assert.assertEquals(4, actualRangerExportPolicyList.getMetaDataInfo().size());
+ Assert.assertEquals("quasar-nnwpyd-1.quasar-nnwpyd.root.hwx.site", actualRangerExportPolicyList
Review comment:
Use some example hostname under apache.org domain
##########
File path: ql/src/test/org/apache/hadoop/hive/ql/exec/repl/TestRangerLoadTask.java
##########
@@ -0,0 +1,107 @@
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.exec.repl;
+
+ import com.google.gson.Gson;
+ import org.apache.hadoop.conf.Configuration;
+ import org.apache.hadoop.fs.Path;
+ import org.apache.hadoop.hive.conf.HiveConf;
+ import org.apache.hadoop.hive.ql.exec.repl.ranger.RangerExportPolicyList;
+ import org.apache.hadoop.hive.ql.exec.repl.ranger.RangerPolicy;
+ import org.apache.hadoop.hive.ql.exec.repl.ranger.RangerRestClientImpl;
+ import org.apache.hadoop.hive.ql.exec.repl.util.ReplUtils;
+ import org.junit.Assert;
+ import org.junit.Before;
+ import org.junit.Test;
+ import org.junit.runner.RunWith;
+ import org.mockito.Mock;
+ import org.mockito.Mockito;
+ import org.mockito.junit.MockitoJUnitRunner;
+ import org.slf4j.Logger;
+ import org.slf4j.LoggerFactory;
+
+ import java.util.ArrayList;
+
+ import static org.apache.hadoop.hive.conf.HiveConf.ConfVars.REPL_AUTHORIZATION_PROVIDER_SERVICE_ENDPOINT;
+ import static org.apache.hadoop.hive.conf.HiveConf.ConfVars.REPL_RANGER_SERVICE_NAME;
+
+ @RunWith(MockitoJUnitRunner.class)
+public class TestRangerLoadTask {
+
+ protected static final Logger LOG = LoggerFactory.getLogger(TestRangerLoadTask.class);
+ private RangerLoadTask task;
+
+ @Mock
+ private RangerRestClientImpl mockClient;
+
+ @Mock
+ private HiveConf conf;
+
+ @Mock
+ private RangerLoadWork work;
+
+ @Before
+ public void setup() throws Exception {
+ task = new RangerLoadTask(mockClient, conf, work);
+ Mockito.when(mockClient.changeDataSet(Mockito.anyList(), Mockito.anyString(), Mockito.anyString()))
+ .thenCallRealMethod();
+ Mockito.when(mockClient.readRangerPoliciesFromJsonFile(Mockito.any())).thenCallRealMethod();
+ Mockito.when(mockClient.checkConnection(Mockito.anyString())).thenReturn(true);
+ }
+
+ @Test
+ public void testFailureInvalidAuthProviderEndpoint() {
+ Mockito.when(conf.getVar(REPL_AUTHORIZATION_PROVIDER_SERVICE_ENDPOINT)).thenReturn(null);
+ int status = task.execute();
+ Assert.assertEquals(40000, status);
+ }
+
+ @Test
+ public void testSuccessValidAuthProviderEndpoint() {
+ Mockito.when(conf.getVar(REPL_AUTHORIZATION_PROVIDER_SERVICE_ENDPOINT)).thenReturn("rangerEndpoint");
+ Mockito.when(work.getSourceDbName()).thenReturn("srcdb");
+ Mockito.when(work.getTargetDbName()).thenReturn("tgtdb");
+ int status = task.execute();
+ Assert.assertEquals(0, status);
+ }
+
+ @Test
+ public void testSuccessNonEmptyRangerPolicies() throws Exception {
+ String rangerResponse = "{\"metaDataInfo\":{\"Host name\":\"quasar-nnwpyd-1.quasar-nnwpyd.root.hwx.site\","
Review comment:
Use some example hostname under apache.org domain
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ranger/RangerRestClientImpl.java
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl.ranger;
+
+import com.google.gson.Gson;
+import com.google.gson.GsonBuilder;
+import com.sun.jersey.api.client.Client;
+import com.sun.jersey.api.client.ClientResponse;
+import com.sun.jersey.api.client.WebResource;
+import com.sun.jersey.api.client.config.ClientConfig;
+import com.sun.jersey.api.client.config.DefaultClientConfig;
+import com.sun.jersey.multipart.FormDataMultiPart;
+import com.sun.jersey.multipart.MultiPart;
+import com.sun.jersey.multipart.file.StreamDataBodyPart;
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.eclipse.jetty.util.MultiPartWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.servlet.http.HttpServletResponse;
+import javax.ws.rs.core.MediaType;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.File;
+import java.io.Reader;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.ArrayList;
+
+/**
+ * RangerRestClientImpl to connect to Ranger and export policies.
+ */
+public class RangerRestClientImpl implements RangerRestClient {
+ private static final Logger LOG = LoggerFactory.getLogger(RangerRestClientImpl.class);
+ private static final String RANGER_REST_URL_EXPORTJSONFILE = "/service/plugins/policies/exportJson";
+ private static final String RANGER_REST_URL_IMPORTJSONFILE =
+ "/service/plugins/policies/importPoliciesFromFile?updateIfExists=true";
+
+ public RangerExportPolicyList exportRangerPolicies(String sourceRangerEndpoint,
+ String dbName, String rangerHiveServiceName) throws Exception {
+ LOG.info("Ranger endpoint for cluster " + sourceRangerEndpoint);
+ ClientResponse clientResp = null;
+ String uri = null;
+ if (!StringUtils.isEmpty(rangerHiveServiceName)) {
Review comment:
It should be error if it is empty.
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ranger/RangerRestClientImpl.java
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl.ranger;
+
+import com.google.gson.Gson;
+import com.google.gson.GsonBuilder;
+import com.sun.jersey.api.client.Client;
+import com.sun.jersey.api.client.ClientResponse;
+import com.sun.jersey.api.client.WebResource;
+import com.sun.jersey.api.client.config.ClientConfig;
+import com.sun.jersey.api.client.config.DefaultClientConfig;
+import com.sun.jersey.multipart.FormDataMultiPart;
+import com.sun.jersey.multipart.MultiPart;
+import com.sun.jersey.multipart.file.StreamDataBodyPart;
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.eclipse.jetty.util.MultiPartWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.servlet.http.HttpServletResponse;
+import javax.ws.rs.core.MediaType;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.File;
+import java.io.Reader;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.ArrayList;
+
+/**
+ * RangerRestClientImpl to connect to Ranger and export policies.
+ */
+public class RangerRestClientImpl implements RangerRestClient {
+ private static final Logger LOG = LoggerFactory.getLogger(RangerRestClientImpl.class);
+ private static final String RANGER_REST_URL_EXPORTJSONFILE = "/service/plugins/policies/exportJson";
+ private static final String RANGER_REST_URL_IMPORTJSONFILE =
+ "/service/plugins/policies/importPoliciesFromFile?updateIfExists=true";
+
+ public RangerExportPolicyList exportRangerPolicies(String sourceRangerEndpoint,
+ String dbName, String rangerHiveServiceName) throws Exception {
+ LOG.info("Ranger endpoint for cluster " + sourceRangerEndpoint);
+ ClientResponse clientResp = null;
+ String uri = null;
+ if (!StringUtils.isEmpty(rangerHiveServiceName)) {
+ uri = RANGER_REST_URL_EXPORTJSONFILE + "?serviceName=" + rangerHiveServiceName + "&polResource="
+ + dbName + "&resource:database=" + dbName
+ + "&serviceType=hive&resourceMatchScope=self_or_ancestor&resourceMatch=full";
+ }
+ if (sourceRangerEndpoint.endsWith("/")) {
+ sourceRangerEndpoint = StringUtils.removePattern(sourceRangerEndpoint, "/+$");
+ }
+ String url = sourceRangerEndpoint + (uri != null ? (uri.startsWith("/") ? uri : ("/" + uri)) : "");
+ LOG.debug("URL to export policies from source Ranger: {}", url);
+ RangerExportPolicyList rangerExportPolicyList = new RangerExportPolicyList();
+ WebResource.Builder builder = getRangerResourceBuilder(url);
+ clientResp = builder.get(ClientResponse.class);
+
+ String response = null;
+ if (clientResp != null) {
+ if (clientResp.getStatus() == HttpServletResponse.SC_OK) {
+ Gson gson = new GsonBuilder().create();
+ response = clientResp.getEntity(String.class);
+ LOG.debug("Response received for ranger export {} ", response);
+ if (StringUtils.isNotEmpty(response)) {
+ rangerExportPolicyList = gson.fromJson(response, RangerExportPolicyList.class);
+ return rangerExportPolicyList;
+ }
+ } else if (clientResp.getStatus() == HttpServletResponse.SC_NO_CONTENT) {
+ LOG.debug("Ranger policy export request returned empty list");
+ return rangerExportPolicyList;
+ } else if (clientResp.getStatus() == HttpServletResponse.SC_UNAUTHORIZED) {
+ throw new Exception("Authentication Failure while communicating to Ranger admin");
+ } else if (clientResp.getStatus() == HttpServletResponse.SC_FORBIDDEN) {
+ throw new Exception("Authorization Failure while communicating to Ranger admin");
+ }
+ }
+ if (StringUtils.isEmpty(response)) {
+ LOG.debug("Ranger policy export request returned empty list or failed, Please refer Ranger admin logs.");
+ }
+ return rangerExportPolicyList;
+ }
+
+ public List<RangerPolicy> removeMultiResourcePolicies(List<RangerPolicy> rangerPolicies) {
+ List<RangerPolicy> rangerPoliciesToImport = new ArrayList<RangerPolicy>();
+ if (CollectionUtils.isNotEmpty(rangerPolicies)) {
+ Map<String, RangerPolicy.RangerPolicyResource> rangerPolicyResourceMap = null;
+ RangerPolicy.RangerPolicyResource rangerPolicyResource = null;
+ List<String> resourceNameList = null;
+ for (RangerPolicy rangerPolicy : rangerPolicies) {
+ if (rangerPolicy != null) {
+ rangerPolicyResourceMap = rangerPolicy.getResources();
+ if (rangerPolicyResourceMap != null) {
+ rangerPolicyResource = rangerPolicyResourceMap.get("database");
+ if (rangerPolicyResource != null) {
+ resourceNameList = rangerPolicyResource.getValues();
+ if (CollectionUtils.isNotEmpty(resourceNameList) && resourceNameList.size() == 1) {
+ rangerPoliciesToImport.add(rangerPolicy);
+ }
+ }
+ }
+ }
+ }
+ }
+ return rangerPoliciesToImport;
+ }
+
+ @Override
+ public RangerExportPolicyList importRangerPolicies(RangerExportPolicyList rangerExportPolicyList, String dbName,
+ String baseUrl,
+ String rangerHiveServiceName)
+ throws Exception {
+ String sourceClusterServiceName = null;
+ String serviceMapJsonFileName = "hive_servicemap.json";
+ String rangerPoliciesJsonFileName = "hive_replicationPolicies.json";
+ String uri = RANGER_REST_URL_IMPORTJSONFILE + "&polResource=" + dbName;
+
+ if (rangerExportPolicyList.getPolicies().size() > 0) {
+ sourceClusterServiceName = rangerExportPolicyList.getPolicies().get(0).getService();
+ }
+
+ if (StringUtils.isEmpty(sourceClusterServiceName)) {
+ sourceClusterServiceName = rangerHiveServiceName;
+ }
+
+ Map<String, String> serviceMap = new LinkedHashMap<String, String>();
+ if (!StringUtils.isEmpty(sourceClusterServiceName) && !StringUtils.isEmpty(rangerHiveServiceName)) {
+ serviceMap.put(sourceClusterServiceName, rangerHiveServiceName);
+ }
+
+ Gson gson = new GsonBuilder().create();
+ String jsonServiceMap = gson.toJson(serviceMap);
+
+ String jsonRangerExportPolicyList = gson.toJson(rangerExportPolicyList);
+
+ String url = baseUrl
+ + (uri.startsWith("/") ? uri : ("/" + uri));
+
+ LOG.debug("URL to import policies on target Ranger: {}", url);
+ ClientResponse clientResp = null;
+
+ StreamDataBodyPart filePartPolicies = new StreamDataBodyPart("file",
+ new ByteArrayInputStream(jsonRangerExportPolicyList.getBytes(StandardCharsets.UTF_8)),
+ rangerPoliciesJsonFileName);
+ StreamDataBodyPart filePartServiceMap = new StreamDataBodyPart("servicesMapJson",
+ new ByteArrayInputStream(jsonServiceMap.getBytes(StandardCharsets.UTF_8)), serviceMapJsonFileName);
+
+ FormDataMultiPart formDataMultiPart = new FormDataMultiPart();
+ MultiPart multipartEntity = null;
+ try {
+ multipartEntity = formDataMultiPart.bodyPart(filePartPolicies).bodyPart(filePartServiceMap);
+ WebResource.Builder builder = getRangerResourceBuilder(url);
+ clientResp = builder.accept(MediaType.APPLICATION_JSON).type(MediaType.MULTIPART_FORM_DATA)
+ .post(ClientResponse.class, multipartEntity);
+ if (clientResp != null) {
+ if (clientResp.getStatus() == HttpServletResponse.SC_NO_CONTENT) {
+ LOG.debug("Ranger policy import finished successfully");
+
+ } else if (clientResp.getStatus() == HttpServletResponse.SC_UNAUTHORIZED) {
+ throw new Exception("Authentication Failure while communicating to Ranger admin");
+ } else {
+ throw new Exception("Ranger policy import failed, Please refer target Ranger admin logs.");
+ }
+ }
+ } finally {
+ try {
+ if (filePartPolicies != null) {
+ filePartPolicies.cleanup();
+ }
+ if (filePartServiceMap != null) {
+ filePartServiceMap.cleanup();
+ }
+ if (formDataMultiPart != null) {
+ formDataMultiPart.close();
+ }
+ if (multipartEntity != null) {
+ multipartEntity.close();
+ }
+ } catch (IOException e) {
+ LOG.error("Exception occurred while closing resources: {}", e);
+ }
+ }
+ return rangerExportPolicyList;
+ }
+
+ private synchronized Client getRangerClient() {
+ Client ret = null;
+ ClientConfig config = new DefaultClientConfig();
+ config.getClasses().add(MultiPartWriter.class);
+ config.getProperties().put(ClientConfig.PROPERTY_FOLLOW_REDIRECTS, true);
+ ret = Client.create(config);
+ return ret;
+ }
+
+ @Override
+ public List<RangerPolicy> changeDataSet(List<RangerPolicy> rangerPolicies, String sourceDbName,
+ String targetDbName) {
+ if (sourceDbName.endsWith("/")) {
+ sourceDbName = StringUtils.removePattern(sourceDbName, "/+$");
+ }
+ if (targetDbName.endsWith("/")) {
+ targetDbName = StringUtils.removePattern(targetDbName, "/+$");
+ }
+ if (targetDbName.equals(sourceDbName)) {
+ return rangerPolicies;
+ }
+ if (CollectionUtils.isNotEmpty(rangerPolicies)) {
+ Map<String, RangerPolicy.RangerPolicyResource> rangerPolicyResourceMap = null;
+ RangerPolicy.RangerPolicyResource rangerPolicyResource = null;
+ List<String> resourceNameList = null;
+ for (RangerPolicy rangerPolicy : rangerPolicies) {
+ if (rangerPolicy != null) {
+ rangerPolicyResourceMap = rangerPolicy.getResources();
+ if (rangerPolicyResourceMap != null) {
+ rangerPolicyResource = rangerPolicyResourceMap.get("database");
+ if (rangerPolicyResource != null) {
+ resourceNameList = rangerPolicyResource.getValues();
+ if (CollectionUtils.isNotEmpty(resourceNameList)) {
+ for (int i = 0; i < resourceNameList.size(); i++) {
+ String resourceName = resourceNameList.get(i);
+ if (resourceName.equals(sourceDbName)) {
+ resourceNameList.set(i, targetDbName);
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return rangerPolicies;
+ }
+
+ private Path writeExportedRangerPoliciesToJsonFile(String jsonString, String fileName, Path stagingDirPath)
+ throws Exception {
+ String filePath = "";
+ Path newPath = null;
+ FSDataOutputStream outStream = null;
+ OutputStreamWriter writer = null;
+ try {
+ if (!StringUtils.isEmpty(jsonString)) {
+ FileSystem fileSystem = getFileSystem(stagingDirPath);
+ if (fileSystem != null) {
+ if (!fileSystem.exists(stagingDirPath)) {
+ fileSystem.mkdirs(stagingDirPath);
+ }
+ newPath = stagingDirPath.suffix(File.separator + fileName);
+ outStream = fileSystem.create(newPath, true);
+ writer = new OutputStreamWriter(outStream, "UTF-8");
+ writer.write(jsonString);
+ }
+ }
+ } catch (IOException ex) {
+ if (newPath != null) {
+ filePath = newPath.toString();
+ }
+ throw new Exception("Failed to write json string to file:" + filePath, ex);
+ } catch (Exception ex) {
+ if (newPath != null) {
+ filePath = newPath.toString();
+ }
+ throw new Exception("Failed to write json string to file:" + filePath, ex);
+ } finally {
+ try {
+ if (writer != null) {
+ writer.close();
+ }
+ if (outStream != null) {
+ outStream.close();
+ }
+ } catch (Exception ex) {
+ LOG.error("Unable to close writer/outStream.", ex);
+ }
+ }
+ return newPath;
+ }
+
+ private FileSystem getFileSystem(Path path) throws IOException {
+ Configuration conf = new Configuration();
+ FileSystem fileSystem = path.getFileSystem(conf);
Review comment:
Shouldn't the conf used here be HiveConf?
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ranger/RangerRestClientImpl.java
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl.ranger;
+
+import com.google.gson.Gson;
+import com.google.gson.GsonBuilder;
+import com.sun.jersey.api.client.Client;
+import com.sun.jersey.api.client.ClientResponse;
+import com.sun.jersey.api.client.WebResource;
+import com.sun.jersey.api.client.config.ClientConfig;
+import com.sun.jersey.api.client.config.DefaultClientConfig;
+import com.sun.jersey.multipart.FormDataMultiPart;
+import com.sun.jersey.multipart.MultiPart;
+import com.sun.jersey.multipart.file.StreamDataBodyPart;
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.eclipse.jetty.util.MultiPartWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.servlet.http.HttpServletResponse;
+import javax.ws.rs.core.MediaType;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.File;
+import java.io.Reader;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.ArrayList;
+
+/**
+ * RangerRestClientImpl to connect to Ranger and export policies.
+ */
+public class RangerRestClientImpl implements RangerRestClient {
+ private static final Logger LOG = LoggerFactory.getLogger(RangerRestClientImpl.class);
+ private static final String RANGER_REST_URL_EXPORTJSONFILE = "/service/plugins/policies/exportJson";
+ private static final String RANGER_REST_URL_IMPORTJSONFILE =
+ "/service/plugins/policies/importPoliciesFromFile?updateIfExists=true";
+
+ public RangerExportPolicyList exportRangerPolicies(String sourceRangerEndpoint,
+ String dbName, String rangerHiveServiceName) throws Exception {
+ LOG.info("Ranger endpoint for cluster " + sourceRangerEndpoint);
+ ClientResponse clientResp = null;
+ String uri = null;
+ if (!StringUtils.isEmpty(rangerHiveServiceName)) {
+ uri = RANGER_REST_URL_EXPORTJSONFILE + "?serviceName=" + rangerHiveServiceName + "&polResource="
+ + dbName + "&resource:database=" + dbName
+ + "&serviceType=hive&resourceMatchScope=self_or_ancestor&resourceMatch=full";
+ }
+ if (sourceRangerEndpoint.endsWith("/")) {
+ sourceRangerEndpoint = StringUtils.removePattern(sourceRangerEndpoint, "/+$");
+ }
+ String url = sourceRangerEndpoint + (uri != null ? (uri.startsWith("/") ? uri : ("/" + uri)) : "");
+ LOG.debug("URL to export policies from source Ranger: {}", url);
+ RangerExportPolicyList rangerExportPolicyList = new RangerExportPolicyList();
+ WebResource.Builder builder = getRangerResourceBuilder(url);
+ clientResp = builder.get(ClientResponse.class);
+
+ String response = null;
+ if (clientResp != null) {
+ if (clientResp.getStatus() == HttpServletResponse.SC_OK) {
+ Gson gson = new GsonBuilder().create();
+ response = clientResp.getEntity(String.class);
+ LOG.debug("Response received for ranger export {} ", response);
+ if (StringUtils.isNotEmpty(response)) {
+ rangerExportPolicyList = gson.fromJson(response, RangerExportPolicyList.class);
+ return rangerExportPolicyList;
+ }
+ } else if (clientResp.getStatus() == HttpServletResponse.SC_NO_CONTENT) {
+ LOG.debug("Ranger policy export request returned empty list");
+ return rangerExportPolicyList;
+ } else if (clientResp.getStatus() == HttpServletResponse.SC_UNAUTHORIZED) {
+ throw new Exception("Authentication Failure while communicating to Ranger admin");
+ } else if (clientResp.getStatus() == HttpServletResponse.SC_FORBIDDEN) {
+ throw new Exception("Authorization Failure while communicating to Ranger admin");
+ }
+ }
+ if (StringUtils.isEmpty(response)) {
+ LOG.debug("Ranger policy export request returned empty list or failed, Please refer Ranger admin logs.");
+ }
+ return rangerExportPolicyList;
+ }
+
+ public List<RangerPolicy> removeMultiResourcePolicies(List<RangerPolicy> rangerPolicies) {
+ List<RangerPolicy> rangerPoliciesToImport = new ArrayList<RangerPolicy>();
+ if (CollectionUtils.isNotEmpty(rangerPolicies)) {
+ Map<String, RangerPolicy.RangerPolicyResource> rangerPolicyResourceMap = null;
+ RangerPolicy.RangerPolicyResource rangerPolicyResource = null;
+ List<String> resourceNameList = null;
+ for (RangerPolicy rangerPolicy : rangerPolicies) {
+ if (rangerPolicy != null) {
+ rangerPolicyResourceMap = rangerPolicy.getResources();
+ if (rangerPolicyResourceMap != null) {
+ rangerPolicyResource = rangerPolicyResourceMap.get("database");
+ if (rangerPolicyResource != null) {
+ resourceNameList = rangerPolicyResource.getValues();
+ if (CollectionUtils.isNotEmpty(resourceNameList) && resourceNameList.size() == 1) {
+ rangerPoliciesToImport.add(rangerPolicy);
+ }
+ }
+ }
+ }
+ }
+ }
+ return rangerPoliciesToImport;
+ }
+
+ @Override
+ public RangerExportPolicyList importRangerPolicies(RangerExportPolicyList rangerExportPolicyList, String dbName,
+ String baseUrl,
+ String rangerHiveServiceName)
+ throws Exception {
+ String sourceClusterServiceName = null;
+ String serviceMapJsonFileName = "hive_servicemap.json";
+ String rangerPoliciesJsonFileName = "hive_replicationPolicies.json";
+ String uri = RANGER_REST_URL_IMPORTJSONFILE + "&polResource=" + dbName;
+
+ if (rangerExportPolicyList.getPolicies().size() > 0) {
+ sourceClusterServiceName = rangerExportPolicyList.getPolicies().get(0).getService();
+ }
+
+ if (StringUtils.isEmpty(sourceClusterServiceName)) {
+ sourceClusterServiceName = rangerHiveServiceName;
+ }
+
+ Map<String, String> serviceMap = new LinkedHashMap<String, String>();
+ if (!StringUtils.isEmpty(sourceClusterServiceName) && !StringUtils.isEmpty(rangerHiveServiceName)) {
+ serviceMap.put(sourceClusterServiceName, rangerHiveServiceName);
+ }
+
+ Gson gson = new GsonBuilder().create();
+ String jsonServiceMap = gson.toJson(serviceMap);
+
+ String jsonRangerExportPolicyList = gson.toJson(rangerExportPolicyList);
+
+ String url = baseUrl
+ + (uri.startsWith("/") ? uri : ("/" + uri));
+
+ LOG.debug("URL to import policies on target Ranger: {}", url);
+ ClientResponse clientResp = null;
+
+ StreamDataBodyPart filePartPolicies = new StreamDataBodyPart("file",
+ new ByteArrayInputStream(jsonRangerExportPolicyList.getBytes(StandardCharsets.UTF_8)),
+ rangerPoliciesJsonFileName);
+ StreamDataBodyPart filePartServiceMap = new StreamDataBodyPart("servicesMapJson",
+ new ByteArrayInputStream(jsonServiceMap.getBytes(StandardCharsets.UTF_8)), serviceMapJsonFileName);
+
+ FormDataMultiPart formDataMultiPart = new FormDataMultiPart();
+ MultiPart multipartEntity = null;
+ try {
+ multipartEntity = formDataMultiPart.bodyPart(filePartPolicies).bodyPart(filePartServiceMap);
+ WebResource.Builder builder = getRangerResourceBuilder(url);
+ clientResp = builder.accept(MediaType.APPLICATION_JSON).type(MediaType.MULTIPART_FORM_DATA)
+ .post(ClientResponse.class, multipartEntity);
+ if (clientResp != null) {
+ if (clientResp.getStatus() == HttpServletResponse.SC_NO_CONTENT) {
+ LOG.debug("Ranger policy import finished successfully");
+
+ } else if (clientResp.getStatus() == HttpServletResponse.SC_UNAUTHORIZED) {
+ throw new Exception("Authentication Failure while communicating to Ranger admin");
+ } else {
+ throw new Exception("Ranger policy import failed, Please refer target Ranger admin logs.");
+ }
+ }
+ } finally {
+ try {
+ if (filePartPolicies != null) {
+ filePartPolicies.cleanup();
+ }
+ if (filePartServiceMap != null) {
+ filePartServiceMap.cleanup();
+ }
+ if (formDataMultiPart != null) {
+ formDataMultiPart.close();
+ }
+ if (multipartEntity != null) {
+ multipartEntity.close();
+ }
+ } catch (IOException e) {
+ LOG.error("Exception occurred while closing resources: {}", e);
+ }
+ }
+ return rangerExportPolicyList;
+ }
+
+ private synchronized Client getRangerClient() {
+ Client ret = null;
+ ClientConfig config = new DefaultClientConfig();
+ config.getClasses().add(MultiPartWriter.class);
+ config.getProperties().put(ClientConfig.PROPERTY_FOLLOW_REDIRECTS, true);
+ ret = Client.create(config);
+ return ret;
+ }
+
+ @Override
+ public List<RangerPolicy> changeDataSet(List<RangerPolicy> rangerPolicies, String sourceDbName,
+ String targetDbName) {
+ if (sourceDbName.endsWith("/")) {
+ sourceDbName = StringUtils.removePattern(sourceDbName, "/+$");
+ }
+ if (targetDbName.endsWith("/")) {
+ targetDbName = StringUtils.removePattern(targetDbName, "/+$");
+ }
+ if (targetDbName.equals(sourceDbName)) {
+ return rangerPolicies;
+ }
+ if (CollectionUtils.isNotEmpty(rangerPolicies)) {
+ Map<String, RangerPolicy.RangerPolicyResource> rangerPolicyResourceMap = null;
+ RangerPolicy.RangerPolicyResource rangerPolicyResource = null;
+ List<String> resourceNameList = null;
+ for (RangerPolicy rangerPolicy : rangerPolicies) {
+ if (rangerPolicy != null) {
+ rangerPolicyResourceMap = rangerPolicy.getResources();
+ if (rangerPolicyResourceMap != null) {
+ rangerPolicyResource = rangerPolicyResourceMap.get("database");
+ if (rangerPolicyResource != null) {
+ resourceNameList = rangerPolicyResource.getValues();
+ if (CollectionUtils.isNotEmpty(resourceNameList)) {
+ for (int i = 0; i < resourceNameList.size(); i++) {
+ String resourceName = resourceNameList.get(i);
+ if (resourceName.equals(sourceDbName)) {
+ resourceNameList.set(i, targetDbName);
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return rangerPolicies;
+ }
+
+ private Path writeExportedRangerPoliciesToJsonFile(String jsonString, String fileName, Path stagingDirPath)
+ throws Exception {
+ String filePath = "";
+ Path newPath = null;
+ FSDataOutputStream outStream = null;
+ OutputStreamWriter writer = null;
+ try {
+ if (!StringUtils.isEmpty(jsonString)) {
+ FileSystem fileSystem = getFileSystem(stagingDirPath);
+ if (fileSystem != null) {
Review comment:
Shouldn't it fail when fileSystem is null?
##########
File path: ql/src/test/org/apache/hadoop/hive/ql/exec/repl/TestRangerDumpTask.java
##########
@@ -0,0 +1,123 @@
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.exec.repl;
+
+import com.google.gson.Gson;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.exec.repl.ranger.RangerExportPolicyList;
+import org.apache.hadoop.hive.ql.exec.repl.ranger.RangerRestClientImpl;
+import org.apache.hadoop.hive.ql.exec.repl.ranger.RangerPolicy;
+import org.apache.hadoop.hive.ql.exec.repl.util.ReplUtils;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.Mock;
+import org.mockito.Mockito;
+import org.mockito.junit.MockitoJUnitRunner;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+
+import static org.apache.hadoop.hive.conf.HiveConf.ConfVars.REPL_AUTHORIZATION_PROVIDER_SERVICE_ENDPOINT;
+import static org.apache.hadoop.hive.conf.HiveConf.ConfVars.REPL_RANGER_SERVICE_NAME;
+
+ @RunWith(MockitoJUnitRunner.class)
+public class TestRangerDumpTask {
+
+ protected static final Logger LOG = LoggerFactory.getLogger(TestRangerDumpTask.class);
+ private RangerDumpTask task;
+
+ @Mock
+ private RangerRestClientImpl mockClient;
+
+ @Mock
+ private HiveConf conf;
+
+ @Mock
+ private RangerDumpWork work;
+
+ @Before
+ public void setup() throws Exception {
+ task = new RangerDumpTask(mockClient, conf, work);
+ Mockito.when(mockClient.removeMultiResourcePolicies(Mockito.anyList())).thenCallRealMethod();
+ Mockito.when(mockClient.saveRangerPoliciesToFile(Mockito.any(), Mockito.any(), Mockito.anyString()))
+ .thenCallRealMethod();
+ Mockito.when(mockClient.readRangerPoliciesFromJsonFile(Mockito.any())).thenCallRealMethod();
+ Mockito.when(mockClient.checkConnection(Mockito.anyString())).thenReturn(true);
+ }
+
+ @Test
+ public void testFailureInvalidAuthProviderEndpoint() throws Exception {
+ Mockito.when(conf.getVar(REPL_AUTHORIZATION_PROVIDER_SERVICE_ENDPOINT)).thenReturn(null);
+ int status = task.execute();
+ Assert.assertEquals(40000, status);
+ }
+
+ @Test
+ public void testSuccessValidAuthProviderEndpoint() throws Exception {
+ RangerExportPolicyList rangerPolicyList = new RangerExportPolicyList();
+ rangerPolicyList.setPolicies(new ArrayList<RangerPolicy>());
+ Mockito.when(mockClient.exportRangerPolicies(Mockito.anyString(), Mockito.anyString(), Mockito.anyString()))
+ .thenReturn(rangerPolicyList);
+ Mockito.when(conf.getVar(REPL_AUTHORIZATION_PROVIDER_SERVICE_ENDPOINT)).thenReturn("rangerEndpoint");
+ Mockito.when(conf.getVar(REPL_RANGER_SERVICE_NAME)).thenReturn("cm_hive");
+ Mockito.when(work.getDbName()).thenReturn("testdb");
+ int status = task.execute();
+ Assert.assertEquals(0, status);
+ }
+
+ @Test
+ public void testSuccessNonEmptyRangerPolicies() throws Exception {
+ String rangerResponse = "{\"metaDataInfo\":{\"Host name\":\"quasar-nnwpyd-1.quasar-nnwpyd.root.hwx.site\","
Review comment:
Use some example hostname under apache.org domain
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ranger/RangerRestClientImpl.java
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl.ranger;
+
+import com.google.gson.Gson;
+import com.google.gson.GsonBuilder;
+import com.sun.jersey.api.client.Client;
+import com.sun.jersey.api.client.ClientResponse;
+import com.sun.jersey.api.client.WebResource;
+import com.sun.jersey.api.client.config.ClientConfig;
+import com.sun.jersey.api.client.config.DefaultClientConfig;
+import com.sun.jersey.multipart.FormDataMultiPart;
+import com.sun.jersey.multipart.MultiPart;
+import com.sun.jersey.multipart.file.StreamDataBodyPart;
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.eclipse.jetty.util.MultiPartWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.servlet.http.HttpServletResponse;
+import javax.ws.rs.core.MediaType;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.File;
+import java.io.Reader;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.ArrayList;
+
+/**
+ * RangerRestClientImpl to connect to Ranger and export policies.
+ */
+public class RangerRestClientImpl implements RangerRestClient {
+ private static final Logger LOG = LoggerFactory.getLogger(RangerRestClientImpl.class);
+ private static final String RANGER_REST_URL_EXPORTJSONFILE = "/service/plugins/policies/exportJson";
+ private static final String RANGER_REST_URL_IMPORTJSONFILE =
+ "/service/plugins/policies/importPoliciesFromFile?updateIfExists=true";
+
+ public RangerExportPolicyList exportRangerPolicies(String sourceRangerEndpoint,
+ String dbName, String rangerHiveServiceName) throws Exception {
+ LOG.info("Ranger endpoint for cluster " + sourceRangerEndpoint);
+ ClientResponse clientResp = null;
+ String uri = null;
+ if (!StringUtils.isEmpty(rangerHiveServiceName)) {
+ uri = RANGER_REST_URL_EXPORTJSONFILE + "?serviceName=" + rangerHiveServiceName + "&polResource="
+ + dbName + "&resource:database=" + dbName
+ + "&serviceType=hive&resourceMatchScope=self_or_ancestor&resourceMatch=full";
+ }
+ if (sourceRangerEndpoint.endsWith("/")) {
+ sourceRangerEndpoint = StringUtils.removePattern(sourceRangerEndpoint, "/+$");
+ }
+ String url = sourceRangerEndpoint + (uri != null ? (uri.startsWith("/") ? uri : ("/" + uri)) : "");
+ LOG.debug("URL to export policies from source Ranger: {}", url);
+ RangerExportPolicyList rangerExportPolicyList = new RangerExportPolicyList();
+ WebResource.Builder builder = getRangerResourceBuilder(url);
+ clientResp = builder.get(ClientResponse.class);
+
+ String response = null;
+ if (clientResp != null) {
+ if (clientResp.getStatus() == HttpServletResponse.SC_OK) {
+ Gson gson = new GsonBuilder().create();
+ response = clientResp.getEntity(String.class);
+ LOG.debug("Response received for ranger export {} ", response);
+ if (StringUtils.isNotEmpty(response)) {
+ rangerExportPolicyList = gson.fromJson(response, RangerExportPolicyList.class);
+ return rangerExportPolicyList;
+ }
+ } else if (clientResp.getStatus() == HttpServletResponse.SC_NO_CONTENT) {
+ LOG.debug("Ranger policy export request returned empty list");
+ return rangerExportPolicyList;
+ } else if (clientResp.getStatus() == HttpServletResponse.SC_UNAUTHORIZED) {
+ throw new Exception("Authentication Failure while communicating to Ranger admin");
+ } else if (clientResp.getStatus() == HttpServletResponse.SC_FORBIDDEN) {
+ throw new Exception("Authorization Failure while communicating to Ranger admin");
+ }
+ }
+ if (StringUtils.isEmpty(response)) {
+ LOG.debug("Ranger policy export request returned empty list or failed, Please refer Ranger admin logs.");
+ }
+ return rangerExportPolicyList;
+ }
+
+ public List<RangerPolicy> removeMultiResourcePolicies(List<RangerPolicy> rangerPolicies) {
+ List<RangerPolicy> rangerPoliciesToImport = new ArrayList<RangerPolicy>();
+ if (CollectionUtils.isNotEmpty(rangerPolicies)) {
+ Map<String, RangerPolicy.RangerPolicyResource> rangerPolicyResourceMap = null;
+ RangerPolicy.RangerPolicyResource rangerPolicyResource = null;
+ List<String> resourceNameList = null;
+ for (RangerPolicy rangerPolicy : rangerPolicies) {
+ if (rangerPolicy != null) {
+ rangerPolicyResourceMap = rangerPolicy.getResources();
+ if (rangerPolicyResourceMap != null) {
+ rangerPolicyResource = rangerPolicyResourceMap.get("database");
+ if (rangerPolicyResource != null) {
+ resourceNameList = rangerPolicyResource.getValues();
+ if (CollectionUtils.isNotEmpty(resourceNameList) && resourceNameList.size() == 1) {
+ rangerPoliciesToImport.add(rangerPolicy);
+ }
+ }
+ }
+ }
+ }
+ }
+ return rangerPoliciesToImport;
+ }
+
+ @Override
+ public RangerExportPolicyList importRangerPolicies(RangerExportPolicyList rangerExportPolicyList, String dbName,
+ String baseUrl,
+ String rangerHiveServiceName)
+ throws Exception {
+ String sourceClusterServiceName = null;
+ String serviceMapJsonFileName = "hive_servicemap.json";
+ String rangerPoliciesJsonFileName = "hive_replicationPolicies.json";
+ String uri = RANGER_REST_URL_IMPORTJSONFILE + "&polResource=" + dbName;
+
+ if (rangerExportPolicyList.getPolicies().size() > 0) {
Review comment:
replace with rangerExportPolicyList.getPolicies().isEmpty()
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java
##########
@@ -100,6 +103,10 @@
// Reserved number of items to accommodate operational files in the dump root dir.
public static final int RESERVED_DIR_ITEMS_COUNT = 10;
+
+ public static final String RANGER_AUTHORIZER = "ranger";
+
+ public static final String HIVE_RANGER_POLICIES_FILE_NAME = "ranger_policies";
Review comment:
Shouldn't we use extension as well, e.g. ranger_policies.json
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ranger/RangerRestClientImpl.java
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl.ranger;
+
+import com.google.gson.Gson;
+import com.google.gson.GsonBuilder;
+import com.sun.jersey.api.client.Client;
+import com.sun.jersey.api.client.ClientResponse;
+import com.sun.jersey.api.client.WebResource;
+import com.sun.jersey.api.client.config.ClientConfig;
+import com.sun.jersey.api.client.config.DefaultClientConfig;
+import com.sun.jersey.multipart.FormDataMultiPart;
+import com.sun.jersey.multipart.MultiPart;
+import com.sun.jersey.multipart.file.StreamDataBodyPart;
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.eclipse.jetty.util.MultiPartWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.servlet.http.HttpServletResponse;
+import javax.ws.rs.core.MediaType;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.File;
+import java.io.Reader;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.ArrayList;
+
+/**
+ * RangerRestClientImpl to connect to Ranger and export policies.
+ */
+public class RangerRestClientImpl implements RangerRestClient {
+ private static final Logger LOG = LoggerFactory.getLogger(RangerRestClientImpl.class);
+ private static final String RANGER_REST_URL_EXPORTJSONFILE = "/service/plugins/policies/exportJson";
+ private static final String RANGER_REST_URL_IMPORTJSONFILE =
+ "/service/plugins/policies/importPoliciesFromFile?updateIfExists=true";
+
+ public RangerExportPolicyList exportRangerPolicies(String sourceRangerEndpoint,
+ String dbName, String rangerHiveServiceName) throws Exception {
+ LOG.info("Ranger endpoint for cluster " + sourceRangerEndpoint);
+ ClientResponse clientResp = null;
+ String uri = null;
+ if (!StringUtils.isEmpty(rangerHiveServiceName)) {
+ uri = RANGER_REST_URL_EXPORTJSONFILE + "?serviceName=" + rangerHiveServiceName + "&polResource="
+ + dbName + "&resource:database=" + dbName
+ + "&serviceType=hive&resourceMatchScope=self_or_ancestor&resourceMatch=full";
+ }
+ if (sourceRangerEndpoint.endsWith("/")) {
+ sourceRangerEndpoint = StringUtils.removePattern(sourceRangerEndpoint, "/+$");
+ }
+ String url = sourceRangerEndpoint + (uri != null ? (uri.startsWith("/") ? uri : ("/" + uri)) : "");
+ LOG.debug("URL to export policies from source Ranger: {}", url);
+ RangerExportPolicyList rangerExportPolicyList = new RangerExportPolicyList();
+ WebResource.Builder builder = getRangerResourceBuilder(url);
+ clientResp = builder.get(ClientResponse.class);
+
+ String response = null;
+ if (clientResp != null) {
+ if (clientResp.getStatus() == HttpServletResponse.SC_OK) {
+ Gson gson = new GsonBuilder().create();
+ response = clientResp.getEntity(String.class);
+ LOG.debug("Response received for ranger export {} ", response);
+ if (StringUtils.isNotEmpty(response)) {
+ rangerExportPolicyList = gson.fromJson(response, RangerExportPolicyList.class);
+ return rangerExportPolicyList;
+ }
+ } else if (clientResp.getStatus() == HttpServletResponse.SC_NO_CONTENT) {
+ LOG.debug("Ranger policy export request returned empty list");
+ return rangerExportPolicyList;
+ } else if (clientResp.getStatus() == HttpServletResponse.SC_UNAUTHORIZED) {
+ throw new Exception("Authentication Failure while communicating to Ranger admin");
+ } else if (clientResp.getStatus() == HttpServletResponse.SC_FORBIDDEN) {
+ throw new Exception("Authorization Failure while communicating to Ranger admin");
+ }
+ }
+ if (StringUtils.isEmpty(response)) {
+ LOG.debug("Ranger policy export request returned empty list or failed, Please refer Ranger admin logs.");
+ }
+ return rangerExportPolicyList;
+ }
+
+ public List<RangerPolicy> removeMultiResourcePolicies(List<RangerPolicy> rangerPolicies) {
+ List<RangerPolicy> rangerPoliciesToImport = new ArrayList<RangerPolicy>();
+ if (CollectionUtils.isNotEmpty(rangerPolicies)) {
+ Map<String, RangerPolicy.RangerPolicyResource> rangerPolicyResourceMap = null;
+ RangerPolicy.RangerPolicyResource rangerPolicyResource = null;
+ List<String> resourceNameList = null;
+ for (RangerPolicy rangerPolicy : rangerPolicies) {
+ if (rangerPolicy != null) {
+ rangerPolicyResourceMap = rangerPolicy.getResources();
+ if (rangerPolicyResourceMap != null) {
+ rangerPolicyResource = rangerPolicyResourceMap.get("database");
+ if (rangerPolicyResource != null) {
+ resourceNameList = rangerPolicyResource.getValues();
+ if (CollectionUtils.isNotEmpty(resourceNameList) && resourceNameList.size() == 1) {
+ rangerPoliciesToImport.add(rangerPolicy);
+ }
+ }
+ }
+ }
+ }
+ }
+ return rangerPoliciesToImport;
+ }
+
+ @Override
+ public RangerExportPolicyList importRangerPolicies(RangerExportPolicyList rangerExportPolicyList, String dbName,
+ String baseUrl,
+ String rangerHiveServiceName)
+ throws Exception {
+ String sourceClusterServiceName = null;
+ String serviceMapJsonFileName = "hive_servicemap.json";
+ String rangerPoliciesJsonFileName = "hive_replicationPolicies.json";
+ String uri = RANGER_REST_URL_IMPORTJSONFILE + "&polResource=" + dbName;
+
+ if (rangerExportPolicyList.getPolicies().size() > 0) {
+ sourceClusterServiceName = rangerExportPolicyList.getPolicies().get(0).getService();
+ }
+
+ if (StringUtils.isEmpty(sourceClusterServiceName)) {
+ sourceClusterServiceName = rangerHiveServiceName;
+ }
+
+ Map<String, String> serviceMap = new LinkedHashMap<String, String>();
+ if (!StringUtils.isEmpty(sourceClusterServiceName) && !StringUtils.isEmpty(rangerHiveServiceName)) {
+ serviceMap.put(sourceClusterServiceName, rangerHiveServiceName);
+ }
+
+ Gson gson = new GsonBuilder().create();
+ String jsonServiceMap = gson.toJson(serviceMap);
+
+ String jsonRangerExportPolicyList = gson.toJson(rangerExportPolicyList);
+
+ String url = baseUrl
+ + (uri.startsWith("/") ? uri : ("/" + uri));
+
+ LOG.debug("URL to import policies on target Ranger: {}", url);
+ ClientResponse clientResp = null;
+
+ StreamDataBodyPart filePartPolicies = new StreamDataBodyPart("file",
+ new ByteArrayInputStream(jsonRangerExportPolicyList.getBytes(StandardCharsets.UTF_8)),
+ rangerPoliciesJsonFileName);
+ StreamDataBodyPart filePartServiceMap = new StreamDataBodyPart("servicesMapJson",
+ new ByteArrayInputStream(jsonServiceMap.getBytes(StandardCharsets.UTF_8)), serviceMapJsonFileName);
+
+ FormDataMultiPart formDataMultiPart = new FormDataMultiPart();
+ MultiPart multipartEntity = null;
+ try {
+ multipartEntity = formDataMultiPart.bodyPart(filePartPolicies).bodyPart(filePartServiceMap);
+ WebResource.Builder builder = getRangerResourceBuilder(url);
+ clientResp = builder.accept(MediaType.APPLICATION_JSON).type(MediaType.MULTIPART_FORM_DATA)
+ .post(ClientResponse.class, multipartEntity);
+ if (clientResp != null) {
+ if (clientResp.getStatus() == HttpServletResponse.SC_NO_CONTENT) {
+ LOG.debug("Ranger policy import finished successfully");
+
+ } else if (clientResp.getStatus() == HttpServletResponse.SC_UNAUTHORIZED) {
+ throw new Exception("Authentication Failure while communicating to Ranger admin");
+ } else {
+ throw new Exception("Ranger policy import failed, Please refer target Ranger admin logs.");
+ }
+ }
+ } finally {
+ try {
+ if (filePartPolicies != null) {
+ filePartPolicies.cleanup();
+ }
+ if (filePartServiceMap != null) {
+ filePartServiceMap.cleanup();
+ }
+ if (formDataMultiPart != null) {
+ formDataMultiPart.close();
+ }
+ if (multipartEntity != null) {
+ multipartEntity.close();
+ }
+ } catch (IOException e) {
+ LOG.error("Exception occurred while closing resources: {}", e);
+ }
+ }
+ return rangerExportPolicyList;
+ }
+
+ private synchronized Client getRangerClient() {
+ Client ret = null;
+ ClientConfig config = new DefaultClientConfig();
+ config.getClasses().add(MultiPartWriter.class);
+ config.getProperties().put(ClientConfig.PROPERTY_FOLLOW_REDIRECTS, true);
+ ret = Client.create(config);
+ return ret;
+ }
+
+ @Override
+ public List<RangerPolicy> changeDataSet(List<RangerPolicy> rangerPolicies, String sourceDbName,
+ String targetDbName) {
+ if (sourceDbName.endsWith("/")) {
+ sourceDbName = StringUtils.removePattern(sourceDbName, "/+$");
+ }
+ if (targetDbName.endsWith("/")) {
+ targetDbName = StringUtils.removePattern(targetDbName, "/+$");
+ }
+ if (targetDbName.equals(sourceDbName)) {
+ return rangerPolicies;
+ }
+ if (CollectionUtils.isNotEmpty(rangerPolicies)) {
+ Map<String, RangerPolicy.RangerPolicyResource> rangerPolicyResourceMap = null;
+ RangerPolicy.RangerPolicyResource rangerPolicyResource = null;
+ List<String> resourceNameList = null;
+ for (RangerPolicy rangerPolicy : rangerPolicies) {
+ if (rangerPolicy != null) {
+ rangerPolicyResourceMap = rangerPolicy.getResources();
+ if (rangerPolicyResourceMap != null) {
+ rangerPolicyResource = rangerPolicyResourceMap.get("database");
+ if (rangerPolicyResource != null) {
+ resourceNameList = rangerPolicyResource.getValues();
+ if (CollectionUtils.isNotEmpty(resourceNameList)) {
+ for (int i = 0; i < resourceNameList.size(); i++) {
+ String resourceName = resourceNameList.get(i);
+ if (resourceName.equals(sourceDbName)) {
+ resourceNameList.set(i, targetDbName);
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return rangerPolicies;
+ }
+
+ private Path writeExportedRangerPoliciesToJsonFile(String jsonString, String fileName, Path stagingDirPath)
Review comment:
Make it as retriable command.
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ranger/RangerRestClientImpl.java
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl.ranger;
+
+import com.google.gson.Gson;
+import com.google.gson.GsonBuilder;
+import com.sun.jersey.api.client.Client;
+import com.sun.jersey.api.client.ClientResponse;
+import com.sun.jersey.api.client.WebResource;
+import com.sun.jersey.api.client.config.ClientConfig;
+import com.sun.jersey.api.client.config.DefaultClientConfig;
+import com.sun.jersey.multipart.FormDataMultiPart;
+import com.sun.jersey.multipart.MultiPart;
+import com.sun.jersey.multipart.file.StreamDataBodyPart;
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.eclipse.jetty.util.MultiPartWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.servlet.http.HttpServletResponse;
+import javax.ws.rs.core.MediaType;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.File;
+import java.io.Reader;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.ArrayList;
+
+/**
+ * RangerRestClientImpl to connect to Ranger and export policies.
+ */
+public class RangerRestClientImpl implements RangerRestClient {
+ private static final Logger LOG = LoggerFactory.getLogger(RangerRestClientImpl.class);
+ private static final String RANGER_REST_URL_EXPORTJSONFILE = "/service/plugins/policies/exportJson";
+ private static final String RANGER_REST_URL_IMPORTJSONFILE =
+ "/service/plugins/policies/importPoliciesFromFile?updateIfExists=true";
+
+ public RangerExportPolicyList exportRangerPolicies(String sourceRangerEndpoint,
+ String dbName, String rangerHiveServiceName) throws Exception {
+ LOG.info("Ranger endpoint for cluster " + sourceRangerEndpoint);
+ ClientResponse clientResp = null;
+ String uri = null;
+ if (!StringUtils.isEmpty(rangerHiveServiceName)) {
+ uri = RANGER_REST_URL_EXPORTJSONFILE + "?serviceName=" + rangerHiveServiceName + "&polResource="
+ + dbName + "&resource:database=" + dbName
+ + "&serviceType=hive&resourceMatchScope=self_or_ancestor&resourceMatch=full";
+ }
+ if (sourceRangerEndpoint.endsWith("/")) {
+ sourceRangerEndpoint = StringUtils.removePattern(sourceRangerEndpoint, "/+$");
+ }
+ String url = sourceRangerEndpoint + (uri != null ? (uri.startsWith("/") ? uri : ("/" + uri)) : "");
+ LOG.debug("URL to export policies from source Ranger: {}", url);
+ RangerExportPolicyList rangerExportPolicyList = new RangerExportPolicyList();
+ WebResource.Builder builder = getRangerResourceBuilder(url);
+ clientResp = builder.get(ClientResponse.class);
+
+ String response = null;
+ if (clientResp != null) {
+ if (clientResp.getStatus() == HttpServletResponse.SC_OK) {
+ Gson gson = new GsonBuilder().create();
+ response = clientResp.getEntity(String.class);
+ LOG.debug("Response received for ranger export {} ", response);
+ if (StringUtils.isNotEmpty(response)) {
+ rangerExportPolicyList = gson.fromJson(response, RangerExportPolicyList.class);
+ return rangerExportPolicyList;
+ }
+ } else if (clientResp.getStatus() == HttpServletResponse.SC_NO_CONTENT) {
+ LOG.debug("Ranger policy export request returned empty list");
+ return rangerExportPolicyList;
+ } else if (clientResp.getStatus() == HttpServletResponse.SC_UNAUTHORIZED) {
Review comment:
Should we introduce new exception MetadatReplException or so, and use that during metadata replication?
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ranger/RangerRestClientImpl.java
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl.ranger;
+
+import com.google.gson.Gson;
+import com.google.gson.GsonBuilder;
+import com.sun.jersey.api.client.Client;
+import com.sun.jersey.api.client.ClientResponse;
+import com.sun.jersey.api.client.WebResource;
+import com.sun.jersey.api.client.config.ClientConfig;
+import com.sun.jersey.api.client.config.DefaultClientConfig;
+import com.sun.jersey.multipart.FormDataMultiPart;
+import com.sun.jersey.multipart.MultiPart;
+import com.sun.jersey.multipart.file.StreamDataBodyPart;
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.eclipse.jetty.util.MultiPartWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.servlet.http.HttpServletResponse;
+import javax.ws.rs.core.MediaType;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.File;
+import java.io.Reader;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.ArrayList;
+
+/**
+ * RangerRestClientImpl to connect to Ranger and export policies.
+ */
+public class RangerRestClientImpl implements RangerRestClient {
+ private static final Logger LOG = LoggerFactory.getLogger(RangerRestClientImpl.class);
+ private static final String RANGER_REST_URL_EXPORTJSONFILE = "/service/plugins/policies/exportJson";
+ private static final String RANGER_REST_URL_IMPORTJSONFILE =
+ "/service/plugins/policies/importPoliciesFromFile?updateIfExists=true";
+
+ public RangerExportPolicyList exportRangerPolicies(String sourceRangerEndpoint,
+ String dbName, String rangerHiveServiceName) throws Exception {
+ LOG.info("Ranger endpoint for cluster " + sourceRangerEndpoint);
+ ClientResponse clientResp = null;
+ String uri = null;
+ if (!StringUtils.isEmpty(rangerHiveServiceName)) {
+ uri = RANGER_REST_URL_EXPORTJSONFILE + "?serviceName=" + rangerHiveServiceName + "&polResource="
+ + dbName + "&resource:database=" + dbName
+ + "&serviceType=hive&resourceMatchScope=self_or_ancestor&resourceMatch=full";
+ }
+ if (sourceRangerEndpoint.endsWith("/")) {
+ sourceRangerEndpoint = StringUtils.removePattern(sourceRangerEndpoint, "/+$");
+ }
+ String url = sourceRangerEndpoint + (uri != null ? (uri.startsWith("/") ? uri : ("/" + uri)) : "");
+ LOG.debug("URL to export policies from source Ranger: {}", url);
+ RangerExportPolicyList rangerExportPolicyList = new RangerExportPolicyList();
+ WebResource.Builder builder = getRangerResourceBuilder(url);
+ clientResp = builder.get(ClientResponse.class);
+
+ String response = null;
+ if (clientResp != null) {
+ if (clientResp.getStatus() == HttpServletResponse.SC_OK) {
+ Gson gson = new GsonBuilder().create();
+ response = clientResp.getEntity(String.class);
+ LOG.debug("Response received for ranger export {} ", response);
+ if (StringUtils.isNotEmpty(response)) {
+ rangerExportPolicyList = gson.fromJson(response, RangerExportPolicyList.class);
+ return rangerExportPolicyList;
+ }
+ } else if (clientResp.getStatus() == HttpServletResponse.SC_NO_CONTENT) {
+ LOG.debug("Ranger policy export request returned empty list");
+ return rangerExportPolicyList;
+ } else if (clientResp.getStatus() == HttpServletResponse.SC_UNAUTHORIZED) {
+ throw new Exception("Authentication Failure while communicating to Ranger admin");
+ } else if (clientResp.getStatus() == HttpServletResponse.SC_FORBIDDEN) {
+ throw new Exception("Authorization Failure while communicating to Ranger admin");
+ }
+ }
+ if (StringUtils.isEmpty(response)) {
+ LOG.debug("Ranger policy export request returned empty list or failed, Please refer Ranger admin logs.");
+ }
+ return rangerExportPolicyList;
+ }
+
+ public List<RangerPolicy> removeMultiResourcePolicies(List<RangerPolicy> rangerPolicies) {
+ List<RangerPolicy> rangerPoliciesToImport = new ArrayList<RangerPolicy>();
+ if (CollectionUtils.isNotEmpty(rangerPolicies)) {
+ Map<String, RangerPolicy.RangerPolicyResource> rangerPolicyResourceMap = null;
+ RangerPolicy.RangerPolicyResource rangerPolicyResource = null;
+ List<String> resourceNameList = null;
+ for (RangerPolicy rangerPolicy : rangerPolicies) {
+ if (rangerPolicy != null) {
+ rangerPolicyResourceMap = rangerPolicy.getResources();
+ if (rangerPolicyResourceMap != null) {
+ rangerPolicyResource = rangerPolicyResourceMap.get("database");
+ if (rangerPolicyResource != null) {
+ resourceNameList = rangerPolicyResource.getValues();
+ if (CollectionUtils.isNotEmpty(resourceNameList) && resourceNameList.size() == 1) {
+ rangerPoliciesToImport.add(rangerPolicy);
+ }
+ }
+ }
+ }
+ }
+ }
+ return rangerPoliciesToImport;
+ }
+
+ @Override
+ public RangerExportPolicyList importRangerPolicies(RangerExportPolicyList rangerExportPolicyList, String dbName,
+ String baseUrl,
+ String rangerHiveServiceName)
+ throws Exception {
+ String sourceClusterServiceName = null;
+ String serviceMapJsonFileName = "hive_servicemap.json";
+ String rangerPoliciesJsonFileName = "hive_replicationPolicies.json";
+ String uri = RANGER_REST_URL_IMPORTJSONFILE + "&polResource=" + dbName;
+
+ if (rangerExportPolicyList.getPolicies().size() > 0) {
+ sourceClusterServiceName = rangerExportPolicyList.getPolicies().get(0).getService();
+ }
+
+ if (StringUtils.isEmpty(sourceClusterServiceName)) {
+ sourceClusterServiceName = rangerHiveServiceName;
+ }
+
+ Map<String, String> serviceMap = new LinkedHashMap<String, String>();
+ if (!StringUtils.isEmpty(sourceClusterServiceName) && !StringUtils.isEmpty(rangerHiveServiceName)) {
+ serviceMap.put(sourceClusterServiceName, rangerHiveServiceName);
+ }
+
+ Gson gson = new GsonBuilder().create();
+ String jsonServiceMap = gson.toJson(serviceMap);
+
+ String jsonRangerExportPolicyList = gson.toJson(rangerExportPolicyList);
+
+ String url = baseUrl
+ + (uri.startsWith("/") ? uri : ("/" + uri));
+
+ LOG.debug("URL to import policies on target Ranger: {}", url);
+ ClientResponse clientResp = null;
+
+ StreamDataBodyPart filePartPolicies = new StreamDataBodyPart("file",
+ new ByteArrayInputStream(jsonRangerExportPolicyList.getBytes(StandardCharsets.UTF_8)),
+ rangerPoliciesJsonFileName);
+ StreamDataBodyPart filePartServiceMap = new StreamDataBodyPart("servicesMapJson",
+ new ByteArrayInputStream(jsonServiceMap.getBytes(StandardCharsets.UTF_8)), serviceMapJsonFileName);
+
+ FormDataMultiPart formDataMultiPart = new FormDataMultiPart();
+ MultiPart multipartEntity = null;
+ try {
+ multipartEntity = formDataMultiPart.bodyPart(filePartPolicies).bodyPart(filePartServiceMap);
+ WebResource.Builder builder = getRangerResourceBuilder(url);
+ clientResp = builder.accept(MediaType.APPLICATION_JSON).type(MediaType.MULTIPART_FORM_DATA)
+ .post(ClientResponse.class, multipartEntity);
+ if (clientResp != null) {
+ if (clientResp.getStatus() == HttpServletResponse.SC_NO_CONTENT) {
+ LOG.debug("Ranger policy import finished successfully");
+
+ } else if (clientResp.getStatus() == HttpServletResponse.SC_UNAUTHORIZED) {
+ throw new Exception("Authentication Failure while communicating to Ranger admin");
+ } else {
+ throw new Exception("Ranger policy import failed, Please refer target Ranger admin logs.");
+ }
+ }
+ } finally {
+ try {
+ if (filePartPolicies != null) {
+ filePartPolicies.cleanup();
+ }
+ if (filePartServiceMap != null) {
+ filePartServiceMap.cleanup();
+ }
+ if (formDataMultiPart != null) {
+ formDataMultiPart.close();
+ }
+ if (multipartEntity != null) {
+ multipartEntity.close();
+ }
+ } catch (IOException e) {
+ LOG.error("Exception occurred while closing resources: {}", e);
+ }
+ }
+ return rangerExportPolicyList;
+ }
+
+ private synchronized Client getRangerClient() {
+ Client ret = null;
+ ClientConfig config = new DefaultClientConfig();
+ config.getClasses().add(MultiPartWriter.class);
+ config.getProperties().put(ClientConfig.PROPERTY_FOLLOW_REDIRECTS, true);
+ ret = Client.create(config);
+ return ret;
+ }
+
+ @Override
+ public List<RangerPolicy> changeDataSet(List<RangerPolicy> rangerPolicies, String sourceDbName,
+ String targetDbName) {
+ if (sourceDbName.endsWith("/")) {
+ sourceDbName = StringUtils.removePattern(sourceDbName, "/+$");
+ }
+ if (targetDbName.endsWith("/")) {
+ targetDbName = StringUtils.removePattern(targetDbName, "/+$");
+ }
+ if (targetDbName.equals(sourceDbName)) {
+ return rangerPolicies;
+ }
+ if (CollectionUtils.isNotEmpty(rangerPolicies)) {
+ Map<String, RangerPolicy.RangerPolicyResource> rangerPolicyResourceMap = null;
+ RangerPolicy.RangerPolicyResource rangerPolicyResource = null;
+ List<String> resourceNameList = null;
+ for (RangerPolicy rangerPolicy : rangerPolicies) {
+ if (rangerPolicy != null) {
+ rangerPolicyResourceMap = rangerPolicy.getResources();
+ if (rangerPolicyResourceMap != null) {
+ rangerPolicyResource = rangerPolicyResourceMap.get("database");
+ if (rangerPolicyResource != null) {
+ resourceNameList = rangerPolicyResource.getValues();
+ if (CollectionUtils.isNotEmpty(resourceNameList)) {
+ for (int i = 0; i < resourceNameList.size(); i++) {
+ String resourceName = resourceNameList.get(i);
+ if (resourceName.equals(sourceDbName)) {
+ resourceNameList.set(i, targetDbName);
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return rangerPolicies;
+ }
+
+ private Path writeExportedRangerPoliciesToJsonFile(String jsonString, String fileName, Path stagingDirPath)
+ throws Exception {
+ String filePath = "";
+ Path newPath = null;
+ FSDataOutputStream outStream = null;
+ OutputStreamWriter writer = null;
+ try {
+ if (!StringUtils.isEmpty(jsonString)) {
+ FileSystem fileSystem = getFileSystem(stagingDirPath);
+ if (fileSystem != null) {
+ if (!fileSystem.exists(stagingDirPath)) {
+ fileSystem.mkdirs(stagingDirPath);
+ }
+ newPath = stagingDirPath.suffix(File.separator + fileName);
+ outStream = fileSystem.create(newPath, true);
+ writer = new OutputStreamWriter(outStream, "UTF-8");
+ writer.write(jsonString);
+ }
+ }
+ } catch (IOException ex) {
+ if (newPath != null) {
+ filePath = newPath.toString();
+ }
+ throw new Exception("Failed to write json string to file:" + filePath, ex);
+ } catch (Exception ex) {
+ if (newPath != null) {
+ filePath = newPath.toString();
+ }
+ throw new Exception("Failed to write json string to file:" + filePath, ex);
+ } finally {
+ try {
+ if (writer != null) {
+ writer.close();
+ }
+ if (outStream != null) {
+ outStream.close();
+ }
+ } catch (Exception ex) {
+ LOG.error("Unable to close writer/outStream.", ex);
Review comment:
In other places, for a failure during stream close, we are throwing exception back. Shouldn't we be consistent in this case as well?
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ranger/RangerPolicy.java
##########
@@ -0,0 +1,1513 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl.ranger;
+
+import org.codehaus.jackson.annotate.JsonAutoDetect;
+import org.codehaus.jackson.annotate.JsonAutoDetect.Visibility;
+import org.codehaus.jackson.annotate.JsonIgnoreProperties;
+import org.codehaus.jackson.map.annotate.JsonSerialize;
+
+import javax.xml.bind.annotation.XmlAccessType;
+import javax.xml.bind.annotation.XmlAccessorType;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * RangerPolicy class to contain Ranger Policy details.
+ */
+@JsonAutoDetect(fieldVisibility = Visibility.ANY)
+@JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+@JsonIgnoreProperties(ignoreUnknown = true)
+@XmlRootElement
+@XmlAccessorType(XmlAccessType.FIELD)
+public class RangerPolicy extends RangerBaseModelObject implements java.io.Serializable {
+ public static final int POLICY_TYPE_ACCESS = 0;
+ public static final int POLICY_TYPE_DATAMASK = 1;
+ public static final int POLICY_TYPE_ROWFILTER = 2;
+
+ public static final int[] POLICY_TYPES = new int[]{
+ POLICY_TYPE_ACCESS,
+ POLICY_TYPE_DATAMASK,
+ POLICY_TYPE_ROWFILTER,
+ };
+
+ public static final String MASK_TYPE_NULL = "MASK_NULL";
+ public static final String MASK_TYPE_NONE = "MASK_NONE";
+ public static final String MASK_TYPE_CUSTOM = "CUSTOM";
+
+ private static final long serialVersionUID = 1L;
+
+ private String service;
+ private String name;
+ private Integer policyType;
+ private String description;
+ private String resourceSignature;
+ private Boolean isAuditEnabled;
+ private Map<String, RangerPolicyResource> resources;
+ private List<RangerPolicyItem> policyItems;
+ private List<RangerPolicyItem> denyPolicyItems;
+ private List<RangerPolicyItem> allowExceptions;
+ private List<RangerPolicyItem> denyExceptions;
+ private List<RangerDataMaskPolicyItem> dataMaskPolicyItems;
+ private List<RangerRowFilterPolicyItem> rowFilterPolicyItems;
+
+
+ /**
+ * Ranger Policy default constructor.
+ */
+ public RangerPolicy() {
+ this(null, null, null, null, null, null, null);
+ }
+
+ /**
+ * @param service
+ * @param name
+ * @param policyType
+ * @param description
+ * @param resources
+ * @param policyItems
+ * @param resourceSignature TODO
+ */
+ public RangerPolicy(String service, String name, Integer policyType, String description,
+ Map<String, RangerPolicyResource> resources, List<RangerPolicyItem> policyItems,
+ String resourceSignature) {
+ super();
+ setService(service);
+ setName(name);
+ setPolicyType(policyType);
+ setDescription(description);
+ setResourceSignature(resourceSignature);
+ setIsAuditEnabled(null);
+ setResources(resources);
+ setPolicyItems(policyItems);
+ setDenyPolicyItems(null);
+ setAllowExceptions(null);
+ setDenyExceptions(null);
+ setDataMaskPolicyItems(null);
+ setRowFilterPolicyItems(null);
+ }
+
+ /**
+ * @param other
+ */
+ public void updateFrom(RangerPolicy other) {
+ super.updateFrom(other);
+
+ setService(other.getService());
+ setName(other.getName());
+ setPolicyType(other.getPolicyType());
+ setDescription(other.getDescription());
+ setResourceSignature(other.getResourceSignature());
+ setIsAuditEnabled(other.getIsAuditEnabled());
+ setResources(other.getResources());
+ setPolicyItems(other.getPolicyItems());
+ setDenyPolicyItems(other.getDenyPolicyItems());
+ setAllowExceptions(other.getAllowExceptions());
+ setDenyExceptions(other.getDenyExceptions());
+ setDataMaskPolicyItems(other.getDataMaskPolicyItems());
+ setRowFilterPolicyItems(other.getRowFilterPolicyItems());
+ }
+
+ /**
+ * @return the type
+ */
+ public String getService() {
+ return service;
+ }
+
+ /**
+ * @param service the type to set
+ */
+ public void setService(String service) {
+ this.service = service;
+ }
+
+ /**
+ * @return the name
+ */
+ public String getName() {
+ return name;
+ }
+
+ /**
+ * @param name the name to set
+ */
+ public void setName(String name) {
+ this.name = name;
+ }
+
+ /**
+ * @return the policyType
+ */
+ public Integer getPolicyType() {
+ return policyType;
+ }
+
+ /**
+ * @param policyType the policyType to set
+ */
+ public void setPolicyType(Integer policyType) {
+ this.policyType = policyType;
+ }
+
+ /**
+ * @return the description
+ */
+ public String getDescription() {
+ return description;
+ }
+
+ /**
+ * @param description the description to set
+ */
+ public void setDescription(String description) {
+ this.description = description;
+ }
+
+ /**
+ * @return the resourceSignature
+ */
+ public String getResourceSignature() {
+ return resourceSignature;
+ }
+
+ /**
+ * @param resourceSignature the resourceSignature to set
+ */
+ public void setResourceSignature(String resourceSignature) {
+ this.resourceSignature = resourceSignature;
+ }
+
+ /**
+ * @return the isAuditEnabled
+ */
+ public Boolean getIsAuditEnabled() {
+ return isAuditEnabled;
+ }
+
+ /**
+ * @param isAuditEnabled the isEnabled to set
+ */
+ public void setIsAuditEnabled(Boolean isAuditEnabled) {
+ this.isAuditEnabled = isAuditEnabled == null ? Boolean.TRUE : isAuditEnabled;
+ }
+
+ /**
+ * @return the resources
+ */
+ public Map<String, RangerPolicyResource> getResources() {
+ return resources;
+ }
+
+ /**
+ * @param resources the resources to set
+ */
+ public void setResources(Map<String, RangerPolicyResource> resources) {
+ if (this.resources == null) {
+ this.resources = new HashMap<>();
+ }
+
+ if (this.resources == resources) {
+ return;
+ }
+
+ this.resources.clear();
+
+ if (resources != null) {
+ for (Map.Entry<String, RangerPolicyResource> e : resources.entrySet()) {
+ this.resources.put(e.getKey(), e.getValue());
+ }
+ }
+ }
+
+ /**
+ * @return the policyItems
+ */
+ public List<RangerPolicyItem> getPolicyItems() {
+ return policyItems;
+ }
+
+ /**
+ * @param policyItems the policyItems to set
+ */
+ public void setPolicyItems(List<RangerPolicyItem> policyItems) {
+ if (this.policyItems == null) {
+ this.policyItems = new ArrayList<>();
+ }
+
+ if (this.policyItems == policyItems) {
+ return;
+ }
+
+ this.policyItems.clear();
+
+ if (policyItems != null) {
+ this.policyItems.addAll(policyItems);
+ }
+ }
+
+ /**
+ * @return the denyPolicyItems
+ */
+ public List<RangerPolicyItem> getDenyPolicyItems() {
+ return denyPolicyItems;
+ }
+
+ /**
+ * @param denyPolicyItems the denyPolicyItems to set
+ */
+ public void setDenyPolicyItems(List<RangerPolicyItem> denyPolicyItems) {
+ if (this.denyPolicyItems == null) {
+ this.denyPolicyItems = new ArrayList<>();
+ }
+
+ if (this.denyPolicyItems == denyPolicyItems) {
+ return;
+ }
+
+ this.denyPolicyItems.clear();
+
+ if (denyPolicyItems != null) {
+ this.denyPolicyItems.addAll(denyPolicyItems);
+ }
+ }
+
+ /**
+ * @return the allowExceptions
+ */
+ public List<RangerPolicyItem> getAllowExceptions() {
+ return allowExceptions;
+ }
+
+ /**
+ * @param allowExceptions the allowExceptions to set
+ */
+ public void setAllowExceptions(List<RangerPolicyItem> allowExceptions) {
+ if (this.allowExceptions == null) {
+ this.allowExceptions = new ArrayList<>();
+ }
+
+ if (this.allowExceptions == allowExceptions) {
+ return;
+ }
+
+ this.allowExceptions.clear();
+
+ if (allowExceptions != null) {
+ this.allowExceptions.addAll(allowExceptions);
+ }
+ }
+
+ /**
+ * @return the denyExceptions
+ */
+ public List<RangerPolicyItem> getDenyExceptions() {
+ return denyExceptions;
+ }
+
+ /**
+ * @param denyExceptions the denyExceptions to set
+ */
+ public void setDenyExceptions(List<RangerPolicyItem> denyExceptions) {
+ if (this.denyExceptions == null) {
+ this.denyExceptions = new ArrayList<>();
+ }
+
+ if (this.denyExceptions == denyExceptions) {
+ return;
+ }
+
+ this.denyExceptions.clear();
+
+ if (denyExceptions != null) {
+ this.denyExceptions.addAll(denyExceptions);
+ }
+ }
+
+ public List<RangerDataMaskPolicyItem> getDataMaskPolicyItems() {
+ return dataMaskPolicyItems;
+ }
+
+ public void setDataMaskPolicyItems(List<RangerDataMaskPolicyItem> dataMaskPolicyItems) {
+ if (this.dataMaskPolicyItems == null) {
+ this.dataMaskPolicyItems = new ArrayList<>();
+ }
+
+ if (this.dataMaskPolicyItems == dataMaskPolicyItems) {
+ return;
+ }
+
+ this.dataMaskPolicyItems.clear();
+
+ if (dataMaskPolicyItems != null) {
+ this.dataMaskPolicyItems.addAll(dataMaskPolicyItems);
+ }
+ }
+
+ public List<RangerRowFilterPolicyItem> getRowFilterPolicyItems() {
+ return rowFilterPolicyItems;
+ }
+
+ public void setRowFilterPolicyItems(List<RangerRowFilterPolicyItem> rowFilterPolicyItems) {
+ if (this.rowFilterPolicyItems == null) {
+ this.rowFilterPolicyItems = new ArrayList<>();
+ }
+
+ if (this.rowFilterPolicyItems == rowFilterPolicyItems) {
+ return;
+ }
+
+ this.rowFilterPolicyItems.clear();
+
+ if (rowFilterPolicyItems != null) {
+ this.rowFilterPolicyItems.addAll(rowFilterPolicyItems);
+ }
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ toString(sb);
+ return sb.toString();
+ }
+
+ public StringBuilder toString(StringBuilder sb) {
+ sb.append("RangerPolicy={");
+
+ super.toString(sb);
+
+ sb.append("service={").append(service).append("} ");
+ sb.append("name={").append(name).append("} ");
+ sb.append("policyType={").append(policyType).append("} ");
+ sb.append("description={").append(description).append("} ");
+ sb.append("resourceSignature={").append(resourceSignature).append("} ");
+ sb.append("isAuditEnabled={").append(isAuditEnabled).append("} ");
+
+ sb.append("resources={");
+ if (resources != null) {
+ for (Map.Entry<String, RangerPolicyResource> e : resources.entrySet()) {
+ sb.append(e.getKey()).append("={");
+ e.getValue().toString(sb);
+ sb.append("} ");
+ }
+ }
+ sb.append("} ");
+
+ sb.append("policyItems={");
+ if (policyItems != null) {
+ for (RangerPolicyItem policyItem : policyItems) {
+ if (policyItem != null) {
+ policyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("denyPolicyItems={");
+ if (denyPolicyItems != null) {
+ for (RangerPolicyItem policyItem : denyPolicyItems) {
+ if (policyItem != null) {
+ policyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("allowExceptions={");
+ if (allowExceptions != null) {
+ for (RangerPolicyItem policyItem : allowExceptions) {
+ if (policyItem != null) {
+ policyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("denyExceptions={");
+ if (denyExceptions != null) {
+ for (RangerPolicyItem policyItem : denyExceptions) {
+ if (policyItem != null) {
+ policyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("dataMaskPolicyItems={");
+ if (dataMaskPolicyItems != null) {
+ for (RangerDataMaskPolicyItem dataMaskPolicyItem : dataMaskPolicyItems) {
+ if (dataMaskPolicyItem != null) {
+ dataMaskPolicyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("rowFilterPolicyItems={");
+ if (rowFilterPolicyItems != null) {
+ for (RangerRowFilterPolicyItem rowFilterPolicyItem : rowFilterPolicyItems) {
+ if (rowFilterPolicyItem != null) {
+ rowFilterPolicyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("}");
+
+ return sb;
+ }
+
+ /**
+ * RangerPolicyResource class to store the resource path values.
+ */
+ @JsonAutoDetect(fieldVisibility = Visibility.ANY)
+ @JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+ @JsonIgnoreProperties(ignoreUnknown = true)
+ @XmlRootElement
+ @XmlAccessorType(XmlAccessType.FIELD)
+ public static class RangerPolicyResource implements java.io.Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private List<String> values;
+ private Boolean isExcludes;
+ private Boolean isRecursive;
+
+ public RangerPolicyResource() {
+ this((List<String>) null, null, null);
+ }
+
+ public RangerPolicyResource(String value) {
+ setValue(value);
+ setIsExcludes(null);
+ setIsRecursive(null);
+ }
+
+ public RangerPolicyResource(String value, Boolean isExcludes, Boolean isRecursive) {
+ setValue(value);
+ setIsExcludes(isExcludes);
+ setIsRecursive(isRecursive);
+ }
+
+ public RangerPolicyResource(List<String> values, Boolean isExcludes, Boolean isRecursive) {
+ setValues(values);
+ setIsExcludes(isExcludes);
+ setIsRecursive(isRecursive);
+ }
+
+ /**
+ * @return the values
+ */
+ public List<String> getValues() {
+ return values;
+ }
+
+ /**
+ * @param values the values to set
+ */
+ public void setValues(List<String> values) {
+ if (this.values == null) {
+ this.values = new ArrayList<>();
+ }
+ if (this.values == values) {
+ return;
+ }
+ this.values.clear();
+ if (values != null) {
+ this.values.addAll(values);
+ }
+ }
+
+ /**
+ * @param value the value to set
+ */
+ public void setValue(String value) {
+ if (this.values == null) {
+ this.values = new ArrayList<>();
+ }
+ this.values.clear();
+ this.values.add(value);
+ }
+
+ /**
+ * @return the isExcludes
+ */
+ public Boolean getIsExcludes() {
+ return isExcludes;
+ }
+
+ /**
+ * @param isExcludes the isExcludes to set
+ */
+ public void setIsExcludes(Boolean isExcludes) {
+ this.isExcludes = isExcludes == null ? Boolean.FALSE : isExcludes;
+ }
+
+ /**
+ * @return the isRecursive
+ */
+ public Boolean getIsRecursive() {
+ return isRecursive;
+ }
+
+ /**
+ * @param isRecursive the isRecursive to set
+ */
+ public void setIsRecursive(Boolean isRecursive) {
+ this.isRecursive = isRecursive == null ? Boolean.FALSE : isRecursive;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ toString(sb);
+ return sb.toString();
+ }
+
+ public StringBuilder toString(StringBuilder sb) {
+ sb.append("RangerPolicyResource={");
+ sb.append("values={");
+ if (values != null) {
+ for (String value : values) {
+ sb.append(value).append(" ");
+ }
+ }
+ sb.append("} ");
+ sb.append("isExcludes={").append(isExcludes).append("} ");
+ sb.append("isRecursive={").append(isRecursive).append("} ");
+ sb.append("}");
+
+ return sb;
+ }
+
+ @Override
+ public int hashCode() {
+ final int prime = 31;
+ int result = 1;
+ result = prime * result
+ + ((isExcludes == null) ? 0 : isExcludes.hashCode());
+ result = prime * result
+ + ((isRecursive == null) ? 0 : isRecursive.hashCode());
+ result = prime * result
+ + ((values == null) ? 0 : values.hashCode());
+ return result;
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (this == obj) {
+ return true;
+ }
+ if (obj == null) {
+ return false;
+ }
+ if (getClass() != obj.getClass()) {
+ return false;
+ }
+ RangerPolicyResource other = (RangerPolicyResource) obj;
+ if (isExcludes == null) {
+ if (other.isExcludes != null) {
+ return false;
+ }
+ } else if (!isExcludes.equals(other.isExcludes)) {
+ return false;
+ }
+ if (isRecursive == null) {
+ if (other.isRecursive != null) {
+ return false;
+ }
+ } else if (!isRecursive.equals(other.isRecursive)) {
+ return false;
+ }
+ if (values == null) {
+ if (other.values != null) {
+ return false;
+ }
+ } else if (!values.equals(other.values)) {
+ return false;
+ }
+ return true;
+ }
+ }
+
+ /**
+ * RangerPolicyItem class contains ranger policy items like access and permissions.
+ */
+ @JsonAutoDetect(fieldVisibility = Visibility.ANY)
+ @JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+ @JsonIgnoreProperties(ignoreUnknown = true)
+ @XmlRootElement
+ @XmlAccessorType(XmlAccessType.FIELD)
+ public static class RangerPolicyItem implements java.io.Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private List<RangerPolicyItemAccess> accesses;
+ private List<String> users;
+ private List<String> groups;
+ private List<RangerPolicyItemCondition> conditions;
+ private Boolean delegateAdmin;
+
+ public RangerPolicyItem() {
+ this(null, null, null, null, null);
+ }
+
+ public RangerPolicyItem(List<RangerPolicyItemAccess> accessTypes, List<String> users, List<String> groups,
+ List<RangerPolicyItemCondition> conditions, Boolean delegateAdmin) {
+ setAccesses(accessTypes);
+ setUsers(users);
+ setGroups(groups);
+ setConditions(conditions);
+ setDelegateAdmin(delegateAdmin);
+ }
+
+ /**
+ * @return the accesses
+ */
+ public List<RangerPolicyItemAccess> getAccesses() {
+ return accesses;
+ }
+
+ /**
+ * @param accesses the accesses to set
+ */
+ public void setAccesses(List<RangerPolicyItemAccess> accesses) {
+ if (this.accesses == null) {
+ this.accesses = new ArrayList<>();
+ }
+
+ if (this.accesses == accesses) {
+ return;
+ }
+
+ this.accesses.clear();
+
+ if (accesses != null) {
+ this.accesses.addAll(accesses);
+ }
+ }
+
+ /**
+ * @return the users
+ */
+ public List<String> getUsers() {
+ return users;
+ }
+
+ /**
+ * @param users the users to set
+ */
+ public void setUsers(List<String> users) {
+ if (this.users == null) {
+ this.users = new ArrayList<>();
+ }
+
+ if (this.users == users) {
+ return;
+ }
+
+ this.users.clear();
+
+ if (users != null) {
+ this.users.addAll(users);
+ }
+ }
+
+ /**
+ * @return the groups
+ */
+ public List<String> getGroups() {
+ return groups;
+ }
+
+ /**
+ * @param groups the groups to set
+ */
+ public void setGroups(List<String> groups) {
+ if (this.groups == null) {
+ this.groups = new ArrayList<>();
+ }
+ if (this.groups == groups) {
+ return;
+ }
+ this.groups.clear();
+ if (groups != null) {
+ this.groups.addAll(groups);
+ }
+ }
+
+ /**
+ * @return the conditions
+ */
+ public List<RangerPolicyItemCondition> getConditions() {
+ return conditions;
+ }
+
+ /**
+ * @param conditions the conditions to set
+ */
+ public void setConditions(List<RangerPolicyItemCondition> conditions) {
+ if (this.conditions == null) {
+ this.conditions = new ArrayList<>();
+ }
+ if (this.conditions == conditions) {
+ return;
+ }
+ this.conditions.clear();
+ if (conditions != null) {
+ this.conditions.addAll(conditions);
+ }
+ }
+
+ /**
+ * @return the delegateAdmin
+ */
+ public Boolean getDelegateAdmin() {
+ return delegateAdmin;
+ }
+
+ /**
+ * @param delegateAdmin the delegateAdmin to set
+ */
+ public void setDelegateAdmin(Boolean delegateAdmin) {
+ this.delegateAdmin = delegateAdmin == null ? Boolean.FALSE : delegateAdmin;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ toString(sb);
+ return sb.toString();
+ }
+
+ public StringBuilder toString(StringBuilder sb) {
+ sb.append("RangerPolicyItem={");
+ sb.append("accessTypes={");
+ if (accesses != null) {
+ for (RangerPolicyItemAccess access : accesses) {
+ if (access != null) {
+ access.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+ sb.append("users={");
+ if (users != null) {
+ for (String user : users) {
+ if (user != null) {
+ sb.append(user).append(" ");
+ }
+ }
+ }
+ sb.append("} ");
+ sb.append("groups={");
+ if (groups != null) {
+ for (String group : groups) {
+ if (group != null) {
+ sb.append(group).append(" ");
+ }
+ }
+ }
+ sb.append("} ");
+ sb.append("conditions={");
+ if (conditions != null) {
+ for (RangerPolicyItemCondition condition : conditions) {
+ if (condition != null) {
+ condition.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+ sb.append("delegateAdmin={").append(delegateAdmin).append("} ");
+ sb.append("}");
+
+ return sb;
+ }
+
+ @Override
+ public int hashCode() {
+ final int prime = 31;
+ int result = 1;
+ result = prime * result
+ + ((accesses == null) ? 0 : accesses.hashCode());
+ result = prime * result
+ + ((conditions == null) ? 0 : conditions.hashCode());
+ result = prime * result
+ + ((delegateAdmin == null) ? 0 : delegateAdmin.hashCode());
+ result = prime * result
+ + ((groups == null) ? 0 : groups.hashCode());
+ result = prime * result + ((users == null) ? 0 : users.hashCode());
+ return result;
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (this == obj) {
+ return true;
+ }
+ if (obj == null) {
+ return false;
+ }
+ if (getClass() != obj.getClass()) {
+ return false;
+ }
+ RangerPolicyItem other = (RangerPolicyItem) obj;
+ if (accesses == null) {
+ if (other.accesses != null) {
+ return false;
+ }
+ } else if (!accesses.equals(other.accesses)) {
+ return false;
+ }
+ if (conditions == null) {
+ if (other.conditions != null) {
+ return false;
+ }
+ } else if (!conditions.equals(other.conditions)) {
+ return false;
+ }
+ if (delegateAdmin == null) {
+ if (other.delegateAdmin != null) {
+ return false;
+ }
+ } else if (!delegateAdmin.equals(other.delegateAdmin)) {
+ return false;
+ }
+ if (groups == null) {
+ if (other.groups != null) {
+ return false;
+ }
+ } else if (!groups.equals(other.groups)) {
+ return false;
+ }
+ if (users == null) {
+ if (other.users != null) {
+ return false;
+ }
+ } else if (!users.equals(other.users)) {
+ return false;
+ }
+ return true;
+
+ }
+ }
+
+ /**
+ * RangerDataMaskPolicyItem class.
+ */
+ @JsonAutoDetect(fieldVisibility = Visibility.ANY)
+ @JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+ @JsonIgnoreProperties(ignoreUnknown = true)
+ @XmlRootElement
+ @XmlAccessorType(XmlAccessType.FIELD)
+ public static class RangerDataMaskPolicyItem extends RangerPolicyItem implements java.io.Serializable {
+ private static final long serialVersionUID = 1L;
+ private RangerPolicyItemDataMaskInfo dataMaskInfo;
+
+ public RangerDataMaskPolicyItem() {
+ this(null, null, null, null, null, null);
+ }
+
+ public RangerDataMaskPolicyItem(List<RangerPolicyItemAccess> accesses,
+ RangerPolicyItemDataMaskInfo dataMaskDetail, List<String> users,
+ List<String> groups,
+ List<RangerPolicyItemCondition> conditions, Boolean delegateAdmin) {
+ super(accesses, users, groups, conditions, delegateAdmin);
+ setDataMaskInfo(dataMaskDetail);
+ }
+
+ /**
+ * @return the dataMaskInfo
+ */
+ public RangerPolicyItemDataMaskInfo getDataMaskInfo() {
+ return dataMaskInfo;
+ }
+
+ /**
+ * @param dataMaskInfo the dataMaskInfo to set
+ */
+ public void setDataMaskInfo(RangerPolicyItemDataMaskInfo dataMaskInfo) {
+ this.dataMaskInfo = dataMaskInfo == null ? new RangerPolicyItemDataMaskInfo() : dataMaskInfo;
+ }
+
+ @Override
+ public int hashCode() {
+ final int prime = 31;
+ int result = super.hashCode();
+ result = prime * result + ((dataMaskInfo == null) ? 0 : dataMaskInfo.hashCode());
+ return result;
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (!super.equals(obj)) {
+ return false;
+ }
+ if (this == obj) {
+ return true;
+ }
+ if (obj == null) {
+ return false;
+ }
+ if (getClass() != obj.getClass()) {
+ return false;
+ }
+ RangerDataMaskPolicyItem other = (RangerDataMaskPolicyItem) obj;
+ if (dataMaskInfo == null) {
+ if (other.dataMaskInfo != null) {
+ return false;
+ }
+ } else if (!dataMaskInfo.equals(other.dataMaskInfo)) {
+ return false;
+ }
+ return true;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ toString(sb);
+ return sb.toString();
+ }
+
+ public StringBuilder toString(StringBuilder sb) {
+ sb.append("RangerDataMaskPolicyItem={");
+ super.toString(sb);
+ sb.append("dataMaskInfo={");
+ if (dataMaskInfo != null) {
+ dataMaskInfo.toString(sb);
+ }
+ sb.append("} ");
+ sb.append("}");
+ return sb;
+ }
+ }
+
+ /**
+ * RangerRowFilterPolicyItem class.
+ */
+ @JsonAutoDetect(fieldVisibility = Visibility.ANY)
+ @JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+ @JsonIgnoreProperties(ignoreUnknown = true)
+ @XmlRootElement
+ @XmlAccessorType(XmlAccessType.FIELD)
+ public static class RangerRowFilterPolicyItem extends RangerPolicyItem implements java.io.Serializable {
+ private static final long serialVersionUID = 1L;
+ private RangerPolicyItemRowFilterInfo rowFilterInfo;
+
+ public RangerRowFilterPolicyItem() {
Review comment:
Unused? Remove it.
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ranger/RangerPolicy.java
##########
@@ -0,0 +1,1513 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl.ranger;
+
+import org.codehaus.jackson.annotate.JsonAutoDetect;
+import org.codehaus.jackson.annotate.JsonAutoDetect.Visibility;
+import org.codehaus.jackson.annotate.JsonIgnoreProperties;
+import org.codehaus.jackson.map.annotate.JsonSerialize;
+
+import javax.xml.bind.annotation.XmlAccessType;
+import javax.xml.bind.annotation.XmlAccessorType;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * RangerPolicy class to contain Ranger Policy details.
+ */
+@JsonAutoDetect(fieldVisibility = Visibility.ANY)
+@JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+@JsonIgnoreProperties(ignoreUnknown = true)
+@XmlRootElement
+@XmlAccessorType(XmlAccessType.FIELD)
+public class RangerPolicy extends RangerBaseModelObject implements java.io.Serializable {
+ public static final int POLICY_TYPE_ACCESS = 0;
+ public static final int POLICY_TYPE_DATAMASK = 1;
+ public static final int POLICY_TYPE_ROWFILTER = 2;
+
+ public static final int[] POLICY_TYPES = new int[]{
+ POLICY_TYPE_ACCESS,
+ POLICY_TYPE_DATAMASK,
+ POLICY_TYPE_ROWFILTER,
+ };
+
+ public static final String MASK_TYPE_NULL = "MASK_NULL";
+ public static final String MASK_TYPE_NONE = "MASK_NONE";
+ public static final String MASK_TYPE_CUSTOM = "CUSTOM";
+
+ private static final long serialVersionUID = 1L;
+
+ private String service;
+ private String name;
+ private Integer policyType;
+ private String description;
+ private String resourceSignature;
+ private Boolean isAuditEnabled;
+ private Map<String, RangerPolicyResource> resources;
+ private List<RangerPolicyItem> policyItems;
+ private List<RangerPolicyItem> denyPolicyItems;
+ private List<RangerPolicyItem> allowExceptions;
+ private List<RangerPolicyItem> denyExceptions;
+ private List<RangerDataMaskPolicyItem> dataMaskPolicyItems;
+ private List<RangerRowFilterPolicyItem> rowFilterPolicyItems;
+
+
+ /**
+ * Ranger Policy default constructor.
+ */
+ public RangerPolicy() {
+ this(null, null, null, null, null, null, null);
+ }
+
+ /**
+ * @param service
+ * @param name
+ * @param policyType
+ * @param description
+ * @param resources
+ * @param policyItems
+ * @param resourceSignature TODO
+ */
+ public RangerPolicy(String service, String name, Integer policyType, String description,
+ Map<String, RangerPolicyResource> resources, List<RangerPolicyItem> policyItems,
+ String resourceSignature) {
+ super();
+ setService(service);
+ setName(name);
+ setPolicyType(policyType);
+ setDescription(description);
+ setResourceSignature(resourceSignature);
+ setIsAuditEnabled(null);
+ setResources(resources);
+ setPolicyItems(policyItems);
+ setDenyPolicyItems(null);
+ setAllowExceptions(null);
+ setDenyExceptions(null);
+ setDataMaskPolicyItems(null);
+ setRowFilterPolicyItems(null);
+ }
+
+ /**
+ * @param other
+ */
+ public void updateFrom(RangerPolicy other) {
+ super.updateFrom(other);
+
+ setService(other.getService());
+ setName(other.getName());
+ setPolicyType(other.getPolicyType());
+ setDescription(other.getDescription());
+ setResourceSignature(other.getResourceSignature());
+ setIsAuditEnabled(other.getIsAuditEnabled());
+ setResources(other.getResources());
+ setPolicyItems(other.getPolicyItems());
+ setDenyPolicyItems(other.getDenyPolicyItems());
+ setAllowExceptions(other.getAllowExceptions());
+ setDenyExceptions(other.getDenyExceptions());
+ setDataMaskPolicyItems(other.getDataMaskPolicyItems());
+ setRowFilterPolicyItems(other.getRowFilterPolicyItems());
+ }
+
+ /**
+ * @return the type
+ */
+ public String getService() {
+ return service;
+ }
+
+ /**
+ * @param service the type to set
+ */
+ public void setService(String service) {
+ this.service = service;
+ }
+
+ /**
+ * @return the name
+ */
+ public String getName() {
+ return name;
+ }
+
+ /**
+ * @param name the name to set
+ */
+ public void setName(String name) {
+ this.name = name;
+ }
+
+ /**
+ * @return the policyType
+ */
+ public Integer getPolicyType() {
+ return policyType;
+ }
+
+ /**
+ * @param policyType the policyType to set
+ */
+ public void setPolicyType(Integer policyType) {
+ this.policyType = policyType;
+ }
+
+ /**
+ * @return the description
+ */
+ public String getDescription() {
+ return description;
+ }
+
+ /**
+ * @param description the description to set
+ */
+ public void setDescription(String description) {
+ this.description = description;
+ }
+
+ /**
+ * @return the resourceSignature
+ */
+ public String getResourceSignature() {
+ return resourceSignature;
+ }
+
+ /**
+ * @param resourceSignature the resourceSignature to set
+ */
+ public void setResourceSignature(String resourceSignature) {
+ this.resourceSignature = resourceSignature;
+ }
+
+ /**
+ * @return the isAuditEnabled
+ */
+ public Boolean getIsAuditEnabled() {
+ return isAuditEnabled;
+ }
+
+ /**
+ * @param isAuditEnabled the isEnabled to set
+ */
+ public void setIsAuditEnabled(Boolean isAuditEnabled) {
+ this.isAuditEnabled = isAuditEnabled == null ? Boolean.TRUE : isAuditEnabled;
+ }
+
+ /**
+ * @return the resources
+ */
+ public Map<String, RangerPolicyResource> getResources() {
+ return resources;
+ }
+
+ /**
+ * @param resources the resources to set
+ */
+ public void setResources(Map<String, RangerPolicyResource> resources) {
+ if (this.resources == null) {
+ this.resources = new HashMap<>();
+ }
+
+ if (this.resources == resources) {
+ return;
+ }
+
+ this.resources.clear();
+
+ if (resources != null) {
+ for (Map.Entry<String, RangerPolicyResource> e : resources.entrySet()) {
+ this.resources.put(e.getKey(), e.getValue());
+ }
+ }
+ }
+
+ /**
+ * @return the policyItems
+ */
+ public List<RangerPolicyItem> getPolicyItems() {
+ return policyItems;
+ }
+
+ /**
+ * @param policyItems the policyItems to set
+ */
+ public void setPolicyItems(List<RangerPolicyItem> policyItems) {
+ if (this.policyItems == null) {
+ this.policyItems = new ArrayList<>();
+ }
+
+ if (this.policyItems == policyItems) {
+ return;
+ }
+
+ this.policyItems.clear();
+
+ if (policyItems != null) {
+ this.policyItems.addAll(policyItems);
+ }
+ }
+
+ /**
+ * @return the denyPolicyItems
+ */
+ public List<RangerPolicyItem> getDenyPolicyItems() {
+ return denyPolicyItems;
+ }
+
+ /**
+ * @param denyPolicyItems the denyPolicyItems to set
+ */
+ public void setDenyPolicyItems(List<RangerPolicyItem> denyPolicyItems) {
+ if (this.denyPolicyItems == null) {
+ this.denyPolicyItems = new ArrayList<>();
+ }
+
+ if (this.denyPolicyItems == denyPolicyItems) {
+ return;
+ }
+
+ this.denyPolicyItems.clear();
+
+ if (denyPolicyItems != null) {
+ this.denyPolicyItems.addAll(denyPolicyItems);
+ }
+ }
+
+ /**
+ * @return the allowExceptions
+ */
+ public List<RangerPolicyItem> getAllowExceptions() {
+ return allowExceptions;
+ }
+
+ /**
+ * @param allowExceptions the allowExceptions to set
+ */
+ public void setAllowExceptions(List<RangerPolicyItem> allowExceptions) {
+ if (this.allowExceptions == null) {
+ this.allowExceptions = new ArrayList<>();
+ }
+
+ if (this.allowExceptions == allowExceptions) {
+ return;
+ }
+
+ this.allowExceptions.clear();
+
+ if (allowExceptions != null) {
+ this.allowExceptions.addAll(allowExceptions);
+ }
+ }
+
+ /**
+ * @return the denyExceptions
+ */
+ public List<RangerPolicyItem> getDenyExceptions() {
+ return denyExceptions;
+ }
+
+ /**
+ * @param denyExceptions the denyExceptions to set
+ */
+ public void setDenyExceptions(List<RangerPolicyItem> denyExceptions) {
+ if (this.denyExceptions == null) {
+ this.denyExceptions = new ArrayList<>();
+ }
+
+ if (this.denyExceptions == denyExceptions) {
+ return;
+ }
+
+ this.denyExceptions.clear();
+
+ if (denyExceptions != null) {
+ this.denyExceptions.addAll(denyExceptions);
+ }
+ }
+
+ public List<RangerDataMaskPolicyItem> getDataMaskPolicyItems() {
+ return dataMaskPolicyItems;
+ }
+
+ public void setDataMaskPolicyItems(List<RangerDataMaskPolicyItem> dataMaskPolicyItems) {
+ if (this.dataMaskPolicyItems == null) {
+ this.dataMaskPolicyItems = new ArrayList<>();
+ }
+
+ if (this.dataMaskPolicyItems == dataMaskPolicyItems) {
+ return;
+ }
+
+ this.dataMaskPolicyItems.clear();
+
+ if (dataMaskPolicyItems != null) {
+ this.dataMaskPolicyItems.addAll(dataMaskPolicyItems);
+ }
+ }
+
+ public List<RangerRowFilterPolicyItem> getRowFilterPolicyItems() {
+ return rowFilterPolicyItems;
+ }
+
+ public void setRowFilterPolicyItems(List<RangerRowFilterPolicyItem> rowFilterPolicyItems) {
+ if (this.rowFilterPolicyItems == null) {
+ this.rowFilterPolicyItems = new ArrayList<>();
+ }
+
+ if (this.rowFilterPolicyItems == rowFilterPolicyItems) {
+ return;
+ }
+
+ this.rowFilterPolicyItems.clear();
+
+ if (rowFilterPolicyItems != null) {
+ this.rowFilterPolicyItems.addAll(rowFilterPolicyItems);
+ }
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ toString(sb);
+ return sb.toString();
+ }
+
+ public StringBuilder toString(StringBuilder sb) {
+ sb.append("RangerPolicy={");
+
+ super.toString(sb);
+
+ sb.append("service={").append(service).append("} ");
+ sb.append("name={").append(name).append("} ");
+ sb.append("policyType={").append(policyType).append("} ");
+ sb.append("description={").append(description).append("} ");
+ sb.append("resourceSignature={").append(resourceSignature).append("} ");
+ sb.append("isAuditEnabled={").append(isAuditEnabled).append("} ");
+
+ sb.append("resources={");
+ if (resources != null) {
+ for (Map.Entry<String, RangerPolicyResource> e : resources.entrySet()) {
+ sb.append(e.getKey()).append("={");
+ e.getValue().toString(sb);
+ sb.append("} ");
+ }
+ }
+ sb.append("} ");
+
+ sb.append("policyItems={");
+ if (policyItems != null) {
+ for (RangerPolicyItem policyItem : policyItems) {
+ if (policyItem != null) {
+ policyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("denyPolicyItems={");
+ if (denyPolicyItems != null) {
+ for (RangerPolicyItem policyItem : denyPolicyItems) {
+ if (policyItem != null) {
+ policyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("allowExceptions={");
+ if (allowExceptions != null) {
+ for (RangerPolicyItem policyItem : allowExceptions) {
+ if (policyItem != null) {
+ policyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("denyExceptions={");
+ if (denyExceptions != null) {
+ for (RangerPolicyItem policyItem : denyExceptions) {
+ if (policyItem != null) {
+ policyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("dataMaskPolicyItems={");
+ if (dataMaskPolicyItems != null) {
+ for (RangerDataMaskPolicyItem dataMaskPolicyItem : dataMaskPolicyItems) {
+ if (dataMaskPolicyItem != null) {
+ dataMaskPolicyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("rowFilterPolicyItems={");
+ if (rowFilterPolicyItems != null) {
+ for (RangerRowFilterPolicyItem rowFilterPolicyItem : rowFilterPolicyItems) {
+ if (rowFilterPolicyItem != null) {
+ rowFilterPolicyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("}");
+
+ return sb;
+ }
+
+ /**
+ * RangerPolicyResource class to store the resource path values.
+ */
+ @JsonAutoDetect(fieldVisibility = Visibility.ANY)
+ @JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+ @JsonIgnoreProperties(ignoreUnknown = true)
+ @XmlRootElement
+ @XmlAccessorType(XmlAccessType.FIELD)
+ public static class RangerPolicyResource implements java.io.Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private List<String> values;
+ private Boolean isExcludes;
+ private Boolean isRecursive;
+
+ public RangerPolicyResource() {
+ this((List<String>) null, null, null);
+ }
+
+ public RangerPolicyResource(String value) {
+ setValue(value);
+ setIsExcludes(null);
+ setIsRecursive(null);
+ }
+
+ public RangerPolicyResource(String value, Boolean isExcludes, Boolean isRecursive) {
+ setValue(value);
+ setIsExcludes(isExcludes);
+ setIsRecursive(isRecursive);
+ }
+
+ public RangerPolicyResource(List<String> values, Boolean isExcludes, Boolean isRecursive) {
+ setValues(values);
+ setIsExcludes(isExcludes);
+ setIsRecursive(isRecursive);
+ }
+
+ /**
+ * @return the values
+ */
+ public List<String> getValues() {
+ return values;
+ }
+
+ /**
+ * @param values the values to set
+ */
+ public void setValues(List<String> values) {
+ if (this.values == null) {
+ this.values = new ArrayList<>();
+ }
+ if (this.values == values) {
+ return;
+ }
+ this.values.clear();
+ if (values != null) {
+ this.values.addAll(values);
+ }
+ }
+
+ /**
+ * @param value the value to set
+ */
+ public void setValue(String value) {
+ if (this.values == null) {
+ this.values = new ArrayList<>();
+ }
+ this.values.clear();
+ this.values.add(value);
+ }
+
+ /**
+ * @return the isExcludes
+ */
+ public Boolean getIsExcludes() {
+ return isExcludes;
+ }
+
+ /**
+ * @param isExcludes the isExcludes to set
+ */
+ public void setIsExcludes(Boolean isExcludes) {
+ this.isExcludes = isExcludes == null ? Boolean.FALSE : isExcludes;
+ }
+
+ /**
+ * @return the isRecursive
+ */
+ public Boolean getIsRecursive() {
+ return isRecursive;
+ }
+
+ /**
+ * @param isRecursive the isRecursive to set
+ */
+ public void setIsRecursive(Boolean isRecursive) {
+ this.isRecursive = isRecursive == null ? Boolean.FALSE : isRecursive;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ toString(sb);
+ return sb.toString();
+ }
+
+ public StringBuilder toString(StringBuilder sb) {
+ sb.append("RangerPolicyResource={");
+ sb.append("values={");
+ if (values != null) {
+ for (String value : values) {
+ sb.append(value).append(" ");
+ }
+ }
+ sb.append("} ");
+ sb.append("isExcludes={").append(isExcludes).append("} ");
+ sb.append("isRecursive={").append(isRecursive).append("} ");
+ sb.append("}");
+
+ return sb;
+ }
+
+ @Override
+ public int hashCode() {
+ final int prime = 31;
+ int result = 1;
+ result = prime * result
+ + ((isExcludes == null) ? 0 : isExcludes.hashCode());
+ result = prime * result
+ + ((isRecursive == null) ? 0 : isRecursive.hashCode());
+ result = prime * result
+ + ((values == null) ? 0 : values.hashCode());
+ return result;
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (this == obj) {
+ return true;
+ }
+ if (obj == null) {
+ return false;
+ }
+ if (getClass() != obj.getClass()) {
+ return false;
+ }
+ RangerPolicyResource other = (RangerPolicyResource) obj;
+ if (isExcludes == null) {
+ if (other.isExcludes != null) {
+ return false;
+ }
+ } else if (!isExcludes.equals(other.isExcludes)) {
+ return false;
+ }
+ if (isRecursive == null) {
+ if (other.isRecursive != null) {
+ return false;
+ }
+ } else if (!isRecursive.equals(other.isRecursive)) {
+ return false;
+ }
+ if (values == null) {
+ if (other.values != null) {
+ return false;
+ }
+ } else if (!values.equals(other.values)) {
+ return false;
+ }
+ return true;
+ }
+ }
+
+ /**
+ * RangerPolicyItem class contains ranger policy items like access and permissions.
+ */
+ @JsonAutoDetect(fieldVisibility = Visibility.ANY)
+ @JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+ @JsonIgnoreProperties(ignoreUnknown = true)
+ @XmlRootElement
+ @XmlAccessorType(XmlAccessType.FIELD)
+ public static class RangerPolicyItem implements java.io.Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private List<RangerPolicyItemAccess> accesses;
+ private List<String> users;
+ private List<String> groups;
+ private List<RangerPolicyItemCondition> conditions;
+ private Boolean delegateAdmin;
+
+ public RangerPolicyItem() {
+ this(null, null, null, null, null);
+ }
+
+ public RangerPolicyItem(List<RangerPolicyItemAccess> accessTypes, List<String> users, List<String> groups,
+ List<RangerPolicyItemCondition> conditions, Boolean delegateAdmin) {
+ setAccesses(accessTypes);
+ setUsers(users);
+ setGroups(groups);
+ setConditions(conditions);
+ setDelegateAdmin(delegateAdmin);
+ }
+
+ /**
+ * @return the accesses
+ */
+ public List<RangerPolicyItemAccess> getAccesses() {
+ return accesses;
+ }
+
+ /**
+ * @param accesses the accesses to set
+ */
+ public void setAccesses(List<RangerPolicyItemAccess> accesses) {
+ if (this.accesses == null) {
+ this.accesses = new ArrayList<>();
+ }
+
+ if (this.accesses == accesses) {
+ return;
+ }
+
+ this.accesses.clear();
+
+ if (accesses != null) {
+ this.accesses.addAll(accesses);
+ }
+ }
+
+ /**
+ * @return the users
+ */
+ public List<String> getUsers() {
+ return users;
+ }
+
+ /**
+ * @param users the users to set
+ */
+ public void setUsers(List<String> users) {
+ if (this.users == null) {
+ this.users = new ArrayList<>();
+ }
+
+ if (this.users == users) {
+ return;
+ }
+
+ this.users.clear();
+
+ if (users != null) {
+ this.users.addAll(users);
+ }
+ }
+
+ /**
+ * @return the groups
+ */
+ public List<String> getGroups() {
+ return groups;
+ }
+
+ /**
+ * @param groups the groups to set
+ */
+ public void setGroups(List<String> groups) {
+ if (this.groups == null) {
+ this.groups = new ArrayList<>();
+ }
+ if (this.groups == groups) {
+ return;
+ }
+ this.groups.clear();
+ if (groups != null) {
+ this.groups.addAll(groups);
+ }
+ }
+
+ /**
+ * @return the conditions
+ */
+ public List<RangerPolicyItemCondition> getConditions() {
+ return conditions;
+ }
+
+ /**
+ * @param conditions the conditions to set
+ */
+ public void setConditions(List<RangerPolicyItemCondition> conditions) {
+ if (this.conditions == null) {
+ this.conditions = new ArrayList<>();
+ }
+ if (this.conditions == conditions) {
+ return;
+ }
+ this.conditions.clear();
+ if (conditions != null) {
+ this.conditions.addAll(conditions);
+ }
+ }
+
+ /**
+ * @return the delegateAdmin
+ */
+ public Boolean getDelegateAdmin() {
+ return delegateAdmin;
+ }
+
+ /**
+ * @param delegateAdmin the delegateAdmin to set
+ */
+ public void setDelegateAdmin(Boolean delegateAdmin) {
+ this.delegateAdmin = delegateAdmin == null ? Boolean.FALSE : delegateAdmin;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ toString(sb);
+ return sb.toString();
+ }
+
+ public StringBuilder toString(StringBuilder sb) {
+ sb.append("RangerPolicyItem={");
+ sb.append("accessTypes={");
+ if (accesses != null) {
+ for (RangerPolicyItemAccess access : accesses) {
+ if (access != null) {
+ access.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+ sb.append("users={");
+ if (users != null) {
+ for (String user : users) {
+ if (user != null) {
+ sb.append(user).append(" ");
+ }
+ }
+ }
+ sb.append("} ");
+ sb.append("groups={");
+ if (groups != null) {
+ for (String group : groups) {
+ if (group != null) {
+ sb.append(group).append(" ");
+ }
+ }
+ }
+ sb.append("} ");
+ sb.append("conditions={");
+ if (conditions != null) {
+ for (RangerPolicyItemCondition condition : conditions) {
+ if (condition != null) {
+ condition.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+ sb.append("delegateAdmin={").append(delegateAdmin).append("} ");
+ sb.append("}");
+
+ return sb;
+ }
+
+ @Override
+ public int hashCode() {
+ final int prime = 31;
+ int result = 1;
+ result = prime * result
+ + ((accesses == null) ? 0 : accesses.hashCode());
+ result = prime * result
+ + ((conditions == null) ? 0 : conditions.hashCode());
+ result = prime * result
+ + ((delegateAdmin == null) ? 0 : delegateAdmin.hashCode());
+ result = prime * result
+ + ((groups == null) ? 0 : groups.hashCode());
+ result = prime * result + ((users == null) ? 0 : users.hashCode());
+ return result;
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (this == obj) {
+ return true;
+ }
+ if (obj == null) {
+ return false;
+ }
+ if (getClass() != obj.getClass()) {
+ return false;
+ }
+ RangerPolicyItem other = (RangerPolicyItem) obj;
+ if (accesses == null) {
+ if (other.accesses != null) {
+ return false;
+ }
+ } else if (!accesses.equals(other.accesses)) {
+ return false;
+ }
+ if (conditions == null) {
+ if (other.conditions != null) {
+ return false;
+ }
+ } else if (!conditions.equals(other.conditions)) {
+ return false;
+ }
+ if (delegateAdmin == null) {
+ if (other.delegateAdmin != null) {
+ return false;
+ }
+ } else if (!delegateAdmin.equals(other.delegateAdmin)) {
+ return false;
+ }
+ if (groups == null) {
+ if (other.groups != null) {
+ return false;
+ }
+ } else if (!groups.equals(other.groups)) {
+ return false;
+ }
+ if (users == null) {
+ if (other.users != null) {
+ return false;
+ }
+ } else if (!users.equals(other.users)) {
+ return false;
+ }
+ return true;
+
+ }
+ }
+
+ /**
+ * RangerDataMaskPolicyItem class.
+ */
+ @JsonAutoDetect(fieldVisibility = Visibility.ANY)
+ @JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+ @JsonIgnoreProperties(ignoreUnknown = true)
+ @XmlRootElement
+ @XmlAccessorType(XmlAccessType.FIELD)
+ public static class RangerDataMaskPolicyItem extends RangerPolicyItem implements java.io.Serializable {
+ private static final long serialVersionUID = 1L;
+ private RangerPolicyItemDataMaskInfo dataMaskInfo;
+
+ public RangerDataMaskPolicyItem() {
+ this(null, null, null, null, null, null);
+ }
+
+ public RangerDataMaskPolicyItem(List<RangerPolicyItemAccess> accesses,
+ RangerPolicyItemDataMaskInfo dataMaskDetail, List<String> users,
+ List<String> groups,
+ List<RangerPolicyItemCondition> conditions, Boolean delegateAdmin) {
+ super(accesses, users, groups, conditions, delegateAdmin);
+ setDataMaskInfo(dataMaskDetail);
+ }
+
+ /**
+ * @return the dataMaskInfo
+ */
+ public RangerPolicyItemDataMaskInfo getDataMaskInfo() {
+ return dataMaskInfo;
+ }
+
+ /**
+ * @param dataMaskInfo the dataMaskInfo to set
+ */
+ public void setDataMaskInfo(RangerPolicyItemDataMaskInfo dataMaskInfo) {
+ this.dataMaskInfo = dataMaskInfo == null ? new RangerPolicyItemDataMaskInfo() : dataMaskInfo;
+ }
+
+ @Override
+ public int hashCode() {
+ final int prime = 31;
+ int result = super.hashCode();
+ result = prime * result + ((dataMaskInfo == null) ? 0 : dataMaskInfo.hashCode());
+ return result;
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (!super.equals(obj)) {
+ return false;
+ }
+ if (this == obj) {
+ return true;
+ }
+ if (obj == null) {
+ return false;
+ }
+ if (getClass() != obj.getClass()) {
+ return false;
+ }
+ RangerDataMaskPolicyItem other = (RangerDataMaskPolicyItem) obj;
+ if (dataMaskInfo == null) {
+ if (other.dataMaskInfo != null) {
+ return false;
+ }
+ } else if (!dataMaskInfo.equals(other.dataMaskInfo)) {
+ return false;
+ }
+ return true;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ toString(sb);
+ return sb.toString();
+ }
+
+ public StringBuilder toString(StringBuilder sb) {
+ sb.append("RangerDataMaskPolicyItem={");
+ super.toString(sb);
+ sb.append("dataMaskInfo={");
+ if (dataMaskInfo != null) {
+ dataMaskInfo.toString(sb);
+ }
+ sb.append("} ");
+ sb.append("}");
+ return sb;
+ }
+ }
+
+ /**
+ * RangerRowFilterPolicyItem class.
+ */
+ @JsonAutoDetect(fieldVisibility = Visibility.ANY)
+ @JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+ @JsonIgnoreProperties(ignoreUnknown = true)
+ @XmlRootElement
+ @XmlAccessorType(XmlAccessType.FIELD)
+ public static class RangerRowFilterPolicyItem extends RangerPolicyItem implements java.io.Serializable {
+ private static final long serialVersionUID = 1L;
+ private RangerPolicyItemRowFilterInfo rowFilterInfo;
+
+ public RangerRowFilterPolicyItem() {
+ this(null, null, null, null, null, null);
+ }
+
+ public RangerRowFilterPolicyItem(RangerPolicyItemRowFilterInfo rowFilterInfo,
+ List<RangerPolicyItemAccess> accesses, List<String> users, List<String> groups,
+ List<RangerPolicyItemCondition> conditions, Boolean delegateAdmin) {
+ super(accesses, users, groups, conditions, delegateAdmin);
+ setRowFilterInfo(rowFilterInfo);
+ }
+
+ /**
+ * @return the rowFilterInfo
+ */
+ public RangerPolicyItemRowFilterInfo getRowFilterInfo() {
+ return rowFilterInfo;
+ }
+
+ /**
+ * @param rowFilterInfo the rowFilterInfo to set
+ */
+ public void setRowFilterInfo(RangerPolicyItemRowFilterInfo rowFilterInfo) {
+ this.rowFilterInfo = rowFilterInfo == null ? new RangerPolicyItemRowFilterInfo() : rowFilterInfo;
+ }
+
+ @Override
+ public int hashCode() {
+ final int prime = 31;
+ int result = super.hashCode();
+ result = prime * result + ((rowFilterInfo == null) ? 0 : rowFilterInfo.hashCode());
+ return result;
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (!super.equals(obj)) {
+ return false;
+ }
+ if (this == obj) {
+ return true;
+ }
+ if (obj == null) {
+ return false;
+ }
+ if (getClass() != obj.getClass()) {
+ return false;
+ }
+ RangerRowFilterPolicyItem other = (RangerRowFilterPolicyItem) obj;
+ if (rowFilterInfo == null) {
+ if (other.rowFilterInfo != null) {
+ return false;
+ }
+ } else if (!rowFilterInfo.equals(other.rowFilterInfo)) {
+ return false;
+ }
+ return true;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ toString(sb);
+ return sb.toString();
+ }
+
+ public StringBuilder toString(StringBuilder sb) {
Review comment:
Mark it private, may change the signature as well. It is toString() but returns StringBuilder
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ranger/RangerPolicy.java
##########
@@ -0,0 +1,1513 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl.ranger;
+
+import org.codehaus.jackson.annotate.JsonAutoDetect;
+import org.codehaus.jackson.annotate.JsonAutoDetect.Visibility;
+import org.codehaus.jackson.annotate.JsonIgnoreProperties;
+import org.codehaus.jackson.map.annotate.JsonSerialize;
+
+import javax.xml.bind.annotation.XmlAccessType;
+import javax.xml.bind.annotation.XmlAccessorType;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * RangerPolicy class to contain Ranger Policy details.
+ */
+@JsonAutoDetect(fieldVisibility = Visibility.ANY)
+@JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+@JsonIgnoreProperties(ignoreUnknown = true)
+@XmlRootElement
+@XmlAccessorType(XmlAccessType.FIELD)
+public class RangerPolicy extends RangerBaseModelObject implements java.io.Serializable {
+ public static final int POLICY_TYPE_ACCESS = 0;
+ public static final int POLICY_TYPE_DATAMASK = 1;
+ public static final int POLICY_TYPE_ROWFILTER = 2;
+
+ public static final int[] POLICY_TYPES = new int[]{
+ POLICY_TYPE_ACCESS,
+ POLICY_TYPE_DATAMASK,
+ POLICY_TYPE_ROWFILTER,
+ };
+
+ public static final String MASK_TYPE_NULL = "MASK_NULL";
+ public static final String MASK_TYPE_NONE = "MASK_NONE";
+ public static final String MASK_TYPE_CUSTOM = "CUSTOM";
+
+ private static final long serialVersionUID = 1L;
+
+ private String service;
+ private String name;
+ private Integer policyType;
+ private String description;
+ private String resourceSignature;
+ private Boolean isAuditEnabled;
+ private Map<String, RangerPolicyResource> resources;
+ private List<RangerPolicyItem> policyItems;
+ private List<RangerPolicyItem> denyPolicyItems;
+ private List<RangerPolicyItem> allowExceptions;
+ private List<RangerPolicyItem> denyExceptions;
+ private List<RangerDataMaskPolicyItem> dataMaskPolicyItems;
+ private List<RangerRowFilterPolicyItem> rowFilterPolicyItems;
+
+
+ /**
+ * Ranger Policy default constructor.
+ */
+ public RangerPolicy() {
+ this(null, null, null, null, null, null, null);
+ }
+
+ /**
+ * @param service
+ * @param name
+ * @param policyType
+ * @param description
+ * @param resources
+ * @param policyItems
+ * @param resourceSignature TODO
+ */
+ public RangerPolicy(String service, String name, Integer policyType, String description,
+ Map<String, RangerPolicyResource> resources, List<RangerPolicyItem> policyItems,
+ String resourceSignature) {
+ super();
+ setService(service);
+ setName(name);
+ setPolicyType(policyType);
+ setDescription(description);
+ setResourceSignature(resourceSignature);
+ setIsAuditEnabled(null);
+ setResources(resources);
+ setPolicyItems(policyItems);
+ setDenyPolicyItems(null);
+ setAllowExceptions(null);
+ setDenyExceptions(null);
+ setDataMaskPolicyItems(null);
+ setRowFilterPolicyItems(null);
+ }
+
+ /**
+ * @param other
+ */
+ public void updateFrom(RangerPolicy other) {
+ super.updateFrom(other);
+
+ setService(other.getService());
+ setName(other.getName());
+ setPolicyType(other.getPolicyType());
+ setDescription(other.getDescription());
+ setResourceSignature(other.getResourceSignature());
+ setIsAuditEnabled(other.getIsAuditEnabled());
+ setResources(other.getResources());
+ setPolicyItems(other.getPolicyItems());
+ setDenyPolicyItems(other.getDenyPolicyItems());
+ setAllowExceptions(other.getAllowExceptions());
+ setDenyExceptions(other.getDenyExceptions());
+ setDataMaskPolicyItems(other.getDataMaskPolicyItems());
+ setRowFilterPolicyItems(other.getRowFilterPolicyItems());
+ }
+
+ /**
+ * @return the type
+ */
+ public String getService() {
+ return service;
+ }
+
+ /**
+ * @param service the type to set
+ */
+ public void setService(String service) {
+ this.service = service;
+ }
+
+ /**
+ * @return the name
+ */
+ public String getName() {
+ return name;
+ }
+
+ /**
+ * @param name the name to set
+ */
+ public void setName(String name) {
+ this.name = name;
+ }
+
+ /**
+ * @return the policyType
+ */
+ public Integer getPolicyType() {
+ return policyType;
+ }
+
+ /**
+ * @param policyType the policyType to set
+ */
+ public void setPolicyType(Integer policyType) {
+ this.policyType = policyType;
+ }
+
+ /**
+ * @return the description
+ */
+ public String getDescription() {
+ return description;
+ }
+
+ /**
+ * @param description the description to set
+ */
+ public void setDescription(String description) {
+ this.description = description;
+ }
+
+ /**
+ * @return the resourceSignature
+ */
+ public String getResourceSignature() {
+ return resourceSignature;
+ }
+
+ /**
+ * @param resourceSignature the resourceSignature to set
+ */
+ public void setResourceSignature(String resourceSignature) {
+ this.resourceSignature = resourceSignature;
+ }
+
+ /**
+ * @return the isAuditEnabled
+ */
+ public Boolean getIsAuditEnabled() {
+ return isAuditEnabled;
+ }
+
+ /**
+ * @param isAuditEnabled the isEnabled to set
+ */
+ public void setIsAuditEnabled(Boolean isAuditEnabled) {
+ this.isAuditEnabled = isAuditEnabled == null ? Boolean.TRUE : isAuditEnabled;
+ }
+
+ /**
+ * @return the resources
+ */
+ public Map<String, RangerPolicyResource> getResources() {
+ return resources;
+ }
+
+ /**
+ * @param resources the resources to set
+ */
+ public void setResources(Map<String, RangerPolicyResource> resources) {
+ if (this.resources == null) {
+ this.resources = new HashMap<>();
+ }
+
+ if (this.resources == resources) {
+ return;
+ }
+
+ this.resources.clear();
+
+ if (resources != null) {
+ for (Map.Entry<String, RangerPolicyResource> e : resources.entrySet()) {
+ this.resources.put(e.getKey(), e.getValue());
+ }
+ }
+ }
+
+ /**
+ * @return the policyItems
+ */
+ public List<RangerPolicyItem> getPolicyItems() {
+ return policyItems;
+ }
+
+ /**
+ * @param policyItems the policyItems to set
+ */
+ public void setPolicyItems(List<RangerPolicyItem> policyItems) {
+ if (this.policyItems == null) {
+ this.policyItems = new ArrayList<>();
+ }
+
+ if (this.policyItems == policyItems) {
+ return;
+ }
+
+ this.policyItems.clear();
+
+ if (policyItems != null) {
+ this.policyItems.addAll(policyItems);
+ }
+ }
+
+ /**
+ * @return the denyPolicyItems
+ */
+ public List<RangerPolicyItem> getDenyPolicyItems() {
+ return denyPolicyItems;
+ }
+
+ /**
+ * @param denyPolicyItems the denyPolicyItems to set
+ */
+ public void setDenyPolicyItems(List<RangerPolicyItem> denyPolicyItems) {
+ if (this.denyPolicyItems == null) {
+ this.denyPolicyItems = new ArrayList<>();
+ }
+
+ if (this.denyPolicyItems == denyPolicyItems) {
+ return;
+ }
+
+ this.denyPolicyItems.clear();
+
+ if (denyPolicyItems != null) {
+ this.denyPolicyItems.addAll(denyPolicyItems);
+ }
+ }
+
+ /**
+ * @return the allowExceptions
+ */
+ public List<RangerPolicyItem> getAllowExceptions() {
+ return allowExceptions;
+ }
+
+ /**
+ * @param allowExceptions the allowExceptions to set
+ */
+ public void setAllowExceptions(List<RangerPolicyItem> allowExceptions) {
+ if (this.allowExceptions == null) {
+ this.allowExceptions = new ArrayList<>();
+ }
+
+ if (this.allowExceptions == allowExceptions) {
+ return;
+ }
+
+ this.allowExceptions.clear();
+
+ if (allowExceptions != null) {
+ this.allowExceptions.addAll(allowExceptions);
+ }
+ }
+
+ /**
+ * @return the denyExceptions
+ */
+ public List<RangerPolicyItem> getDenyExceptions() {
+ return denyExceptions;
+ }
+
+ /**
+ * @param denyExceptions the denyExceptions to set
+ */
+ public void setDenyExceptions(List<RangerPolicyItem> denyExceptions) {
+ if (this.denyExceptions == null) {
+ this.denyExceptions = new ArrayList<>();
+ }
+
+ if (this.denyExceptions == denyExceptions) {
+ return;
+ }
+
+ this.denyExceptions.clear();
+
+ if (denyExceptions != null) {
+ this.denyExceptions.addAll(denyExceptions);
+ }
+ }
+
+ public List<RangerDataMaskPolicyItem> getDataMaskPolicyItems() {
+ return dataMaskPolicyItems;
+ }
+
+ public void setDataMaskPolicyItems(List<RangerDataMaskPolicyItem> dataMaskPolicyItems) {
+ if (this.dataMaskPolicyItems == null) {
+ this.dataMaskPolicyItems = new ArrayList<>();
+ }
+
+ if (this.dataMaskPolicyItems == dataMaskPolicyItems) {
+ return;
+ }
+
+ this.dataMaskPolicyItems.clear();
+
+ if (dataMaskPolicyItems != null) {
+ this.dataMaskPolicyItems.addAll(dataMaskPolicyItems);
+ }
+ }
+
+ public List<RangerRowFilterPolicyItem> getRowFilterPolicyItems() {
+ return rowFilterPolicyItems;
+ }
+
+ public void setRowFilterPolicyItems(List<RangerRowFilterPolicyItem> rowFilterPolicyItems) {
+ if (this.rowFilterPolicyItems == null) {
+ this.rowFilterPolicyItems = new ArrayList<>();
+ }
+
+ if (this.rowFilterPolicyItems == rowFilterPolicyItems) {
+ return;
+ }
+
+ this.rowFilterPolicyItems.clear();
+
+ if (rowFilterPolicyItems != null) {
+ this.rowFilterPolicyItems.addAll(rowFilterPolicyItems);
+ }
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ toString(sb);
+ return sb.toString();
+ }
+
+ public StringBuilder toString(StringBuilder sb) {
+ sb.append("RangerPolicy={");
+
+ super.toString(sb);
+
+ sb.append("service={").append(service).append("} ");
+ sb.append("name={").append(name).append("} ");
+ sb.append("policyType={").append(policyType).append("} ");
+ sb.append("description={").append(description).append("} ");
+ sb.append("resourceSignature={").append(resourceSignature).append("} ");
+ sb.append("isAuditEnabled={").append(isAuditEnabled).append("} ");
+
+ sb.append("resources={");
+ if (resources != null) {
+ for (Map.Entry<String, RangerPolicyResource> e : resources.entrySet()) {
+ sb.append(e.getKey()).append("={");
+ e.getValue().toString(sb);
+ sb.append("} ");
+ }
+ }
+ sb.append("} ");
+
+ sb.append("policyItems={");
+ if (policyItems != null) {
+ for (RangerPolicyItem policyItem : policyItems) {
+ if (policyItem != null) {
+ policyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("denyPolicyItems={");
+ if (denyPolicyItems != null) {
+ for (RangerPolicyItem policyItem : denyPolicyItems) {
+ if (policyItem != null) {
+ policyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("allowExceptions={");
+ if (allowExceptions != null) {
+ for (RangerPolicyItem policyItem : allowExceptions) {
+ if (policyItem != null) {
+ policyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("denyExceptions={");
+ if (denyExceptions != null) {
+ for (RangerPolicyItem policyItem : denyExceptions) {
+ if (policyItem != null) {
+ policyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("dataMaskPolicyItems={");
+ if (dataMaskPolicyItems != null) {
+ for (RangerDataMaskPolicyItem dataMaskPolicyItem : dataMaskPolicyItems) {
+ if (dataMaskPolicyItem != null) {
+ dataMaskPolicyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("rowFilterPolicyItems={");
+ if (rowFilterPolicyItems != null) {
+ for (RangerRowFilterPolicyItem rowFilterPolicyItem : rowFilterPolicyItems) {
+ if (rowFilterPolicyItem != null) {
+ rowFilterPolicyItem.toString(sb);
+ }
+ }
+ }
+ sb.append("} ");
+
+ sb.append("}");
+
+ return sb;
+ }
+
+ /**
+ * RangerPolicyResource class to store the resource path values.
+ */
+ @JsonAutoDetect(fieldVisibility = Visibility.ANY)
+ @JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+ @JsonIgnoreProperties(ignoreUnknown = true)
+ @XmlRootElement
+ @XmlAccessorType(XmlAccessType.FIELD)
+ public static class RangerPolicyResource implements java.io.Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private List<String> values;
+ private Boolean isExcludes;
+ private Boolean isRecursive;
+
+ public RangerPolicyResource() {
+ this((List<String>) null, null, null);
+ }
+
+ public RangerPolicyResource(String value) {
+ setValue(value);
+ setIsExcludes(null);
+ setIsRecursive(null);
+ }
+
+ public RangerPolicyResource(String value, Boolean isExcludes, Boolean isRecursive) {
+ setValue(value);
+ setIsExcludes(isExcludes);
+ setIsRecursive(isRecursive);
+ }
+
+ public RangerPolicyResource(List<String> values, Boolean isExcludes, Boolean isRecursive) {
+ setValues(values);
+ setIsExcludes(isExcludes);
+ setIsRecursive(isRecursive);
+ }
+
+ /**
+ * @return the values
+ */
+ public List<String> getValues() {
+ return values;
+ }
+
+ /**
+ * @param values the values to set
+ */
+ public void setValues(List<String> values) {
+ if (this.values == null) {
+ this.values = new ArrayList<>();
+ }
+ if (this.values == values) {
+ return;
+ }
+ this.values.clear();
+ if (values != null) {
+ this.values.addAll(values);
+ }
+ }
+
+ /**
+ * @param value the value to set
+ */
+ public void setValue(String value) {
+ if (this.values == null) {
+ this.values = new ArrayList<>();
+ }
+ this.values.clear();
+ this.values.add(value);
+ }
+
+ /**
+ * @return the isExcludes
+ */
+ public Boolean getIsExcludes() {
+ return isExcludes;
+ }
+
+ /**
+ * @param isExcludes the isExcludes to set
+ */
+ public void setIsExcludes(Boolean isExcludes) {
+ this.isExcludes = isExcludes == null ? Boolean.FALSE : isExcludes;
+ }
+
+ /**
+ * @return the isRecursive
+ */
+ public Boolean getIsRecursive() {
+ return isRecursive;
+ }
+
+ /**
+ * @param isRecursive the isRecursive to set
+ */
+ public void setIsRecursive(Boolean isRecursive) {
+ this.isRecursive = isRecursive == null ? Boolean.FALSE : isRecursive;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ toString(sb);
+ return sb.toString();
+ }
+
+ public StringBuilder toString(StringBuilder sb) {
+ sb.append("RangerPolicyResource={");
+ sb.append("values={");
+ if (values != null) {
+ for (String value : values) {
+ sb.append(value).append(" ");
+ }
+ }
+ sb.append("} ");
+ sb.append("isExcludes={").append(isExcludes).append("} ");
+ sb.append("isRecursive={").append(isRecursive).append("} ");
+ sb.append("}");
+
+ return sb;
+ }
+
+ @Override
+ public int hashCode() {
+ final int prime = 31;
+ int result = 1;
+ result = prime * result
+ + ((isExcludes == null) ? 0 : isExcludes.hashCode());
+ result = prime * result
+ + ((isRecursive == null) ? 0 : isRecursive.hashCode());
+ result = prime * result
+ + ((values == null) ? 0 : values.hashCode());
+ return result;
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (this == obj) {
+ return true;
+ }
+ if (obj == null) {
+ return false;
+ }
+ if (getClass() != obj.getClass()) {
+ return false;
+ }
+ RangerPolicyResource other = (RangerPolicyResource) obj;
+ if (isExcludes == null) {
+ if (other.isExcludes != null) {
+ return false;
+ }
+ } else if (!isExcludes.equals(other.isExcludes)) {
+ return false;
+ }
+ if (isRecursive == null) {
+ if (other.isRecursive != null) {
+ return false;
+ }
+ } else if (!isRecursive.equals(other.isRecursive)) {
+ return false;
+ }
+ if (values == null) {
+ if (other.values != null) {
+ return false;
+ }
+ } else if (!values.equals(other.values)) {
+ return false;
+ }
+ return true;
+ }
+ }
+
+ /**
+ * RangerPolicyItem class contains ranger policy items like access and permissions.
+ */
+ @JsonAutoDetect(fieldVisibility = Visibility.ANY)
+ @JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+ @JsonIgnoreProperties(ignoreUnknown = true)
+ @XmlRootElement
+ @XmlAccessorType(XmlAccessType.FIELD)
+ public static class RangerPolicyItem implements java.io.Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private List<RangerPolicyItemAccess> accesses;
+ private List<String> users;
+ private List<String> groups;
+ private List<RangerPolicyItemCondition> conditions;
+ private Boolean delegateAdmin;
+
+ public RangerPolicyItem() {
Review comment:
Unused? Remove it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aasha commented on a change in pull request #1004: HIVE-23351 : Ranger Replication Scheduling
Posted by GitBox <gi...@apache.org>.
aasha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r422153226
##########
File path: ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/StageType.java
##########
@@ -26,7 +26,9 @@
REPL_TXN(15),
REPL_INCREMENTAL_LOAD(16),
SCHEDULED_QUERY_MAINT(17),
- ACK(18);
+ ACK(18),
+ RANGER_DUMP(19),
+ RANGER_LOAD(20);
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aasha commented on a change in pull request #1004: HIVE-23351 : Ranger Replication Scheduling
Posted by GitBox <gi...@apache.org>.
aasha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r422155515
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##########
@@ -170,6 +175,26 @@ public int execute() {
return 0;
}
+ private void initiateAuthorizationDumpTask(Path currentDumpPath) throws SemanticException {
+ if (RANGER_AUTHORIZER.equalsIgnoreCase(conf.getVar(HiveConf.ConfVars.REPL_AUTHORIZATION_PROVIDER_SERVICE))) {
+ Path rangerDumpRoot = new Path(currentDumpPath, ReplUtils.REPL_RANGER_BASE_DIR);
+ LOG.info("Exporting Authorization Metadata at {} ", rangerDumpRoot);
+ RangerDumpWork rangerDumpWork = new RangerDumpWork(rangerDumpRoot, work.dbNameOrPattern);
+ Task<RangerDumpWork> rangerDumpTask = TaskFactory.get(rangerDumpWork, conf);
+ if (childTasks == null) {
Review comment:
Not necessary, once we add Atlas childtask will already contain Atlas.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] closed pull request #1004: HIVE-23351 : Ranger Replication Scheduling
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #1004:
URL: https://github.com/apache/hive/pull/1004
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aasha commented on a change in pull request #1004: HIVE-23351 : Ranger Replication Scheduling
Posted by GitBox <gi...@apache.org>.
aasha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r422285774
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##########
@@ -103,13 +105,43 @@ public int execute() {
}
work.setRootTask(this);
this.parentTasks = null;
+ if (shouldLoadAuthorizationMetadata()) {
+ LOG.info("Loading authorization data");
+ try {
+ initiateAuthorizationLoadTask(work.dumpDirectory);
+ } catch (Exception e) {
+ LOG.error("failed", e);
+ setException(e);
+ return ErrorMsg.getErrorMsg(e.getMessage()).getErrorCode();
+ }
+ }
if (work.isIncrementalLoad()) {
return executeIncrementalLoad();
} else {
return executeBootStrapLoad();
}
}
+ private boolean shouldLoadAuthorizationMetadata() {
+ return conf.getBoolVar(HiveConf.ConfVars.REPL_INCLUDE_AUTHORIZATION_METADATA);
+ }
+
+ private void initiateAuthorizationLoadTask(String hiveDumpDirectory) throws SemanticException {
+ if (RANGER_AUTHORIZER.equalsIgnoreCase(conf.getVar(HiveConf.ConfVars.REPL_AUTHORIZATION_PROVIDER_SERVICE))) {
+ Path rangerLoadRoot = new Path(new Path(hiveDumpDirectory).getParent(), ReplUtils.REPL_RANGER_BASE_DIR);
+ LOG.info("Importing Authorization Metadata from {} ", rangerLoadRoot);
+ RangerLoadWork rangerLoadWork = new RangerLoadWork(rangerLoadRoot, work.getSourceDbName(), work.dbNameToLoadIn);
+ Task<RangerLoadWork> rangerLoadTask = TaskFactory.get(rangerLoadWork, conf);
+ if (childTasks == null) {
+ childTasks = new ArrayList<>();
+ }
+ childTasks.add(rangerLoadTask);
Review comment:
Separate tasks are created.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aasha commented on a change in pull request #1004: HIVE-23351 : Ranger Replication Scheduling
Posted by GitBox <gi...@apache.org>.
aasha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r422131096
##########
File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
##########
@@ -526,6 +526,22 @@ private static void populateLlapDaemonVarsSet(Set<String> llapDaemonVarsSetLocal
"This is the base directory on the target/replica warehouse under which data for "
+ "external tables is stored. This is relative base path and hence prefixed to the source "
+ "external table path on target cluster."),
+ REPL_INCLUDE_AUTHORIZATION_METADATA("hive.repl.include.authorization.metadata", false,
+ "This configuration will enable security and authorization related metadata along "
+ + "with the hive data and metadata replication. "),
+ REPL_AUTHORIZATION_PROVIDER_SERVICE("hive.repl.authorization.provider.service", "ranger",
+ "This configuration will define which service will provide the security and authorization "
+ + "related metadata that needs to be replicated along "
+ + "with the hive data and metadata replication. Set the configuration "
+ + "hive.repl.include.authorization.metadata to false to disable "
+ + "security policies being replicated "),
+ REPL_AUTHORIZATION_PROVIDER_SERVICE_ENDPOINT("hive.repl.authorization.provider.service.endpoint",
+ "",
+ "This configuration will define the authorization service endpoint"),
+ REPL_RANGER_SERVICE_NAME("hive.repl.ranger.service.name",
+ "cm_hive",
Review comment:
from cdp cluster
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pkumarsinha commented on a change in pull request #1004: HIVE-23351 : Ranger Replication Scheduling
Posted by GitBox <gi...@apache.org>.
pkumarsinha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r422172542
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##########
@@ -198,7 +223,12 @@ private void initiateDataCopyTasks() throws SemanticException {
finishRemainingTasks();
} else {
DAGTraversal.traverse(childTasks, new AddDependencyToLeaves(TaskFactory.get(work, conf)));
- this.childTasks = childTasks;
Review comment:
Let's say, during an incremental run, you don't have any new table to bootstrap and you don't have even any external table in DB.
List<Task<?>> childTasks = new ArrayList<>();
childTasks.addAll(work.externalTableCopyTasks(taskTracker, conf));
childTasks.addAll(work.managedTableCopyTasks(taskTracker, conf));
if (childTasks.isEmpty()) {
//All table data copy work finished.
finishRemainingTasks();
}
This will go to finishRemainingTasks() immediately even though you have rangerDumpTask as childTask of ReplDumpTask
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##########
@@ -170,6 +175,26 @@ public int execute() {
return 0;
}
+ private void initiateAuthorizationDumpTask(Path currentDumpPath) throws SemanticException {
+ if (RANGER_AUTHORIZER.equalsIgnoreCase(conf.getVar(HiveConf.ConfVars.REPL_AUTHORIZATION_PROVIDER_SERVICE))) {
+ Path rangerDumpRoot = new Path(currentDumpPath, ReplUtils.REPL_RANGER_BASE_DIR);
+ LOG.info("Exporting Authorization Metadata at {} ", rangerDumpRoot);
+ RangerDumpWork rangerDumpWork = new RangerDumpWork(rangerDumpRoot, work.dbNameOrPattern);
+ Task<RangerDumpWork> rangerDumpTask = TaskFactory.get(rangerDumpWork, conf);
+ if (childTasks == null) {
Review comment:
If it is considering Atlas task then it may be fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] commented on pull request #1004: HIVE-23351 : Ranger Replication Scheduling
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #1004:
URL: https://github.com/apache/hive/pull/1004#issuecomment-656949812
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aasha commented on a change in pull request #1004: HIVE-23351 : Ranger Replication Scheduling
Posted by GitBox <gi...@apache.org>.
aasha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r422156406
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##########
@@ -198,7 +223,12 @@ private void initiateDataCopyTasks() throws SemanticException {
finishRemainingTasks();
} else {
DAGTraversal.traverse(childTasks, new AddDependencyToLeaves(TaskFactory.get(work, conf)));
- this.childTasks = childTasks;
Review comment:
If Ranger dump fails, the task execution will be stopped there itself with a failure and we will get an error. It won't even come to the second ReplDump task execution to execute the finishRemainingTasks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pkumarsinha commented on a change in pull request #1004: HIVE-23351 : Ranger Replication Scheduling
Posted by GitBox <gi...@apache.org>.
pkumarsinha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r422175531
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##########
@@ -103,13 +105,43 @@ public int execute() {
}
work.setRootTask(this);
this.parentTasks = null;
+ if (shouldLoadAuthorizationMetadata()) {
+ LOG.info("Loading authorization data");
+ try {
+ initiateAuthorizationLoadTask(work.dumpDirectory);
+ } catch (Exception e) {
+ LOG.error("failed", e);
+ setException(e);
+ return ErrorMsg.getErrorMsg(e.getMessage()).getErrorCode();
+ }
+ }
if (work.isIncrementalLoad()) {
return executeIncrementalLoad();
} else {
return executeBootStrapLoad();
}
}
+ private boolean shouldLoadAuthorizationMetadata() {
+ return conf.getBoolVar(HiveConf.ConfVars.REPL_INCLUDE_AUTHORIZATION_METADATA);
+ }
+
+ private void initiateAuthorizationLoadTask(String hiveDumpDirectory) throws SemanticException {
+ if (RANGER_AUTHORIZER.equalsIgnoreCase(conf.getVar(HiveConf.ConfVars.REPL_AUTHORIZATION_PROVIDER_SERVICE))) {
+ Path rangerLoadRoot = new Path(new Path(hiveDumpDirectory).getParent(), ReplUtils.REPL_RANGER_BASE_DIR);
+ LOG.info("Importing Authorization Metadata from {} ", rangerLoadRoot);
+ RangerLoadWork rangerLoadWork = new RangerLoadWork(rangerLoadRoot, work.getSourceDbName(), work.dbNameToLoadIn);
+ Task<RangerLoadWork> rangerLoadTask = TaskFactory.get(rangerLoadWork, conf);
+ if (childTasks == null) {
+ childTasks = new ArrayList<>();
+ }
+ childTasks.add(rangerLoadTask);
Review comment:
Metadata will run as a part of current task.
if (work.isIncrementalLoad()) {
return executeIncrementalLoad();
} else {
return executeBootStrapLoad();
}
in either bootstrap or in incremental, this will run it as part of current ReplLoadTask
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pkumarsinha commented on a change in pull request #1004: HIVE-23351 : Ranger Replication Scheduling
Posted by GitBox <gi...@apache.org>.
pkumarsinha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r420680172
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##########
@@ -144,6 +145,11 @@ public int execute() {
if (shouldDump(previousValidHiveDumpPath)) {
Path currentDumpPath = getCurrentDumpPath(dumpRoot, isBootstrap);
Path hiveDumpRoot = new Path(currentDumpPath, ReplUtils.REPL_HIVE_BASE_DIR);
+ work.setCurrentDumpPath(currentDumpPath);
+ if (shouldDumpAuthorizationMetadata()) {
+ LOG.info("Dumping authorization data");
+ initiateAuthorizationDumpTask(currentDumpPath);
Review comment:
If you don't return here, this will run only after current repl dump execution?
What is the order of execution in the current case?
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##########
@@ -170,6 +175,31 @@ public int execute() {
return 0;
}
+ private void initiateAuthorizationDumpTask(Path currentDumpPath) throws SemanticException, IOException {
+ if (RANGER_AUTHORIZER.equalsIgnoreCase(conf.getVar(HiveConf.ConfVars.REPL_AUTHORIZATION_PROVIDER_SERVICE))) {
+ Path rangerDumpRoot = new Path(currentDumpPath, ReplUtils.REPL_RANGER_BASE_DIR);
+ FileSystem fs = rangerDumpRoot.getFileSystem(conf);
+ if (fs.exists(new Path(rangerDumpRoot, ReplAck.RANGER_DUMP_ACKNOWLEDGEMENT.toString()))) {
+ LOG.info("Ranger Authorization Metadata is already exported at {} ", rangerDumpRoot);
+ } else {
+ LOG.info("Exporting Authorization Metadata at {} ", rangerDumpRoot);
+ RangerDumpWork rangerDumpWork = new RangerDumpWork(rangerDumpRoot, work.dbNameOrPattern);
+ Task<RangerDumpWork> rangerDumpTask = TaskFactory.get(rangerDumpWork, conf);
+ if (childTasks == null) {
+ childTasks = new ArrayList<>();
+ }
+ childTasks.add(rangerDumpTask);
Review comment:
Not getting added to the DAG, how does it get executed?
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplAck.java
##########
@@ -23,7 +23,9 @@
public enum ReplAck {
DUMP_ACKNOWLEDGEMENT("_finished_dump"),
EVENTS_DUMP("_events_dump"),
- LOAD_ACKNOWLEDGEMENT("_finished_load");
+ LOAD_ACKNOWLEDGEMENT("_finished_load"),
+ RANGER_DUMP_ACKNOWLEDGEMENT("_finished_ranger_dump"),
Review comment:
We can use same DUMP_ACKNOWLEDGEMENT LOAD_ACKNOWLEDGEMENT
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ranger/RangerBaseModelObject.java
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl.ranger;
+
+import org.codehaus.jackson.annotate.JsonAutoDetect;
+import org.codehaus.jackson.annotate.JsonAutoDetect.Visibility;
+import org.codehaus.jackson.annotate.JsonIgnoreProperties;
+import org.codehaus.jackson.map.annotate.JsonSerialize;
+
+import javax.xml.bind.annotation.XmlAccessType;
+import javax.xml.bind.annotation.XmlAccessorType;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Date;
+
+/** RangerBaseModelObject class to contain common attributes of Ranger Base object.
+*
+*/
+@JsonAutoDetect(getterVisibility=Visibility.NONE, setterVisibility=Visibility.NONE, fieldVisibility=Visibility.ANY)
+@JsonSerialize(include=JsonSerialize.Inclusion.NON_NULL)
+@JsonIgnoreProperties(ignoreUnknown=true)
+@XmlRootElement
+@XmlAccessorType(XmlAccessType.FIELD)
+public class RangerBaseModelObject implements java.io.Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private Long id;
+ private String guid;
+ private Boolean isEnabled;
+ private String createdBy;
+ private String updatedBy;
+ private Date createTime;
+ private Date updateTime;
+ private Long version;
+
+ public RangerBaseModelObject() {
+ setIsEnabled(null);
+ }
+
+ public void updateFrom(RangerBaseModelObject other) {
+ setIsEnabled(other.getIsEnabled());
+ }
+
+ /**
+ * @return the id
+ */
+ public Long getId() {
+ return id;
+ }
+ /**
+ * @param id the id to set
+ */
+ public void setId(Long id) {
+ this.id = id;
+ }
+ /**
+ * @return the guid
+ */
+ public String getGuid() {
+ return guid;
+ }
+ /**
+ * @param guid the guid to set
+ */
+ public void setGuid(String guid) {
+ this.guid = guid;
+ }
+ /**
+ * @return the isEnabled
+ */
+ public Boolean getIsEnabled() {
+ return isEnabled;
+ }
+ /**
+ * @param isEnabled the isEnabled to set
+ */
+ public void setIsEnabled(Boolean isEnabled) {
+ this.isEnabled = isEnabled == null ? Boolean.TRUE : isEnabled;
+ }
+ /**
+ * @return the createdBy
+ */
+ public String getCreatedBy() {
+ return createdBy;
+ }
+ /**
+ * @param createdBy the createdBy to set
+ */
+ public void setCreatedBy(String createdBy) {
+ this.createdBy = createdBy;
+ }
+ /**
+ * @return the updatedBy
+ */
+ public String getUpdatedBy() {
+ return updatedBy;
+ }
+ /**
+ * @param updatedBy the updatedBy to set
+ */
+ public void setUpdatedBy(String updatedBy) {
+ this.updatedBy = updatedBy;
+ }
+ /**
+ * @return the createTime
+ */
+ public Date getCreateTime() {
+ return new Date(createTime.getTime());
+ }
+ /**
+ * @param createTime the createTime to set
+ */
+ public void setCreateTime(Date createTime) {
+ this.createTime = new Date(createTime.getTime());
+ }
+ /**
+ * @return the updateTime
+ */
+ public Date getUpdateTime() {
+ return new Date(updateTime.getTime());
+ }
+ /**
+ * @param updateTime the updateTime to set
+ */
+ public void setUpdateTime(Date updateTime) {
+ this.updateTime = new Date(updateTime.getTime());
+ }
+ /**
+ * @return the version
+ */
+ public Long getVersion() {
+ return version;
+ }
+ /**
+ * @param version the version to set
+ */
+ public void setVersion(Long version) {
+ this.version = version;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ toString(sb);
+ return sb.toString();
+ }
+
+ public StringBuilder toString(StringBuilder sb) {
+ sb.append("id={").append(id).append("} ");
+ sb.append("guid={").append(guid).append("} ");
+ sb.append("isEnabled={").append(isEnabled).append("} ");
+ sb.append("createdBy={").append(createdBy).append("} ");
+ sb.append("updatedBy={").append(updatedBy).append("} ");
+ sb.append("createTime={").append(createTime).append("} ");
+ sb.append("updateTime={").append(updateTime).append("} ");
+ sb.append("version={").append(version).append("} ");
+
+ return sb;
Review comment:
Format the code to align with existing code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pkumarsinha commented on a change in pull request #1004: HIVE-23351 : Ranger Replication Scheduling
Posted by GitBox <gi...@apache.org>.
pkumarsinha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r422175531
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##########
@@ -103,13 +105,43 @@ public int execute() {
}
work.setRootTask(this);
this.parentTasks = null;
+ if (shouldLoadAuthorizationMetadata()) {
+ LOG.info("Loading authorization data");
+ try {
+ initiateAuthorizationLoadTask(work.dumpDirectory);
+ } catch (Exception e) {
+ LOG.error("failed", e);
+ setException(e);
+ return ErrorMsg.getErrorMsg(e.getMessage()).getErrorCode();
+ }
+ }
if (work.isIncrementalLoad()) {
return executeIncrementalLoad();
} else {
return executeBootStrapLoad();
}
}
+ private boolean shouldLoadAuthorizationMetadata() {
+ return conf.getBoolVar(HiveConf.ConfVars.REPL_INCLUDE_AUTHORIZATION_METADATA);
+ }
+
+ private void initiateAuthorizationLoadTask(String hiveDumpDirectory) throws SemanticException {
+ if (RANGER_AUTHORIZER.equalsIgnoreCase(conf.getVar(HiveConf.ConfVars.REPL_AUTHORIZATION_PROVIDER_SERVICE))) {
+ Path rangerLoadRoot = new Path(new Path(hiveDumpDirectory).getParent(), ReplUtils.REPL_RANGER_BASE_DIR);
+ LOG.info("Importing Authorization Metadata from {} ", rangerLoadRoot);
+ RangerLoadWork rangerLoadWork = new RangerLoadWork(rangerLoadRoot, work.getSourceDbName(), work.dbNameToLoadIn);
+ Task<RangerLoadWork> rangerLoadTask = TaskFactory.get(rangerLoadWork, conf);
+ if (childTasks == null) {
+ childTasks = new ArrayList<>();
+ }
+ childTasks.add(rangerLoadTask);
Review comment:
Metadata will run as a part of current task.
if (work.isIncrementalLoad()) {
return executeIncrementalLoad();
} else {
return executeBootStrapLoad();
}
in either bootstrap or in incremental, this will run it as part of current ReplLoadTask
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org