You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/05/16 02:04:14 UTC
[GitHub] [incubator-doris] stalary opened a new pull request, #9583: [Feature-wip] Support es external table not assign schema
stalary opened a new pull request, #9583:
URL: https://github.com/apache/incubator-doris/pull/9583
# Proposed changes
Issue Number: close https://github.com/apache/incubator-doris/issues/9578
## Problem Summary:
Describe the overview of changes.
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #9583: [Feature] Support es external table not assign schema
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #9583:
URL: https://github.com/apache/doris/pull/9583#issuecomment-1172382095
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] stalary commented on a diff in pull request #9583: [Feature] Support es external table not assign schema
Posted by GitBox <gi...@apache.org>.
stalary commented on code in PR #9583:
URL: https://github.com/apache/doris/pull/9583#discussion_r910794922
##########
fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/EsUtil.java:
##########
@@ -87,12 +93,129 @@ public static JSONObject getJsonObject(JSONObject jsonObject, String key, int fr
}
}
+ /**
+ * Get boolean field, if parse error throw DdlException.
+ **/
public static boolean getBoolean(Map<String, String> properties, String name) throws DdlException {
String property = properties.get(name).trim();
try {
return Boolean.parseBoolean(property);
} catch (Exception e) {
- throw new DdlException(String.format("fail to parse %s, %s = %s, `%s` should be like 'true' or 'false'， value should be double quotation marks", name, name, property, name));
+ throw new DdlException(String.format(
+ "fail to parse %s, %s = %s, `%s` should be like 'true' or 'false',"
+ + " value should be double quotation marks",
+ name, name, property, name));
+ }
+ }
+
+ /**
+ * Parse the required field information from the json.
+ *
+ * @param searchContext the current associated column searchContext
+ * @param indexMapping the return value of _mapping
+ */
+ public static void resolveFields(SearchContext searchContext, String indexMapping) throws DorisEsException {
+ JSONObject properties =
+ getMappingProps(searchContext.sourceIndex(), searchContext.esTable().getName(), indexMapping,
+ searchContext.type());
+ for (Column col : searchContext.columns()) {
+ String colName = col.getName();
+ // if column exists in Doris Table but no found in ES's mapping, we choose to ignore this situation?
+ if (!properties.containsKey(colName)) {
+ continue;
+ }
+ JSONObject fieldObject = (JSONObject) properties.get(colName);
+ resolveKeywordFields(searchContext, fieldObject, colName);
+ resolveDocValuesFields(searchContext, fieldObject, colName);
+ }
+ }
+
+ /**
+ * Get mapping properties JSONObject.
+ **/
+ public static JSONObject getMappingProps(String sourceIndex, String tableName, String indexMapping,
+ String mappingType) {
+ JSONObject jsonObject = (JSONObject) JSONValue.parse(indexMapping);
+ // the indexName use alias takes the first mapping
+ Iterator<String> keys = jsonObject.keySet().iterator();
+ String docKey = keys.next();
+ JSONObject docData = (JSONObject) jsonObject.get(docKey);
+ JSONObject mappings = (JSONObject) docData.get("mappings");
+ JSONObject rootSchema = (JSONObject) mappings.get(mappingType);
+ JSONObject properties;
+ // After (include) 7.x, type was removed from ES mapping, default type is `_doc`
+ // https://www.elastic.co/guide/en/elasticsearch/reference/7.0/removal-of-types.html
+ if (rootSchema == null) {
+ if (!"_doc".equals(mappingType)) {
Review Comment:
I will rebase later. This has been modified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #9583: [Feature] Support es external table not assign schema
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #9583:
URL: https://github.com/apache/doris/pull/9583#issuecomment-1172382063
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman merged pull request #9583: [Feature] Support es external table not assign schema
Posted by GitBox <gi...@apache.org>.
morningman merged PR #9583:
URL: https://github.com/apache/doris/pull/9583
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] stalary commented on a diff in pull request #9583: [Feature] Support es external table not assign schema
Posted by GitBox <gi...@apache.org>.
stalary commented on code in PR #9583:
URL: https://github.com/apache/doris/pull/9583#discussion_r911516109
##########
fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/EsUtil.java:
##########
@@ -87,12 +93,129 @@ public static JSONObject getJsonObject(JSONObject jsonObject, String key, int fr
}
}
+ /**
+ * Get boolean field, if parse error throw DdlException.
+ **/
public static boolean getBoolean(Map<String, String> properties, String name) throws DdlException {
String property = properties.get(name).trim();
try {
return Boolean.parseBoolean(property);
} catch (Exception e) {
- throw new DdlException(String.format("fail to parse %s, %s = %s, `%s` should be like 'true' or 'false'， value should be double quotation marks", name, name, property, name));
+ throw new DdlException(String.format(
+ "fail to parse %s, %s = %s, `%s` should be like 'true' or 'false',"
+ + " value should be double quotation marks",
+ name, name, property, name));
+ }
+ }
+
+ /**
+ * Parse the required field information from the json.
+ *
+ * @param searchContext the current associated column searchContext
+ * @param indexMapping the return value of _mapping
+ */
+ public static void resolveFields(SearchContext searchContext, String indexMapping) throws DorisEsException {
+ JSONObject properties =
+ getMappingProps(searchContext.sourceIndex(), searchContext.esTable().getName(), indexMapping,
+ searchContext.type());
+ for (Column col : searchContext.columns()) {
+ String colName = col.getName();
+ // if column exists in Doris Table but no found in ES's mapping, we choose to ignore this situation?
Review Comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qidaye commented on a diff in pull request #9583: [Feature] Support es external table not assign schema
Posted by GitBox <gi...@apache.org>.
qidaye commented on code in PR #9583:
URL: https://github.com/apache/doris/pull/9583#discussion_r910788339
##########
fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/EsUtil.java:
##########
@@ -87,12 +93,129 @@ public static JSONObject getJsonObject(JSONObject jsonObject, String key, int fr
}
}
+ /**
+ * Get boolean field, if parse error throw DdlException.
+ **/
public static boolean getBoolean(Map<String, String> properties, String name) throws DdlException {
String property = properties.get(name).trim();
try {
return Boolean.parseBoolean(property);
} catch (Exception e) {
- throw new DdlException(String.format("fail to parse %s, %s = %s, `%s` should be like 'true' or 'false'， value should be double quotation marks", name, name, property, name));
+ throw new DdlException(String.format(
+ "fail to parse %s, %s = %s, `%s` should be like 'true' or 'false',"
+ + " value should be double quotation marks",
+ name, name, property, name));
+ }
+ }
+
+ /**
+ * Parse the required field information from the json.
+ *
+ * @param searchContext the current associated column searchContext
+ * @param indexMapping the return value of _mapping
+ */
+ public static void resolveFields(SearchContext searchContext, String indexMapping) throws DorisEsException {
+ JSONObject properties =
+ getMappingProps(searchContext.sourceIndex(), searchContext.esTable().getName(), indexMapping,
+ searchContext.type());
+ for (Column col : searchContext.columns()) {
+ String colName = col.getName();
+ // if column exists in Doris Table but no found in ES's mapping, we choose to ignore this situation?
+ if (!properties.containsKey(colName)) {
+ continue;
+ }
+ JSONObject fieldObject = (JSONObject) properties.get(colName);
+ resolveKeywordFields(searchContext, fieldObject, colName);
+ resolveDocValuesFields(searchContext, fieldObject, colName);
+ }
+ }
+
+ /**
+ * Get mapping properties JSONObject.
+ **/
+ public static JSONObject getMappingProps(String sourceIndex, String tableName, String indexMapping,
+ String mappingType) {
+ JSONObject jsonObject = (JSONObject) JSONValue.parse(indexMapping);
+ // the indexName use alias takes the first mapping
+ Iterator<String> keys = jsonObject.keySet().iterator();
+ String docKey = keys.next();
+ JSONObject docData = (JSONObject) jsonObject.get(docKey);
+ JSONObject mappings = (JSONObject) docData.get("mappings");
+ JSONObject rootSchema = (JSONObject) mappings.get(mappingType);
+ JSONObject properties;
+ // After (include) 7.x, type was removed from ES mapping, default type is `_doc`
+ // https://www.elastic.co/guide/en/elasticsearch/reference/7.0/removal-of-types.html
+ if (rootSchema == null) {
+ if (!"_doc".equals(mappingType)) {
Review Comment:
Should we consider ES 8.x?
##########
fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/EsUtil.java:
##########
@@ -87,12 +93,129 @@ public static JSONObject getJsonObject(JSONObject jsonObject, String key, int fr
}
}
+ /**
+ * Get boolean field, if parse error throw DdlException.
+ **/
public static boolean getBoolean(Map<String, String> properties, String name) throws DdlException {
String property = properties.get(name).trim();
try {
return Boolean.parseBoolean(property);
} catch (Exception e) {
- throw new DdlException(String.format("fail to parse %s, %s = %s, `%s` should be like 'true' or 'false'， value should be double quotation marks", name, name, property, name));
+ throw new DdlException(String.format(
+ "fail to parse %s, %s = %s, `%s` should be like 'true' or 'false',"
+ + " value should be double quotation marks",
+ name, name, property, name));
+ }
+ }
+
+ /**
+ * Parse the required field information from the json.
+ *
+ * @param searchContext the current associated column searchContext
+ * @param indexMapping the return value of _mapping
+ */
+ public static void resolveFields(SearchContext searchContext, String indexMapping) throws DorisEsException {
+ JSONObject properties =
+ getMappingProps(searchContext.sourceIndex(), searchContext.esTable().getName(), indexMapping,
+ searchContext.type());
+ for (Column col : searchContext.columns()) {
+ String colName = col.getName();
+ // if column exists in Doris Table but no found in ES's mapping, we choose to ignore this situation?
Review Comment:
I think it would be better to throw an exception to user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org