You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/08/05 03:35:19 UTC
[GitHub] [doris] stalary opened a new pull request, #11547: [Bug](es) Fix es not support aliases error
stalary opened a new pull request, #11547:
URL: https://github.com/apache/doris/pull/11547
# Proposed changes
Issue Number: close #xxx
## Problem summary
1. Fix es not support aliases error
2. Fix multicatalog query es error
3. add ut
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
4. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
5. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
6. Does it need to update dependencies:
- [ ] Yes
- [ ] No
7. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] stalary commented on a diff in pull request #11547: [Bug](es) Fix es not support aliases error
Posted by GitBox <gi...@apache.org>.
stalary commented on code in PR #11547:
URL: https://github.com/apache/doris/pull/11547#discussion_r939838763
##########
fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/EsUtil.java:
##########
@@ -168,7 +171,7 @@ public static JSONObject getMappingProps(String sourceIndex, String indexMapping
if (rootSchema == null) {
properties = (JSONObject) mappings.get("properties");
// Compatible es6 with no type passed in.
- if (mappingType == null) {
+ if (mappingType == null && properties == null) {
Review Comment:
Okay, so I'm going to refactor this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] stalary commented on a diff in pull request #11547: [Bug](es) Fix es not support aliases error
Posted by GitBox <gi...@apache.org>.
stalary commented on code in PR #11547:
URL: https://github.com/apache/doris/pull/11547#discussion_r939811286
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/EsExternalDataSource.java:
##########
@@ -37,14 +37,15 @@
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
+import java.util.stream.Collectors;
/**
* External data source for elasticsearch
*/
@Getter
public class EsExternalDataSource extends ExternalDataSource {
- public static final String DEFAULT_DB = "default";
+ public static final String DEFAULT_DB = "default_es_db";
Review Comment:
A catalog can only support one cluster, so it would be easier to use a static name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] stalary commented on a diff in pull request #11547: [Bug](es) Fix es not support aliases error
Posted by GitBox <gi...@apache.org>.
stalary commented on code in PR #11547:
URL: https://github.com/apache/doris/pull/11547#discussion_r939679536
##########
fe/fe-core/src/test/resources/data/es/es7_aliases_mapping.json:
##########
@@ -0,0 +1,50 @@
+{
+ "test2_202207": {
+ "mappings": {
Review Comment:
My problem, I'm going to replace it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman merged pull request #11547: [Bug](Doris On ES) Fix es not support aliases error
Posted by GitBox <gi...@apache.org>.
morningman merged PR #11547:
URL: https://github.com/apache/doris/pull/11547
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11547: [Bug](es) Fix es not support aliases error
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #11547:
URL: https://github.com/apache/doris/pull/11547#issuecomment-1207576118
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wuyunfeng commented on a diff in pull request #11547: [Bug](es) Fix es not support aliases error
Posted by GitBox <gi...@apache.org>.
wuyunfeng commented on code in PR #11547:
URL: https://github.com/apache/doris/pull/11547#discussion_r939828810
##########
fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/EsUtil.java:
##########
@@ -168,7 +171,7 @@ public static JSONObject getMappingProps(String sourceIndex, String indexMapping
if (rootSchema == null) {
properties = (JSONObject) mappings.get("properties");
// Compatible es6 with no type passed in.
- if (mappingType == null) {
+ if (mappingType == null && properties == null) {
Review Comment:
OK,we should change the location
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wuyunfeng commented on a diff in pull request #11547: [Bug](es) Fix es not support aliases error
Posted by GitBox <gi...@apache.org>.
wuyunfeng commented on code in PR #11547:
URL: https://github.com/apache/doris/pull/11547#discussion_r939795388
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/EsExternalDataSource.java:
##########
@@ -166,7 +167,10 @@ public List<String> listDatabaseNames(SessionContext ctx) {
@Override
public List<String> listTableNames(SessionContext ctx, String dbName) {
- return esRestClient.getIndexes();
+ List<String> indexes = esRestClient.getIndexes().stream().distinct().collect(Collectors.toList());
Review Comment:
I think maybe package this line into esRestClient is better, provide `listTable` method
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/EsExternalDataSource.java:
##########
@@ -37,14 +37,15 @@
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
+import java.util.stream.Collectors;
/**
* External data source for elasticsearch
*/
@Getter
public class EsExternalDataSource extends ExternalDataSource {
- public static final String DEFAULT_DB = "default";
+ public static final String DEFAULT_DB = "default_es_db";
Review Comment:
can we use the `cluster_name`_`uuid` for DB?
##########
fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/EsUtil.java:
##########
@@ -168,7 +171,7 @@ public static JSONObject getMappingProps(String sourceIndex, String indexMapping
if (rootSchema == null) {
properties = (JSONObject) mappings.get("properties");
// Compatible es6 with no type passed in.
- if (mappingType == null) {
+ if (mappingType == null && properties == null) {
Review Comment:
&& properties == null maybe not necessary, you check the context again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wuyunfeng commented on a diff in pull request #11547: [Bug](es) Fix es not support aliases error
Posted by GitBox <gi...@apache.org>.
wuyunfeng commented on code in PR #11547:
URL: https://github.com/apache/doris/pull/11547#discussion_r939990635
##########
fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/EsRestClient.java:
##########
@@ -158,6 +162,38 @@ public List<String> getIndexes() {
return ret;
}
+ /**
+ * Get all aliases.
+ **/
+ public Map<String, List<String>> getAliases() {
+ String res = execute("_aliases");
+ Map<String, List<String>> ret = new HashMap<>();
+ JsonNode root = JsonUtil.readTree(res);
+ if (root == null) {
+ return ret;
+ }
+ Iterator<Map.Entry<String, JsonNode>> elements = root.fields();
+ while (elements.hasNext()) {
+ Map.Entry<String, JsonNode> element = elements.next();
+ JsonNode aliases = element.getValue().get("aliases");
+ Iterator<String> aliasNames = aliases.fieldNames();
+ if (aliasNames.hasNext()) {
+ ret.put(element.getKey(), ImmutableList.copyOf(aliasNames));
+ }
+ }
+ return ret;
+ }
+
+ /**
+ * Returns the merge of index and alias
+ **/
+ public List<String> listTable() {
+ List<String> indexes = getIndices().stream().distinct().collect(Collectors.toList());
Review Comment:
```suggestion
List<String> indices = getIndices().stream().distinct().collect(Collectors.toList());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] stalary commented on a diff in pull request #11547: [Bug](es) Fix es not support aliases error
Posted by GitBox <gi...@apache.org>.
stalary commented on code in PR #11547:
URL: https://github.com/apache/doris/pull/11547#discussion_r939734077
##########
fe/fe-core/src/test/resources/data/es/es7_aliases_mapping.json:
##########
@@ -0,0 +1,50 @@
+{
+ "test2_202207": {
+ "mappings": {
Review Comment:
`JSONObject test2Aliases = EsUtil.getMappingProps("test2", loadJsonFromFile("data/es/es7_aliases_mapping.json"),
null);` _mapping not contains aliases, I use test2 to get real index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11547: [Bug](es) Fix es not support aliases error
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #11547:
URL: https://github.com/apache/doris/pull/11547#issuecomment-1207848799
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] stalary commented on a diff in pull request #11547: [Bug](es) Fix es not support aliases error
Posted by GitBox <gi...@apache.org>.
stalary commented on code in PR #11547:
URL: https://github.com/apache/doris/pull/11547#discussion_r939679754
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/EsExternalDataSource.java:
##########
@@ -166,7 +167,10 @@ public List<String> listDatabaseNames(SessionContext ctx) {
@Override
public List<String> listTableNames(SessionContext ctx, String dbName) {
- return esRestClient.getIndexes();
+ List<String> indexes = esRestClient.getIndexes().stream().distinct().collect(Collectors.toList());
+ esRestClient.getAliases().entrySet().stream().filter(e -> indexes.contains(e.getKey()))
Review Comment:
Currently, the method of geting the mapping is compatible. Alias and index can be queried. Therefore, the mapping relationship is not maintained.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] stalary commented on a diff in pull request #11547: [Bug](es) Fix es not support aliases error
Posted by GitBox <gi...@apache.org>.
stalary commented on code in PR #11547:
URL: https://github.com/apache/doris/pull/11547#discussion_r939811382
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/EsExternalDataSource.java:
##########
@@ -166,7 +167,10 @@ public List<String> listDatabaseNames(SessionContext ctx) {
@Override
public List<String> listTableNames(SessionContext ctx, String dbName) {
- return esRestClient.getIndexes();
+ List<String> indexes = esRestClient.getIndexes().stream().distinct().collect(Collectors.toList());
Review Comment:
Good idea. Let me modify it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] stalary commented on a diff in pull request #11547: [Bug](es) Fix es not support aliases error
Posted by GitBox <gi...@apache.org>.
stalary commented on code in PR #11547:
URL: https://github.com/apache/doris/pull/11547#discussion_r939679536
##########
fe/fe-core/src/test/resources/data/es/es7_aliases_mapping.json:
##########
@@ -0,0 +1,50 @@
+{
+ "test2_202207": {
+ "mappings": {
Review Comment:
My problem, I'm going to replace it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qidaye commented on a diff in pull request #11547: [Bug](es) Fix es not support aliases error
Posted by GitBox <gi...@apache.org>.
qidaye commented on code in PR #11547:
URL: https://github.com/apache/doris/pull/11547#discussion_r939676905
##########
fe/fe-core/src/test/resources/data/es/es7_aliases_mapping.json:
##########
@@ -0,0 +1,50 @@
+{
+ "test2_202207": {
+ "mappings": {
Review Comment:
Where is `aliases` field?
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/EsExternalDataSource.java:
##########
@@ -166,7 +167,10 @@ public List<String> listDatabaseNames(SessionContext ctx) {
@Override
public List<String> listTableNames(SessionContext ctx, String dbName) {
- return esRestClient.getIndexes();
+ List<String> indexes = esRestClient.getIndexes().stream().distinct().collect(Collectors.toList());
+ esRestClient.getAliases().entrySet().stream().filter(e -> indexes.contains(e.getKey()))
Review Comment:
Do we treat `alias` as an independent index?
Should we keep the correspondence between the alias and the real index?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11547: [Bug](es) Fix es not support aliases error
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #11547:
URL: https://github.com/apache/doris/pull/11547#issuecomment-1207576162
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] stalary commented on a diff in pull request #11547: [Bug](es) Fix es not support aliases error
Posted by GitBox <gi...@apache.org>.
stalary commented on code in PR #11547:
URL: https://github.com/apache/doris/pull/11547#discussion_r939811117
##########
fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/EsUtil.java:
##########
@@ -168,7 +171,7 @@ public static JSONObject getMappingProps(String sourceIndex, String indexMapping
if (rootSchema == null) {
properties = (JSONObject) mappings.get("properties");
// Compatible es6 with no type passed in.
- if (mappingType == null) {
+ if (mappingType == null && properties == null) {
Review Comment:
The logic that es6 does not pass type is special, and this should not be entered in es7 or later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org