You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "hubgeter (via GitHub)" <gi...@apache.org> on 2024/10/10 08:33:29 UTC
[PR] [Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable [doris]
hubgeter opened a new pull request, #41659:
URL: https://github.com/apache/doris/pull/41659
## Proposed changes
Because ExternalTable will initialize the previously uninitialized table when `getCachedRowCount()`, which is unnecessary. So for the uninitialized table, we directly return -2.
This will increase the speed of our query `information_schema.tables`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable [doris]
Posted by "hubgeter (via GitHub)" <gi...@apache.org>.
hubgeter commented on code in PR #41659:
URL: https://github.com/apache/doris/pull/41659#discussion_r1796782517
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java:
##########
@@ -665,4 +668,16 @@ public static String dataLocation(Table table) {
}
return dataLocation;
}
+
+ public static HiveCatalog createIcebergHiveCatalog(ExternalCatalog externalCatalog, String name) {
+ HiveCatalog hiveCatalog = new org.apache.iceberg.hive.HiveCatalog();
+ hiveCatalog.setConf(externalCatalog.getConfiguration());
+
+ Map<String, String> catalogProperties = externalCatalog.getProperties();
+ String metastoreUris = catalogProperties.getOrDefault(HMSProperties.HIVE_METASTORE_URIS, "");
+ catalogProperties.put(CatalogProperties.URI, metastoreUris);
+
+ hiveCatalog.initialize(name, catalogProperties);
+ return hiveCatalog;
Review Comment:
you can ref : https://github.com/apache/doris/pull/41659/files#diff-e6e9c0730301ce99354da7a68c322f1c94d656033b25ab525eab6f17eb9dce7dL41
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable [doris]
Posted by "hubgeter (via GitHub)" <gi...@apache.org>.
hubgeter commented on code in PR #41659:
URL: https://github.com/apache/doris/pull/41659#discussion_r1796781504
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java:
##########
@@ -665,4 +668,16 @@ public static String dataLocation(Table table) {
}
return dataLocation;
}
+
+ public static HiveCatalog createIcebergHiveCatalog(ExternalCatalog externalCatalog, String name) {
+ HiveCatalog hiveCatalog = new org.apache.iceberg.hive.HiveCatalog();
+ hiveCatalog.setConf(externalCatalog.getConfiguration());
+
+ Map<String, String> catalogProperties = externalCatalog.getProperties();
+ String metastoreUris = catalogProperties.getOrDefault(HMSProperties.HIVE_METASTORE_URIS, "");
+ catalogProperties.put(CatalogProperties.URI, metastoreUris);
+
+ hiveCatalog.initialize(name, catalogProperties);
+ return hiveCatalog;
Review Comment:
yes . so now IcebergHMSExternalCatalog.initCatalog() method use this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #41659:
URL: https://github.com/apache/doris/pull/41659
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #41659:
URL: https://github.com/apache/doris/pull/41659#discussion_r1796396660
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java:
##########
@@ -85,6 +88,8 @@ public class HMSExternalCatalog extends ExternalCatalog {
private int hmsEventsBatchSizePerRpc = -1;
private boolean enableHmsEventsIncrementalSync = false;
+ //for "type" = "hms" , but is iceberg table.
+ HiveCatalog icebergHiveCatalog;
Review Comment:
```suggestion
private HiveCatalog icebergHiveCatalog;
```
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalTable.java:
##########
@@ -203,14 +208,15 @@ public long getRowCount() {
@Override
public long getCachedRowCount() {
- // Return -1 if makeSureInitialized throw exception.
- // For example, init hive table may throw NotSupportedException.
- try {
- makeSureInitialized();
- } catch (Exception e) {
- LOG.warn("Failed to initialize table {}.{}.{}", catalog.getName(), dbName, name, e);
- return -1;
+ // Return -2 if uninitialized.
+ // Before this, for uninitialized tables, we would call makeSureInitialized(), just like the implementation of
+ // ExternalTable.getRowCount(), but this is not very meaningful and time-consuming.
+ // The getCachedRowCount() function is only used when `show table` and querying `information_schema.tables`.
+ if (!checkInitialized()) {
+ return -2;
Review Comment:
Use -1
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java:
##########
@@ -331,6 +336,13 @@ public boolean isEnableHmsEventsIncrementalSync() {
return enableHmsEventsIncrementalSync;
}
+ public Catalog getIcebergHiveCatalog() {
+ if (icebergHiveCatalog == null) {
Review Comment:
This method may be called concurrently
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataCache.java:
##########
@@ -177,29 +170,6 @@ public void invalidateDbCache(long catalogId, String dbName) {
});
}
- private Catalog createIcebergHiveCatalog(String uri, Map<String, String> hdfsConf, Map<String, String> props) {
- // set hdfs configure
- Configuration conf = DFSFileSystem.getHdfsConf(
- hdfsConf.getOrDefault(DFSFileSystem.PROP_ALLOW_FALLBACK_TO_SIMPLE_AUTH, "").isEmpty());
- for (Map.Entry<String, String> entry : hdfsConf.entrySet()) {
- conf.set(entry.getKey(), entry.getValue());
- }
- HiveCatalog hiveCatalog = new HiveCatalog();
- hiveCatalog.setConf(conf);
-
- if (props.containsKey(HMSExternalCatalog.BIND_BROKER_NAME)) {
Review Comment:
This part is missing?
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalTable.java:
##########
@@ -114,6 +114,11 @@ public boolean isView() {
return false;
}
+ public boolean checkInitialized() {
Review Comment:
No need this method, we can use check `objectCreated`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41659:
URL: https://github.com/apache/doris/pull/41659#issuecomment-2404452185
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable [doris]
Posted by "hubgeter (via GitHub)" <gi...@apache.org>.
hubgeter commented on code in PR #41659:
URL: https://github.com/apache/doris/pull/41659#discussion_r1796475626
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalTable.java:
##########
@@ -114,6 +114,11 @@ public boolean isView() {
return false;
}
+ public boolean checkInitialized() {
Review Comment:
i will remove this method.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalTable.java:
##########
@@ -203,14 +208,15 @@ public long getRowCount() {
@Override
public long getCachedRowCount() {
- // Return -1 if makeSureInitialized throw exception.
- // For example, init hive table may throw NotSupportedException.
- try {
- makeSureInitialized();
- } catch (Exception e) {
- LOG.warn("Failed to initialize table {}.{}.{}", catalog.getName(), dbName, name, e);
- return -1;
+ // Return -2 if uninitialized.
+ // Before this, for uninitialized tables, we would call makeSureInitialized(), just like the implementation of
+ // ExternalTable.getRowCount(), but this is not very meaningful and time-consuming.
+ // The getCachedRowCount() function is only used when `show table` and querying `information_schema.tables`.
+ if (!checkInitialized()) {
+ return -2;
Review Comment:
i will fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable [doris]
Posted by "hubgeter (via GitHub)" <gi...@apache.org>.
hubgeter commented on code in PR #41659:
URL: https://github.com/apache/doris/pull/41659#discussion_r1796472371
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataCache.java:
##########
@@ -177,29 +170,6 @@ public void invalidateDbCache(long catalogId, String dbName) {
});
}
- private Catalog createIcebergHiveCatalog(String uri, Map<String, String> hdfsConf, Map<String, String> props) {
- // set hdfs configure
- Configuration conf = DFSFileSystem.getHdfsConf(
- hdfsConf.getOrDefault(DFSFileSystem.PROP_ALLOW_FALLBACK_TO_SIMPLE_AUTH, "").isEmpty());
- for (Map.Entry<String, String> entry : hdfsConf.entrySet()) {
- conf.set(entry.getKey(), entry.getValue());
- }
- HiveCatalog hiveCatalog = new HiveCatalog();
- hiveCatalog.setConf(conf);
-
- if (props.containsKey(HMSExternalCatalog.BIND_BROKER_NAME)) {
Review Comment:
I kept this part and removed the `else` part. As it seemed to me that the two were duplicates. The `else` part just initialize hivecatalog use `uri`property.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable [doris]
Posted by "hubgeter (via GitHub)" <gi...@apache.org>.
hubgeter commented on code in PR #41659:
URL: https://github.com/apache/doris/pull/41659#discussion_r1796474210
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java:
##########
@@ -331,6 +336,13 @@ public boolean isEnableHmsEventsIncrementalSync() {
return enableHmsEventsIncrementalSync;
}
+ public Catalog getIcebergHiveCatalog() {
+ if (icebergHiveCatalog == null) {
Review Comment:
for prevent this, i will move icebergHiveCatalog init to initLocalObjectsImpl()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable [doris]
Posted by "hubgeter (via GitHub)" <gi...@apache.org>.
hubgeter commented on PR #41659:
URL: https://github.com/apache/doris/pull/41659#issuecomment-2405225772
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41659:
URL: https://github.com/apache/doris/pull/41659#issuecomment-2407100125
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable [doris]
Posted by "hubgeter (via GitHub)" <gi...@apache.org>.
hubgeter commented on code in PR #41659:
URL: https://github.com/apache/doris/pull/41659#discussion_r1796472371
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataCache.java:
##########
@@ -177,29 +170,6 @@ public void invalidateDbCache(long catalogId, String dbName) {
});
}
- private Catalog createIcebergHiveCatalog(String uri, Map<String, String> hdfsConf, Map<String, String> props) {
- // set hdfs configure
- Configuration conf = DFSFileSystem.getHdfsConf(
- hdfsConf.getOrDefault(DFSFileSystem.PROP_ALLOW_FALLBACK_TO_SIMPLE_AUTH, "").isEmpty());
- for (Map.Entry<String, String> entry : hdfsConf.entrySet()) {
- conf.set(entry.getKey(), entry.getValue());
- }
- HiveCatalog hiveCatalog = new HiveCatalog();
- hiveCatalog.setConf(conf);
-
- if (props.containsKey(HMSExternalCatalog.BIND_BROKER_NAME)) {
Review Comment:
I kept this part and removed the `else` part. As it seemed to me that the two were duplicates. The `else` part just initialize hivecatalog use `uri` propertie.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable [doris]
Posted by "hubgeter (via GitHub)" <gi...@apache.org>.
hubgeter commented on PR #41659:
URL: https://github.com/apache/doris/pull/41659#issuecomment-2406656441
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable [doris]
Posted by "hubgeter (via GitHub)" <gi...@apache.org>.
hubgeter commented on PR #41659:
URL: https://github.com/apache/doris/pull/41659#issuecomment-2404452439
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable [doris]
Posted by "hubgeter (via GitHub)" <gi...@apache.org>.
hubgeter commented on PR #41659:
URL: https://github.com/apache/doris/pull/41659#issuecomment-2404778028
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41659:
URL: https://github.com/apache/doris/pull/41659#issuecomment-2407100178
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable [doris]
Posted by "hubgeter (via GitHub)" <gi...@apache.org>.
hubgeter commented on code in PR #41659:
URL: https://github.com/apache/doris/pull/41659#discussion_r1796475890
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java:
##########
@@ -85,6 +88,8 @@ public class HMSExternalCatalog extends ExternalCatalog {
private int hmsEventsBatchSizePerRpc = -1;
private boolean enableHmsEventsIncrementalSync = false;
+ //for "type" = "hms" , but is iceberg table.
+ HiveCatalog icebergHiveCatalog;
Review Comment:
i will fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable [doris]
Posted by "wuwenchi (via GitHub)" <gi...@apache.org>.
wuwenchi commented on code in PR #41659:
URL: https://github.com/apache/doris/pull/41659#discussion_r1796767594
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java:
##########
@@ -665,4 +668,16 @@ public static String dataLocation(Table table) {
}
return dataLocation;
}
+
+ public static HiveCatalog createIcebergHiveCatalog(ExternalCatalog externalCatalog, String name) {
+ HiveCatalog hiveCatalog = new org.apache.iceberg.hive.HiveCatalog();
+ hiveCatalog.setConf(externalCatalog.getConfiguration());
+
+ Map<String, String> catalogProperties = externalCatalog.getProperties();
+ String metastoreUris = catalogProperties.getOrDefault(HMSProperties.HIVE_METASTORE_URIS, "");
+ catalogProperties.put(CatalogProperties.URI, metastoreUris);
+
+ hiveCatalog.initialize(name, catalogProperties);
+ return hiveCatalog;
Review Comment:
Is same as `IcebergHMSExternalCatalog.initCatalog` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org