You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "heguanhui (via GitHub)" <gi...@apache.org> on 2025/05/30 04:17:38 UTC
[PR] support iceberg view query [doris]
heguanhui opened a new pull request, #51376:
URL: https://github.com/apache/doris/pull/51376
### What problem does this PR solve?
This feature addresses the issue that Doris cannot query Iceberg views.
1. Prerequisite Dependencies
Iceberg version 1.7.x+
2. Design Approach
- Enhanced View Loading in IcebergMetadataCache:
Added view loading capability during loadTable operations for Iceberg.
Introduced getIcebergView() method to retrieve Iceberg views.
- New Methods in IcebergExternalTable:
isView(): Determines if the current Iceberg table is a view.
getViewText(): Retrieves the SQL definition query of the view.
- Execution Plan Generation Adjustment:
Checks isView() during plan generation to identify Iceberg views.
For views, generates a logical subquery execution plan by flattening the view into a subquery.
- Cache Invalidation Enhancement:
Added view cache invalidation in IcebergMetadataCache when invalidating catalog/database/table caches.
### Problem Summary:
This PR aims to resolve the issue where querying an Iceberg view in Doris results in an error indicating the table does not exist. Specifically, it addresses the problem that Doris currently cannot recognize and query Iceberg views, causing exceptions when users attempt to access view data.
### Check List (For Author)
Test (At least one of them must be included):
Manual test scripts and report as below:
[回归测试用例.docx](https://github.com/user-attachments/files/20514382/default.docx)
### Behavior changed:
Yes.
- BindRelation.java
Modified getLogicalPlan method: Adjusted logic for Iceberg external tables to generate subquery execution plans when encountering views.
Renamed parseAndAnalyzeHiveView to parseAndAnalyzeExternalView: Unified handling for external views (Hive/Iceberg).
- Config.java
**(1)** Added enable_query_iceberg_views: Configuration flag to enable/disable Iceberg view query support.
- IcebergExternalTable.java
**(1)** Enhanced initSchema method: Added schema initialization compatibility for both Iceberg tables and views.
**(2)** New getViewText method: Retrieves the SQL definition of an Iceberg view.
- IcebergMetadataCache.java
**(1)** Constructor Initialization: Added cache loading for Iceberg views.
**(2)** Invalidation Methods: Extended invalidateCatalogCache, invalidateTableCache, invalidateDbCache to include view cache invalidation.
New Methods:
**(1)** loadView: Loads Iceberg view metadata.
**(2)** getIcebergView: Retrieves cached Iceberg view information.
- IcebergMetadataOps.java
**(1)** Modified listTableNames: Returns combined list of tables and views.
New Methods:
**(1)** viewExists: Checks if an Iceberg view exists.
**(2)** loadView: Loads Iceberg view metadata.
- IcebergUtils.java
Modified Methods:
**(1)** getSchema: Supports schema retrieval for both tables and views.
**(2)** loadSchemaCacheValue: Handles schema caching for both types.
New Methods:
**(1)** getIcebergView/getIcebergViewInternal: Retrieve Iceberg view objects.
**(2)** getConvertedSchema: Converts Iceberg schema to Doris-compatible format.
- ExternalMetadataOps.java
New Interface Methods:
**(1)** loadView: Load external view metadata.
**(2)** - viewExists: Check view existence.
**(3)** listViewNames: List available views.
- IcebergExternalCatalog.java
**(1)** Modified listTableNames: Returns combined tables and views.
**(2)** Overridden viewExists: Implements view existence check.
- ExternalCatalog.java
**(1)** New Interface Method: viewExists to check view existence.
- ExternalTable.java
New Methods:
**(1)** isTable: Determines if the object is a table (vs. view).
**(2)** getRemoteDbName: Retrieves the remote database name.
- HMSExternalTable.java
Modified Methods:
**(1)** initSchema: Added view compatibility.
**(2)** getIcebergSchema: Supports both tables and views.
**(3)** Overridden isTable: Implements table type check.
- IcebergApiSource.java
**(1)** Constructor Adjustment: Added view type validation to prevent invalid operations.
- IcebergTableSink.java
**(1)** Constructor Adjustment: Added view type validation to prevent invalid operations.
Does this need documentation?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #51376:
URL: https://github.com/apache/doris/pull/51376#issuecomment-2983645772
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #51376:
URL: https://github.com/apache/doris/pull/51376#issuecomment-2982623367
# FE UT Coverage Report
Increment line coverage ` 4.29% (9/210)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/51376_ff0969c50c761c36cd31b4690d39acd0d0e997b5/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/51376_ff0969c50c761c36cd31b4690d39acd0d0e997b5/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #51376:
URL: https://github.com/apache/doris/pull/51376#issuecomment-2982500877
<details>
<summary>ClickBench: <b>Total hot run time: 29.74 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ff0969c50c761c36cd31b4690d39acd0d0e997b5, data reload: false
query1 0.04 0.04 0.03
query2 0.07 0.04 0.04
query3 0.23 0.07 0.07
query4 1.62 0.10 0.10
query5 0.45 0.42 0.42
query6 1.20 0.64 0.67
query7 0.03 0.01 0.01
query8 0.05 0.04 0.04
query9 0.56 0.52 0.51
query10 0.57 0.58 0.56
query11 0.15 0.11 0.11
query12 0.15 0.12 0.12
query13 0.63 0.61 0.62
query14 0.80 0.82 0.85
query15 0.89 0.88 0.86
query16 0.40 0.37 0.38
query17 1.10 1.04 1.04
query18 0.23 0.22 0.21
query19 1.93 1.85 1.86
query20 0.02 0.01 0.01
query21 15.40 0.90 0.56
query22 0.77 1.22 0.66
query23 14.90 1.37 0.63
query24 7.40 1.43 0.91
query25 0.50 0.21 0.08
query26 0.61 0.16 0.14
query27 0.07 0.05 0.05
query28 9.69 0.87 0.45
query29 12.56 4.02 3.34
query30 0.26 0.09 0.06
query31 2.83 0.60 0.40
query32 3.26 0.56 0.48
query33 3.15 3.12 3.09
query34 16.00 5.38 4.81
query35 4.85 4.83 4.90
query36 0.68 0.50 0.49
query37 0.08 0.06 0.06
query38 0.05 0.04 0.04
query39 0.03 0.02 0.02
query40 0.18 0.15 0.15
query41 0.09 0.02 0.02
query42 0.03 0.03 0.02
query43 0.04 0.04 0.03
Total cold run time: 104.55 s
Total hot run time: 29.74 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #51376:
URL: https://github.com/apache/doris/pull/51376#issuecomment-2979861583
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #51376:
URL: https://github.com/apache/doris/pull/51376#issuecomment-2989885291
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #51376:
URL: https://github.com/apache/doris/pull/51376#issuecomment-2927428427
And please add necessary UT and regression test for these feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #51376:
URL: https://github.com/apache/doris/pull/51376#issuecomment-2991703523
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on code in PR #51376:
URL: https://github.com/apache/doris/pull/51376#discussion_r2115222879
##########
fe/fe-core/src/main/java/org/apache/doris/planner/IcebergTableSink.java:
##########
@@ -83,6 +86,9 @@ public void bindDataSink(Optional<InsertCommandContext> insertCtx)
throws AnalysisException {
TIcebergTableSink tSink = new TIcebergTableSink();
+ if (targetTable.isView()) {
+ throw new UnsupportedOperationException("IcebergTableSink does not support view");
+ }
Review Comment:
already fixed and committed
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java:
##########
@@ -372,6 +389,48 @@ public Table loadTable(String dbName, String tblName) {
}
}
+ @Override
+ public boolean viewExists(String dbName, String viewName) {
Review Comment:
already fixed and committed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "wuwenchi (via GitHub)" <gi...@apache.org>.
wuwenchi commented on code in PR #51376:
URL: https://github.com/apache/doris/pull/51376#discussion_r2123825479
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java:
##########
@@ -274,4 +291,64 @@ public List<SysTable> getSupportedSysTables() {
makeSureInitialized();
return SupportedSysTables.ICEBERG_SUPPORTED_SYS_TABLES;
}
+
+ @Override
+ public boolean isTable() {
+ return isTable;
+ }
+
+ @Override
+ public boolean isView() {
+ return isView;
+ }
+
+ public String getViewText() {
+ try {
+ return catalog.getPreExecutionAuthenticator().execute(() -> {
+ View icebergView = IcebergUtils.getIcebergView(getCatalog(), getRemoteDbName(), getRemoteName());
+ ViewVersion viewVersion = icebergView.currentVersion();
+ if (viewVersion == null) {
+ throw new RuntimeException(String.format("Cannot get view version for view '%s'", icebergView));
+ }
+ Map<String, String> summary = viewVersion.summary();
+ if (summary == null) {
+ throw new RuntimeException(String.format("Cannot get summary for view '%s'", icebergView));
+ }
+ String engineName = summary.get("engine-name");
+ if (StringUtils.isEmpty(engineName)) {
+ throw new RuntimeException(String.format("Cannot get engine-name for view '%s'", icebergView));
+ }
+ SQLViewRepresentation sqlViewRepresentation = icebergView.sqlFor(engineName.toLowerCase());
+ if (sqlViewRepresentation == null) {
+ throw new UnsupportedOperationException("Cannot get view text from iceberg view");
+ }
+ return sqlViewRepresentation.sql();
+ });
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public String getSqlDialect() {
+ try {
+ return catalog.getPreExecutionAuthenticator().execute(() -> {
+ View icebergView = IcebergUtils.getIcebergView(getCatalog(), getRemoteDbName(), getRemoteName());
+ ViewVersion viewVersion = icebergView.currentVersion();
+ if (viewVersion == null) {
+ throw new RuntimeException(String.format("Cannot get view version for view '%s'", icebergView));
+ }
+ Map<String, String> summary = viewVersion.summary();
+ if (summary == null) {
+ throw new RuntimeException(String.format("Cannot get summary for view '%s'", icebergView));
+ }
+ String engineName = summary.get("engine-name");
Review Comment:
Here, you can use a constant to represent "engine-name".
The same applies to other places.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java:
##########
@@ -274,4 +291,64 @@ public List<SysTable> getSupportedSysTables() {
makeSureInitialized();
return SupportedSysTables.ICEBERG_SUPPORTED_SYS_TABLES;
}
+
+ @Override
+ public boolean isTable() {
+ return isTable;
+ }
+
+ @Override
+ public boolean isView() {
+ return isView;
+ }
+
+ public String getViewText() {
+ try {
+ return catalog.getPreExecutionAuthenticator().execute(() -> {
+ View icebergView = IcebergUtils.getIcebergView(getCatalog(), getRemoteDbName(), getRemoteName());
+ ViewVersion viewVersion = icebergView.currentVersion();
+ if (viewVersion == null) {
+ throw new RuntimeException(String.format("Cannot get view version for view '%s'", icebergView));
+ }
+ Map<String, String> summary = viewVersion.summary();
+ if (summary == null) {
+ throw new RuntimeException(String.format("Cannot get summary for view '%s'", icebergView));
+ }
+ String engineName = summary.get("engine-name");
+ if (StringUtils.isEmpty(engineName)) {
+ throw new RuntimeException(String.format("Cannot get engine-name for view '%s'", icebergView));
+ }
+ SQLViewRepresentation sqlViewRepresentation = icebergView.sqlFor(engineName.toLowerCase());
Review Comment:
If there are multiple dialects, will there be any issues here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #51376:
URL: https://github.com/apache/doris/pull/51376#issuecomment-2983775705
<details>
<summary>TPC-DS: <b>Total hot run time: 187047 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 79097b1953e6a9f086a82d25d8683f1abf48adfe, data reload: false
query1 1006 401 393 393
query2 6542 1929 1931 1929
query3 6750 229 221 221
query4 26337 24092 23710 23710
query5 4343 643 497 497
query6 306 214 203 203
query7 4634 495 305 305
query8 270 237 220 220
query9 8613 2650 2660 2650
query10 483 338 273 273
query11 15882 15152 14874 14874
query12 171 111 110 110
query13 1658 538 409 409
query14 9068 6223 6019 6019
query15 208 198 186 186
query16 7164 608 461 461
query17 1192 732 567 567
query18 1981 402 314 314
query19 192 190 182 182
query20 129 122 114 114
query21 214 121 117 117
query22 4079 4083 4043 4043
query23 34013 33095 33124 33095
query24 8431 2371 2434 2371
query25 564 490 402 402
query26 1220 266 153 153
query27 2764 510 352 352
query28 4318 2139 2122 2122
query29 780 560 455 455
query30 288 223 194 194
query31 925 855 746 746
query32 74 63 93 63
query33 546 365 315 315
query34 808 856 559 559
query35 787 810 767 767
query36 973 998 886 886
query37 111 99 82 82
query38 4093 4252 4000 4000
query39 1475 1415 1412 1412
query40 215 118 107 107
query41 63 61 59 59
query42 132 111 123 111
query43 510 528 491 491
query44 1362 822 824 822
query45 205 173 171 171
query46 850 1039 633 633
query47 1755 1755 1717 1717
query48 386 440 318 318
query49 768 480 401 401
query50 682 689 407 407
query51 4172 4264 4011 4011
query52 114 108 97 97
query53 226 255 180 180
query54 577 574 508 508
query55 88 80 83 80
query56 300 305 299 299
query57 1183 1204 1130 1130
query58 277 258 266 258
query59 2641 2727 2733 2727
query60 338 332 317 317
query61 130 124 126 124
query62 800 729 648 648
query63 238 197 197 197
query64 4458 1102 765 765
query65 4284 4216 4216 4216
query66 1171 410 311 311
query67 15849 15663 15470 15470
query68 7766 885 532 532
query69 472 318 265 265
query70 1151 1133 1136 1133
query71 429 335 293 293
query72 5511 4783 4792 4783
query73 649 633 351 351
query74 9270 9174 8785 8785
query75 3508 3213 2736 2736
query76 3372 1212 785 785
query77 751 420 313 313
query78 10064 10098 9365 9365
query79 2292 858 595 595
query80 599 519 454 454
query81 501 273 227 227
query82 449 131 101 101
query83 258 255 240 240
query84 248 112 91 91
query85 798 348 318 318
query86 383 306 279 279
query87 4485 4577 4430 4430
query88 3981 2280 2286 2280
query89 387 353 287 287
query90 1853 210 213 210
query91 141 146 116 116
query92 79 66 61 61
query93 1875 958 594 594
query94 695 416 314 314
query95 375 299 293 293
query96 501 575 283 283
query97 2744 2766 2686 2686
query98 238 217 205 205
query99 1469 1383 1296 1296
Total cold run time: 274460 ms
Total hot run time: 187047 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #51376:
URL: https://github.com/apache/doris/pull/51376#issuecomment-2980113798
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #51376:
URL: https://github.com/apache/doris/pull/51376#discussion_r2119245161
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalTable.java:
##########
@@ -466,4 +466,13 @@ public boolean equals(Object o) {
public int hashCode() {
return Objects.hashCode(name, db);
}
+
+ public boolean isTable() {
+ return true;
+ }
+
+ public String getRemoteDbName() {
Review Comment:
```suggestion
protected String getRemoteDbName() {
```
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java:
##########
@@ -274,4 +287,41 @@ public List<SysTable> getSupportedSysTables() {
makeSureInitialized();
return SupportedSysTables.ICEBERG_SUPPORTED_SYS_TABLES;
}
+
+ @Override
+ public boolean isTable() {
+ return catalog.tableExist(null, getRemoteDbName(), getRemoteName());
+ }
+
+ @Override
+ public boolean isView() {
+ return catalog.viewExists(getRemoteDbName(), getRemoteName());
+ }
+
+ public String getViewText() {
Review Comment:
Can we cache the view text somewhere?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java:
##########
@@ -422,6 +423,23 @@ private LogicalPlan getLogicalPlan(TableIf table, UnboundRelation unboundRelatio
unboundRelation.getTableSnapshot(), ImmutableList.of());
}
case ICEBERG_EXTERNAL_TABLE:
+ IcebergExternalTable icebergExternalTable = (IcebergExternalTable) table;
+ if (Config.enable_query_iceberg_views && icebergExternalTable.isView()) {
+ isView = true;
+ String hiveCatalog = icebergExternalTable.getCatalog().getName();
Review Comment:
Change name of `hiveCatalog` and `hiveDb`
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java:
##########
@@ -1304,4 +1304,9 @@ public void notifyPropertiesUpdated(Map<String, String> updatedProps) {
Env.getCurrentEnv().getExtMetaCacheMgr().invalidSchemaCache(id);
}
}
+
+ public boolean viewExists(String dbName, String viewName) {
Review Comment:
Add comment for this interface
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java:
##########
@@ -274,4 +287,41 @@ public List<SysTable> getSupportedSysTables() {
makeSureInitialized();
return SupportedSysTables.ICEBERG_SUPPORTED_SYS_TABLES;
}
+
+ @Override
+ public boolean isTable() {
+ return catalog.tableExist(null, getRemoteDbName(), getRemoteName());
Review Comment:
Do we need to visit catalog service to check if this is a table or view?
I think we can cache this info, such as a `isView` field in this class and only visit catalog service once?
Same suggestion for `isView()` method.
And I think these operation can be done when initializing this object.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java:
##########
@@ -654,6 +654,9 @@ public long getLastDdlTime() {
public Optional<SchemaCacheValue> initSchema(SchemaCacheKey key) {
makeSureInitialized();
if (dlaType.equals(DLAType.ICEBERG)) {
+ if (isView()) {
+ throw new RuntimeException("Cannot use ICEBERG schema when view is used");
Review Comment:
So if I execute `describe iceberg_view`, I will get this error?
And I see in `IcebergExternalTable`, the `initSchema()` can work with view, which is not consistent with logic here.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalTable.java:
##########
@@ -466,4 +466,13 @@ public boolean equals(Object o) {
public int hashCode() {
return Objects.hashCode(name, db);
}
+
+ public boolean isTable() {
Review Comment:
```suggestion
protected boolean isTable() {
```
And add comment for this interface
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataCache.java:
##########
@@ -270,4 +290,25 @@ public Map<String, Map<String, String>> getCacheStats() {
snapshotCache.estimatedSize()));
return res;
}
+
+ private View loadView(IcebergMetadataCacheKey key) {
+ IcebergMetadataOps ops;
+ if (key.catalog instanceof IcebergExternalCatalog) {
Review Comment:
What about `HMSExternalCatlaog`?
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java:
##########
@@ -1107,4 +1093,47 @@ public static IcebergSnapshotCacheValue getOrFetchSnapshotCacheValue(
return IcebergUtils.getIcebergSnapshotCacheValue(Optional.empty(), catalog, dbName, tbName);
}
}
+
+ public static org.apache.iceberg.view.View getIcebergView(ExternalCatalog catalog, String dbName, String tblName) {
+ return getIcebergViewInternal(catalog, dbName, tblName);
+ }
+
+ private static org.apache.iceberg.view.View getIcebergViewInternal(ExternalCatalog catalog, String dbName,
+ String tblName) {
+ IcebergMetadataCache metadataCache = Env.getCurrentEnv().getExtMetaCacheMgr().getIcebergMetadataCache();
+ return metadataCache.getIcebergView(catalog, dbName, tblName);
+ }
+
+ public static Optional<SchemaCacheValue> loadSchemaCacheValue(
+ ExternalCatalog catalog, String dbName, String tbName, long schemaId, boolean isView) {
+ List<Column> schema = IcebergUtils.getSchema(catalog, dbName, tbName, schemaId, isView);
+ List<Column> tmpColumns = Lists.newArrayList();
+ if (!isView) {
+ // get table partition column info
+ Table table = IcebergUtils.getIcebergTable(catalog, dbName, tbName);
+ PartitionSpec spec = table.spec();
+ for (PartitionField field : spec.fields()) {
+ Types.NestedField col = table.schema().findField(field.sourceId());
+ for (Column c : schema) {
+ if (c.getName().equalsIgnoreCase(col.name())) {
+ tmpColumns.add(c);
+ break;
+ }
+ }
+ }
+ }
+ return Optional.of(new IcebergSchemaCacheValue(schema, tmpColumns));
+ }
+
+ public static List<Column> getConvertedSchema(Schema schema) {
Review Comment:
```suggestion
private static List<Column> getConvertedSchema(Schema schema) {
```
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java:
##########
@@ -274,4 +287,41 @@ public List<SysTable> getSupportedSysTables() {
makeSureInitialized();
return SupportedSysTables.ICEBERG_SUPPORTED_SYS_TABLES;
}
+
+ @Override
+ public boolean isTable() {
+ return catalog.tableExist(null, getRemoteDbName(), getRemoteName());
+ }
+
+ @Override
+ public boolean isView() {
+ return catalog.viewExists(getRemoteDbName(), getRemoteName());
+ }
+
+ public String getViewText() {
+ try {
+ return catalog.getPreExecutionAuthenticator().execute(() -> {
+ View icebergView = IcebergUtils.getIcebergView(getCatalog(), getRemoteDbName(), getRemoteName());
+ ViewVersion viewVersion = icebergView.currentVersion();
+ if (viewVersion == null) {
+ throw new RuntimeException(String.format("Cannot get view version for view '%s'", icebergView));
+ }
+ Map<String, String> summary = viewVersion.summary();
+ if (summary == null) {
+ throw new RuntimeException(String.format("Cannot get summary for view '%s'", icebergView));
+ }
+ String engineName = summary.get("engine-name");
+ if (StringUtils.isEmpty(engineName)) {
+ throw new RuntimeException(String.format("Cannot get engine-name for view '%s'", icebergView));
+ }
+ SQLViewRepresentation sqlViewRepresentation = icebergView.sqlFor(engineName.toLowerCase());
+ if (sqlViewRepresentation == null) {
+ throw new UnsupportedOperationException("Cannot get view text from iceberg view");
+ }
+ return sqlViewRepresentation.sql();
Review Comment:
I think we should consider the dialect of the view sql?
At least add add a placeholder interface for further extend.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java:
##########
@@ -126,7 +130,20 @@ public List<String> listTableNames(String dbName) {
try {
return preExecutionAuthenticator.execute(() -> {
List<TableIdentifier> tableIdentifiers = catalog.listTables(getNamespace(dbName));
- return tableIdentifiers.stream().map(TableIdentifier::name).collect(Collectors.toList());
+ List<String> views;
+ if (catalog instanceof ViewCatalog) {
+ views = ((ViewCatalog) catalog).listViews(getNamespace(dbName))
+ .stream().map(TableIdentifier::name).collect(Collectors.toList());
+ } else {
+ views = Collections.emptyList();
+ }
+ if (views.isEmpty()) {
+ return tableIdentifiers.stream().map(TableIdentifier::name).collect(Collectors.toList());
+ } else {
+ return tableIdentifiers.stream()
Review Comment:
Add comment to explain this logic
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataCache.java:
##########
@@ -52,6 +53,7 @@ public class IcebergMetadataCache {
private final LoadingCache<IcebergMetadataCacheKey, List<Snapshot>> snapshotListCache;
private final LoadingCache<IcebergMetadataCacheKey, Table> tableCache;
private final LoadingCache<IcebergMetadataCacheKey, IcebergSnapshotCacheValue> snapshotCache;
+ private final LoadingCache<IcebergMetadataCacheKey, View> viewCache;
Review Comment:
And comment to explain why need this cache and the difference between `viewCache` and `tableCache`
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java:
##########
@@ -460,8 +478,8 @@ private LogicalPlan getLogicalPlan(TableIf table, UnboundRelation unboundRelatio
}
}
- private Plan parseAndAnalyzeHiveView(
- HMSExternalTable table, String hiveCatalog, String hiveDb, String ddlSql, CascadesContext cascadesContext) {
+ private Plan parseAndAnalyzeExternalView(
Review Comment:
Here we should consider the dialect info in iceberg view's definition and dialect info user set in session variable.
If they are different, need to do something to handle it, like report an error.
For example, if the view's dialect is spark, and user specified session dialect is presto, then it will confuse the sql convertor about which dialect should it use to convert the view sql.
##########
fe/pom.xml:
##########
@@ -319,7 +319,7 @@ under the License.
<!-- ATTN: avro version must be consistent with Iceberg version -->
<!-- Please modify iceberg.version and avro.version together,
you can find avro version info in iceberg mvn repository -->
- <iceberg.version>1.6.1</iceberg.version>
+ <iceberg.version>1.7.1</iceberg.version>
Review Comment:
we should also upgrade the version in `https://github.com/apache/doris-shade`
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/source/IcebergApiSource.java:
##########
@@ -42,6 +42,9 @@ public class IcebergApiSource implements IcebergSource {
public IcebergApiSource(IcebergExternalTable table, TupleDescriptor desc,
Map<String, ColumnRange> columnNameToRange) {
+ if (table.isView()) {
Review Comment:
`IcebergApiSource` is created in `IcebergScanNode`.
So if this is a view, I can not query it?
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/operations/ExternalMetadataOps.java:
##########
@@ -173,4 +176,17 @@ default Object loadTable(String dbName, String tblName) {
* close the connection, eg, to hms
*/
void close();
+
+ default View loadView(String dbName, String viewName) {
Review Comment:
Add comment for all newly added interfaces
##########
fe/fe-core/src/main/java/org/apache/doris/planner/IcebergTableSink.java:
##########
@@ -59,6 +59,9 @@ public class IcebergTableSink extends BaseExternalTableDataSink {
public IcebergTableSink(IcebergExternalTable targetTable) {
super();
+ if (targetTable.isView()) {
+ throw new UnsupportedOperationException("IcebergTableSink does not support view");
Review Comment:
```suggestion
throw new UnsupportedOperationException("Write data to iceberg view is not supported");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #51376:
URL: https://github.com/apache/doris/pull/51376#issuecomment-2983928514
# FE UT Coverage Report
Increment line coverage ` 4.57% (9/197)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/51376_79097b1953e6a9f086a82d25d8683f1abf48adfe/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/51376_79097b1953e6a9f086a82d25d8683f1abf48adfe/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #51376:
URL: https://github.com/apache/doris/pull/51376#discussion_r2124254672
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java:
##########
@@ -1304,4 +1304,15 @@ public void notifyPropertiesUpdated(Map<String, String> updatedProps) {
Env.getCurrentEnv().getExtMetaCacheMgr().invalidSchemaCache(id);
}
}
+
+ /**
+ * Check if an Iceberg view exists.
Review Comment:
This is a interface in `ExternalCatalog`, so it should not only for `Iceberg`
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java:
##########
@@ -63,10 +68,14 @@ public class IcebergExternalTable extends ExternalTable implements MTMVRelatedTa
private Table table;
private boolean isValidRelatedTableCached = false;
private boolean isValidRelatedTable = false;
+ private boolean isTable;
+ private boolean isView;
public IcebergExternalTable(long id, String name, String remoteName, IcebergExternalCatalog catalog,
IcebergExternalDatabase db) {
super(id, name, remoteName, catalog, db, TableType.ICEBERG_EXTERNAL_TABLE);
+ isTable = catalog.tableExist(null, getRemoteDbName(), getRemoteName());
Review Comment:
I think we should move these 2 lines to `makeSureInitialized()`
And in `isTable()` and `isView()` method, call `makeSureInitialized()` first
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java:
##########
@@ -87,7 +96,11 @@ public void setTable(Table table) {
@Override
public Optional<SchemaCacheValue> initSchema(SchemaCacheKey key) {
- return IcebergUtils.loadSchemaCacheValue(catalog, dbName, name, ((IcebergSchemaCacheKey) key).getSchemaId());
+ boolean isView = isView();
+ String remoteDbName = getRemoteDbName();
Review Comment:
Why should we using remote name?
Is this a bug that unrelated to this iceberg view feature?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #51376:
URL: https://github.com/apache/doris/pull/51376#issuecomment-2979918547
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/b08455a93fbe13dfa44da0564d07a7fa780c940d_b08455a93fbe13dfa44da0564d07a7fa780c940d_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/b08455a93fbe13dfa44da0564d07a7fa780c940d_b08455a93fbe13dfa44da0564d07a7fa780c940d_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 83.33% (1120/1344) |
| Line Coverage | 66.82% (19318/28909) |
| Region Coverage | 66.53% (9574/14390) |
| Branch Coverage | 56.53% (5210/9216) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #51376:
URL: https://github.com/apache/doris/pull/51376#issuecomment-2982439284
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "wuwenchi (via GitHub)" <gi...@apache.org>.
wuwenchi commented on code in PR #51376:
URL: https://github.com/apache/doris/pull/51376#discussion_r2123782661
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalCatalog.java:
##########
@@ -94,7 +94,10 @@ public boolean tableExist(SessionContext ctx, String dbName, String tblName) {
@Override
public List<String> listTableNames(SessionContext ctx, String dbName) {
makeSureInitialized();
- return metadataOps.listTableNames(dbName);
+ List<String> tableNames = metadataOps.listTableNames(dbName);
+ List<String> viewNames = metadataOps.listViewNames(dbName);
+ tableNames.addAll(viewNames);
Review Comment:
Why do we need to distinguish between tables and views here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] support iceberg view query [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #51376:
URL: https://github.com/apache/doris/pull/51376#issuecomment-2921181690
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #51376:
URL: https://github.com/apache/doris/pull/51376#discussion_r2139105823
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java:
##########
@@ -87,7 +96,11 @@ public void setTable(Table table) {
@Override
public Optional<SchemaCacheValue> initSchema(SchemaCacheKey key) {
- return IcebergUtils.loadSchemaCacheValue(catalog, dbName, name, ((IcebergSchemaCacheKey) key).getSchemaId());
+ boolean isView = isView();
+ String remoteDbName = getRemoteDbName();
Review Comment:
can fix in other PR
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java:
##########
@@ -1188,4 +1189,10 @@ public List<SysTable> getSupportedSysTables() {
return Lists.newArrayList();
}
}
+
+ @Override
+ public boolean isTable() {
Review Comment:
can remove this
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/source/IcebergApiSource.java:
##########
@@ -42,6 +42,9 @@ public class IcebergApiSource implements IcebergSource {
public IcebergApiSource(IcebergExternalTable table, TupleDescriptor desc,
Map<String, ColumnRange> columnNameToRange) {
+ if (table.isView()) {
Review Comment:
Plz add comment
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java:
##########
@@ -1188,4 +1189,10 @@ public List<SysTable> getSupportedSysTables() {
return Lists.newArrayList();
}
}
+
+ @Override
+ public boolean isTable() {
Review Comment:
Looks like no one call this method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #51376:
URL: https://github.com/apache/doris/pull/51376#issuecomment-2983742222
<details>
<summary>TPC-H: <b>Total hot run time: 33918 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 79097b1953e6a9f086a82d25d8683f1abf48adfe, data reload: false
------ Round 1 ----------------------------------
q1 17583 5231 4997 4997
q2 1929 279 187 187
q3 10380 1284 728 728
q4 10287 1005 523 523
q5 8399 2450 2339 2339
q6 187 162 132 132
q7 922 751 632 632
q8 9346 1334 1064 1064
q9 6808 5037 5079 5037
q10 6887 2384 1955 1955
q11 494 297 277 277
q12 343 353 212 212
q13 17775 3708 3134 3134
q14 231 227 212 212
q15 564 474 479 474
q16 424 440 378 378
q17 583 860 350 350
q18 7689 7128 7128 7128
q19 1650 942 563 563
q20 338 340 216 216
q21 3872 3191 2433 2433
q22 1051 1017 947 947
Total cold run time: 107742 ms
Total hot run time: 33918 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5086 5030 5071 5030
q2 242 325 223 223
q3 2197 2705 2312 2312
q4 1346 1872 1355 1355
q5 4261 4129 4491 4129
q6 219 171 129 129
q7 2030 1934 1785 1785
q8 2654 2634 2503 2503
q9 7146 7099 7154 7099
q10 3110 3334 2809 2809
q11 588 517 502 502
q12 690 773 638 638
q13 3494 3910 3281 3281
q14 287 297 268 268
q15 509 483 471 471
q16 455 487 431 431
q17 1143 1577 1345 1345
q18 7828 7452 7314 7314
q19 794 794 978 794
q20 1981 2132 1892 1892
q21 5445 4520 4183 4183
q22 1018 1001 973 973
Total cold run time: 52523 ms
Total hot run time: 49466 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #51376:
URL: https://github.com/apache/doris/pull/51376#discussion_r2115112982
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java:
##########
@@ -372,6 +389,48 @@ public Table loadTable(String dbName, String tblName) {
}
}
+ @Override
+ public boolean viewExists(String dbName, String viewName) {
Review Comment:
The viewExists method always returns false regardless of the actual existence check. Capture and return the result from the preExecutionAuthenticator execute block to properly reflect whether the view exists.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/IcebergTableSink.java:
##########
@@ -83,6 +86,9 @@ public void bindDataSink(Optional<InsertCommandContext> insertCtx)
throws AnalysisException {
TIcebergTableSink tSink = new TIcebergTableSink();
+ if (targetTable.isView()) {
+ throw new UnsupportedOperationException("IcebergTableSink does not support view");
+ }
Review Comment:
[nitpick] The view check is performed both in the constructor and again in bindDataSink. Consider centralizing the view validation in the constructor to avoid duplicate checks.
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #51376:
URL: https://github.com/apache/doris/pull/51376#issuecomment-2991703441
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #51376:
URL: https://github.com/apache/doris/pull/51376#issuecomment-2982477340
<details>
<summary>TPC-H: <b>Total hot run time: 34056 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ff0969c50c761c36cd31b4690d39acd0d0e997b5, data reload: false
------ Round 1 ----------------------------------
q1 17639 5220 4989 4989
q2 1920 295 200 200
q3 10293 1306 738 738
q4 10234 1000 521 521
q5 7537 2344 2384 2344
q6 179 180 131 131
q7 904 751 606 606
q8 9324 1525 1128 1128
q9 6812 5116 5142 5116
q10 6897 2385 1958 1958
q11 499 280 271 271
q12 350 356 216 216
q13 17756 3652 3078 3078
q14 241 240 225 225
q15 542 480 470 470
q16 428 432 368 368
q17 591 851 371 371
q18 7579 7242 7127 7127
q19 1218 942 558 558
q20 337 337 224 224
q21 3760 3145 2443 2443
q22 1101 1030 974 974
Total cold run time: 106141 ms
Total hot run time: 34056 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5462 5035 5041 5035
q2 249 324 226 226
q3 2150 2667 2289 2289
q4 1333 1798 1349 1349
q5 4259 4092 4447 4092
q6 209 167 130 130
q7 2022 1910 1780 1780
q8 2628 2533 2548 2533
q9 7143 7188 7083 7083
q10 3034 3293 2888 2888
q11 570 507 489 489
q12 686 760 625 625
q13 3446 3856 3325 3325
q14 295 294 264 264
q15 527 479 480 479
q16 435 477 449 449
q17 1177 1481 1421 1421
q18 7766 7610 7384 7384
q19 812 825 884 825
q20 1974 1980 1888 1888
q21 4989 4491 4335 4335
q22 1092 1061 994 994
Total cold run time: 52258 ms
Total hot run time: 49883 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #51376:
URL: https://github.com/apache/doris/pull/51376#issuecomment-2982446933
> which TABLE_TYPE will return by iceberg's view when query `information.tables`?
it will return `iceberg`, same as iceberg table.
And also, for hive view, it return `hive`, same as hive table.
I will rethink of this logic since currently there is no standard
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #51376:
URL: https://github.com/apache/doris/pull/51376#issuecomment-2989916950
<details>
<summary>TPC-H: <b>Total hot run time: 34185 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f105dffb4303ac09a24b860f39a5338d1bf55837, data reload: false
------ Round 1 ----------------------------------
q1 17624 5195 5096 5096
q2 1938 308 197 197
q3 10257 1303 731 731
q4 10217 1012 533 533
q5 7497 2363 2369 2363
q6 185 160 137 137
q7 899 770 601 601
q8 9315 1296 1164 1164
q9 6625 5070 5115 5070
q10 6917 2376 1964 1964
q11 496 290 278 278
q12 351 344 219 219
q13 17785 3669 3112 3112
q14 242 234 218 218
q15 553 476 482 476
q16 439 425 380 380
q17 624 905 386 386
q18 7735 7177 7012 7012
q19 1615 959 591 591
q20 345 338 234 234
q21 4052 3358 2452 2452
q22 1032 1039 971 971
Total cold run time: 106743 ms
Total hot run time: 34185 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5236 5126 5178 5126
q2 249 327 223 223
q3 2258 2700 2359 2359
q4 1353 1821 1376 1376
q5 4204 4161 4390 4161
q6 256 179 137 137
q7 1975 1943 1758 1758
q8 2637 2592 2538 2538
q9 7063 6976 7035 6976
q10 3092 3268 2816 2816
q11 581 504 498 498
q12 702 758 613 613
q13 3949 3928 3320 3320
q14 278 304 281 281
q15 519 475 465 465
q16 435 501 433 433
q17 1147 1504 1373 1373
q18 7284 7028 6942 6942
q19 786 792 943 792
q20 1973 1928 1802 1802
q21 4739 4335 4281 4281
q22 1078 1065 998 998
Total cold run time: 51794 ms
Total hot run time: 49268 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #51376:
URL: https://github.com/apache/doris/pull/51376#issuecomment-2989936604
<details>
<summary>TPC-DS: <b>Total hot run time: 185875 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f105dffb4303ac09a24b860f39a5338d1bf55837, data reload: false
query1 979 404 391 391
query2 6511 1884 1873 1873
query3 6744 231 224 224
query4 26463 23718 23031 23031
query5 4383 652 488 488
query6 306 215 211 211
query7 4629 487 305 305
query8 268 232 214 214
query9 8623 2623 2640 2623
query10 479 328 284 284
query11 15754 15190 14973 14973
query12 181 116 107 107
query13 1656 549 438 438
query14 9781 6134 6294 6134
query15 205 198 173 173
query16 7643 632 448 448
query17 1195 686 569 569
query18 2005 404 300 300
query19 190 187 157 157
query20 117 113 115 113
query21 216 133 109 109
query22 3983 4255 4053 4053
query23 34025 32986 33217 32986
query24 8392 2368 2413 2368
query25 535 469 395 395
query26 1231 278 154 154
query27 2742 515 374 374
query28 4316 2113 2085 2085
query29 738 541 438 438
query30 283 220 197 197
query31 928 829 754 754
query32 69 69 61 61
query33 561 391 339 339
query34 806 849 543 543
query35 802 822 764 764
query36 959 1004 862 862
query37 114 107 83 83
query38 4072 4086 4040 4040
query39 1489 1423 1390 1390
query40 217 121 118 118
query41 64 61 63 61
query42 132 111 113 111
query43 496 519 482 482
query44 1360 847 829 829
query45 182 172 169 169
query46 864 1040 627 627
query47 1728 1755 1675 1675
query48 385 425 317 317
query49 732 485 410 410
query50 662 693 423 423
query51 4108 4156 4141 4141
query52 116 111 108 108
query53 231 261 191 191
query54 582 585 527 527
query55 90 92 91 91
query56 325 307 321 307
query57 1211 1176 1117 1117
query58 260 259 257 257
query59 2686 2713 2583 2583
query60 333 332 317 317
query61 130 123 121 121
query62 817 709 683 683
query63 234 196 187 187
query64 4243 1026 681 681
query65 4225 4177 4179 4177
query66 1084 410 335 335
query67 15730 15447 15331 15331
query68 6550 899 539 539
query69 491 305 280 280
query70 1197 1108 1111 1108
query71 419 337 314 314
query72 5153 4655 4564 4564
query73 622 573 357 357
query74 9331 9094 8977 8977
query75 3180 3194 2733 2733
query76 3275 1192 780 780
query77 483 383 293 293
query78 10023 10176 9256 9256
query79 1700 784 599 599
query80 675 502 461 461
query81 510 265 220 220
query82 184 128 97 97
query83 257 251 237 237
query84 254 111 91 91
query85 754 363 360 360
query86 356 331 298 298
query87 4364 4506 4344 4344
query88 2910 2280 2306 2280
query89 399 313 280 280
query90 1738 210 209 209
query91 145 144 115 115
query92 68 62 58 58
query93 1040 945 599 599
query94 624 416 319 319
query95 381 288 295 288
query96 492 568 284 284
query97 2704 2760 2645 2645
query98 233 204 210 204
query99 1323 1399 1259 1259
Total cold run time: 269435 ms
Total hot run time: 185875 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #51376:
URL: https://github.com/apache/doris/pull/51376
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on PR #51376:
URL: https://github.com/apache/doris/pull/51376#issuecomment-2982361647
> which TABLE_TYPE will return by iceberg's view when query `information.tables`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui closed pull request #51376: [feature](iceberg catalog) support iceberg view query
URL: https://github.com/apache/doris/pull/51376
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg catalog) support iceberg view query [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #51376:
URL: https://github.com/apache/doris/pull/51376#issuecomment-2982493481
<details>
<summary>TPC-DS: <b>Total hot run time: 185762 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ff0969c50c761c36cd31b4690d39acd0d0e997b5, data reload: false
query1 999 394 382 382
query2 6532 1832 1866 1832
query3 6745 219 232 219
query4 26341 23481 23509 23481
query5 4353 628 467 467
query6 317 226 199 199
query7 4637 486 288 288
query8 267 234 223 223
query9 8647 2637 2648 2637
query10 491 358 279 279
query11 15802 15287 14868 14868
query12 161 113 107 107
query13 1664 520 412 412
query14 10063 6123 6102 6102
query15 200 190 176 176
query16 7204 654 496 496
query17 1206 726 592 592
query18 1996 436 310 310
query19 200 190 171 171
query20 128 122 130 122
query21 220 133 108 108
query22 4062 4135 3990 3990
query23 34115 33102 33023 33023
query24 7747 2383 2365 2365
query25 551 459 393 393
query26 1244 267 151 151
query27 2645 478 345 345
query28 4379 2138 2114 2114
query29 762 576 425 425
query30 280 222 192 192
query31 946 870 762 762
query32 73 65 64 64
query33 568 372 317 317
query34 805 829 529 529
query35 778 822 745 745
query36 940 1021 883 883
query37 111 96 83 83
query38 4076 4079 4036 4036
query39 1499 1429 1416 1416
query40 214 123 110 110
query41 64 60 61 60
query42 134 108 108 108
query43 503 504 480 480
query44 1319 812 825 812
query45 182 167 168 167
query46 847 1016 620 620
query47 1782 1804 1706 1706
query48 398 412 330 330
query49 747 490 395 395
query50 628 674 405 405
query51 4101 4142 4197 4142
query52 111 109 96 96
query53 218 248 182 182
query54 576 569 505 505
query55 82 86 81 81
query56 307 295 278 278
query57 1193 1200 1130 1130
query58 278 266 249 249
query59 2697 2736 2641 2641
query60 322 317 302 302
query61 126 156 126 126
query62 797 717 685 685
query63 229 191 191 191
query64 4388 1036 667 667
query65 4250 4175 4229 4175
query66 1161 406 320 320
query67 15662 15446 15218 15218
query68 8254 877 511 511
query69 472 298 271 271
query70 1189 1117 1082 1082
query71 485 336 308 308
query72 5698 4725 4647 4647
query73 696 583 353 353
query74 9209 9071 8721 8721
query75 3972 3191 2711 2711
query76 3725 1200 764 764
query77 869 369 291 291
query78 10034 10053 9329 9329
query79 2664 784 575 575
query80 627 537 441 441
query81 501 265 234 234
query82 684 131 101 101
query83 294 244 236 236
query84 293 113 93 93
query85 781 352 364 352
query86 395 311 309 309
query87 4393 4446 4450 4446
query88 3678 2279 2272 2272
query89 393 310 287 287
query90 1846 210 206 206
query91 148 147 115 115
query92 75 66 58 58
query93 1753 935 572 572
query94 669 414 304 304
query95 373 297 284 284
query96 509 560 284 284
query97 2719 2812 2608 2608
query98 244 205 206 205
query99 1460 1421 1307 1307
Total cold run time: 276172 ms
Total hot run time: 185762 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org