You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org> on 2025/03/03 12:28:05 UTC
[PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
XiaoHongbo-Hope opened a new pull request, #5200:
URL: https://github.com/apache/paimon/pull/5200
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
[core] RestCatalog: set up paged list tables/views/partitions
<!-- Linking this pull request to the issue -->
Linked issue: close #xxx
<!-- What is the purpose of the change -->
Check Iceberg & Unity Catalog, they support paged list for tables & views both, so suggest adding paged list method into paimon
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #5200:
URL: https://github.com/apache/paimon/pull/5200#discussion_r1979003762
##########
paimon-core/src/main/java/org/apache/paimon/catalog/AbstractCatalog.java:
##########
@@ -240,8 +254,69 @@ public List<String> listTables(String databaseName) throws DatabaseNotExistExcep
return listTablesImpl(databaseName).stream().sorted().collect(Collectors.toList());
}
+ @Override
+ public PagedList<String> listTablesPaged(
+ String databaseName, Integer maxResults, String pageToken)
+ throws DatabaseNotExistException {
+ return new PagedList<>(listTables(databaseName), null);
+ }
+
+ @Override
+ public PagedList<Table> listTableDetailsPaged(
+ String databaseName, Integer maxResults, String pageToken)
+ throws DatabaseNotExistException {
+ if (isSystemDatabase(databaseName)) {
+ List<Table> systemTables =
+ SystemTableLoader.loadGlobalTableNames().stream()
+ .map(
+ tableName -> {
+ try {
+ return getTable(
+ Identifier.create(databaseName, tableName));
+ } catch (TableNotExistException ignored) {
+ LOG.warn(
+ "system table {}.{} does not exist",
+ databaseName,
+ tableName);
+ return null;
+ }
+ })
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList());
+ return new PagedList<>(systemTables, null);
+ }
+
+ // check db exists
+ getDatabase(databaseName);
+
+ return listTableDetailsPagedImpl(databaseName, maxResults, pageToken);
+ }
+
protected abstract List<String> listTablesImpl(String databaseName);
+ protected PagedList<Table> listTableDetailsPagedImpl(
+ String databaseName, Integer maxResults, String pageToken)
+ throws DatabaseNotExistException {
+ PagedList<String> pagedTableNames = listTablesPaged(databaseName, maxResults, pageToken);
Review Comment:
will get table details directly in RestCatalog, this is the default implements for other catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #5200:
URL: https://github.com/apache/paimon/pull/5200#discussion_r1979001190
##########
paimon-common/src/main/java/org/apache/paimon/PagedList.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon;
+
+import java.util.List;
+
+/** Paged List which supports request data from page streaming. */
+public class PagedList<T> {
+ private final List<T> pagedLists;
Review Comment:
will update pagedLists to pagedList.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on PR #5200:
URL: https://github.com/apache/paimon/pull/5200#issuecomment-2696372929
Can you please help review again? I have fixed all the comment,optimized the test case assertion and cleaned code for the mock server. Note the catalog client logic should be same as last review.
@sundapeng @JingsongLi @jerry-024
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #5200:
URL: https://github.com/apache/paimon/pull/5200#discussion_r1978568747
##########
paimon-core/src/main/java/org/apache/paimon/catalog/Catalog.java:
##########
@@ -130,6 +131,44 @@ void alterDatabase(String name, List<PropertyChange> changes, boolean ignoreIfNo
*/
List<String> listTables(String databaseName) throws DatabaseNotExistException;
+ /**
+ * Get paged list names of tables under this database. An empty list is returned if none exists.
+ *
+ * <p>NOTE: System tables will not be listed.
+ *
+ * @param databaseName Name of the database to list tables.
+ * @param maxResults Optional parameter indicating the maximum number of results to include in
+ * the result. If maxResults is not specified or set to 0, will return the default number of
+ * max results.
+ * @param pageToken Optional parameter indicating the next page token allows list to be start
+ * from a specific point.
+ * @return a list of the names of tables with provided page size in this database and next page
+ * token
+ * @throws DatabaseNotExistException if the database does not exist
+ */
+ PagedList<String> listTablesPaged(String databaseName, Integer maxResults, String pageToken)
Review Comment:
ok, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #5200:
URL: https://github.com/apache/paimon/pull/5200#discussion_r1979569003
##########
paimon-core/src/main/java/org/apache/paimon/rest/HttpClient.java:
##########
@@ -191,7 +209,23 @@ private static RequestBody buildRequestBody(String body) throws JsonProcessingEx
}
private String getRequestUrl(String path) {
- return StringUtils.isNullOrWhitespaceOnly(path) ? uri : uri + path;
+ return getRequestUrl(path, null);
+ }
+
+ private String getRequestUrl(String path, Map<String, String> queryParams) {
Review Comment:
added, and test case of catalog will cover this too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #5200:
URL: https://github.com/apache/paimon/pull/5200#discussion_r1979401681
##########
paimon-common/src/main/java/org/apache/paimon/PagedList.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon;
+
+import java.util.List;
+
+/** Paged List which supports request data from page streaming. */
Review Comment:
👌
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #5200:
URL: https://github.com/apache/paimon/pull/5200#discussion_r1979197023
##########
paimon-core/src/main/java/org/apache/paimon/rest/RESTCatalog.java:
##########
@@ -257,16 +271,117 @@ public void alterDatabase(String name, List<PropertyChange> changes, boolean ign
@Override
public List<String> listTables(String databaseName) throws DatabaseNotExistException {
+ try {
+ String pageToken = null;
+ Map<String, String> queryParams = Maps.newHashMap();
+ Integer maxResults = context.options().get(RESTCatalogOptions.REST_PAGE_MAX_RESULTS);
+ if (Objects.nonNull(maxResults) && maxResults > 0) {
+ queryParams.put(MAX_RESULTS, String.valueOf(maxResults));
+ }
+ List<String> tables = new ArrayList<>();
+ do {
+ queryParams.put(PAGE_TOKEN, pageToken);
+ ListTablesResponse response =
+ client.get(
+ resourcePaths.tables(databaseName),
+ queryParams,
+ ListTablesResponse.class,
+ restAuthFunction);
+ if (Objects.nonNull(response)) {
Review Comment:
I think we can gain more and loss less by double checking
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "sundapeng (via GitHub)" <gi...@apache.org>.
sundapeng commented on code in PR #5200:
URL: https://github.com/apache/paimon/pull/5200#discussion_r1977467443
##########
paimon-open-api/src/main/java/org/apache/paimon/open/api/RESTCatalogController.java:
##########
@@ -226,8 +228,47 @@ public AlterDatabaseResponse alterDatabase(
})
@GetMapping("/v1/{prefix}/databases/{database}/tables")
public ListTablesResponse listTables(
- @PathVariable String prefix, @PathVariable String database) {
- return new ListTablesResponse(ImmutableList.of("user"));
+ @PathVariable String prefix,
+ @PathVariable String database,
+ @PathVariable Integer maxResults,
Review Comment:
Please update API spec `rest-catalog-open-api.yaml`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "sundapeng (via GitHub)" <gi...@apache.org>.
sundapeng commented on PR #5200:
URL: https://github.com/apache/paimon/pull/5200#issuecomment-2694485231
Looks good overall, but minor comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #5200:
URL: https://github.com/apache/paimon/pull/5200#discussion_r1978566426
##########
paimon-common/src/main/java/org/apache/paimon/PagedList.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon;
+
+import java.io.Serializable;
+import java.util.List;
+
+/** Paged List which supports request data from page streaming. */
+public class PagedList<T> implements Serializable {
+ private final List<T> pagedLists;
Review Comment:
Since we will hold some un Serializable data in the PagedList, such as Table/View, I remove the Serializable implements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #5200:
URL: https://github.com/apache/paimon/pull/5200#discussion_r1978566426
##########
paimon-common/src/main/java/org/apache/paimon/PagedList.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon;
+
+import java.io.Serializable;
+import java.util.List;
+
+/** Paged List which supports request data from page streaming. */
+public class PagedList<T> implements Serializable {
+ private final List<T> pagedLists;
Review Comment:
Since we will hold some un Serializable data in the PagedList, such as Table/View, which means PagedList is just a special List. I remove the Serializable implements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope closed pull request #5200: [core] RestCatalog: set up paged list tables/views/partitions
URL: https://github.com/apache/paimon/pull/5200
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "sundapeng (via GitHub)" <gi...@apache.org>.
sundapeng commented on code in PR #5200:
URL: https://github.com/apache/paimon/pull/5200#discussion_r1977535263
##########
paimon-core/src/main/java/org/apache/paimon/rest/RESTCatalog.java:
##########
@@ -560,16 +690,34 @@ public void markDonePartitions(Identifier identifier, List<Map<String, String>>
@Override
public List<Partition> listPartitions(Identifier identifier) throws TableNotExistException {
try {
- ListPartitionsResponse response =
- client.get(
- resourcePaths.partitions(
- identifier.getDatabaseName(), identifier.getTableName()),
- ListPartitionsResponse.class,
- restAuthFunction);
- if (response == null || response.getPartitions() == null) {
- return Collections.emptyList();
+ String pageToken = null;
+ Map<String, String> queryParams = Maps.newHashMap();
+ Integer maxResults = context.options().get(RESTCatalogOptions.REST_PAGE_MAX_RESULTS);
+ if (Objects.nonNull(maxResults) && maxResults > 0) {
+ queryParams.put(MAX_RESULTS, String.valueOf(maxResults));
}
- return response.getPartitions();
+ List<Partition> partitions = new ArrayList<>();
+ do {
+ queryParams.put(PAGE_TOKEN, pageToken);
+ ListPartitionsResponse response =
+ client.get(
+ resourcePaths.partitions(
+ identifier.getDatabaseName(), identifier.getTableName()),
+ queryParams,
+ ListPartitionsResponse.class,
+ restAuthFunction);
+ if (Objects.nonNull(response)) {
+ pageToken = response.getNextPageToken();
+ partitions.addAll(response.getPartitions());
+ } else {
+ LOG.warn(
+ "response of listPartitions for {}.{} is null with params {}",
+ identifier.getDatabaseName(),
+ identifier.getTableName(),
+ queryParams);
+ }
+ } while (pageToken != null);
Review Comment:
It is better to check `pageToken.isNotEmpty()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #5200:
URL: https://github.com/apache/paimon/pull/5200#discussion_r1979416348
##########
paimon-core/src/main/java/org/apache/paimon/rest/RESTCatalogOptions.java:
##########
@@ -85,4 +85,10 @@ public class RESTCatalogOptions {
.booleanType()
.defaultValue(false)
.withDescription("Whether to support data token provided by the REST server.");
+
+ public static final ConfigOption<Integer> REST_PAGE_MAX_RESULTS =
+ ConfigOptions.key("rest.page.max.results")
Review Comment:
👌
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #5200:
URL: https://github.com/apache/paimon/pull/5200#discussion_r1979008389
##########
paimon-core/src/main/java/org/apache/paimon/catalog/Catalog.java:
##########
@@ -326,6 +385,46 @@ default List<String> listViews(String databaseName) throws DatabaseNotExistExcep
return Collections.emptyList();
}
+ /**
+ * Get paged list names of views under this database. An empty list is returned if none view
+ * exists.
+ *
+ * @param databaseName Name of the database to list views.
+ * @param maxResults Optional parameter indicating the maximum number of results to include in
+ * the result. If maxResults is not specified or set to 0, will return the default number of
+ * max results.
+ * @param pageToken Optional parameter indicating the next page token allows list to be start
+ * from a specific point.
+ * @return a list of the names of views with provided page size in this database and next page
+ * token
+ * @throws DatabaseNotExistException if the database does not exist
+ */
+ default PagedList<String> listViewsPaged(
+ String databaseName, @Nullable Integer maxResults, @Nullable String pageToken)
+ throws DatabaseNotExistException {
+ return new PagedList<>(listViews(databaseName), null);
Review Comment:
just follow the behavior of listViews in interface Catalog, maybe because most catalog does not support view functions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "jerry-024 (via GitHub)" <gi...@apache.org>.
jerry-024 commented on code in PR #5200:
URL: https://github.com/apache/paimon/pull/5200#discussion_r1978867026
##########
paimon-core/src/main/java/org/apache/paimon/catalog/Catalog.java:
##########
@@ -130,6 +133,45 @@ void alterDatabase(String name, List<PropertyChange> changes, boolean ignoreIfNo
*/
List<String> listTables(String databaseName) throws DatabaseNotExistException;
+ /**
+ * Get paged list names of tables under this database. An empty list is returned if none exists.
+ *
+ * <p>NOTE: System tables will not be listed.
+ *
+ * @param databaseName Name of the database to list tables.
+ * @param maxResults Optional parameter indicating the maximum number of results to include in
+ * the result. If maxResults is not specified or set to 0, will return the default number of
+ * max results.
+ * @param pageToken Optional parameter indicating the next page token allows list to be start
+ * from a specific point.
+ * @return a list of the names of tables with provided page size in this database and next page
+ * token
+ * @throws DatabaseNotExistException if the database does not exist
+ */
+ PagedList<String> listTablesPaged(
+ String databaseName, @Nullable Integer maxResults, @Nullable String pageToken)
+ throws DatabaseNotExistException;
+
+ /**
+ * Get paged list of table details under this database. An empty list is returned if none
+ * exists.
+ *
+ * <p>NOTE: System tables will not be listed.
+ *
+ * @param databaseName Name of the database to list table details.
+ * @param maxResults Optional parameter indicating the maximum number of results to include in
+ * the result. If maxResults is not specified or set to 0, will return the default number of
+ * max results.
+ * @param pageToken Optional parameter indicating the next page token allows list to be start
+ * from a specific point.
Review Comment:
from a specific table name?
##########
paimon-core/src/main/java/org/apache/paimon/catalog/Catalog.java:
##########
@@ -326,6 +385,46 @@ default List<String> listViews(String databaseName) throws DatabaseNotExistExcep
return Collections.emptyList();
}
+ /**
+ * Get paged list names of views under this database. An empty list is returned if none view
+ * exists.
+ *
+ * @param databaseName Name of the database to list views.
+ * @param maxResults Optional parameter indicating the maximum number of results to include in
+ * the result. If maxResults is not specified or set to 0, will return the default number of
+ * max results.
+ * @param pageToken Optional parameter indicating the next page token allows list to be start
+ * from a specific point.
+ * @return a list of the names of views with provided page size in this database and next page
+ * token
+ * @throws DatabaseNotExistException if the database does not exist
+ */
+ default PagedList<String> listViewsPaged(
+ String databaseName, @Nullable Integer maxResults, @Nullable String pageToken)
+ throws DatabaseNotExistException {
+ return new PagedList<>(listViews(databaseName), null);
Review Comment:
Why is there a default implementation?
##########
paimon-core/src/main/java/org/apache/paimon/rest/HttpClient.java:
##########
@@ -191,7 +209,23 @@ private static RequestBody buildRequestBody(String body) throws JsonProcessingEx
}
private String getRequestUrl(String path) {
- return StringUtils.isNullOrWhitespaceOnly(path) ? uri : uri + path;
+ return getRequestUrl(path, null);
+ }
+
+ private String getRequestUrl(String path, Map<String, String> queryParams) {
Review Comment:
Is there any test case for this method?
##########
paimon-core/src/main/java/org/apache/paimon/rest/RESTCatalog.java:
##########
@@ -257,16 +271,117 @@ public void alterDatabase(String name, List<PropertyChange> changes, boolean ign
@Override
public List<String> listTables(String databaseName) throws DatabaseNotExistException {
+ try {
+ String pageToken = null;
+ Map<String, String> queryParams = Maps.newHashMap();
+ Integer maxResults = context.options().get(RESTCatalogOptions.REST_PAGE_MAX_RESULTS);
+ if (Objects.nonNull(maxResults) && maxResults > 0) {
+ queryParams.put(MAX_RESULTS, String.valueOf(maxResults));
+ }
+ List<String> tables = new ArrayList<>();
+ do {
+ queryParams.put(PAGE_TOKEN, pageToken);
+ ListTablesResponse response =
+ client.get(
+ resourcePaths.tables(databaseName),
+ queryParams,
+ ListTablesResponse.class,
+ restAuthFunction);
+ if (Objects.nonNull(response)) {
Review Comment:
Could be response null here?
##########
paimon-core/src/main/java/org/apache/paimon/rest/RESTCatalog.java:
##########
@@ -257,16 +271,117 @@ public void alterDatabase(String name, List<PropertyChange> changes, boolean ign
@Override
public List<String> listTables(String databaseName) throws DatabaseNotExistException {
+ try {
+ String pageToken = null;
+ Map<String, String> queryParams = Maps.newHashMap();
+ Integer maxResults = context.options().get(RESTCatalogOptions.REST_PAGE_MAX_RESULTS);
+ if (Objects.nonNull(maxResults) && maxResults > 0) {
+ queryParams.put(MAX_RESULTS, String.valueOf(maxResults));
+ }
+ List<String> tables = new ArrayList<>();
+ do {
+ queryParams.put(PAGE_TOKEN, pageToken);
+ ListTablesResponse response =
+ client.get(
+ resourcePaths.tables(databaseName),
+ queryParams,
+ ListTablesResponse.class,
+ restAuthFunction);
+ if (Objects.nonNull(response)) {
+ pageToken = response.getNextPageToken();
+ tables.addAll(response.getTables());
+ } else {
+ LOG.warn(
+ "response of listTables for {} is null with params {}",
+ databaseName,
+ queryParams);
+ }
+ } while (StringUtils.isNotEmpty(pageToken));
+ return tables;
+ } catch (NoSuchResourceException e) {
+ throw new DatabaseNotExistException(databaseName);
+ }
+ }
+
+ @Override
+ public PagedList<String> listTablesPaged(
+ String databaseName, Integer maxResults, String pageToken)
+ throws DatabaseNotExistException {
+ Map<String, String> queryParams = Maps.newHashMap();
+ if (Objects.nonNull(maxResults) && maxResults > 0) {
+ queryParams.put(MAX_RESULTS, maxResults.toString());
+ }
+ if (Objects.nonNull(pageToken)) {
+ queryParams.put(PAGE_TOKEN, pageToken);
+ }
try {
ListTablesResponse response =
client.get(
resourcePaths.tables(databaseName),
+ queryParams,
ListTablesResponse.class,
restAuthFunction);
- if (response.getTables() != null) {
- return response.getTables();
+ if (Objects.nonNull(response) && Objects.nonNull(response.getTables())) {
Review Comment:
Could be response null here?
##########
paimon-common/src/main/java/org/apache/paimon/PagedList.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon;
+
+import java.util.List;
+
+/** Paged List which supports request data from page streaming. */
+public class PagedList<T> {
+ private final List<T> pagedLists;
Review Comment:
just pagedList or data?
##########
paimon-core/src/main/java/org/apache/paimon/catalog/AbstractCatalog.java:
##########
@@ -240,8 +254,69 @@ public List<String> listTables(String databaseName) throws DatabaseNotExistExcep
return listTablesImpl(databaseName).stream().sorted().collect(Collectors.toList());
}
+ @Override
+ public PagedList<String> listTablesPaged(
+ String databaseName, Integer maxResults, String pageToken)
+ throws DatabaseNotExistException {
+ return new PagedList<>(listTables(databaseName), null);
+ }
+
+ @Override
+ public PagedList<Table> listTableDetailsPaged(
+ String databaseName, Integer maxResults, String pageToken)
+ throws DatabaseNotExistException {
+ if (isSystemDatabase(databaseName)) {
+ List<Table> systemTables =
+ SystemTableLoader.loadGlobalTableNames().stream()
+ .map(
+ tableName -> {
+ try {
+ return getTable(
+ Identifier.create(databaseName, tableName));
+ } catch (TableNotExistException ignored) {
+ LOG.warn(
+ "system table {}.{} does not exist",
+ databaseName,
+ tableName);
+ return null;
+ }
+ })
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList());
+ return new PagedList<>(systemTables, null);
+ }
+
+ // check db exists
+ getDatabase(databaseName);
+
+ return listTableDetailsPagedImpl(databaseName, maxResults, pageToken);
+ }
+
protected abstract List<String> listTablesImpl(String databaseName);
+ protected PagedList<Table> listTableDetailsPagedImpl(
+ String databaseName, Integer maxResults, String pageToken)
+ throws DatabaseNotExistException {
+ PagedList<String> pagedTableNames = listTablesPaged(databaseName, maxResults, pageToken);
Review Comment:
Is it possible to get table detail directly?
##########
paimon-core/src/main/java/org/apache/paimon/catalog/Catalog.java:
##########
@@ -326,6 +385,46 @@ default List<String> listViews(String databaseName) throws DatabaseNotExistExcep
return Collections.emptyList();
}
+ /**
+ * Get paged list names of views under this database. An empty list is returned if none view
+ * exists.
+ *
+ * @param databaseName Name of the database to list views.
+ * @param maxResults Optional parameter indicating the maximum number of results to include in
+ * the result. If maxResults is not specified or set to 0, will return the default number of
+ * max results.
+ * @param pageToken Optional parameter indicating the next page token allows list to be start
+ * from a specific point.
+ * @return a list of the names of views with provided page size in this database and next page
+ * token
+ * @throws DatabaseNotExistException if the database does not exist
+ */
+ default PagedList<String> listViewsPaged(
+ String databaseName, @Nullable Integer maxResults, @Nullable String pageToken)
+ throws DatabaseNotExistException {
+ return new PagedList<>(listViews(databaseName), null);
+ }
+
+ /**
+ * Get paged list view details under this database. An empty list is returned if none view
+ * exists.
+ *
+ * @param databaseName Name of the database to list views.
+ * @param maxResults Optional parameter indicating the maximum number of results to include in
+ * the result. If maxResults is not specified or set to 0, will return the default number of
+ * max results.
+ * @param pageToken Optional parameter indicating the next page token allows list to be start
+ * from a specific point.
+ * @return a list of the view details with provided page size (@param maxResults) in this
+ * database and next page token
+ * @throws DatabaseNotExistException if the database does not exist
+ */
+ default PagedList<View> listViewDetailsPaged(
+ String databaseName, @Nullable Integer maxResults, @Nullable String pageToken)
+ throws DatabaseNotExistException {
+ return new PagedList<>(Collections.emptyList(), null);
Review Comment:
Why is there a default implementation?
##########
paimon-core/src/main/java/org/apache/paimon/rest/RESTCatalog.java:
##########
@@ -257,16 +271,117 @@ public void alterDatabase(String name, List<PropertyChange> changes, boolean ign
@Override
public List<String> listTables(String databaseName) throws DatabaseNotExistException {
+ try {
+ String pageToken = null;
+ Map<String, String> queryParams = Maps.newHashMap();
+ Integer maxResults = context.options().get(RESTCatalogOptions.REST_PAGE_MAX_RESULTS);
+ if (Objects.nonNull(maxResults) && maxResults > 0) {
+ queryParams.put(MAX_RESULTS, String.valueOf(maxResults));
+ }
+ List<String> tables = new ArrayList<>();
+ do {
+ queryParams.put(PAGE_TOKEN, pageToken);
+ ListTablesResponse response =
+ client.get(
+ resourcePaths.tables(databaseName),
+ queryParams,
+ ListTablesResponse.class,
+ restAuthFunction);
+ if (Objects.nonNull(response)) {
+ pageToken = response.getNextPageToken();
+ tables.addAll(response.getTables());
+ } else {
+ LOG.warn(
+ "response of listTables for {} is null with params {}",
+ databaseName,
+ queryParams);
+ }
+ } while (StringUtils.isNotEmpty(pageToken));
+ return tables;
+ } catch (NoSuchResourceException e) {
+ throw new DatabaseNotExistException(databaseName);
+ }
+ }
+
+ @Override
+ public PagedList<String> listTablesPaged(
+ String databaseName, Integer maxResults, String pageToken)
+ throws DatabaseNotExistException {
+ Map<String, String> queryParams = Maps.newHashMap();
+ if (Objects.nonNull(maxResults) && maxResults > 0) {
+ queryParams.put(MAX_RESULTS, maxResults.toString());
+ }
+ if (Objects.nonNull(pageToken)) {
+ queryParams.put(PAGE_TOKEN, pageToken);
+ }
try {
ListTablesResponse response =
client.get(
resourcePaths.tables(databaseName),
+ queryParams,
ListTablesResponse.class,
restAuthFunction);
- if (response.getTables() != null) {
- return response.getTables();
+ if (Objects.nonNull(response) && Objects.nonNull(response.getTables())) {
+ return new PagedList<>(response.getTables(), response.getNextPageToken());
+ } else {
+ LOG.warn(
+ "response of listTablesPaged for {} is null with maxResults {} and pageToken {}",
+ databaseName,
+ maxResults,
+ pageToken);
+ return new PagedList<>(Collections.emptyList(), null);
+ }
+ } catch (NoSuchResourceException e) {
+ throw new DatabaseNotExistException(databaseName);
+ }
+ }
+
+ @Override
+ public PagedList<Table> listTableDetailsPaged(
+ String databaseName, Integer maxResults, String pageToken)
+ throws DatabaseNotExistException {
+ Map<String, String> queryParams = Maps.newHashMap();
+ if (Objects.nonNull(maxResults) && maxResults > 0) {
+ queryParams.put(MAX_RESULTS, maxResults.toString());
+ }
+ if (Objects.nonNull(pageToken)) {
+ queryParams.put(PAGE_TOKEN, pageToken);
+ }
+ try {
+ ListTableDetailsResponse response =
+ client.get(
+ resourcePaths.tableDetails(databaseName),
+ queryParams,
+ ListTableDetailsResponse.class,
+ restAuthFunction);
+ if (Objects.nonNull(response) && Objects.nonNull(response.getTableDetails())) {
Review Comment:
Could be response null here?
##########
paimon-core/src/main/java/org/apache/paimon/rest/RESTCatalog.java:
##########
@@ -560,16 +675,34 @@ public void markDonePartitions(Identifier identifier, List<Map<String, String>>
@Override
public List<Partition> listPartitions(Identifier identifier) throws TableNotExistException {
try {
- ListPartitionsResponse response =
- client.get(
- resourcePaths.partitions(
- identifier.getDatabaseName(), identifier.getTableName()),
- ListPartitionsResponse.class,
- restAuthFunction);
- if (response == null || response.getPartitions() == null) {
- return Collections.emptyList();
+ String pageToken = null;
+ Map<String, String> queryParams = Maps.newHashMap();
+ Integer maxResults = context.options().get(RESTCatalogOptions.REST_PAGE_MAX_RESULTS);
+ if (Objects.nonNull(maxResults) && maxResults > 0) {
+ queryParams.put(MAX_RESULTS, String.valueOf(maxResults));
}
- return response.getPartitions();
+ List<Partition> partitions = new ArrayList<>();
+ do {
+ queryParams.put(PAGE_TOKEN, pageToken);
+ ListPartitionsResponse response =
+ client.get(
+ resourcePaths.partitions(
+ identifier.getDatabaseName(), identifier.getTableName()),
+ queryParams,
+ ListPartitionsResponse.class,
+ restAuthFunction);
+ if (Objects.nonNull(response)) {
Review Comment:
Could be response null here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #5200:
URL: https://github.com/apache/paimon/pull/5200
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #5200:
URL: https://github.com/apache/paimon/pull/5200#discussion_r1977605255
##########
paimon-common/src/main/java/org/apache/paimon/PagedList.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon;
+
+import java.io.Serializable;
+import java.util.List;
+
+/** Paged List which supports request data from page streaming. */
+public class PagedList<T> implements Serializable {
+ private final List<T> pagedLists;
Review Comment:
Please add ser id for Serializable class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #5200:
URL: https://github.com/apache/paimon/pull/5200#discussion_r1979384320
##########
paimon-common/src/main/java/org/apache/paimon/PagedList.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon;
+
+import java.util.List;
+
+/** Paged List which supports request data from page streaming. */
+public class PagedList<T> {
+ private final List<T> pagedList;
Review Comment:
rename this to `elements`? it is not a paged list without `nextPageToken`.
##########
paimon-common/src/main/java/org/apache/paimon/PagedList.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon;
+
+import java.util.List;
+
+/** Paged List which supports request data from page streaming. */
+public class PagedList<T> {
+ private final List<T> pagedList;
+
+ private final String nextPageToken;
+
+ public PagedList(List<T> pagedList, String nextPageToken) {
Review Comment:
`@Nullable`
##########
paimon-core/src/main/java/org/apache/paimon/rest/RESTCatalogOptions.java:
##########
@@ -85,4 +85,10 @@ public class RESTCatalogOptions {
.booleanType()
.defaultValue(false)
.withDescription("Whether to support data token provided by the REST server.");
+
+ public static final ConfigOption<Integer> REST_PAGE_MAX_RESULTS =
+ ConfigOptions.key("rest.page.max.results")
Review Comment:
This should be defined by server, we should just pass no max results to server.
##########
paimon-common/src/main/java/org/apache/paimon/PagedList.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon;
+
+import java.util.List;
+
+/** Paged List which supports request data from page streaming. */
+public class PagedList<T> {
+ private final List<T> pagedList;
+
+ private final String nextPageToken;
Review Comment:
`@Nullable`
##########
paimon-common/src/main/java/org/apache/paimon/PagedList.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon;
+
+import java.util.List;
+
+/** Paged List which supports request data from page streaming. */
+public class PagedList<T> {
+ private final List<T> pagedList;
+
+ private final String nextPageToken;
+
+ public PagedList(List<T> pagedList, String nextPageToken) {
+ this.pagedList = pagedList;
+ this.nextPageToken = nextPageToken;
+ }
+
+ public List<T> getPagedList() {
+ return this.pagedList;
+ }
+
+ public String getNextPageToken() {
Review Comment:
`@Nullable`
##########
paimon-common/src/main/java/org/apache/paimon/PagedList.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon;
+
+import java.util.List;
+
+/** Paged List which supports request data from page streaming. */
Review Comment:
Add:
```
* @since 1.1.0
*/
@Public
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "sundapeng (via GitHub)" <gi...@apache.org>.
sundapeng commented on PR #5200:
URL: https://github.com/apache/paimon/pull/5200#issuecomment-2696604057
LGTM, thank @XiaoHongbo-Hope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #5200:
URL: https://github.com/apache/paimon/pull/5200#discussion_r1979405264
##########
paimon-common/src/main/java/org/apache/paimon/PagedList.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon;
+
+import java.util.List;
+
+/** Paged List which supports request data from page streaming. */
+public class PagedList<T> {
+ private final List<T> pagedList;
+
+ private final String nextPageToken;
Review Comment:
👌
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #5200:
URL: https://github.com/apache/paimon/pull/5200#discussion_r1979403194
##########
paimon-common/src/main/java/org/apache/paimon/PagedList.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon;
+
+import java.util.List;
+
+/** Paged List which supports request data from page streaming. */
+public class PagedList<T> {
+ private final List<T> pagedList;
Review Comment:
👌
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #5200:
URL: https://github.com/apache/paimon/pull/5200#discussion_r1977470819
##########
paimon-open-api/src/main/java/org/apache/paimon/open/api/RESTCatalogController.java:
##########
@@ -226,8 +228,47 @@ public AlterDatabaseResponse alterDatabase(
})
@GetMapping("/v1/{prefix}/databases/{database}/tables")
public ListTablesResponse listTables(
- @PathVariable String prefix, @PathVariable String database) {
- return new ListTablesResponse(ImmutableList.of("user"));
+ @PathVariable String prefix,
+ @PathVariable String database,
+ @PathVariable Integer maxResults,
Review Comment:
ok, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "sundapeng (via GitHub)" <gi...@apache.org>.
sundapeng commented on code in PR #5200:
URL: https://github.com/apache/paimon/pull/5200#discussion_r1977541986
##########
paimon-core/src/main/java/org/apache/paimon/rest/RESTCatalog.java:
##########
@@ -257,16 +270,133 @@ public void alterDatabase(String name, List<PropertyChange> changes, boolean ign
@Override
public List<String> listTables(String databaseName) throws DatabaseNotExistException {
+ try {
+ String pageToken = null;
+ Map<String, String> queryParams = Maps.newHashMap();
+ Integer maxResults = context.options().get(RESTCatalogOptions.REST_PAGE_MAX_RESULTS);
+ if (Objects.nonNull(maxResults) && maxResults > 0) {
+ queryParams.put(MAX_RESULTS, String.valueOf(maxResults));
+ }
+ List<String> tables = new ArrayList<>();
+ do {
+ queryParams.put(PAGE_TOKEN, pageToken);
+ ListTablesResponse response =
+ client.get(
+ resourcePaths.tables(databaseName),
+ queryParams,
+ ListTablesResponse.class,
+ restAuthFunction);
+ if (Objects.nonNull(response)) {
+ pageToken = response.getNextPageToken();
+ tables.addAll(response.getTables());
+ } else {
+ LOG.warn(
+ "response of listTables for {} is null with params {}",
+ databaseName,
+ queryParams);
+ }
+ } while (pageToken != null);
+ return tables;
+ } catch (NoSuchResourceException e) {
+ throw new DatabaseNotExistException(databaseName);
+ }
+ }
+
+ @Override
+ public PagedList<String> listTablesPaged(
+ String databaseName, Integer maxResults, String pageToken)
+ throws DatabaseNotExistException {
+ Map<String, String> queryParams = Maps.newHashMap();
+ if (Objects.nonNull(maxResults) && maxResults > 0) {
+ queryParams.put(MAX_RESULTS, maxResults.toString());
+ }
+ if (Objects.nonNull(pageToken)) {
+ queryParams.put(PAGE_TOKEN, pageToken);
+ }
try {
ListTablesResponse response =
client.get(
resourcePaths.tables(databaseName),
+ queryParams,
ListTablesResponse.class,
restAuthFunction);
- if (response.getTables() != null) {
- return response.getTables();
+ if (Objects.nonNull(response) && Objects.nonNull(response.getTables())) {
+ return new PagedList<>(response.getTables(), response.getNextPageToken());
+ } else {
+ LOG.warn(
+ "response of listTablesPaged for {} is null with maxResults {} and pageToken {}",
+ databaseName,
+ maxResults,
+ pageToken);
+ return new PagedList<>(Collections.emptyList(), null);
+ }
+ } catch (NoSuchResourceException e) {
+ throw new DatabaseNotExistException(databaseName);
+ }
+ }
+
+ @Override
+ public PagedList<Table> listTableDetailsPaged(
+ String databaseName, Integer maxResults, String pageToken)
+ throws DatabaseNotExistException {
+ Map<String, String> queryParams = Maps.newHashMap();
+ if (Objects.nonNull(maxResults) && maxResults > 0) {
+ queryParams.put(MAX_RESULTS, maxResults.toString());
+ }
+ if (Objects.nonNull(pageToken)) {
+ queryParams.put(PAGE_TOKEN, pageToken);
+ }
+ try {
+ ListTableDetailsResponse response =
+ client.get(
+ resourcePaths.tableDetails(databaseName),
+ queryParams,
+ ListTableDetailsResponse.class,
+ restAuthFunction);
+ if (Objects.nonNull(response) && Objects.nonNull(response.getTableDetails())) {
+ return new PagedList<>(
+ response.getTableDetails().stream()
+ .map(
+ getTableResponse -> {
+ Identifier identifier =
+ Identifier.create(
+ databaseName,
+ getTableResponse.getName());
+ try {
Review Comment:
Is it possible to reuse some utility methods, such as `getTable(Identifier identifier)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #5200:
URL: https://github.com/apache/paimon/pull/5200#discussion_r1977607478
##########
paimon-core/src/main/java/org/apache/paimon/catalog/Catalog.java:
##########
@@ -130,6 +131,44 @@ void alterDatabase(String name, List<PropertyChange> changes, boolean ignoreIfNo
*/
List<String> listTables(String databaseName) throws DatabaseNotExistException;
+ /**
+ * Get paged list names of tables under this database. An empty list is returned if none exists.
+ *
+ * <p>NOTE: System tables will not be listed.
+ *
+ * @param databaseName Name of the database to list tables.
+ * @param maxResults Optional parameter indicating the maximum number of results to include in
+ * the result. If maxResults is not specified or set to 0, will return the default number of
+ * max results.
+ * @param pageToken Optional parameter indicating the next page token allows list to be start
+ * from a specific point.
+ * @return a list of the names of tables with provided page size in this database and next page
+ * token
+ * @throws DatabaseNotExistException if the database does not exist
+ */
+ PagedList<String> listTablesPaged(String databaseName, Integer maxResults, String pageToken)
Review Comment:
`@Nullable` for nullable arguements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #5200:
URL: https://github.com/apache/paimon/pull/5200#discussion_r1979407159
##########
paimon-common/src/main/java/org/apache/paimon/PagedList.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon;
+
+import java.util.List;
+
+/** Paged List which supports request data from page streaming. */
+public class PagedList<T> {
+ private final List<T> pagedList;
+
+ private final String nextPageToken;
+
+ public PagedList(List<T> pagedList, String nextPageToken) {
+ this.pagedList = pagedList;
+ this.nextPageToken = nextPageToken;
+ }
+
+ public List<T> getPagedList() {
+ return this.pagedList;
+ }
+
+ public String getNextPageToken() {
Review Comment:
👌
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] RestCatalog: set up paged list tables/views/partitions [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #5200:
URL: https://github.com/apache/paimon/pull/5200#discussion_r1979405972
##########
paimon-common/src/main/java/org/apache/paimon/PagedList.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon;
+
+import java.util.List;
+
+/** Paged List which supports request data from page streaming. */
+public class PagedList<T> {
+ private final List<T> pagedList;
+
+ private final String nextPageToken;
+
+ public PagedList(List<T> pagedList, String nextPageToken) {
Review Comment:
👌
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org