You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "yuxuan-luo (via GitHub)" <gi...@apache.org> on 2023/05/16 02:44:49 UTC
[GitHub] [doris] yuxuan-luo opened a new pull request, #19681: [Enhancement](catalog) paimon catalog
yuxuan-luo opened a new pull request, #19681:
URL: https://github.com/apache/doris/pull/19681
# Proposed changes
Issue Number: close #xxx
## Problem summary
Describe your changes.
## Checklist(Required)
* [ ] Does it affect the original behavior
* [ ] Has unit tests been added
* [ ] Has document been added or modified
* [ ] Does it need to update dependencies
* [ ] Is this PR support rollback (If NO, please explain WHY)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1567725024
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] AshinGau commented on a diff in pull request #19681: [Enhancement](catalog) paimon catalog
Posted by "AshinGau (via GitHub)" <gi...@apache.org>.
AshinGau commented on code in PR #19681:
URL: https://github.com/apache/doris/pull/19681#discussion_r1198784696
##########
gensrc/thrift/PlanNodes.thrift:
##########
@@ -554,7 +569,7 @@ struct TSortInfo {
4: optional list<Exprs.TExpr> sort_tuple_slot_exprs
// Indicates the nullable info of sort_tuple_slot_exprs is changed after substitute by child's smap
- 5: optional list<bool> slot_exprs_nullability_changed_flags
Review Comment:
Some unknown changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1569400626
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [Feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1573423162
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman merged pull request #19681: [Feature](multi-catalog)support paimon catalog
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #19681:
URL: https://github.com/apache/doris/pull/19681
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yuxuan-luo commented on a diff in pull request #19681: [Enhancement](catalog) paimon catalog
Posted by "yuxuan-luo (via GitHub)" <gi...@apache.org>.
yuxuan-luo commented on code in PR #19681:
URL: https://github.com/apache/doris/pull/19681#discussion_r1198757150
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonHMSExternalCatalog.java:
##########
@@ -0,0 +1,165 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.paimon;
+
+import org.apache.doris.datasource.CatalogProperty;
+import org.apache.doris.datasource.property.PropertyConverter;
+import org.apache.doris.datasource.property.constants.HMSProperties;
+import org.apache.doris.datasource.property.constants.PaimonProperties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.fs.Path;
+import org.apache.paimon.hive.HiveCatalog;
+import org.apache.paimon.hive.HiveCatalogOptions;
+import org.apache.paimon.options.CatalogOptions;
+import org.apache.paimon.options.ConfigOption;
+import org.apache.paimon.options.ConfigOptions;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.utils.StringUtils;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URL;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+
+
+
+public class PaimonHMSExternalCatalog extends PaimonExternalCatalog {
+
+ private static final ConfigOption<String> METASTORE_CLIENT_CLASS =
+ ConfigOptions.key("metastore.client.class")
+ .stringType()
+ .defaultValue("org.apache.hadoop.hive.metastore.HiveMetaStoreClient")
+ .withDescription(
+ "Class name of Hive metastore client.\n"
+ + "NOTE: This class must directly implements "
+ + "org.apache.hadoop.hive.metastore.IMetaStoreClient.");
+
+ public PaimonHMSExternalCatalog(long catalogId, String name, String resource, Map<String, String> props) {
+ super(catalogId, name);
+ props = PropertyConverter.convertToMetaProperties(props);
+ catalogProperty = new CatalogProperty(resource, props);
+ }
+
+ @Override
+ protected void initLocalObjectsImpl() {
+ paimonCatalogType = PAIMON_HMS;
+ String metastoreUris = catalogProperty.getOrDefault(HMSProperties.HIVE_METASTORE_URIS, "");
+ String warehouse = catalogProperty.getOrDefault(PaimonProperties.PAIMON_WAREHOUSE, "");
+ Options options = new Options();
+ options.set(PaimonProperties.WAREHOUSE, warehouse);
+ // Currently, only supports hive
+ options.set(PaimonProperties.METASTORE, PaimonProperties.METASTORE_HIVE);
+ options.set(PaimonProperties.URI, metastoreUris);
+ CatalogContext context = CatalogContext.create(options, getConfiguration());
+ try {
+ catalog = create(context);
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ }
+
+ public Catalog create(CatalogContext context) throws IOException {
+ Path warehousePath = new Path(context.options().get(CatalogOptions.WAREHOUSE));
+ FileIO fileIO;
+ fileIO = FileIO.get(warehousePath, context);
+ String uri = context.options().get(CatalogOptions.URI);
+ String hiveConfDir = context.options().get(HiveCatalogOptions.HIVE_CONF_DIR);
+ String hadoopConfDir = context.options().get(HiveCatalogOptions.HADOOP_CONF_DIR);
+ HiveConf hiveConf = createHiveConf(hiveConfDir, hadoopConfDir);
+
+ // always using user-set parameters overwrite hive-site.xml parameters
+ context.options().toMap().forEach(hiveConf::set);
+ hiveConf.set(HiveConf.ConfVars.METASTOREURIS.varname, uri);
+ // set the warehouse location to the hiveConf
+ hiveConf.set(HiveConf.ConfVars.METASTOREWAREHOUSE.varname, context.options().get(CatalogOptions.WAREHOUSE));
+
+ String clientClassName = context.options().get(METASTORE_CLIENT_CLASS);
+
+ return new HiveCatalog(fileIO, hiveConf, clientClassName, context.options().toMap());
+ }
+
+ public static HiveConf createHiveConf(@Nullable String hiveConfDir, @Nullable String hadoopConfDir) {
+ Configuration hadoopConf = null;
+ if (!StringUtils.isNullOrWhitespaceOnly(hadoopConfDir)) {
+ hadoopConf = HiveCatalog.getHadoopConfiguration(hadoopConfDir);
+ if (hadoopConf == null) {
+ String possiableUsedConfFiles = "core-site.xml | hdfs-site.xml | yarn-site.xml | mapred-site.xml";
+ throw new RuntimeException("Failed to load the hadoop conf from specified path:" + hadoopConfDir,
+ new FileNotFoundException("Please check the path none of the conf files ("
+ + possiableUsedConfFiles + ") exist in the folder."));
+ }
+ }
+
+ if (hadoopConf == null) {
+ hadoopConf = new Configuration();
+ }
+
+ if (hiveConfDir == null) {
+ return new HiveConf(hadoopConf, HiveConf.class);
+ } else {
+ HiveConf.setHiveSiteLocation((URL) null);
Review Comment:
Reading hive-site.xml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1567806561
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1567812430
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [Feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1573426016
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on pull request #19681: [Feature](multi-catalog)support paimon catalog
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1576405915
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wsjz commented on a diff in pull request #19681: [Enhancement](catalog) paimon catalog
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on code in PR #19681:
URL: https://github.com/apache/doris/pull/19681#discussion_r1198757265
##########
be/src/vec/exec/scan/vfile_scanner.cpp:
##########
@@ -641,6 +642,14 @@ Status VFileScanner::_get_next_reader() {
init_status = ((NewJsonReader*)(_cur_reader.get()))->init_reader();
break;
}
+ case TFileFormatType::FORMAT_JNI: {
+ // test
+ LOG(INFO) << "FORMAT_JNI ";
+ _cur_reader = PaimonJniReader::create_unique(_file_slot_descs, _state, _profile, range);
Review Comment:
JNI not only paimon, can you judge this in any other way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1556862324
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1567145443
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #19681: [Enhancement](catalog) paimon catalog
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #19681:
URL: https://github.com/apache/doris/pull/19681#discussion_r1196724094
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonHMSExternalCatalog.java:
##########
@@ -0,0 +1,165 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.paimon;
+
+import org.apache.doris.datasource.CatalogProperty;
+import org.apache.doris.datasource.property.PropertyConverter;
+import org.apache.doris.datasource.property.constants.HMSProperties;
+import org.apache.doris.datasource.property.constants.PaimonProperties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.fs.Path;
+import org.apache.paimon.hive.HiveCatalog;
+import org.apache.paimon.hive.HiveCatalogOptions;
+import org.apache.paimon.options.CatalogOptions;
+import org.apache.paimon.options.ConfigOption;
+import org.apache.paimon.options.ConfigOptions;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.utils.StringUtils;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URL;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+
+
+
+public class PaimonHMSExternalCatalog extends PaimonExternalCatalog {
+
+ private static final ConfigOption<String> METASTORE_CLIENT_CLASS =
+ ConfigOptions.key("metastore.client.class")
+ .stringType()
+ .defaultValue("org.apache.hadoop.hive.metastore.HiveMetaStoreClient")
+ .withDescription(
+ "Class name of Hive metastore client.\n"
+ + "NOTE: This class must directly implements "
+ + "org.apache.hadoop.hive.metastore.IMetaStoreClient.");
+
+ public PaimonHMSExternalCatalog(long catalogId, String name, String resource, Map<String, String> props) {
+ super(catalogId, name);
+ props = PropertyConverter.convertToMetaProperties(props);
+ catalogProperty = new CatalogProperty(resource, props);
+ }
+
+ @Override
+ protected void initLocalObjectsImpl() {
+ paimonCatalogType = PAIMON_HMS;
+ String metastoreUris = catalogProperty.getOrDefault(HMSProperties.HIVE_METASTORE_URIS, "");
+ String warehouse = catalogProperty.getOrDefault(PaimonProperties.PAIMON_WAREHOUSE, "");
+ Options options = new Options();
+ options.set(PaimonProperties.WAREHOUSE, warehouse);
+ // Currently, only supports hive
+ options.set(PaimonProperties.METASTORE, PaimonProperties.METASTORE_HIVE);
+ options.set(PaimonProperties.URI, metastoreUris);
+ CatalogContext context = CatalogContext.create(options, getConfiguration());
+ try {
+ catalog = create(context);
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ }
+
+ public Catalog create(CatalogContext context) throws IOException {
+ Path warehousePath = new Path(context.options().get(CatalogOptions.WAREHOUSE));
+ FileIO fileIO;
+ fileIO = FileIO.get(warehousePath, context);
+ String uri = context.options().get(CatalogOptions.URI);
+ String hiveConfDir = context.options().get(HiveCatalogOptions.HIVE_CONF_DIR);
+ String hadoopConfDir = context.options().get(HiveCatalogOptions.HADOOP_CONF_DIR);
+ HiveConf hiveConf = createHiveConf(hiveConfDir, hadoopConfDir);
+
+ // always using user-set parameters overwrite hive-site.xml parameters
+ context.options().toMap().forEach(hiveConf::set);
+ hiveConf.set(HiveConf.ConfVars.METASTOREURIS.varname, uri);
+ // set the warehouse location to the hiveConf
+ hiveConf.set(HiveConf.ConfVars.METASTOREWAREHOUSE.varname, context.options().get(CatalogOptions.WAREHOUSE));
+
+ String clientClassName = context.options().get(METASTORE_CLIENT_CLASS);
+
+ return new HiveCatalog(fileIO, hiveConf, clientClassName, context.options().toMap());
+ }
+
+ public static HiveConf createHiveConf(@Nullable String hiveConfDir, @Nullable String hadoopConfDir) {
+ Configuration hadoopConf = null;
+ if (!StringUtils.isNullOrWhitespaceOnly(hadoopConfDir)) {
+ hadoopConf = HiveCatalog.getHadoopConfiguration(hadoopConfDir);
+ if (hadoopConf == null) {
+ String possiableUsedConfFiles = "core-site.xml | hdfs-site.xml | yarn-site.xml | mapred-site.xml";
+ throw new RuntimeException("Failed to load the hadoop conf from specified path:" + hadoopConfDir,
+ new FileNotFoundException("Please check the path none of the conf files ("
+ + possiableUsedConfFiles + ") exist in the folder."));
+ }
+ }
+
+ if (hadoopConf == null) {
+ hadoopConf = new Configuration();
+ }
+
+ if (hiveConfDir == null) {
+ return new HiveConf(hadoopConf, HiveConf.class);
+ } else {
+ HiveConf.setHiveSiteLocation((URL) null);
Review Comment:
What is these code for? Reading hive-site.xml?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on pull request #19681: [Feature](multi-catalog)support paimon catalog
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1576858895
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on pull request #19681: [Feature](multi-catalog)support paimon catalog
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1577867301
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [Feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1576229636
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [Feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1577853735
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] SteNicholas commented on pull request #19681: [Enhancement](catalog) paimon catalog
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1554330964
@yuxuan-luo, could you please add the test cases for fe changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [Feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1574589859
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [Feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1577787023
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [Feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1577778778
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [Feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1575906929
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wsjz commented on a diff in pull request #19681: [Enhancement](catalog) paimon catalog
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on code in PR #19681:
URL: https://github.com/apache/doris/pull/19681#discussion_r1198771894
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/property/constants/PaimonProperties.java:
##########
@@ -0,0 +1,26 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.property.constants;
+
+public class PaimonProperties {
Review Comment:
Properties classes at property/constants are used for cloud storage, and they are provided to the users. So I think paimon doesn't need new extra properties class, you can just **put these properties to paimon catalog if these properties are not exposed to the users.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wsjz commented on a diff in pull request #19681: [Enhancement](catalog) paimon catalog
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on code in PR #19681:
URL: https://github.com/apache/doris/pull/19681#discussion_r1198762923
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalCatalog.java:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.paimon;
+
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.external.ExternalDatabase;
+import org.apache.doris.catalog.external.PaimonExternalDatabase;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.FeNameFormat;
+import org.apache.doris.common.util.Util;
+import org.apache.doris.datasource.ExternalCatalog;
+import org.apache.doris.datasource.InitCatalogLog;
+import org.apache.doris.datasource.SessionContext;
+
+import com.google.common.collect.Maps;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.HdfsConfiguration;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.Identifier;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public abstract class PaimonExternalCatalog extends ExternalCatalog {
Review Comment:
register into fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/external/PaimonExternalDatabase.java:
##########
@@ -0,0 +1,181 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.catalog.external;
+
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.TableIf;
+import org.apache.doris.datasource.ExternalCatalog;
+import org.apache.doris.datasource.InitDatabaseLog;
+import org.apache.doris.datasource.paimon.PaimonExternalCatalog;
+import org.apache.doris.persist.gson.GsonPostProcessable;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import com.google.gson.annotations.SerializedName;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.IOException;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.stream.Collectors;
+
+public class PaimonExternalDatabase extends ExternalDatabase<PaimonExternalTable> implements GsonPostProcessable {
+
+ private static final Logger LOG = LogManager.getLogger(PaimonExternalDatabase.class);
+ // Cache of table name to table id.
+ private Map<String, Long> tableNameToId = Maps.newConcurrentMap();
+ @SerializedName(value = "idToTbl")
+ private Map<Long, PaimonExternalTable> idToTbl = Maps.newConcurrentMap();
+
+ public PaimonExternalDatabase(ExternalCatalog extCatalog, Long id, String name) {
+ super(extCatalog, id, name);
+ }
+
+ public void replayInitDb(InitDatabaseLog log, ExternalCatalog catalog) {
Review Comment:
rebase master and extends ExternalCatalog to avoid repeated code.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalCatalog.java:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.paimon;
+
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.external.ExternalDatabase;
+import org.apache.doris.catalog.external.PaimonExternalDatabase;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.FeNameFormat;
+import org.apache.doris.common.util.Util;
+import org.apache.doris.datasource.ExternalCatalog;
+import org.apache.doris.datasource.InitCatalogLog;
+import org.apache.doris.datasource.SessionContext;
+
+import com.google.common.collect.Maps;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.HdfsConfiguration;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.Identifier;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public abstract class PaimonExternalCatalog extends ExternalCatalog {
+
+ private static final Logger LOG = LogManager.getLogger(PaimonExternalCatalog.class);
+ public static final String PAIMON_CATALOG_TYPE = "paimon.catalog.type";
+ public static final String PAIMON_HMS = "hms";
+ protected String paimonCatalogType;
+ protected Catalog catalog;
+
+ public PaimonExternalCatalog(long catalogId, String name) {
+ super(catalogId, name);
+ }
+
+ @Override
+ protected void init() {
Review Comment:
rebase master and extends ExternalCatalog to avoid repeated code.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonHMSExternalCatalog.java:
##########
@@ -0,0 +1,165 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.paimon;
+
+import org.apache.doris.datasource.CatalogProperty;
+import org.apache.doris.datasource.property.PropertyConverter;
+import org.apache.doris.datasource.property.constants.HMSProperties;
+import org.apache.doris.datasource.property.constants.PaimonProperties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.fs.Path;
+import org.apache.paimon.hive.HiveCatalog;
+import org.apache.paimon.hive.HiveCatalogOptions;
+import org.apache.paimon.options.CatalogOptions;
+import org.apache.paimon.options.ConfigOption;
+import org.apache.paimon.options.ConfigOptions;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.utils.StringUtils;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URL;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+
+
+
+public class PaimonHMSExternalCatalog extends PaimonExternalCatalog {
Review Comment:
why not use PaimonExternalCatalog directly? Are there any other clients can access paimon besides HMS?
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/external/PaimonExternalDatabase.java:
##########
@@ -0,0 +1,181 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.catalog.external;
+
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.TableIf;
+import org.apache.doris.datasource.ExternalCatalog;
+import org.apache.doris.datasource.InitDatabaseLog;
+import org.apache.doris.datasource.paimon.PaimonExternalCatalog;
+import org.apache.doris.persist.gson.GsonPostProcessable;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import com.google.gson.annotations.SerializedName;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.IOException;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.stream.Collectors;
+
+public class PaimonExternalDatabase extends ExternalDatabase<PaimonExternalTable> implements GsonPostProcessable {
Review Comment:
register into fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/external/PaimonExternalTable.java:
##########
@@ -0,0 +1,132 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.catalog.external;
+
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.catalog.Type;
+import org.apache.doris.datasource.paimon.PaimonExternalCatalog;
+import org.apache.doris.thrift.THiveTable;
+import org.apache.doris.thrift.TTableDescriptor;
+import org.apache.doris.thrift.TTableType;
+
+import com.google.common.collect.Lists;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.paimon.schema.TableSchema;
+import org.apache.paimon.table.AbstractFileStoreTable;
+import org.apache.paimon.table.Table;
+import org.apache.paimon.types.DataField;
+import org.apache.paimon.types.DecimalType;
+
+import java.util.HashMap;
+import java.util.List;
+
+public class PaimonExternalTable extends ExternalTable {
Review Comment:
register into fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wsjz commented on a diff in pull request #19681: [Enhancement](catalog) paimon catalog
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on code in PR #19681:
URL: https://github.com/apache/doris/pull/19681#discussion_r1198762923
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalCatalog.java:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.paimon;
+
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.external.ExternalDatabase;
+import org.apache.doris.catalog.external.PaimonExternalDatabase;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.FeNameFormat;
+import org.apache.doris.common.util.Util;
+import org.apache.doris.datasource.ExternalCatalog;
+import org.apache.doris.datasource.InitCatalogLog;
+import org.apache.doris.datasource.SessionContext;
+
+import com.google.common.collect.Maps;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.HdfsConfiguration;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.Identifier;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public abstract class PaimonExternalCatalog extends ExternalCatalog {
Review Comment:
register into fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java
when add a new catalog, we need use gson to persist catalog, or if restart the FE, catalog properties will be lost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [Feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1573609178
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [Feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1577853920
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wsjz commented on a diff in pull request #19681: [Enhancement](catalog) paimon catalog
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on code in PR #19681:
URL: https://github.com/apache/doris/pull/19681#discussion_r1198767824
##########
fe/fe-core/src/main/java/org/apache/doris/planner/external/paimon/PaimonSource.java:
##########
@@ -0,0 +1,64 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.planner.external.paimon;
+
+import org.apache.doris.analysis.TupleDescriptor;
+import org.apache.doris.catalog.TableIf;
+import org.apache.doris.catalog.external.PaimonExternalTable;
+import org.apache.doris.common.UserException;
+import org.apache.doris.datasource.ExternalCatalog;
+import org.apache.doris.planner.ColumnRange;
+import org.apache.doris.thrift.TFileAttributes;
+
+import org.apache.paimon.table.Table;
+
+import java.util.Map;
+
+public class PaimonSource {
Review Comment:
Are there any other sources besides HMS?
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/property/constants/PaimonProperties.java:
##########
@@ -0,0 +1,26 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.property.constants;
+
+public class PaimonProperties {
Review Comment:
Properties classes at property/constants are used for cloud storage, I think paimon doesn't need new extra properties class, you can just **put these properties to paimon catalog if these properties are not exposed to the users.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wsjz commented on a diff in pull request #19681: [Enhancement](catalog) paimon catalog
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on code in PR #19681:
URL: https://github.com/apache/doris/pull/19681#discussion_r1198779960
##########
fe/java-udf/src/main/java/org/apache/doris/jni/vec/PaimonColumnValue.java:
##########
@@ -0,0 +1,121 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.jni.vec;
+
+import org.apache.paimon.utils.OffsetRow;
+
+import java.math.BigDecimal;
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.List;
+
+public class PaimonColumnValue implements ColumnValue {
+ private int idx;
+ private OffsetRow record;
+
+ public PaimonColumnValue() {
+ }
+
+ public void setIdx(int idx) {
+ this.record = record;
+ this.idx = idx;
+ }
+
+ public void setOffsetRow(OffsetRow record) {
+ this.record = record;
+ }
+
+ @Override
+ public boolean getBoolean() {
+ return record.getBoolean(idx);
+ }
+
+ @Override
+ public byte getByte() {
+ return record.getByte(idx);
+ }
+
+ @Override
+ public short getShort() {
+ return record.getShort(idx);
+ }
+
+ @Override
+ public int getInt() {
+ return record.getInt(idx);
+ }
+
+ @Override
+ public float getFloat() {
+ return record.getFloat(idx);
+ }
+
+ @Override
+ public long getLong() {
+ return record.getLong(idx);
+ }
+
+ @Override
+ public double getDouble() {
+ return record.getDouble(idx);
+ }
+
+ @Override
+ public BigDecimal getDecimal() {
+ return BigDecimal.valueOf(getDouble());
+ }
+
+ @Override
+ public String getString() {
+ return record.getString(idx).toString();
+ }
+
+ @Override
+ public LocalDate getDate() {
+ return Instant.ofEpochMilli(record.getTimestamp(idx, 3)
+ .getMillisecond()).atZone(ZoneOffset.ofHours(8)).toLocalDate();
+ }
+
+ @Override
+ public LocalDateTime getDateTime() {
+ return Instant.ofEpochMilli(record.getTimestamp(idx, 3)
+ .getMillisecond()).atZone(ZoneOffset.ofHours(8)).toLocalDateTime();
+ }
+
+ @Override
+ public byte[] getBytes() {
+ return record.getBinary(idx);
+ }
+
+ @Override
+ public void unpackArray(List<ColumnValue> values) {
+
Review Comment:
When can these nested types be supported？ And is this easy or hard?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] SteNicholas commented on a diff in pull request #19681: [Enhancement](catalog) paimon catalog
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #19681:
URL: https://github.com/apache/doris/pull/19681#discussion_r1198776851
##########
gensrc/thrift/PlanNodes.thrift:
##########
@@ -694,7 +709,7 @@ enum TAggregationOp {
DENSE_RANK,
ROW_NUMBER,
LAG,
- HLL_C,
+ HLL_C,
Review Comment:
No necessary change in this line. Revert.
##########
gensrc/thrift/PlanNodes.thrift:
##########
@@ -760,8 +775,8 @@ struct TSortNode {
// This is the number of rows to skip before returning results
3: optional i64 offset
- // Indicates whether the imposed limit comes DEFAULT_ORDER_BY_LIMIT.
- 6: optional bool is_default_limit
+ // Indicates whether the imposed limit comes DEFAULT_ORDER_BY_LIMIT.
Review Comment:
No necessary change in this line. Revert.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1567725325
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [Enhancement](catalog) paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1554298534
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] SteNicholas commented on a diff in pull request #19681: [Enhancement](catalog) paimon catalog
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #19681:
URL: https://github.com/apache/doris/pull/19681#discussion_r1198777157
##########
gensrc/thrift/PlanNodes.thrift:
##########
@@ -931,7 +946,7 @@ struct TBackendResourceProfile {
// The maximum reservation for this plan node in bytes. MAX_INT64 means effectively
// unlimited.
-2: required i64 max_reservation = 12188490189880; // no max reservation limit
Review Comment:
No necessary change in this line. Revert.
##########
gensrc/thrift/PlanNodes.thrift:
##########
@@ -1037,9 +1052,9 @@ struct TPlanNode {
14: optional TMergeNode merge_node
15: optional TExchangeNode exchange_node
17: optional TMySQLScanNode mysql_scan_node
- 18: optional TOlapScanNode olap_scan_node
- 19: optional TCsvScanNode csv_scan_node
- 20: optional TBrokerScanNode broker_scan_node
+ 18: optional TOlapScanNode olap_scan_node
Review Comment:
No necessary change in this line. Revert.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1556668987
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [Feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1573601826
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on pull request #19681: [Feature](multi-catalog)support paimon catalog
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1577792233
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #19681: [Feature](multi-catalog)support paimon catalog
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #19681:
URL: https://github.com/apache/doris/pull/19681#discussion_r1214108063
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalCatalog.java:
##########
@@ -0,0 +1,117 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.paimon;
+
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.FeNameFormat;
+import org.apache.doris.common.util.Util;
+import org.apache.doris.datasource.ExternalCatalog;
+import org.apache.doris.datasource.InitCatalogLog;
+import org.apache.doris.datasource.SessionContext;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.HdfsConfiguration;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.Identifier;
+
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public abstract class PaimonExternalCatalog extends ExternalCatalog {
+
+ private static final Logger LOG = LogManager.getLogger(PaimonExternalCatalog.class);
+ public static final String PAIMON_HMS = "hms";
+ protected String paimonCatalogType;
+ protected Catalog catalog;
+
+ public PaimonExternalCatalog(long catalogId, String name) {
+ super(catalogId, name, InitCatalogLog.Type.PAIMON);
+ this.type = "paimon";
+ }
+
+ @Override
+ protected void init() {
+ super.init();
+ }
+
+ protected Configuration getConfiguration() {
+ Configuration conf = new HdfsConfiguration();
+ Map<String, String> catalogProperties = catalogProperty.getHadoopProperties();
+ for (Map.Entry<String, String> entry : catalogProperties.entrySet()) {
+ conf.set(entry.getKey(), entry.getValue());
+ }
+ return conf;
+ }
+
+ public Catalog getCatalog() {
+ makeSureInitialized();
+ return catalog;
+ }
+
+ public String getPaimonCatalogType() {
+ makeSureInitialized();
+ return paimonCatalogType;
+ }
+
+ protected List<String> listDatabaseNames() {
+ return catalog.listDatabases().stream()
+ .map(e -> {
+ String dbName = e.toString();
+ try {
+ FeNameFormat.checkDbName(dbName);
Review Comment:
I think we don't need to check it
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalCatalog.java:
##########
@@ -0,0 +1,117 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.paimon;
+
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.FeNameFormat;
+import org.apache.doris.common.util.Util;
+import org.apache.doris.datasource.ExternalCatalog;
+import org.apache.doris.datasource.InitCatalogLog;
+import org.apache.doris.datasource.SessionContext;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.HdfsConfiguration;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.Identifier;
+
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public abstract class PaimonExternalCatalog extends ExternalCatalog {
+
+ private static final Logger LOG = LogManager.getLogger(PaimonExternalCatalog.class);
+ public static final String PAIMON_HMS = "hms";
+ protected String paimonCatalogType;
+ protected Catalog catalog;
+
+ public PaimonExternalCatalog(long catalogId, String name) {
+ super(catalogId, name, InitCatalogLog.Type.PAIMON);
+ this.type = "paimon";
+ }
+
+ @Override
+ protected void init() {
+ super.init();
+ }
+
+ protected Configuration getConfiguration() {
+ Configuration conf = new HdfsConfiguration();
+ Map<String, String> catalogProperties = catalogProperty.getHadoopProperties();
+ for (Map.Entry<String, String> entry : catalogProperties.entrySet()) {
+ conf.set(entry.getKey(), entry.getValue());
+ }
+ return conf;
+ }
+
+ public Catalog getCatalog() {
+ makeSureInitialized();
+ return catalog;
+ }
+
+ public String getPaimonCatalogType() {
+ makeSureInitialized();
+ return paimonCatalogType;
+ }
+
+ protected List<String> listDatabaseNames() {
+ return catalog.listDatabases().stream()
+ .map(e -> {
+ String dbName = e.toString();
+ try {
+ FeNameFormat.checkDbName(dbName);
+ } catch (AnalysisException ex) {
+ Util.logAndThrowRuntimeException(LOG,
+ String.format("Not a supported name format: %s", dbName), ex);
+ }
+ return dbName;
+ })
+ .collect(Collectors.toList());
+ }
+
+ @Override
+ public boolean tableExist(SessionContext ctx, String dbName, String tblName) {
+ makeSureInitialized();
+ return catalog.tableExists(Identifier.create(dbName, tblName));
+ }
+
+ @Override
+ public List<String> listTableNames(SessionContext ctx, String dbName) {
+ makeSureInitialized();
+ List<String> tableNames = null;
+ try {
+ tableNames = catalog.listTables(dbName);
+ } catch (Catalog.DatabaseNotExistException e) {
+ LOG.warn("DatabaseNotExistException", e);
Review Comment:
print catalog and db name in log
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonHMSExternalCatalog.java:
##########
@@ -0,0 +1,168 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.paimon;
+
+import org.apache.doris.datasource.CatalogProperty;
+import org.apache.doris.datasource.property.PropertyConverter;
+import org.apache.doris.datasource.property.constants.HMSProperties;
+import org.apache.doris.datasource.property.constants.PaimonProperties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.fs.Path;
+import org.apache.paimon.hive.HiveCatalog;
+import org.apache.paimon.hive.HiveCatalogOptions;
+import org.apache.paimon.options.CatalogOptions;
+import org.apache.paimon.options.ConfigOption;
+import org.apache.paimon.options.ConfigOptions;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.utils.StringUtils;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URL;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+
+
+
+public class PaimonHMSExternalCatalog extends PaimonExternalCatalog {
+
+ public static final String METASTORE = "metastore";
+ public static final String METASTORE_HIVE = "hive";
+ public static final String URI = "uri";
+ private static final ConfigOption<String> METASTORE_CLIENT_CLASS =
+ ConfigOptions.key("metastore.client.class")
+ .stringType()
+ .defaultValue("org.apache.hadoop.hive.metastore.HiveMetaStoreClient")
+ .withDescription(
+ "Class name of Hive metastore client.\n"
+ + "NOTE: This class must directly implements "
+ + "org.apache.hadoop.hive.metastore.IMetaStoreClient.");
+
+ public PaimonHMSExternalCatalog(long catalogId, String name, String resource, Map<String, String> props) {
+ super(catalogId, name);
+ props = PropertyConverter.convertToMetaProperties(props);
+ catalogProperty = new CatalogProperty(resource, props);
+ }
+
+ @Override
+ protected void initLocalObjectsImpl() {
+ paimonCatalogType = PAIMON_HMS;
Review Comment:
This should be done in constructor
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalCatalog.java:
##########
@@ -0,0 +1,117 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.paimon;
+
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.FeNameFormat;
+import org.apache.doris.common.util.Util;
+import org.apache.doris.datasource.ExternalCatalog;
+import org.apache.doris.datasource.InitCatalogLog;
+import org.apache.doris.datasource.SessionContext;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.HdfsConfiguration;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.Identifier;
+
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public abstract class PaimonExternalCatalog extends ExternalCatalog {
+
+ private static final Logger LOG = LogManager.getLogger(PaimonExternalCatalog.class);
+ public static final String PAIMON_HMS = "hms";
+ protected String paimonCatalogType;
+ protected Catalog catalog;
+
+ public PaimonExternalCatalog(long catalogId, String name) {
+ super(catalogId, name, InitCatalogLog.Type.PAIMON);
+ this.type = "paimon";
+ }
+
+ @Override
+ protected void init() {
+ super.init();
+ }
+
+ protected Configuration getConfiguration() {
+ Configuration conf = new HdfsConfiguration();
+ Map<String, String> catalogProperties = catalogProperty.getHadoopProperties();
+ for (Map.Entry<String, String> entry : catalogProperties.entrySet()) {
+ conf.set(entry.getKey(), entry.getValue());
+ }
+ return conf;
+ }
+
+ public Catalog getCatalog() {
+ makeSureInitialized();
+ return catalog;
+ }
+
+ public String getPaimonCatalogType() {
+ makeSureInitialized();
+ return paimonCatalogType;
+ }
+
+ protected List<String> listDatabaseNames() {
+ return catalog.listDatabases().stream()
+ .map(e -> {
+ String dbName = e.toString();
+ try {
+ FeNameFormat.checkDbName(dbName);
+ } catch (AnalysisException ex) {
+ Util.logAndThrowRuntimeException(LOG,
+ String.format("Not a supported name format: %s", dbName), ex);
+ }
+ return dbName;
+ })
+ .collect(Collectors.toList());
+ }
+
+ @Override
+ public boolean tableExist(SessionContext ctx, String dbName, String tblName) {
+ makeSureInitialized();
+ return catalog.tableExists(Identifier.create(dbName, tblName));
+ }
+
+ @Override
+ public List<String> listTableNames(SessionContext ctx, String dbName) {
+ makeSureInitialized();
+ List<String> tableNames = null;
+ try {
+ tableNames = catalog.listTables(dbName);
+ } catch (Catalog.DatabaseNotExistException e) {
+ LOG.warn("DatabaseNotExistException", e);
+ }
+ return tableNames;
+ }
+
+ public org.apache.paimon.table.Table getPaimonTable(String dbName, String tblName) {
+ makeSureInitialized();
+ org.apache.paimon.table.Table table = null;
+ try {
+ table = catalog.getTable(Identifier.create(dbName, tblName));
+ } catch (Catalog.TableNotExistException e) {
+ LOG.warn("TableNotExistException", e);
Review Comment:
ditto
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonHMSExternalCatalog.java:
##########
@@ -0,0 +1,168 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.paimon;
+
+import org.apache.doris.datasource.CatalogProperty;
+import org.apache.doris.datasource.property.PropertyConverter;
+import org.apache.doris.datasource.property.constants.HMSProperties;
+import org.apache.doris.datasource.property.constants.PaimonProperties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.fs.Path;
+import org.apache.paimon.hive.HiveCatalog;
+import org.apache.paimon.hive.HiveCatalogOptions;
+import org.apache.paimon.options.CatalogOptions;
+import org.apache.paimon.options.ConfigOption;
+import org.apache.paimon.options.ConfigOptions;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.utils.StringUtils;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URL;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+
+
+
+public class PaimonHMSExternalCatalog extends PaimonExternalCatalog {
+
+ public static final String METASTORE = "metastore";
+ public static final String METASTORE_HIVE = "hive";
+ public static final String URI = "uri";
+ private static final ConfigOption<String> METASTORE_CLIENT_CLASS =
+ ConfigOptions.key("metastore.client.class")
+ .stringType()
+ .defaultValue("org.apache.hadoop.hive.metastore.HiveMetaStoreClient")
+ .withDescription(
+ "Class name of Hive metastore client.\n"
+ + "NOTE: This class must directly implements "
+ + "org.apache.hadoop.hive.metastore.IMetaStoreClient.");
+
+ public PaimonHMSExternalCatalog(long catalogId, String name, String resource, Map<String, String> props) {
+ super(catalogId, name);
+ props = PropertyConverter.convertToMetaProperties(props);
+ catalogProperty = new CatalogProperty(resource, props);
+ }
+
+ @Override
+ protected void initLocalObjectsImpl() {
+ paimonCatalogType = PAIMON_HMS;
+ String metastoreUris = catalogProperty.getOrDefault(HMSProperties.HIVE_METASTORE_URIS, "");
+ String warehouse = catalogProperty.getOrDefault(PaimonProperties.WAREHOUSE, "");
+ Options options = new Options();
+ options.set(PaimonProperties.WAREHOUSE, warehouse);
+ // Currently, only supports hive
+ options.set(METASTORE, METASTORE_HIVE);
+ options.set(URI, metastoreUris);
+ CatalogContext context = CatalogContext.create(options, getConfiguration());
+ try {
+ catalog = create(context);
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ }
+
+ public Catalog create(CatalogContext context) throws IOException {
Review Comment:
```suggestion
private Catalog create(CatalogContext context) throws IOException {
```
##########
fe/java-udf/src/main/java/org/apache/doris/jni/PaimonJniScanner.java:
##########
@@ -0,0 +1,187 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.jni;
+
+import org.apache.doris.jni.utils.OffHeap;
+import org.apache.doris.jni.vec.ColumnType;
+import org.apache.doris.jni.vec.PaimonColumnValue;
+import org.apache.doris.jni.vec.ScanPredicate;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.log4j.Logger;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.catalog.Identifier;
+import org.apache.paimon.data.InternalRow;
+import org.apache.paimon.data.columnar.ColumnarRow;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.fs.Path;
+import org.apache.paimon.hive.HiveCatalog;
+import org.apache.paimon.hive.HiveCatalogOptions;
+import org.apache.paimon.hive.mapred.PaimonInputSplit;
+import org.apache.paimon.options.CatalogOptions;
+import org.apache.paimon.options.ConfigOption;
+import org.apache.paimon.options.ConfigOptions;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.reader.RecordReader;
+import org.apache.paimon.table.Table;
+import org.apache.paimon.table.source.ReadBuilder;
+import org.apache.paimon.table.source.TableRead;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
+
+
+public class PaimonJniScanner extends JniScanner {
+ private static final Logger LOG = Logger.getLogger(PaimonJniScanner.class);
+
+ private final String metastoreUris;
+ private final String warehouse;
+ private final String dbName;
+ private final String tblName;
+ private final String[] ids;
+ private final long splitAddress;
+ private final int lengthByte;
+ private PaimonInputSplit paimonInputSplit;
+ private Table table;
+ private RecordReader<InternalRow> reader;
+ private final PaimonColumnValue columnValue = new PaimonColumnValue();
+
+ public PaimonJniScanner(int batchSize, Map<String, String> params) {
+ metastoreUris = params.get("hive.metastore.uris");
+ warehouse = params.get("warehouse");
+ splitAddress = Long.parseLong(params.get("split_byte"));
+ lengthByte = Integer.parseInt(params.get("length_byte"));
+ LOG.info("splitAddress:" + splitAddress);
+ LOG.info("lengthByte:" + lengthByte);
+ dbName = params.get("db_name");
+ tblName = params.get("table_name");
+ String[] requiredFields = params.get("required_fields").split(",");
+ String[] types = params.get("columns_types").split(",");
+ ids = params.get("columns_id").split(",");
+ ColumnType[] columnTypes = new ColumnType[types.length];
+ for (int i = 0; i < types.length; i++) {
+ columnTypes[i] = ColumnType.parseType(requiredFields[i], types[i]);
+ }
+ ScanPredicate[] predicates = new ScanPredicate[0];
+ if (params.containsKey("push_down_predicates")) {
+ long predicatesAddress = Long.parseLong(params.get("push_down_predicates"));
+ if (predicatesAddress != 0) {
+ predicates = ScanPredicate.parseScanPredicates(predicatesAddress, columnTypes);
+ LOG.info("MockJniScanner gets pushed-down predicates: " + ScanPredicate.dump(predicates));
+ }
+ }
+ initTableInfo(columnTypes, requiredFields, predicates, batchSize);
+ }
+
+ @Override
+ public void open() throws IOException {
+ getCatalog();
+ // 拿 []byte 反序列化成 split
Review Comment:
In English
##########
fe/java-udf/src/main/java/org/apache/doris/jni/PaimonJniScanner.java:
##########
@@ -0,0 +1,187 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.jni;
+
+import org.apache.doris.jni.utils.OffHeap;
+import org.apache.doris.jni.vec.ColumnType;
+import org.apache.doris.jni.vec.PaimonColumnValue;
+import org.apache.doris.jni.vec.ScanPredicate;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.log4j.Logger;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.catalog.Identifier;
+import org.apache.paimon.data.InternalRow;
+import org.apache.paimon.data.columnar.ColumnarRow;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.fs.Path;
+import org.apache.paimon.hive.HiveCatalog;
+import org.apache.paimon.hive.HiveCatalogOptions;
+import org.apache.paimon.hive.mapred.PaimonInputSplit;
+import org.apache.paimon.options.CatalogOptions;
+import org.apache.paimon.options.ConfigOption;
+import org.apache.paimon.options.ConfigOptions;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.reader.RecordReader;
+import org.apache.paimon.table.Table;
+import org.apache.paimon.table.source.ReadBuilder;
+import org.apache.paimon.table.source.TableRead;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
+
+
+public class PaimonJniScanner extends JniScanner {
+ private static final Logger LOG = Logger.getLogger(PaimonJniScanner.class);
+
+ private final String metastoreUris;
+ private final String warehouse;
+ private final String dbName;
+ private final String tblName;
+ private final String[] ids;
+ private final long splitAddress;
+ private final int lengthByte;
+ private PaimonInputSplit paimonInputSplit;
+ private Table table;
+ private RecordReader<InternalRow> reader;
+ private final PaimonColumnValue columnValue = new PaimonColumnValue();
+
+ public PaimonJniScanner(int batchSize, Map<String, String> params) {
+ metastoreUris = params.get("hive.metastore.uris");
+ warehouse = params.get("warehouse");
+ splitAddress = Long.parseLong(params.get("split_byte"));
+ lengthByte = Integer.parseInt(params.get("length_byte"));
+ LOG.info("splitAddress:" + splitAddress);
+ LOG.info("lengthByte:" + lengthByte);
+ dbName = params.get("db_name");
+ tblName = params.get("table_name");
+ String[] requiredFields = params.get("required_fields").split(",");
+ String[] types = params.get("columns_types").split(",");
+ ids = params.get("columns_id").split(",");
+ ColumnType[] columnTypes = new ColumnType[types.length];
+ for (int i = 0; i < types.length; i++) {
+ columnTypes[i] = ColumnType.parseType(requiredFields[i], types[i]);
+ }
+ ScanPredicate[] predicates = new ScanPredicate[0];
+ if (params.containsKey("push_down_predicates")) {
+ long predicatesAddress = Long.parseLong(params.get("push_down_predicates"));
+ if (predicatesAddress != 0) {
+ predicates = ScanPredicate.parseScanPredicates(predicatesAddress, columnTypes);
+ LOG.info("MockJniScanner gets pushed-down predicates: " + ScanPredicate.dump(predicates));
+ }
+ }
+ initTableInfo(columnTypes, requiredFields, predicates, batchSize);
+ }
+
+ @Override
+ public void open() throws IOException {
+ getCatalog();
+ // 拿 []byte 反序列化成 split
+ byte[] splitByte = new byte[lengthByte];
+ OffHeap.copyMemory(null, splitAddress, splitByte, OffHeap.BYTE_ARRAY_OFFSET, lengthByte);
+ LOG.info("splitBytes:" + Arrays.toString(splitByte));
Review Comment:
```suggestion
LOG.debug("splitBytes: {}", Arrays.toString(splitByte));
```
And if `Arrays.toString(splitByte)` is expensive, better if `if LOG.isDebugEnable()`
##########
fe/java-udf/src/main/java/org/apache/doris/jni/PaimonJniScanner.java:
##########
@@ -0,0 +1,187 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.jni;
+
+import org.apache.doris.jni.utils.OffHeap;
+import org.apache.doris.jni.vec.ColumnType;
+import org.apache.doris.jni.vec.PaimonColumnValue;
+import org.apache.doris.jni.vec.ScanPredicate;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.log4j.Logger;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.catalog.Identifier;
+import org.apache.paimon.data.InternalRow;
+import org.apache.paimon.data.columnar.ColumnarRow;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.fs.Path;
+import org.apache.paimon.hive.HiveCatalog;
+import org.apache.paimon.hive.HiveCatalogOptions;
+import org.apache.paimon.hive.mapred.PaimonInputSplit;
+import org.apache.paimon.options.CatalogOptions;
+import org.apache.paimon.options.ConfigOption;
+import org.apache.paimon.options.ConfigOptions;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.reader.RecordReader;
+import org.apache.paimon.table.Table;
+import org.apache.paimon.table.source.ReadBuilder;
+import org.apache.paimon.table.source.TableRead;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
+
+
+public class PaimonJniScanner extends JniScanner {
+ private static final Logger LOG = Logger.getLogger(PaimonJniScanner.class);
+
+ private final String metastoreUris;
+ private final String warehouse;
+ private final String dbName;
+ private final String tblName;
+ private final String[] ids;
+ private final long splitAddress;
+ private final int lengthByte;
+ private PaimonInputSplit paimonInputSplit;
+ private Table table;
+ private RecordReader<InternalRow> reader;
+ private final PaimonColumnValue columnValue = new PaimonColumnValue();
+
+ public PaimonJniScanner(int batchSize, Map<String, String> params) {
+ metastoreUris = params.get("hive.metastore.uris");
+ warehouse = params.get("warehouse");
+ splitAddress = Long.parseLong(params.get("split_byte"));
+ lengthByte = Integer.parseInt(params.get("length_byte"));
+ LOG.info("splitAddress:" + splitAddress);
+ LOG.info("lengthByte:" + lengthByte);
+ dbName = params.get("db_name");
+ tblName = params.get("table_name");
+ String[] requiredFields = params.get("required_fields").split(",");
+ String[] types = params.get("columns_types").split(",");
+ ids = params.get("columns_id").split(",");
+ ColumnType[] columnTypes = new ColumnType[types.length];
+ for (int i = 0; i < types.length; i++) {
+ columnTypes[i] = ColumnType.parseType(requiredFields[i], types[i]);
+ }
+ ScanPredicate[] predicates = new ScanPredicate[0];
+ if (params.containsKey("push_down_predicates")) {
+ long predicatesAddress = Long.parseLong(params.get("push_down_predicates"));
+ if (predicatesAddress != 0) {
+ predicates = ScanPredicate.parseScanPredicates(predicatesAddress, columnTypes);
+ LOG.info("MockJniScanner gets pushed-down predicates: " + ScanPredicate.dump(predicates));
+ }
+ }
+ initTableInfo(columnTypes, requiredFields, predicates, batchSize);
+ }
+
+ @Override
+ public void open() throws IOException {
+ getCatalog();
+ // 拿 []byte 反序列化成 split
+ byte[] splitByte = new byte[lengthByte];
+ OffHeap.copyMemory(null, splitAddress, splitByte, OffHeap.BYTE_ARRAY_OFFSET, lengthByte);
+ LOG.info("splitBytes:" + Arrays.toString(splitByte));
+ ByteArrayInputStream bais = new ByteArrayInputStream(splitByte);
+ DataInputStream input = new DataInputStream(bais);
+ LOG.info("input:" + input);
Review Comment:
ditto
##########
fe/java-udf/src/main/java/org/apache/doris/jni/PaimonJniScanner.java:
##########
@@ -0,0 +1,187 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.jni;
+
+import org.apache.doris.jni.utils.OffHeap;
+import org.apache.doris.jni.vec.ColumnType;
+import org.apache.doris.jni.vec.PaimonColumnValue;
+import org.apache.doris.jni.vec.ScanPredicate;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.log4j.Logger;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.catalog.Identifier;
+import org.apache.paimon.data.InternalRow;
+import org.apache.paimon.data.columnar.ColumnarRow;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.fs.Path;
+import org.apache.paimon.hive.HiveCatalog;
+import org.apache.paimon.hive.HiveCatalogOptions;
+import org.apache.paimon.hive.mapred.PaimonInputSplit;
+import org.apache.paimon.options.CatalogOptions;
+import org.apache.paimon.options.ConfigOption;
+import org.apache.paimon.options.ConfigOptions;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.reader.RecordReader;
+import org.apache.paimon.table.Table;
+import org.apache.paimon.table.source.ReadBuilder;
+import org.apache.paimon.table.source.TableRead;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
+
+
+public class PaimonJniScanner extends JniScanner {
+ private static final Logger LOG = Logger.getLogger(PaimonJniScanner.class);
+
+ private final String metastoreUris;
+ private final String warehouse;
+ private final String dbName;
+ private final String tblName;
+ private final String[] ids;
+ private final long splitAddress;
+ private final int lengthByte;
+ private PaimonInputSplit paimonInputSplit;
+ private Table table;
+ private RecordReader<InternalRow> reader;
+ private final PaimonColumnValue columnValue = new PaimonColumnValue();
+
+ public PaimonJniScanner(int batchSize, Map<String, String> params) {
+ metastoreUris = params.get("hive.metastore.uris");
+ warehouse = params.get("warehouse");
+ splitAddress = Long.parseLong(params.get("split_byte"));
+ lengthByte = Integer.parseInt(params.get("length_byte"));
+ LOG.info("splitAddress:" + splitAddress);
+ LOG.info("lengthByte:" + lengthByte);
+ dbName = params.get("db_name");
+ tblName = params.get("table_name");
+ String[] requiredFields = params.get("required_fields").split(",");
+ String[] types = params.get("columns_types").split(",");
+ ids = params.get("columns_id").split(",");
+ ColumnType[] columnTypes = new ColumnType[types.length];
+ for (int i = 0; i < types.length; i++) {
+ columnTypes[i] = ColumnType.parseType(requiredFields[i], types[i]);
+ }
+ ScanPredicate[] predicates = new ScanPredicate[0];
+ if (params.containsKey("push_down_predicates")) {
+ long predicatesAddress = Long.parseLong(params.get("push_down_predicates"));
+ if (predicatesAddress != 0) {
+ predicates = ScanPredicate.parseScanPredicates(predicatesAddress, columnTypes);
+ LOG.info("MockJniScanner gets pushed-down predicates: " + ScanPredicate.dump(predicates));
+ }
+ }
+ initTableInfo(columnTypes, requiredFields, predicates, batchSize);
+ }
+
+ @Override
+ public void open() throws IOException {
+ getCatalog();
+ // 拿 []byte 反序列化成 split
+ byte[] splitByte = new byte[lengthByte];
+ OffHeap.copyMemory(null, splitAddress, splitByte, OffHeap.BYTE_ARRAY_OFFSET, lengthByte);
+ LOG.info("splitBytes:" + Arrays.toString(splitByte));
+ ByteArrayInputStream bais = new ByteArrayInputStream(splitByte);
+ DataInputStream input = new DataInputStream(bais);
+ LOG.info("input:" + input);
+ try {
+ paimonInputSplit.readFields(input);
+ } catch (IOException e) {
+ e.printStackTrace();
Review Comment:
throw it
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonHMSExternalCatalog.java:
##########
@@ -0,0 +1,168 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.paimon;
+
+import org.apache.doris.datasource.CatalogProperty;
+import org.apache.doris.datasource.property.PropertyConverter;
+import org.apache.doris.datasource.property.constants.HMSProperties;
+import org.apache.doris.datasource.property.constants.PaimonProperties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.fs.Path;
+import org.apache.paimon.hive.HiveCatalog;
+import org.apache.paimon.hive.HiveCatalogOptions;
+import org.apache.paimon.options.CatalogOptions;
+import org.apache.paimon.options.ConfigOption;
+import org.apache.paimon.options.ConfigOptions;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.utils.StringUtils;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URL;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+
+
+
+public class PaimonHMSExternalCatalog extends PaimonExternalCatalog {
+
+ public static final String METASTORE = "metastore";
+ public static final String METASTORE_HIVE = "hive";
+ public static final String URI = "uri";
+ private static final ConfigOption<String> METASTORE_CLIENT_CLASS =
+ ConfigOptions.key("metastore.client.class")
+ .stringType()
+ .defaultValue("org.apache.hadoop.hive.metastore.HiveMetaStoreClient")
+ .withDescription(
+ "Class name of Hive metastore client.\n"
+ + "NOTE: This class must directly implements "
+ + "org.apache.hadoop.hive.metastore.IMetaStoreClient.");
+
+ public PaimonHMSExternalCatalog(long catalogId, String name, String resource, Map<String, String> props) {
+ super(catalogId, name);
+ props = PropertyConverter.convertToMetaProperties(props);
+ catalogProperty = new CatalogProperty(resource, props);
+ }
+
+ @Override
+ protected void initLocalObjectsImpl() {
+ paimonCatalogType = PAIMON_HMS;
+ String metastoreUris = catalogProperty.getOrDefault(HMSProperties.HIVE_METASTORE_URIS, "");
+ String warehouse = catalogProperty.getOrDefault(PaimonProperties.WAREHOUSE, "");
+ Options options = new Options();
+ options.set(PaimonProperties.WAREHOUSE, warehouse);
+ // Currently, only supports hive
+ options.set(METASTORE, METASTORE_HIVE);
+ options.set(URI, metastoreUris);
+ CatalogContext context = CatalogContext.create(options, getConfiguration());
+ try {
+ catalog = create(context);
+ } catch (IOException e) {
+ e.printStackTrace();
Review Comment:
use `LOG` to print it. And throw exception
##########
fe/java-udf/src/main/java/org/apache/doris/jni/PaimonJniScanner.java:
##########
@@ -0,0 +1,187 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.jni;
+
+import org.apache.doris.jni.utils.OffHeap;
+import org.apache.doris.jni.vec.ColumnType;
+import org.apache.doris.jni.vec.PaimonColumnValue;
+import org.apache.doris.jni.vec.ScanPredicate;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.log4j.Logger;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.catalog.Identifier;
+import org.apache.paimon.data.InternalRow;
+import org.apache.paimon.data.columnar.ColumnarRow;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.fs.Path;
+import org.apache.paimon.hive.HiveCatalog;
+import org.apache.paimon.hive.HiveCatalogOptions;
+import org.apache.paimon.hive.mapred.PaimonInputSplit;
+import org.apache.paimon.options.CatalogOptions;
+import org.apache.paimon.options.ConfigOption;
+import org.apache.paimon.options.ConfigOptions;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.reader.RecordReader;
+import org.apache.paimon.table.Table;
+import org.apache.paimon.table.source.ReadBuilder;
+import org.apache.paimon.table.source.TableRead;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
+
+
+public class PaimonJniScanner extends JniScanner {
+ private static final Logger LOG = Logger.getLogger(PaimonJniScanner.class);
+
+ private final String metastoreUris;
+ private final String warehouse;
+ private final String dbName;
+ private final String tblName;
+ private final String[] ids;
+ private final long splitAddress;
+ private final int lengthByte;
+ private PaimonInputSplit paimonInputSplit;
+ private Table table;
+ private RecordReader<InternalRow> reader;
+ private final PaimonColumnValue columnValue = new PaimonColumnValue();
+
+ public PaimonJniScanner(int batchSize, Map<String, String> params) {
+ metastoreUris = params.get("hive.metastore.uris");
+ warehouse = params.get("warehouse");
+ splitAddress = Long.parseLong(params.get("split_byte"));
+ lengthByte = Integer.parseInt(params.get("length_byte"));
+ LOG.info("splitAddress:" + splitAddress);
+ LOG.info("lengthByte:" + lengthByte);
+ dbName = params.get("db_name");
+ tblName = params.get("table_name");
+ String[] requiredFields = params.get("required_fields").split(",");
+ String[] types = params.get("columns_types").split(",");
+ ids = params.get("columns_id").split(",");
+ ColumnType[] columnTypes = new ColumnType[types.length];
+ for (int i = 0; i < types.length; i++) {
+ columnTypes[i] = ColumnType.parseType(requiredFields[i], types[i]);
+ }
+ ScanPredicate[] predicates = new ScanPredicate[0];
+ if (params.containsKey("push_down_predicates")) {
+ long predicatesAddress = Long.parseLong(params.get("push_down_predicates"));
+ if (predicatesAddress != 0) {
+ predicates = ScanPredicate.parseScanPredicates(predicatesAddress, columnTypes);
+ LOG.info("MockJniScanner gets pushed-down predicates: " + ScanPredicate.dump(predicates));
+ }
+ }
+ initTableInfo(columnTypes, requiredFields, predicates, batchSize);
+ }
+
+ @Override
+ public void open() throws IOException {
+ getCatalog();
+ // 拿 []byte 反序列化成 split
+ byte[] splitByte = new byte[lengthByte];
+ OffHeap.copyMemory(null, splitAddress, splitByte, OffHeap.BYTE_ARRAY_OFFSET, lengthByte);
+ LOG.info("splitBytes:" + Arrays.toString(splitByte));
+ ByteArrayInputStream bais = new ByteArrayInputStream(splitByte);
+ DataInputStream input = new DataInputStream(bais);
+ LOG.info("input:" + input);
+ try {
+ paimonInputSplit.readFields(input);
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ ReadBuilder readBuilder = table.newReadBuilder();
+ TableRead read = readBuilder.newRead();
+ reader = read.createReader(paimonInputSplit.split());
+ }
+
+ @Override
+ public void close() throws IOException {
+ reader.close();
+ }
+
+ @Override
+ protected int getNext() throws IOException {
+ int rows = 0;
+ try {
+ RecordReader.RecordIterator batch;
+ while ((batch = reader.readBatch()) != null) {
+ Object record;
+ while ((record = batch.next()) != null) {
+ columnValue.setOffsetRow((ColumnarRow) record);
+ for (int i = 0; i < ids.length; i++) {
+ columnValue.setIdx(Integer.parseInt(ids[i]));
+ appendData(i, columnValue);
+ }
+ rows++;
+ }
+ batch.releaseBatch();
+ }
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ return rows;
+ }
+
+ public Catalog create(CatalogContext context) throws IOException {
+ Path warehousePath = new Path(context.options().get(CatalogOptions.WAREHOUSE));
+ FileIO fileIO;
+ fileIO = FileIO.get(warehousePath, context);
+ String uri = context.options().get(CatalogOptions.URI);
+ String hiveConfDir = context.options().get(HiveCatalogOptions.HIVE_CONF_DIR);
+ String hadoopConfDir = context.options().get(HiveCatalogOptions.HADOOP_CONF_DIR);
+ HiveConf hiveConf = HiveCatalog.createHiveConf(hiveConfDir, hadoopConfDir);
+
+ // always using user-set parameters overwrite hive-site.xml parameters
+ context.options().toMap().forEach(hiveConf::set);
+ hiveConf.set(HiveConf.ConfVars.METASTOREURIS.varname, uri);
+ // set the warehouse location to the hiveConf
+ hiveConf.set(HiveConf.ConfVars.METASTOREWAREHOUSE.varname, context.options().get(CatalogOptions.WAREHOUSE));
+
+ String clientClassName = context.options().get(METASTORE_CLIENT_CLASS);
+
+ return new HiveCatalog(fileIO, hiveConf, clientClassName, context.options().toMap());
+ }
+
+ public void getCatalog() {
Review Comment:
private
##########
fe/fe-core/src/main/java/org/apache/doris/planner/external/paimon/PaimonScanNode.java:
##########
@@ -0,0 +1,176 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.planner.external.paimon;
+
+import org.apache.doris.analysis.SlotDescriptor;
+import org.apache.doris.analysis.TupleDescriptor;
+import org.apache.doris.catalog.TableIf;
+import org.apache.doris.catalog.external.ExternalTable;
+import org.apache.doris.catalog.external.PaimonExternalTable;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.FeConstants;
+import org.apache.doris.common.MetaNotFoundException;
+import org.apache.doris.common.UserException;
+import org.apache.doris.common.util.S3Util;
+import org.apache.doris.datasource.property.constants.PaimonProperties;
+import org.apache.doris.planner.PlanNodeId;
+import org.apache.doris.planner.external.FileQueryScanNode;
+import org.apache.doris.planner.external.TableFormatType;
+import org.apache.doris.spi.Split;
+import org.apache.doris.statistics.StatisticalType;
+import org.apache.doris.thrift.TFileAttributes;
+import org.apache.doris.thrift.TFileFormatType;
+import org.apache.doris.thrift.TFileRangeDesc;
+import org.apache.doris.thrift.TFileType;
+import org.apache.doris.thrift.TPaimonFileDesc;
+import org.apache.doris.thrift.TTableFormatFileDesc;
+
+import avro.shaded.com.google.common.base.Preconditions;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.paimon.hive.mapred.PaimonInputSplit;
+import org.apache.paimon.table.AbstractFileStoreTable;
+import org.apache.paimon.table.source.DataSplit;
+import org.apache.paimon.table.source.ReadBuilder;
+import org.apache.paimon.types.DataField;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class PaimonScanNode extends FileQueryScanNode {
+ private static PaimonSource source = null;
+
+ public PaimonScanNode(PlanNodeId id, TupleDescriptor desc, boolean needCheckColumnPriv) {
+ super(id, desc, "PAIMON_SCAN_NODE", StatisticalType.PAIMON_SCAN_NODE, needCheckColumnPriv);
+ }
+
+ @Override
+ protected void doInitialize() throws UserException {
+ ExternalTable table = (ExternalTable) desc.getTable();
+ if (table.isView()) {
+ throw new AnalysisException(
+ String.format("Querying external view '%s.%s' is not supported", table.getDbName(), table.getName()));
+ }
+ computeColumnFilter();
+ initBackendPolicy();
+ source = new PaimonSource((PaimonExternalTable) table, desc, columnNameToRange);
+ Preconditions.checkNotNull(source);
+ initSchemaParams();
+ }
+
+ public static void setPaimonParams(TFileRangeDesc rangeDesc, PaimonSplit paimonSplit) {
+ TTableFormatFileDesc tableFormatFileDesc = new TTableFormatFileDesc();
+ tableFormatFileDesc.setTableFormatType(paimonSplit.getTableFormatType().value());
+ TPaimonFileDesc fileDesc = new TPaimonFileDesc();
+ fileDesc.setPaimonSplit(paimonSplit.getSerializableSplit());
+ fileDesc.setLengthByte(Integer.toString(paimonSplit.getSerializableSplit().length));
+ StringBuilder columnNamesBuilder = new StringBuilder();
+ StringBuilder columnTypesBuilder = new StringBuilder();
+ StringBuilder columnIdsBuilder = new StringBuilder();
+ Map<String, Integer> paimonFieldsId = new HashMap<>();
+ Map<String, String> paimonFieldsName = new HashMap<>();
+ for (DataField field : ((AbstractFileStoreTable) source.getPaimonTable()).schema().fields()) {
+ paimonFieldsId.put(field.name(), field.id());
+ paimonFieldsName.put(field.name(), field.type().toString());
+ }
+ boolean isFirst = true;
+ for (SlotDescriptor slot : source.getDesc().getSlots()) {
+ if (!isFirst) {
+ columnNamesBuilder.append(",");
Review Comment:
Could you add a comment in code, to explain these formats?
##########
fe/java-udf/src/main/java/org/apache/doris/jni/PaimonJniScanner.java:
##########
@@ -0,0 +1,187 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.jni;
+
+import org.apache.doris.jni.utils.OffHeap;
+import org.apache.doris.jni.vec.ColumnType;
+import org.apache.doris.jni.vec.PaimonColumnValue;
+import org.apache.doris.jni.vec.ScanPredicate;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.log4j.Logger;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.catalog.Identifier;
+import org.apache.paimon.data.InternalRow;
+import org.apache.paimon.data.columnar.ColumnarRow;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.fs.Path;
+import org.apache.paimon.hive.HiveCatalog;
+import org.apache.paimon.hive.HiveCatalogOptions;
+import org.apache.paimon.hive.mapred.PaimonInputSplit;
+import org.apache.paimon.options.CatalogOptions;
+import org.apache.paimon.options.ConfigOption;
+import org.apache.paimon.options.ConfigOptions;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.reader.RecordReader;
+import org.apache.paimon.table.Table;
+import org.apache.paimon.table.source.ReadBuilder;
+import org.apache.paimon.table.source.TableRead;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
+
+
+public class PaimonJniScanner extends JniScanner {
+ private static final Logger LOG = Logger.getLogger(PaimonJniScanner.class);
+
+ private final String metastoreUris;
+ private final String warehouse;
+ private final String dbName;
+ private final String tblName;
+ private final String[] ids;
+ private final long splitAddress;
+ private final int lengthByte;
+ private PaimonInputSplit paimonInputSplit;
+ private Table table;
+ private RecordReader<InternalRow> reader;
+ private final PaimonColumnValue columnValue = new PaimonColumnValue();
+
+ public PaimonJniScanner(int batchSize, Map<String, String> params) {
+ metastoreUris = params.get("hive.metastore.uris");
+ warehouse = params.get("warehouse");
+ splitAddress = Long.parseLong(params.get("split_byte"));
+ lengthByte = Integer.parseInt(params.get("length_byte"));
+ LOG.info("splitAddress:" + splitAddress);
+ LOG.info("lengthByte:" + lengthByte);
+ dbName = params.get("db_name");
+ tblName = params.get("table_name");
+ String[] requiredFields = params.get("required_fields").split(",");
+ String[] types = params.get("columns_types").split(",");
+ ids = params.get("columns_id").split(",");
+ ColumnType[] columnTypes = new ColumnType[types.length];
+ for (int i = 0; i < types.length; i++) {
+ columnTypes[i] = ColumnType.parseType(requiredFields[i], types[i]);
+ }
+ ScanPredicate[] predicates = new ScanPredicate[0];
+ if (params.containsKey("push_down_predicates")) {
+ long predicatesAddress = Long.parseLong(params.get("push_down_predicates"));
+ if (predicatesAddress != 0) {
+ predicates = ScanPredicate.parseScanPredicates(predicatesAddress, columnTypes);
+ LOG.info("MockJniScanner gets pushed-down predicates: " + ScanPredicate.dump(predicates));
+ }
+ }
+ initTableInfo(columnTypes, requiredFields, predicates, batchSize);
+ }
+
+ @Override
+ public void open() throws IOException {
+ getCatalog();
+ // 拿 []byte 反序列化成 split
+ byte[] splitByte = new byte[lengthByte];
+ OffHeap.copyMemory(null, splitAddress, splitByte, OffHeap.BYTE_ARRAY_OFFSET, lengthByte);
+ LOG.info("splitBytes:" + Arrays.toString(splitByte));
+ ByteArrayInputStream bais = new ByteArrayInputStream(splitByte);
+ DataInputStream input = new DataInputStream(bais);
+ LOG.info("input:" + input);
+ try {
+ paimonInputSplit.readFields(input);
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ ReadBuilder readBuilder = table.newReadBuilder();
+ TableRead read = readBuilder.newRead();
+ reader = read.createReader(paimonInputSplit.split());
+ }
+
+ @Override
+ public void close() throws IOException {
+ reader.close();
+ }
+
+ @Override
+ protected int getNext() throws IOException {
+ int rows = 0;
+ try {
+ RecordReader.RecordIterator batch;
+ while ((batch = reader.readBatch()) != null) {
+ Object record;
+ while ((record = batch.next()) != null) {
+ columnValue.setOffsetRow((ColumnarRow) record);
+ for (int i = 0; i < ids.length; i++) {
+ columnValue.setIdx(Integer.parseInt(ids[i]));
+ appendData(i, columnValue);
+ }
+ rows++;
+ }
+ batch.releaseBatch();
+ }
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ return rows;
+ }
+
+ public Catalog create(CatalogContext context) throws IOException {
Review Comment:
```suggestion
private Catalog create(CatalogContext context) throws IOException {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wsjz commented on a diff in pull request #19681: [Enhancement](catalog) paimon catalog
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on code in PR #19681:
URL: https://github.com/apache/doris/pull/19681#discussion_r1194737043
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/property/constants/PaimonProperties.java:
##########
@@ -0,0 +1,26 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.property.constants;
+
+public class PaimonProperties {
+ public static final String PAIMON_WAREHOUSE = "paimon.warehouse";
+ public static final String URI = "uri";
+ public static final String METASTORE = "metastore";
+ public static final String METASTORE_HIVE = "hive";
+ public static final String WAREHOUSE = "warehouse";
Review Comment:
about the PaimonProperties, I think we should not give the paimon directly, use paimon.xxx for external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wsjz commented on pull request #19681: [Enhancement](catalog) paimon catalog
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1554309539
the class: IcebergDLFExternalCatalog, IcebergExternalDatabase and IcebergExternalTable, should be register in org.apache.doris.persist.gson.GsonUtils
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1567718477
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [Feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1569793220
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [Feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1576827072
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [Feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1577786958
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1556480061
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1556479880
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [Feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1573612401
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [Feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1573617409
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [Feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1574569580
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [Feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1576824854
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wsjz commented on pull request #19681: [Enhancement](catalog) paimon catalog
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1549162183
when add new catalog, we need use gson to persist catalog, you can consider the class: fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1558932304
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] SteNicholas commented on a diff in pull request #19681: [Enhancement](catalog) paimon catalog
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #19681:
URL: https://github.com/apache/doris/pull/19681#discussion_r1198775706
##########
gensrc/thrift/PlanNodes.thrift:
##########
@@ -203,7 +204,7 @@ struct TBrokerScanRangeParams {
// If partition_ids is set, data that doesn't in this partition will be filtered.
8: optional list<i64> partition_ids
-
Review Comment:
No necessary change in this line. Revert.
##########
gensrc/thrift/PlanNodes.thrift:
##########
@@ -232,7 +233,7 @@ struct TEsScanRange {
1: required list<Types.TNetworkAddress> es_hosts // es hosts is used by be scan node to connect to es
// has to set index and type here, could not set it in scannode
// because on scan node maybe scan an es alias then it contains one or more indices
- 2: required string index
Review Comment:
No necessary change in this line. Revert.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] SteNicholas commented on a diff in pull request #19681: [Enhancement](catalog) paimon catalog
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #19681:
URL: https://github.com/apache/doris/pull/19681#discussion_r1198775971
##########
gensrc/thrift/PlanNodes.thrift:
##########
@@ -184,8 +185,8 @@ struct TBrokerScanRangeParams {
1: required i8 column_separator;
2: required i8 line_delimiter;
- // We construct one line in file to a tuple. And each field of line
- // correspond to a slot in this tuple.
+ // We construct one line in file to a tuple. And each field of line
Review Comment:
No necessary change in this line. Revert.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [Enhancement](catalog) paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1554315324
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wsjz commented on a diff in pull request #19681: [Enhancement](catalog) paimon catalog
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on code in PR #19681:
URL: https://github.com/apache/doris/pull/19681#discussion_r1198757265
##########
be/src/vec/exec/scan/vfile_scanner.cpp:
##########
@@ -641,6 +642,14 @@ Status VFileScanner::_get_next_reader() {
init_status = ((NewJsonReader*)(_cur_reader.get()))->init_reader();
break;
}
+ case TFileFormatType::FORMAT_JNI: {
+ // test
+ LOG(INFO) << "FORMAT_JNI ";
+ _cur_reader = PaimonJniReader::create_unique(_file_slot_descs, _state, _profile, range);
Review Comment:
JNI type is not only paimon, can you judge this in any other way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] SteNicholas commented on a diff in pull request #19681: [Enhancement](catalog) paimon catalog
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #19681:
URL: https://github.com/apache/doris/pull/19681#discussion_r1198776515
##########
gensrc/thrift/PlanNodes.thrift:
##########
@@ -554,7 +569,7 @@ struct TSortInfo {
4: optional list<Exprs.TExpr> sort_tuple_slot_exprs
// Indicates the nullable info of sort_tuple_slot_exprs is changed after substitute by child's smap
- 5: optional list<bool> slot_exprs_nullability_changed_flags
+ 5: optional list<bool> slot_exprs_nullability_changed_flags
Review Comment:
No necessary change in this line. Revert.
##########
gensrc/thrift/PlanNodes.thrift:
##########
@@ -593,7 +608,7 @@ struct TEqJoinCondition {
// right-hand side of "<a> = <b>"
2: required Exprs.TExpr right;
// operator of equal join
- 3: optional Opcodes.TExprOpcode opcode;
Review Comment:
No necessary change in this line. Revert.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19681: [feature](multi-catalog)support paimon catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19681:
URL: https://github.com/apache/doris/pull/19681#issuecomment-1558922649
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org