You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "XingChenBigData (via GitHub)" <gi...@apache.org> on 2025/06/08 03:29:39 UTC

[I] [Bug]: encountered the error "Multiple sources found for iceberg" when creating the Iceberg table. [kyuubi]

XingChenBigData opened a new issue, #7088:
URL: https://github.com/apache/kyuubi/issues/7088

   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   
   
   ### Search before asking
   
   - [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### Describe the bug
   
   What happened?
   The amoro web page, using Kyuubi as the Terminal, encountered the error "Multiple sources found for iceberg" when creating the Iceberg table.
   
   The error log is:
   Caused by: org.apache.kyuubi.jdbc.hive.KyuubiSQLException: org.apache.kyuubi.KyuubiSQLException: org.apache.kyuubi.KyuubiSQLException: Error operating ExecuteStatement: org.apache.spark.sql.AnalysisException:
   Multiple sources found for iceberg (org.apache.amoro.shade.org.apache.iceberg.spark.source.IcebergSource, org.apache.iceberg.spark.source.IcebergSource),
   please specify the fully qualified class name.
   
   The reason is probably due to Spark's extension mechanism. If Iceberg and Amoro extensions (org.apache.amoro.shade.org.apache.iceberg.spark.source.IcebergSource, org.apache.iceberg.spark.source.IcebergSource) are configured in spark.sql.extensions at the same time, there will be conflicts.
   
   | spark.sql.extensions  | org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions |
   
   ### Affects Version(s)
   
   1.10.0
   
   ### Kyuubi Server Log Output
   
   ```logtalk
   
   ```
   
   ### Kyuubi Engine Log Output
   
   ```logtalk
   
   ```
   
   ### Kyuubi Server Configurations
   
   ```yaml
   
   ```
   
   ### Kyuubi Engine Configurations
   
   ```yaml
   
   ```
   
   ### Additional context
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
   - [ ] No. I cannot submit a PR at this time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [I] [Bug]: encountered the error "Multiple sources found for iceberg" when creating the Iceberg table. [kyuubi]

Posted by "XingChenBigData (via GitHub)" <gi...@apache.org>.
XingChenBigData commented on issue #7088:
URL: https://github.com/apache/kyuubi/issues/7088#issuecomment-2953508572

   在配置文件/etc/kyuubi/conf/kyuubi-defaults.conf中,只有一部分生效的。
   
   在命令行查看:
   SET spark.sql.extensions; 生效
   SET spark.sql.catalog.kyuubi_mixed_iceberg_catalog; 没有生效
   SET spark.sql.catalog.kyuubi_mixed_iceberg_catalog.url; 没有生效
   
   在命令行设置:
   SET spark.sql.catalog.kyuubi_mixed_iceberg_catalog = org.apache.amoro.spark.MixedFormatSparkCatalog;
   SET spark.sql.catalog.kyuubi_mixed_iceberg_catalog.url = thrift://127.0.0.1:1260/kyuubi_mixed_iceberg_catalog;


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [I] [Bug]: encountered the error "Multiple sources found for iceberg" when creating the Iceberg table. [kyuubi]

Posted by "XingChenBigData (via GitHub)" <gi...@apache.org>.
XingChenBigData commented on issue #7088:
URL: https://github.com/apache/kyuubi/issues/7088#issuecomment-2953494089

   ### 编辑配置文件
   vi /etc/kyuubi/conf/kyuubi-defaults.conf
   
   ![Image](https://github.com/user-attachments/assets/ec7960b4-2e2d-4c20-8968-5363aa3e18eb)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [I] [Bug]: encountered the error "Multiple sources found for iceberg" when creating the Iceberg table. [kyuubi]

Posted by "XingChenBigData (via GitHub)" <gi...@apache.org>.
XingChenBigData commented on issue #7088:
URL: https://github.com/apache/kyuubi/issues/7088#issuecomment-2953562348

   Amoro 服务运行在 Master 节点,但通过 127.0.0.1 仅在本地可访问,导致 Worker 节点连接失败​​。
   server-expose-host: "127.0.0.1"  # 仅限本地访问
   
   ams:
     server-bind-host: "0.0.0.0"    # 保持监听所有接口 ✓
     server-expose-host: "hadoop-master1.orb.local"  # 修正为集群可达地址 ↓
     # 或使用IP: "172.18.0.4"
   
   
   # 使用域名方案
   spark.sql.catalog.mixed_catalog.url=thrift://hadoop-master1.orb.local:1260/mixed_catalog
   
   # 或使用IP方案
   spark.sql.catalog.mixed_catalog.url=thrift://172.18.0.4:1260/mixed_catalog


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [I] [Bug]: encountered the error "Multiple sources found for iceberg" when creating the Iceberg table. [kyuubi]

Posted by "XingChenBigData (via GitHub)" <gi...@apache.org>.
XingChenBigData commented on issue #7088:
URL: https://github.com/apache/kyuubi/issues/7088#issuecomment-2953565683

   ![Image](https://github.com/user-attachments/assets/43702360-5790-4f30-81f4-adf043e53ca2)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [I] [Bug]: encountered the error "Multiple sources found for iceberg" when creating the Iceberg table. [kyuubi]

Posted by "XingChenBigData (via GitHub)" <gi...@apache.org>.
XingChenBigData commented on issue #7088:
URL: https://github.com/apache/kyuubi/issues/7088#issuecomment-2953492003

   ![Image](https://github.com/user-attachments/assets/fe0f5d91-65b9-422c-9b7d-4db8273eb155)
   
   ![Image](https://github.com/user-attachments/assets/c41c4688-de86-4ca1-b3ee-016443aaabc4)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [I] [Bug]: encountered the error "Multiple sources found for iceberg" when creating the Iceberg table. [kyuubi]

Posted by "XingChenBigData (via GitHub)" <gi...@apache.org>.
XingChenBigData commented on issue #7088:
URL: https://github.com/apache/kyuubi/issues/7088#issuecomment-2953505425

   AMS_HOST=127.0.0.1
   AMS_PORT=1260
   AMS_CATALOG_NAME=kyuubi_mixed_iceberg_catalog
   ${SPARK_HOME}/bin/spark-sql \
       --conf spark.sql.extensions=org.apache.amoro.spark.MixedFormatSparkExtensions \
       --conf spark.sql.catalog.${AMS_CATALOG_NAME}=org.apache.amoro.spark.MixedFormatSparkCatalog \
       --conf spark.sql.catalog.${AMS_CATALOG_NAME}.url=thrift://${AMS_HOST}:${AMS_PORT}/${AMS_CATALOG_NAME}
    
   
   vi /etc/kyuubi/conf/kyuubi-defaults.conf
   /opt/kyuubi/bin/kyuubi restart   
   
   # 应用 Spark Extensions
   spark.sql.extensions = org.apache.amoro.spark.MixedFormatSparkExtensions
   
   # 配置 Amoro Catalog
   spark.sql.catalog.kyuubi_mixed_iceberg_catalog = org.apache.amoro.spark.MixedFormatSparkCatalog
   spark.sql.catalog.kyuubi_mixed_iceberg_catalog.url = thrift://127.0.0.1:1260/kyuubi_mixed_iceberg_catalog    
       


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [I] [Bug]: encountered the error "Multiple sources found for iceberg" when creating the Iceberg table. [kyuubi]

Posted by "XingChenBigData (via GitHub)" <gi...@apache.org>.
XingChenBigData commented on issue #7088:
URL: https://github.com/apache/kyuubi/issues/7088#issuecomment-2953560760

   telnet 127.0.0.1 1260
   telnet 172.18.0.4 1260
   telnet hadoop-master1.orb.local 1260
   hadoop-master1.orb.local


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [I] [Bug]: encountered the error "Multiple sources found for iceberg" when creating the Iceberg table. [kyuubi]

Posted by "XingChenBigData (via GitHub)" <gi...@apache.org>.
XingChenBigData commented on issue #7088:
URL: https://github.com/apache/kyuubi/issues/7088#issuecomment-2953488104

   kyuubi-beeline -u "jdbc:hive2://127.0.0.1:10009/;spark.sql.extensions=org.apache.amoro.spark.MixedFormatSparkExtensions"
   进入命令行之后,执行 SET spark.sql.extensions;
   结果为:| spark.sql.extensions  | org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [I] [Bug]: encountered the error "Multiple sources found for iceberg" when creating the Iceberg table. [kyuubi]

Posted by "XingChenBigData (via GitHub)" <gi...@apache.org>.
XingChenBigData commented on issue #7088:
URL: https://github.com/apache/kyuubi/issues/7088#issuecomment-2953513575

   create table test1 (id int, data string, ts timestamp) using mixed_iceberg;
   create table kybbui (id int, data string, ts timestamp) using mixed_iceberg;
   insert into kybbui values 
   ( 1, "aaa", timestamp('2022-1-1 00:00:00')),
   ( 2, "bbb", timestamp('2022-1-2 00:00:00')),
   ( 3, "bbb", timestamp('2022-1-3 00:00:00'));


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [I] [Bug]: encountered the error "Multiple sources found for iceberg" when creating the Iceberg table. [kyuubi]

Posted by "XingChenBigData (via GitHub)" <gi...@apache.org>.
XingChenBigData commented on issue #7088:
URL: https://github.com/apache/kyuubi/issues/7088#issuecomment-2953516775

   /opt/kyuubi/bin/kyuubi stop
   /opt/kyuubi/bin/kyuubi start \
     --conf spark.sql.extensions=org.apache.amoro.spark.MixedFormatSparkExtensions \
     --conf spark.sql.catalog.kyuubi_mixed_iceberg_catalog = org.apache.amoro.spark.MixedFormatSparkCatalog; \
     --conf spark.sql.catalog.kyuubi_mixed_iceberg_catalog.url = thrift://127.0.0.1:1260/kyuubi_mixed_iceberg_catalog
   
   
   --conf spark.sql.extensions=org.apache.amoro.spark.MixedFormatSparkExtensions
   --conf spark.sql.catalog.${AMS_CATALOG_NAME}=org.apache.amoro.spark.MixedFormatSparkCatalog
   --conf spark.sql.catalog.${AMS_CATALOG_NAME}.url=thrift://${AMS_HOST}:${AMS_PORT}/${AMS_CATALOG_NAME}


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [I] [Resolved]: By configuring spark.sql.extensions in the kyuubi configuration file, to solve the error "Multiple sources found for iceberg" when creating the Iceberg table. [kyuubi]

Posted by "XingChenBigData (via GitHub)" <gi...@apache.org>.
XingChenBigData commented on issue #7088:
URL: https://github.com/apache/kyuubi/issues/7088#issuecomment-2953740143

   By configuring spark.sql.extensions in the kyuubi configuration file, to solve the error "Multiple sources found for iceberg" when creating the Iceberg table.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [I] [Resolved]: By configuring spark.sql.extensions in the kyuubi configuration file, to solve the error "Multiple sources found for iceberg" when creating the Iceberg table. [kyuubi]

Posted by "XingChenBigData (via GitHub)" <gi...@apache.org>.
XingChenBigData closed issue #7088: [Resolved]: By configuring spark.sql.extensions in the kyuubi configuration file, to solve the error "Multiple sources found for iceberg" when creating the Iceberg table.
URL: https://github.com/apache/kyuubi/issues/7088


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [I] [Bug]: encountered the error "Multiple sources found for iceberg" when creating the Iceberg table. [kyuubi]

Posted by "XingChenBigData (via GitHub)" <gi...@apache.org>.
XingChenBigData commented on issue #7088:
URL: https://github.com/apache/kyuubi/issues/7088#issuecomment-2953555617

   🔍 根本原因剖析
   ​​本地与集群环境的网络差异​​
   ✅ ​​Spark-shell本地测试成功​​
   您通过spark-sql --conf直接运行在​​Amoro服务同一节点​​(127.0.0.1指向本机),网络直连无阻碍:
   复制
   use mixed_catalog;  // 成功切换Catalog
   show catalogs;       // 正确显示mixed_catalog
   ❌ ​​Kyuubi在YARN集群失败​​
   Kyuubi提交的任务运行在​​YARN容器内​​,容器中的127.0.0.1指向容器自身而非宿主机,导致无法访问宿主机的Amoro服务16。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [I] [Bug]: encountered the error "Multiple sources found for iceberg" when creating the Iceberg table. [kyuubi]

Posted by "XingChenBigData (via GitHub)" <gi...@apache.org>.
XingChenBigData commented on issue #7088:
URL: https://github.com/apache/kyuubi/issues/7088#issuecomment-2953556919

   查看机器有哪些 yarn 节点? 
   sh-5.0# yarn node -list -all
   WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
   2025-06-08 13:03:16,042 - INFO  [main:DefaultNoHARMFailoverProxyProvider@64] - Connecting to ResourceManager at hadoop-master1.orb.local/172.18.0.4:8032
   Total Nodes:3
            Node-Id             Node-State Node-Http-Address       Number-of-Running-Containers
   hadoop-worker3.orb.local:36881          RUNNING hadoop-worker3.orb.local:8042                              0
   hadoop-worker1.orb.local:36181          RUNNING hadoop-worker1.orb.local:8042                              0
   hadoop-worker2.orb.local:43579          RUNNING hadoop-worker2.orb.local:8042                              0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [I] [Bug]: encountered the error "Multiple sources found for iceberg" when creating the Iceberg table. [kyuubi]

Posted by "XingChenBigData (via GitHub)" <gi...@apache.org>.
XingChenBigData commented on issue #7088:
URL: https://github.com/apache/kyuubi/issues/7088#issuecomment-2953720087

   telnet hadoop-master1.orb.local 1260
   telnet 172.18.0.4 1260
   
   SET spark.sql.catalog.kyuubi_mixed_iceberg_catalog.url;
   SET spark.sql.catalog.mixed_catalog.url=thrift://hadoop-master1.orb.local:1260/mixed_catalog;
   SET spark.sql.catalog.mixed_catalog.url=thrift://hadoop-master1.orb.local:1260/kyuubi_mixed_iceberg_catalog;


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org