You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@doris.apache.org by "cst4049 (via GitHub)" <gi...@apache.org> on 2024/01/11 08:45:49 UTC

[I] [Bug] routine load 导入数据失败，物化视图统计字段不存在 [doris]

cst4049 opened a new issue, #29862:
URL: https://github.com/apache/doris/issues/29862

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### Version
   
   2.0.3
   
   ### What's Wrong?
   
   物化视图根据 _source_id 字段进行聚合计数
   所有的数据都有该_source_id字段， 运行一段时间后， routineload 自动停止
   报错
   errCode = 2, detailMessage = column has no source field, column=mv__source_id
   
   实际上所有数据都是有该_source_id字段的。 
   
   临时解决 只能删除物化视图后 重启routineload, 然后重建物化视图后正常
   
   十天半个月就会出现一次
   
   
   ### What You Expected?
   
   服务正常
   
   ### How to Reproduce?
   
   _No response_
   
   ### Anything Else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

Re: [I] [Bug] routine load 导入数据失败，物化视图统计字段不存在 [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] closed issue #29862: [Bug] routine load 导入数据失败，物化视图统计字段不存在
URL: https://github.com/apache/doris/issues/29862


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

Re: [I] [Bug] routine load 导入数据失败，物化视图统计字段不存在 [doris]

Posted by "cst4049 (via GitHub)" <gi...@apache.org>.

cst4049 commented on issue #29862:
URL: https://github.com/apache/doris/issues/29862#issuecomment-1888698940

   建表语句
   CREATE TABLE `原始日志` (
     `_time` datetime NOT NULL COMMENT '日志发生时间',
     `_uuid` varchar(64) NOT NULL COMMENT '原始数据ID',
     `_raw` text NOT NULL COMMENT '原始数据',
     `_host` varchar(64) NOT NULL COMMENT '数据源IP',
     `_source_name` varchar(256) NOT NULL COMMENT '数据源名称',
     `_source_id` varchar(*) NOT NULL COMMENT '数据源ID',
     `_model_id` varchar(64) NOT NULL COMMENT '数据模型ID',
     `_received_time` datetime NOT NULL COMMENT '数据入库时间'
   ) ENGINE=OLAP
   DUPLICATE KEY(`_time`)
   COMMENT 'OLAP'
   PARTITION BY RANGE(`_time`)
   (PARTITION p20231229 VALUES [('2023-12-29 00:00:00'), ('2023-12-30 00:00:00')),
   PARTITION p20231230 VALUES [('2023-12-30 00:00:00'), ('2023-12-31 00:00:00')),
   PARTITION p20231231 VALUES [('2023-12-31 00:00:00'), ('2024-01-01 00:00:00')),
   PARTITION p20240101 VALUES [('2024-01-01 00:00:00'), ('2024-01-02 00:00:00')),
   PARTITION p20240102 VALUES [('2024-01-02 00:00:00'), ('2024-01-03 00:00:00')),
   PARTITION p20240103 VALUES [('2024-01-03 00:00:00'), ('2024-01-04 00:00:00')),
   PARTITION p20240104 VALUES [('2024-01-04 00:00:00'), ('2024-01-05 00:00:00')),
   PARTITION p20240105 VALUES [('2024-01-05 00:00:00'), ('2024-01-06 00:00:00')),
   PARTITION p20240106 VALUES [('2024-01-06 00:00:00'), ('2024-01-07 00:00:00')),
   PARTITION p20240107 VALUES [('2024-01-07 00:00:00'), ('2024-01-08 00:00:00')),
   PARTITION p20240108 VALUES [('2024-01-08 00:00:00'), ('2024-01-09 00:00:00'))
   DISTRIBUTED BY HASH(`_time`) BUCKETS AUTO
   PROPERTIES (
   "replication_allocation" = "tag.location.default: 1",
   "is_being_synced" = "false",
   "dynamic_partition.enable" = "true",
   "dynamic_partition.time_unit" = "DAY",
   "dynamic_partition.time_zone" = "Etc/UTC",
   "dynamic_partition.start" = "-14",
   "dynamic_partition.end" = "1",
   "dynamic_partition.prefix" = "p",
   "dynamic_partition.replication_allocation" = "tag.location.default: 1",
   "dynamic_partition.buckets" = "10",
   "dynamic_partition.create_history_partition" = "true",
   "dynamic_partition.history_partition_num" = "14",
   "dynamic_partition.hot_partition_num" = "0",
   "dynamic_partition.reserved_history_periods" = "NULL",
   "dynamic_partition.storage_policy" = "",
   "dynamic_partition.storage_medium" = "HDD",
   "storage_format" = "V2",
   "estimate_partition_size" = "50G",
   "light_schema_change" = "true",
   "disable_auto_compaction" = "false",
   "enable_single_replica_compaction" = "false"
   ); 
   
   
   CREATE ROUTINE LOAD raw_log ON 原始日志
   WITH APPEND
   COLUMNS(_model_id='raw_log')
   PROPERTIES
   (
   "desired_concurrent_number" = "5",
   "max_error_number" = "10000000",
   "max_batch_interval" = "20",
   "max_batch_rows" = "200000",
   "max_batch_size" = "419430400",
   "format" = "json",
   "strip_outer_array" = "false",
   "num_as_string" = "false",
   "fuzzy_parse" = "false",
   "strict_mode" = "false",
   "timezone" = "Etc/UTC",
   "exec_mem_limit" = "2147483648"
   )
   FROM KAFKA
   (
   "kafka_broker_list" = "broker:9092",
   "kafka_topic" = "raw_log",
   "property.group.id" = "raw_log",
   "property.client.id" = "raw_log",
   "kafka_partitions" = "0, 1, 2, 3, 4",
   "kafka_offsets" = "9226034810, 9226237000, 9226193894, 9226101268, 9226818883"
   ); 
   
   
   持续运行一段时间后就会出现该问题


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

Re: [I] [Bug] routine load 导入数据失败，物化视图统计字段不存在 [doris]

Posted by "Liupb511 (via GitHub)" <gi...@apache.org>.

Liupb511 commented on issue #29862:
URL: https://github.com/apache/doris/issues/29862#issuecomment-3095054886

   我也遇到了这个问题，有什么进展了吗？
   
   <img width="1117" height="28" alt="Image" src="https://github.com/user-attachments/assets/dea114ff-78a7-43fb-a0cb-3f9da8299994" />


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

Re: [I] [Bug] routine load 导入数据失败，物化视图统计字段不存在 [doris]

Posted by "Liupb511 (via GitHub)" <gi...@apache.org>.

Liupb511 commented on issue #29862:
URL: https://github.com/apache/doris/issues/29862#issuecomment-3100237714

   好的，感谢


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

Re: [I] [Bug] routine load 导入数据失败，物化视图统计字段不存在 [doris]

Posted by "chenshuai1995 (via GitHub)" <gi...@apache.org>.

chenshuai1995 commented on issue #29862:
URL: https://github.com/apache/doris/issues/29862#issuecomment-1888373105

   能提供能复现的步骤吗？


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

Re: [I] [Bug] routine load 导入数据失败，物化视图统计字段不存在 [doris]

Posted by "cst4049 (via GitHub)" <gi...@apache.org>.

cst4049 commented on issue #29862:
URL: https://github.com/apache/doris/issues/29862#issuecomment-3095080326

   删除物化视图，然后重建吧。还有其他bug，很多问题。已经没用doris了
   
   
   
   ---原始邮件---
   发件人: ***@***.***&gt;
   发送时间: 2025年7月21日(周一) 上午10:29
   收件人: ***@***.***&gt;;
   抄送: ***@***.******@***.***&gt;;
   主题: Re: [apache/doris] [Bug] routine load 导入数据失败，物化视图统计字段不存在 (Issue #29862)
   
   
   Liupb511 left a comment (apache/doris#29862)
    
   我也遇到了这个问题，有什么进展了吗？
    image.png (view on web) 
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you authored the thread.Message ID: ***@***.***&gt;


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

Re: [I] [Bug] routine load 导入数据失败，物化视图统计字段不存在 [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] commented on issue #29862:
URL: https://github.com/apache/doris/issues/29862#issuecomment-3765922680

   We're closing this issue because it hasn't been updated in a while.
   If you'd like to revive this issue, please reopen it and feel free a maintainer to remove the Stale tag!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org