You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@iotdb.apache.org by "Jialin Qiao (Jira)" <ji...@apache.org> on 2022/06/02 10:40:00 UTC

[jira] [Assigned] (IOTDB-3379) Align by device: Error in calling method sendFragmentInstance

     [ https://issues.apache.org/jira/browse/IOTDB-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jialin Qiao reassigned IOTDB-3379:
----------------------------------

    Assignee: Yuan Tian

> Align by device: Error in calling method sendFragmentInstance
> -------------------------------------------------------------
>
>                 Key: IOTDB-3379
>                 URL: https://issues.apache.org/jira/browse/IOTDB-3379
>             Project: Apache IoTDB
>          Issue Type: Bug
>    Affects Versions: 0.14.0-SNAPSHOT
>            Reporter: Jialin Qiao
>            Assignee: Yuan Tian
>            Priority: Major
>              Labels: pull-request-available
>
>  
> datanode1
> {code:java}
> 2022-06-02 14:45:44,182 [pool-7-IoTDB-RPC-Client-1] INFO  o.a.i.d.m.p.e.QueryExecution:186 - Query[20220602_064544_00093_0]: start to analyze query 
> 2022-06-02 14:45:44,182 [pool-7-IoTDB-RPC-Client-1] INFO  o.a.i.d.m.p.a.Analyzer$AnalyzeVisitor:159 - Query[20220602_064544_00093_0]: fetch query schema... 
> 2022-06-02 14:45:44,184 [pool-7-IoTDB-RPC-Client-1] INFO  o.a.i.d.m.p.e.QueryExecution:186 - Query[20220602_064544_00094_0]: start to analyze query 
> 2022-06-02 14:45:44,184 [pool-7-IoTDB-RPC-Client-1] INFO  o.a.i.d.m.p.e.QueryExecution:216 - Query[20220602_064544_00094_0]: do logical plan... 
> 2022-06-02 14:45:44,184 [pool-7-IoTDB-RPC-Client-1] INFO  o.a.i.d.m.p.e.QueryExecution:219 - Query[20220602_064544_00094_0]: logical plan is: 
>  SchemaFetchNode-1
>   └──org.apache.iotdb.db.mpp.plan.planner.plan.node.metedata.read.SchemaFetchScanNode@51
>  
> 2022-06-02 14:45:44,184 [pool-7-IoTDB-RPC-Client-1] INFO  o.a.i.d.m.p.e.QueryExecution:227 - Query[20220602_064544_00094_0]: do distribution plan... 
> SchemaFetchNode-1
>   └──org.apache.iotdb.db.mpp.plan.planner.plan.node.metedata.read.SchemaFetchScanNode@52
> SchemaFetchNode-1
>   └──org.apache.iotdb.db.mpp.plan.planner.plan.node.metedata.read.SchemaFetchScanNode@52
> 2022-06-02 14:45:44,185 [pool-7-IoTDB-RPC-Client-1] INFO  o.a.i.d.m.p.e.QueryExecution:230 - Query[20220602_064544_00094_0]: distribution plan done. Fragment instance count is 1, details is: 
>  [FragmentInstance-20220602_064544_00094_0.0.0:Host: 2 Region: TConsensusGroupId(type:SchemaRegion, id:0) 
> ---- Plan Node Tree ----
> FragmentSinkNode-4:[SendTo: (127.0.0.1/20220602_064544_00094_0.0.0/0)]
>   └──SchemaFetchNode-1
>       └──org.apache.iotdb.db.mpp.plan.planner.plan.node.metedata.read.SchemaFetchScanNode@52
> ] 
> 2022-06-02 14:45:44,185 [pool-7-IoTDB-RPC-Client-1] INFO  o.a.i.d.m.p.s.ClusterScheduler:97 - Query[20220602_064544_00094_0] transit to DISPATCHING 
> 2022-06-02 14:45:44,186 [pool-7-IoTDB-RPC-Client-1] INFO  o.a.i.d.m.p.s.ClusterScheduler:127 - Query[20220602_064544_00094_0] transit to RUNNING 
> 2022-06-02 14:45:44,187 [pool-7-IoTDB-RPC-Client-1] INFO  o.a.i.d.m.p.s.ClusterScheduler:135 - Query[20220602_064544_00094_0] state tracker starts 
> 2022-06-02 14:45:44,187 [pool-6-IoTDB-MPPCoordinatorScheduled-1] INFO  o.a.i.d.m.p.s.FixedRateFragInsStateTracker:75 - Instance 20220602_064544_00094_0.0.0's State is RUNNING 
> 2022-06-02 14:45:44,188 [pool-20-IoTDB-DataBlockManagerRPC-Client-1] INFO  o.a.i.d.m.e.d.SourceHandle:207 - Query[20220602_064544_00094_0]-[0-0-SourceHandle-0]: receive newDataBlockEvent. [0, 1) 
> 2022-06-02 14:45:44,188 [pool-19-IoTDB-data-block-manager-task-executors-1] INFO  o.a.i.d.m.e.d.SourceHandle$GetDataBlocksTask:318 - Query[20220602_064544_00094_0]-[0-0-SourceHandle-0]: try to get data blocks [0, 1) 
> 2022-06-02 14:45:44,188 [pool-20-IoTDB-DataBlockManagerRPC-Client-2] INFO  o.a.i.d.m.e.d.SourceHandle:196 - Query[20220602_064544_00094_0]-[0-0-SourceHandle-0]: receive NoMoreTsBlock event. 
> 2022-06-02 14:45:44,188 [pool-19-IoTDB-data-block-manager-task-executors-1] INFO  o.a.i.d.m.e.d.SourceHandle$GetDataBlocksTask:336 - Query[20220602_064544_00094_0]-[0-0-SourceHandle-0]: got data blocks. count: 1 
> 2022-06-02 14:45:44,188 [pool-19-IoTDB-data-block-manager-task-executors-1] INFO  o.a.i.d.m.e.d.SourceHandle$SendAcknowledgeDataBlockEventTask:391 - Query[20220602_064544_00094_0]-[0-0-SourceHandle-0]: send ack data block event [0, 1). 
> 2022-06-02 14:45:44,188 [pool-7-IoTDB-RPC-Client-1] INFO  o.a.i.d.m.e.d.DataBlockManager$SourceHandleListenerImpl:188 - Query[20220602_064544_00094_0]-[0-0-SourceHandle-0] finished and release resources 
> 2022-06-02 14:45:44,189 [pool-7-IoTDB-RPC-Client-1] INFO  o.a.i.d.m.p.a.Analyzer$AnalyzeVisitor:161 - Query[20220602_064544_00093_0]: fetch schema done 
> 2022-06-02 14:45:44,190 [pool-7-IoTDB-RPC-Client-1] INFO  o.a.i.d.m.p.e.QueryExecution:216 - Query[20220602_064544_00093_0]: do logical plan... 
> 2022-06-02 14:45:44,191 [pool-7-IoTDB-RPC-Client-1] INFO  o.a.i.d.m.p.e.QueryExecution:219 - Query[20220602_064544_00093_0]: logical plan is: 
>  DeviceView-8
>   ├──AlignedSeriesScanNode-1:[SeriesPath: root.sg1.d1, DataRegion: null]
>   └──TimeJoinNode-7
>       ├──SeriesScanNode-2:[SeriesPath: root.sg1.d2.s3, DataRegion: null]
>       ├──SeriesScanNode-3:[SeriesPath: root.sg1.d2.s4, DataRegion: null]
>       ├──SeriesScanNode-4:[SeriesPath: root.sg1.d2.s5, DataRegion: null]
>       ├──SeriesScanNode-5:[SeriesPath: root.sg1.d2.s1, DataRegion: null]
>       └──SeriesScanNode-6:[SeriesPath: root.sg1.d2.s2, DataRegion: null]
>  
> 2022-06-02 14:45:44,191 [pool-7-IoTDB-RPC-Client-1] INFO  o.a.i.d.m.p.e.QueryExecution:227 - Query[20220602_064544_00093_0]: do distribution plan... 
> DeviceMerge-9
>   ├──DeviceView-16
>   │   └──TimeJoinNode-15
>   │       ├──SeriesScanNode-10:[SeriesPath: root.sg1.d2.s3, DataRegion: TRegionReplicaSet(regionId:TConsensusGroupId(type:DataRegion, id:2), dataNodeLocations:[TDataNodeLocation(dataNodeId:1, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6668), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9004), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8778), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40011), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50011)), TDataNodeLocation(dataNodeId:2, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6669), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9005), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8779), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40012), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50012)), TDataNodeLocation(dataNodeId:0, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6667), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9003), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8777), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40010), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50010))])]
>   │       ├──SeriesScanNode-11:[SeriesPath: root.sg1.d2.s4, DataRegion: TRegionReplicaSet(regionId:TConsensusGroupId(type:DataRegion, id:2), dataNodeLocations:[TDataNodeLocation(dataNodeId:1, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6668), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9004), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8778), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40011), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50011)), TDataNodeLocation(dataNodeId:2, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6669), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9005), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8779), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40012), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50012)), TDataNodeLocation(dataNodeId:0, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6667), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9003), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8777), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40010), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50010))])]
>   │       ├──SeriesScanNode-12:[SeriesPath: root.sg1.d2.s5, DataRegion: TRegionReplicaSet(regionId:TConsensusGroupId(type:DataRegion, id:2), dataNodeLocations:[TDataNodeLocation(dataNodeId:1, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6668), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9004), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8778), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40011), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50011)), TDataNodeLocation(dataNodeId:2, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6669), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9005), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8779), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40012), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50012)), TDataNodeLocation(dataNodeId:0, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6667), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9003), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8777), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40010), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50010))])]
>   │       ├──SeriesScanNode-13:[SeriesPath: root.sg1.d2.s1, DataRegion: TRegionReplicaSet(regionId:TConsensusGroupId(type:DataRegion, id:2), dataNodeLocations:[TDataNodeLocation(dataNodeId:1, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6668), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9004), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8778), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40011), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50011)), TDataNodeLocation(dataNodeId:2, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6669), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9005), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8779), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40012), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50012)), TDataNodeLocation(dataNodeId:0, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6667), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9003), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8777), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40010), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50010))])]
>   │       └──SeriesScanNode-14:[SeriesPath: root.sg1.d2.s2, DataRegion: TRegionReplicaSet(regionId:TConsensusGroupId(type:DataRegion, id:2), dataNodeLocations:[TDataNodeLocation(dataNodeId:1, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6668), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9004), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8778), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40011), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50011)), TDataNodeLocation(dataNodeId:2, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6669), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9005), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8779), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40012), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50012)), TDataNodeLocation(dataNodeId:0, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6667), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9003), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8777), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40010), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50010))])]
>   └──DeviceView-18
>       └──AlignedSeriesScanNode-17:[SeriesPath: root.sg1.d1, DataRegion: TRegionReplicaSet(regionId:TConsensusGroupId(type:DataRegion, id:1), dataNodeLocations:[TDataNodeLocation(dataNodeId:0, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6667), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9003), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8777), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40010), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50010)), TDataNodeLocation(dataNodeId:1, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6668), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9004), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8778), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40011), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50011)), TDataNodeLocation(dataNodeId:2, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6669), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9005), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8779), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40012), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50012))])]
> DeviceMerge-9
>   ├──DeviceView-16
>   │   └──TimeJoinNode-15
>   │       ├──SeriesScanNode-10:[SeriesPath: root.sg1.d2.s3, DataRegion: TRegionReplicaSet(regionId:TConsensusGroupId(type:DataRegion, id:2), dataNodeLocations:[TDataNodeLocation(dataNodeId:1, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6668), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9004), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8778), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40011), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50011)), TDataNodeLocation(dataNodeId:2, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6669), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9005), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8779), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40012), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50012)), TDataNodeLocation(dataNodeId:0, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6667), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9003), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8777), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40010), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50010))])]
>   │       ├──SeriesScanNode-11:[SeriesPath: root.sg1.d2.s4, DataRegion: TRegionReplicaSet(regionId:TConsensusGroupId(type:DataRegion, id:2), dataNodeLocations:[TDataNodeLocation(dataNodeId:1, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6668), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9004), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8778), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40011), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50011)), TDataNodeLocation(dataNodeId:2, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6669), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9005), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8779), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40012), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50012)), TDataNodeLocation(dataNodeId:0, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6667), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9003), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8777), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40010), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50010))])]
>   │       ├──SeriesScanNode-12:[SeriesPath: root.sg1.d2.s5, DataRegion: TRegionReplicaSet(regionId:TConsensusGroupId(type:DataRegion, id:2), dataNodeLocations:[TDataNodeLocation(dataNodeId:1, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6668), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9004), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8778), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40011), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50011)), TDataNodeLocation(dataNodeId:2, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6669), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9005), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8779), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40012), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50012)), TDataNodeLocation(dataNodeId:0, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6667), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9003), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8777), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40010), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50010))])]
>   │       ├──SeriesScanNode-13:[SeriesPath: root.sg1.d2.s1, DataRegion: TRegionReplicaSet(regionId:TConsensusGroupId(type:DataRegion, id:2), dataNodeLocations:[TDataNodeLocation(dataNodeId:1, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6668), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9004), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8778), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40011), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50011)), TDataNodeLocation(dataNodeId:2, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6669), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9005), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8779), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40012), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50012)), TDataNodeLocation(dataNodeId:0, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6667), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9003), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8777), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40010), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50010))])]
>   │       └──SeriesScanNode-14:[SeriesPath: root.sg1.d2.s2, DataRegion: TRegionReplicaSet(regionId:TConsensusGroupId(type:DataRegion, id:2), dataNodeLocations:[TDataNodeLocation(dataNodeId:1, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6668), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9004), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8778), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40011), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50011)), TDataNodeLocation(dataNodeId:2, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6669), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9005), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8779), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40012), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50012)), TDataNodeLocation(dataNodeId:0, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6667), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9003), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8777), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40010), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50010))])]
>   └──ExchangeNode-19: [SourceAddress:Not assigned]
>       └──DeviceView-18
>           └──AlignedSeriesScanNode-17:[SeriesPath: root.sg1.d1, DataRegion: TRegionReplicaSet(regionId:TConsensusGroupId(type:DataRegion, id:1), dataNodeLocations:[TDataNodeLocation(dataNodeId:0, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6667), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9003), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8777), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40010), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50010)), TDataNodeLocation(dataNodeId:1, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6668), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9004), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8778), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40011), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50011)), TDataNodeLocation(dataNodeId:2, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6669), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9005), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8779), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40012), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50012))])]
> 2022-06-02 14:45:44,192 [pool-7-IoTDB-RPC-Client-1] INFO  o.a.i.d.m.p.e.QueryExecution:230 - Query[20220602_064544_00093_0]: distribution plan done. Fragment instance count is 2, details is: 
>  [FragmentInstance-20220602_064544_00093_0.0.0:Host: 1 Region: TConsensusGroupId(type:DataRegion, id:2) 
> ---- Plan Node Tree ----
> FragmentSinkNode-21:[SendTo: (127.0.0.1/20220602_064544_00093_0.0.0/0)]
>   └──DeviceMerge-9
>       ├──DeviceView-16
>       │   └──TimeJoinNode-15
>       │       ├──SeriesScanNode-10:[SeriesPath: root.sg1.d2.s3, DataRegion: TRegionReplicaSet(regionId:TConsensusGroupId(type:DataRegion, id:2), dataNodeLocations:[TDataNodeLocation(dataNodeId:1, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6668), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9004), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8778), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40011), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50011)), TDataNodeLocation(dataNodeId:2, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6669), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9005), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8779), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40012), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50012)), TDataNodeLocation(dataNodeId:0, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6667), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9003), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8777), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40010), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50010))])]
>       │       ├──SeriesScanNode-11:[SeriesPath: root.sg1.d2.s4, DataRegion: TRegionReplicaSet(regionId:TConsensusGroupId(type:DataRegion, id:2), dataNodeLocations:[TDataNodeLocation(dataNodeId:1, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6668), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9004), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8778), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40011), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50011)), TDataNodeLocation(dataNodeId:2, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6669), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9005), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8779), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40012), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50012)), TDataNodeLocation(dataNodeId:0, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6667), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9003), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8777), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40010), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50010))])]
>       │       ├──SeriesScanNode-12:[SeriesPath: root.sg1.d2.s5, DataRegion: TRegionReplicaSet(regionId:TConsensusGroupId(type:DataRegion, id:2), dataNodeLocations:[TDataNodeLocation(dataNodeId:1, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6668), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9004), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8778), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40011), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50011)), TDataNodeLocation(dataNodeId:2, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6669), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9005), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8779), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40012), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50012)), TDataNodeLocation(dataNodeId:0, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6667), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9003), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8777), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40010), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50010))])]
>       │       ├──SeriesScanNode-13:[SeriesPath: root.sg1.d2.s1, DataRegion: TRegionReplicaSet(regionId:TConsensusGroupId(type:DataRegion, id:2), dataNodeLocations:[TDataNodeLocation(dataNodeId:1, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6668), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9004), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8778), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40011), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50011)), TDataNodeLocation(dataNodeId:2, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6669), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9005), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8779), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40012), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50012)), TDataNodeLocation(dataNodeId:0, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6667), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9003), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8777), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40010), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50010))])]
>       │       └──SeriesScanNode-14:[SeriesPath: root.sg1.d2.s2, DataRegion: TRegionReplicaSet(regionId:TConsensusGroupId(type:DataRegion, id:2), dataNodeLocations:[TDataNodeLocation(dataNodeId:1, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6668), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9004), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8778), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40011), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50011)), TDataNodeLocation(dataNodeId:2, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6669), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9005), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8779), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40012), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50012)), TDataNodeLocation(dataNodeId:0, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6667), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9003), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8777), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40010), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50010))])]
>       └──ExchangeNode-19: [SourceAddress:127.0.0.1/20220602_064544_00093_0.1.0/20]
> , FragmentInstance-20220602_064544_00093_0.1.0:Host: 0 Region: TConsensusGroupId(type:DataRegion, id:1) 
> ---- Plan Node Tree ----
> FragmentSinkNode-20:[SendTo: (127.0.0.1/20220602_064544_00093_0.0.0/19)]
>   └──DeviceView-18
>       └──AlignedSeriesScanNode-17:[SeriesPath: root.sg1.d1, DataRegion: TRegionReplicaSet(regionId:TConsensusGroupId(type:DataRegion, id:1), dataNodeLocations:[TDataNodeLocation(dataNodeId:0, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6667), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9003), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8777), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40010), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50010)), TDataNodeLocation(dataNodeId:1, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6668), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9004), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8778), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40011), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50011)), TDataNodeLocation(dataNodeId:2, externalEndPoint:TEndPoint(ip:127.0.0.1, port:6669), internalEndPoint:TEndPoint(ip:127.0.0.1, port:9005), dataBlockManagerEndPoint:TEndPoint(ip:127.0.0.1, port:8779), dataRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:40012), schemaRegionConsensusEndPoint:TEndPoint(ip:127.0.0.1, port:50012))])]
> ] 
> 2022-06-02 14:45:44,193 [pool-7-IoTDB-RPC-Client-1] INFO  o.a.i.d.m.p.s.ClusterScheduler:97 - Query[20220602_064544_00093_0] transit to DISPATCHING 
> 2022-06-02 14:45:44,195 [pool-4-IoTDB-MPPCoordinator-5] ERROR o.a.i.c.c.s.SyncThriftClientWithErrorHandler:65 - level-0 Exception class org.apache.thrift.TApplicationException, message Internal error processing sendFragmentInstance 
> 2022-06-02 14:45:44,195 [pool-4-IoTDB-MPPCoordinator-5] ERROR o.a.i.c.c.s.SyncThriftClientWithErrorHandler:80 - root cause message Internal error processing sendFragmentInstance, LocalizedMessage Internal error processing sendFragmentInstance, 
> org.apache.thrift.TApplicationException: Internal error processing sendFragmentInstance
> 	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e.CGLIB$receiveBase$60(<generated>)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e$$FastClassByCGLIB$$f5ef5929.invoke(<generated>)
> 	at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
> 	at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e.receiveBase(<generated>)
> 	at org.apache.iotdb.mpp.rpc.thrift.InternalService$Client.recv_sendFragmentInstance(InternalService.java:205)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e.CGLIB$recv_sendFragmentInstance$10(<generated>)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e$$FastClassByCGLIB$$f5ef5929.invoke(<generated>)
> 	at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
> 	at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e.recv_sendFragmentInstance(<generated>)
> 	at org.apache.iotdb.mpp.rpc.thrift.InternalService$Client.sendFragmentInstance(InternalService.java:192)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e.CGLIB$sendFragmentInstance$40(<generated>)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e$$FastClassByCGLIB$$f5ef5929.invoke(<generated>)
> 	at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
> 	at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e.sendFragmentInstance(<generated>)
> 	at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchRemote(FragmentInstanceDispatcherImpl.java:160)
> 	at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchOneInstance(FragmentInstanceDispatcherImpl.java:141)
> 	at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.lambda$dispatchRead$0(FragmentInstanceDispatcherImpl.java:100)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 	at java.lang.Thread.run(Thread.java:750)
> 2022-06-02 14:45:44,195 [pool-4-IoTDB-MPPCoordinator-5] ERROR o.a.i.c.c.s.SyncThriftClientWithErrorHandler:65 - level-0 Exception class org.apache.thrift.TException, message Error in calling method receiveBase 
> 2022-06-02 14:45:44,195 [pool-4-IoTDB-MPPCoordinator-5] ERROR o.a.i.c.c.s.SyncThriftClientWithErrorHandler:65 - level-1 Exception class org.apache.thrift.TApplicationException, message Internal error processing sendFragmentInstance 
> 2022-06-02 14:45:44,195 [pool-4-IoTDB-MPPCoordinator-5] ERROR o.a.i.c.c.s.SyncThriftClientWithErrorHandler:80 - root cause message Internal error processing sendFragmentInstance, LocalizedMessage Internal error processing sendFragmentInstance, 
> org.apache.thrift.TApplicationException: Internal error processing sendFragmentInstance
> 	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e.CGLIB$receiveBase$60(<generated>)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e$$FastClassByCGLIB$$f5ef5929.invoke(<generated>)
> 	at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
> 	at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e.receiveBase(<generated>)
> 	at org.apache.iotdb.mpp.rpc.thrift.InternalService$Client.recv_sendFragmentInstance(InternalService.java:205)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e.CGLIB$recv_sendFragmentInstance$10(<generated>)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e$$FastClassByCGLIB$$f5ef5929.invoke(<generated>)
> 	at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
> 	at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e.recv_sendFragmentInstance(<generated>)
> 	at org.apache.iotdb.mpp.rpc.thrift.InternalService$Client.sendFragmentInstance(InternalService.java:192)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e.CGLIB$sendFragmentInstance$40(<generated>)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e$$FastClassByCGLIB$$f5ef5929.invoke(<generated>)
> 	at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
> 	at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e.sendFragmentInstance(<generated>)
> 	at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchRemote(FragmentInstanceDispatcherImpl.java:160)
> 	at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchOneInstance(FragmentInstanceDispatcherImpl.java:141)
> 	at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.lambda$dispatchRead$0(FragmentInstanceDispatcherImpl.java:100)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 	at java.lang.Thread.run(Thread.java:750)
> 2022-06-02 14:45:44,196 [pool-4-IoTDB-MPPCoordinator-5] ERROR o.a.i.c.c.s.SyncThriftClientWithErrorHandler:65 - level-0 Exception class org.apache.thrift.TException, message Error in calling method recv_sendFragmentInstance 
> 2022-06-02 14:45:44,196 [pool-4-IoTDB-MPPCoordinator-5] ERROR o.a.i.c.c.s.SyncThriftClientWithErrorHandler:65 - level-1 Exception class org.apache.thrift.TException, message Error in calling method receiveBase 
> 2022-06-02 14:45:44,196 [pool-4-IoTDB-MPPCoordinator-5] ERROR o.a.i.c.c.s.SyncThriftClientWithErrorHandler:65 - level-2 Exception class org.apache.thrift.TApplicationException, message Internal error processing sendFragmentInstance 
> 2022-06-02 14:45:44,196 [pool-4-IoTDB-MPPCoordinator-5] ERROR o.a.i.c.c.s.SyncThriftClientWithErrorHandler:80 - root cause message Internal error processing sendFragmentInstance, LocalizedMessage Internal error processing sendFragmentInstance, 
> org.apache.thrift.TApplicationException: Internal error processing sendFragmentInstance
> 	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e.CGLIB$receiveBase$60(<generated>)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e$$FastClassByCGLIB$$f5ef5929.invoke(<generated>)
> 	at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
> 	at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e.receiveBase(<generated>)
> 	at org.apache.iotdb.mpp.rpc.thrift.InternalService$Client.recv_sendFragmentInstance(InternalService.java:205)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e.CGLIB$recv_sendFragmentInstance$10(<generated>)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e$$FastClassByCGLIB$$f5ef5929.invoke(<generated>)
> 	at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
> 	at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e.recv_sendFragmentInstance(<generated>)
> 	at org.apache.iotdb.mpp.rpc.thrift.InternalService$Client.sendFragmentInstance(InternalService.java:192)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e.CGLIB$sendFragmentInstance$40(<generated>)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e$$FastClassByCGLIB$$f5ef5929.invoke(<generated>)
> 	at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
> 	at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e.sendFragmentInstance(<generated>)
> 	at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchRemote(FragmentInstanceDispatcherImpl.java:160)
> 	at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchOneInstance(FragmentInstanceDispatcherImpl.java:141)
> 	at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.lambda$dispatchRead$0(FragmentInstanceDispatcherImpl.java:100)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 	at java.lang.Thread.run(Thread.java:750)
> 2022-06-02 14:45:44,196 [pool-4-IoTDB-MPPCoordinator-5] ERROR o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:163 - can't connect to node TEndPoint(ip:127.0.0.1, port:9004) 
> org.apache.thrift.TException: Error in calling method sendFragmentInstance
> 	at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:94)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e.sendFragmentInstance(<generated>)
> 	at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchRemote(FragmentInstanceDispatcherImpl.java:160)
> 	at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchOneInstance(FragmentInstanceDispatcherImpl.java:141)
> 	at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.lambda$dispatchRead$0(FragmentInstanceDispatcherImpl.java:100)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 	at java.lang.Thread.run(Thread.java:750)
> Caused by: org.apache.thrift.TException: Error in calling method recv_sendFragmentInstance
> 	at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:94)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e.recv_sendFragmentInstance(<generated>)
> 	at org.apache.iotdb.mpp.rpc.thrift.InternalService$Client.sendFragmentInstance(InternalService.java:192)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e.CGLIB$sendFragmentInstance$40(<generated>)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e$$FastClassByCGLIB$$f5ef5929.invoke(<generated>)
> 	at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
> 	at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
> 	... 8 common frames omitted
> Caused by: org.apache.thrift.TException: Error in calling method receiveBase
> 	at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:94)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e.receiveBase(<generated>)
> 	at org.apache.iotdb.mpp.rpc.thrift.InternalService$Client.recv_sendFragmentInstance(InternalService.java:205)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e.CGLIB$recv_sendFragmentInstance$10(<generated>)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e$$FastClassByCGLIB$$f5ef5929.invoke(<generated>)
> 	at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
> 	at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
> 	... 14 common frames omitted
> Caused by: org.apache.thrift.TApplicationException: Internal error processing sendFragmentInstance
> 	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e.CGLIB$receiveBase$60(<generated>)
> 	at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$35b0e10e$$FastClassByCGLIB$$f5ef5929.invoke(<generated>)
> 	at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
> 	at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
> 	... 20 common frames omitted
> 2022-06-02 14:45:44,197 [pool-7-IoTDB-RPC-Client-1] WARN  o.a.i.d.u.ErrorHandlingUtils:62 - Status code: INTERNAL_SERVER_ERROR(500), operation: "select * from root.sg1.** where time >= 9 and time <= 33 align by device". executeStatement failed 
> java.lang.RuntimeException: error code: TSStatus(code:411, message:org.apache.iotdb.db.exception.mpp.FragmentInstanceDispatchException: org.apache.thrift.TException: Error in calling method sendFragmentInstance)
> 	at org.apache.iotdb.db.service.thrift.impl.DataNodeTSIServiceImpl.executeStatement(DataNodeTSIServiceImpl.java:561)
> 	at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$executeStatement.getResult(TSIService.java:2853)
> 	at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$executeStatement.getResult(TSIService.java:2833)
> 	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> 	at org.apache.iotdb.db.service.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:64)
> 	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 	at java.lang.Thread.run(Thread.java:750)
> 2022-06-02 14:45:44,691 [pool-6-IoTDB-MPPCoordinatorScheduled-1] INFO  o.a.i.d.m.p.s.FixedRateFragInsStateTracker:75 - Instance 20220602_064544_00094_0.0.0's State is FINISHED 
> 2022-06-02 14:45:45,201 [pool-4-IoTDB-MPPCoordinator-10] INFO  o.a.i.d.m.e.d.DataBlockManager$SourceHandleListenerImpl:207 - Query[20220602_064544_00093_0]-[0-0-SourceHandle-0]: onAborted is invoked 
> 2022-06-02 14:45:45,202 [pool-4-IoTDB-MPPCoordinator-10] INFO  o.a.i.d.m.e.d.DataBlockManager$SourceHandleListenerImpl:188 - Query[20220602_064544_00093_0]-[0-0-SourceHandle-0] finished and release resources  {code}
> datanode2
> {code:java}
> 2022-06-02 14:45:44,194 [pool-1-IoTDB-InternalServiceRPC-Client-6] INFO  o.a.i.d.s.t.i.InternalServiceImpl:117 - receive FragmentInstance to group[TConsensusGroupId(type:DataRegion, id:2)] 
> 2022-06-02 14:45:44,194 [pool-1-IoTDB-InternalServiceRPC-Client-6] ERROR o.a.i.d.c.s.DataRegionStateMachine:146 - Invalid node type: 35 
> 2022-06-02 14:45:44,194 [pool-1-IoTDB-InternalServiceRPC-Client-6] ERROR o.a.t.ProcessFunction:47 - Internal error processing sendFragmentInstance 
> java.lang.NullPointerException: null
> 	at org.apache.iotdb.db.service.thrift.impl.InternalServiceImpl.sendFragmentInstance(InternalServiceImpl.java:141)
> 	at org.apache.iotdb.mpp.rpc.thrift.InternalService$Processor$sendFragmentInstance.getResult(InternalService.java:1195)
> 	at org.apache.iotdb.mpp.rpc.thrift.InternalService$Processor$sendFragmentInstance.getResult(InternalService.java:1175)
> 	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> 	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
> 	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 	at java.lang.Thread.run(Thread.java:750) {code}
> Confignode-2
> {code:java}
> 2022-06-02 14:42:11,677 [pool-2-IoTDB-ConfigNodeRPC-Client-1] ERROR o.a.t.s.TThreadPoolServer$WorkerProcess:258 - Thrift Error occurred during processing of message. 
> org.apache.thrift.protocol.TProtocolException: Required field 'userInfo' was not present! Struct: TPermissionInfoResp(userInfo:null, roleInfo:null, status:TSStatus(code:707, message:The current ConfigNode is not leader. And ConfigNodeGroup is in leader election. Please redirect with a random ConfigNode.))
> 	at org.apache.iotdb.confignode.rpc.thrift.TPermissionInfoResp.validate(TPermissionInfoResp.java:458)
> 	at org.apache.iotdb.confignode.rpc.thrift.ConfigIService$login_result.validate(ConfigIService.java:18415)
> 	at org.apache.iotdb.confignode.rpc.thrift.ConfigIService$login_result$login_resultStandardScheme.write(ConfigIService.java:18474)
> 	at org.apache.iotdb.confignode.rpc.thrift.ConfigIService$login_result$login_resultStandardScheme.write(ConfigIService.java:18441)
> 	at org.apache.iotdb.confignode.rpc.thrift.ConfigIService$login_result.write(ConfigIService.java:18392)
> 	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:58)
> 	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
> 	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 	at java.lang.Thread.run(Thread.java:750)
> 2022-06-02 14:42:23,449 [pool-2-IoTDB-ConfigNodeRPC-Client-2] INFO  o.a.i.c.s.t.ConfigNodeRPCServiceProcessor:193 - Execute SetStorageGroupRequest TSetStorageGroupReq(storageGroup:TStorageGroupSchema(name:root.sg1, TTL:9223372036854775807, schemaReplicationFactor:3, dataReplicationFactor:3, timePartitionInterval:604800, maximumSchemaRegionCount:-1, maximumDataRegionCount:-1, dataRegionGroupIds:[], schemaRegionGroupIds:[])) with result TSStatus(code:707, message:The current ConfigNode is not leader. And ConfigNodeGroup is in leader election. Please redirect with a random ConfigNode.) 
> 2022-06-02 14:42:23,478 [0.0.0.0_22280@group-000000000000-StateMachineUpdater] WARN  o.a.i.d.m.r.MemoryStatistics:80 - Current series number 224 is too large... 
> 2022-06-02 14:42:23,479 [0.0.0.0_22280@group-000000000000-StateMachineUpdater] INFO  o.a.i.c.p.ClusterSchemaInfo:113 - Successfully set StorageGroup: TStorageGroupSchema(name:root.sg1, TTL:9223372036854775807, schemaReplicationFactor:3, dataReplicationFactor:3, timePartitionInterval:604800, maximumSchemaRegionCount:-1, maximumDataRegionCount:-1, dataRegionGroupIds:[], schemaRegionGroupIds:[])  {code}
>  
> confignode3
> {code:java}
> type:DataRegion, id:2)] 
> 2022-06-02 14:42:50,243 [pool-1-IoTDB-InternalServiceRPC-Client-5] INFO  o.a.i.d.s.t.i.InternalServiceImpl:117 - receive FragmentInstance to group[TConsensusGroupId(type:DataRegion, id:2)] 
> 2022-06-02 14:42:53,739 [Service Thread] INFO  o.a.i.m.m.MicrometerMetricManager:200 - create getOrCreateTimer jvm.gc.pause 
> 2022-06-02 14:43:03,344 [pool-1-IoTDB-InternalServiceRPC-Client-5] INFO  o.a.i.d.s.t.i.InternalServiceImpl:117 - receive FragmentInstance to group[TConsensusGroupId(type:DataRegion, id:2)] 
> 2022-06-02 14:43:03,346 [pool-1-IoTDB-InternalServiceRPC-Client-5] ERROR o.a.i.d.c.s.DataRegionStateMachine:146 - Invalid node type: 35 
> 2022-06-02 14:43:03,348 [pool-1-IoTDB-InternalServiceRPC-Client-5] ERROR o.a.t.ProcessFunction:47 - Internal error processing sendFragmentInstance 
> java.lang.NullPointerException: null
> 	at org.apache.iotdb.db.service.thrift.impl.InternalServiceImpl.sendFragmentInstance(InternalServiceImpl.java:141)
> 	at org.apache.iotdb.mpp.rpc.thrift.InternalService$Processor$sendFragmentInstance.getResult(InternalService.java:1195)
> 	at org.apache.iotdb.mpp.rpc.thrift.InternalService$Processor$sendFragmentInstance.getResult(InternalService.java:1175)
> 	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> 	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
> 	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 	at java.lang.Thread.run(Thread.java:750)
>  {code}
>  



--
This message was sent by Atlassian Jira
(v8.20.7#820007)