You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2020/09/17 06:18:28 UTC
[GitHub] [shardingsphere] taojintianxia opened a new issue #7223: does scaling support resume ?
taojintianxia opened a new issue #7223:
URL: https://github.com/apache/shardingsphere/issues/7223
hi there
this issue for discuss the current situation of sharding scaling resume function and related.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] dmbj commented on issue #7223: does scaling support resume ?
Posted by GitBox <gi...@apache.org>.
dmbj commented on issue #7223:
URL: https://github.com/apache/shardingsphere/issues/7223#issuecomment-685452592
here my case for example:
1.6000w+ data intent to sharding
2.mysql version 5.6.45-log base VIP mod,binlog `expire_logs_days:7 max_binlog_size:500m`
3.scaling & proxy version:5.0.0.RC.snapshot
use less then 2 day to sync INVENTORY_DATA, when INVENTORY_DATA sync finished and begin to sync increase Data, occur exceptions
```
[ERROR] 13:25:47.613 [nioEventLoopGroup-4-1] o.a.s.s.mysql.client.MySQLClient - protocol resolution error
io.netty.handler.codec.DecoderException: java.lang.RuntimeException: Decode binlog event failed, errorCode: 1236, sqlState: HY000, errorMessage: Could not find first log file name in binary log index file
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Decode binlog event failed, errorCode: 1236, sqlState: HY000, errorMessage: Could not find first log file name in binary log index file
at org.apache.shardingsphere.scaling.mysql.client.netty.MySQLBinlogEventPacketDecoder.checkError(MySQLBinlogEventPacketDecoder.java:101)
at org.apache.shardingsphere.scaling.mysql.client.netty.MySQLBinlogEventPacketDecoder.decode(MySQLBinlogEventPacketDecoder.java:57)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
... 23 common frames omitted
```
then `SHOW BINARY LOGS;` to check binlog filelist, found scaling job target binlog file is not exists.
now here is the quesion:
does scaling support resume or start increase data from Specify binlog file position?
so could not restart a new job sync INVENTORY_DATA again
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] Lucas-307 commented on issue #7223: does scaling support resume ?
Posted by GitBox <gi...@apache.org>.
Lucas-307 commented on issue #7223:
URL: https://github.com/apache/shardingsphere/issues/7223#issuecomment-693774460
@dmbj Sorry, I can't understand, what is "file document read" mean?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] dmbj commented on issue #7223: does scaling support resume ?
Posted by GitBox <gi...@apache.org>.
dmbj commented on issue #7223:
URL: https://github.com/apache/shardingsphere/issues/7223#issuecomment-691757293
> @dmbj , Scaling support resume from break-point, which depends on zookeeper service. You should setup an zookeeper service and add config.
>
> ${scaling_home}/conf/server.yaml
>
> ```yaml
> name: scalingjob
> registryCenter:
> type: zookeeper
> serverLists: localhost:2181 # your zk host
> props:
> ```
>
> restart scaling service.
>
> Focus on your exceptions: `Could not find first log file name in binary log index file`, which means binlog has been removed before execute increase job.
>
> So you should turn up binlog expire days.
appreciate to answer the question! In my case,as you said binlog has been removed is the exact reason cause the exception,I contact ops found their has a strategy to ensure disk space,their will not intent to change that.So i need to think another solution to solve my problem:
can scaling surpport file document read to solve increase data sync?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] menghaoranss closed issue #7223: does scaling support resume ?
Posted by GitBox <gi...@apache.org>.
menghaoranss closed issue #7223:
URL: https://github.com/apache/shardingsphere/issues/7223
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] dmbj commented on issue #7223: does scaling support resume ?
Posted by GitBox <gi...@apache.org>.
dmbj commented on issue #7223:
URL: https://github.com/apache/shardingsphere/issues/7223#issuecomment-693779366
> @dmbj Sorry, I can't understand, what is "file document read" mean?
in my case for some reason,mysql binlog couldn't keep such long time(about 2 days). however, our ops can get binlog file offline,so i think is any possibility that scaling surpport read target file, not only mysql slave client sync read binlog?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] Lucas-307 commented on issue #7223: does scaling support resume ?
Posted by GitBox <gi...@apache.org>.
Lucas-307 commented on issue #7223:
URL: https://github.com/apache/shardingsphere/issues/7223#issuecomment-693780401
> > @dmbj Sorry, I can't understand, what is "file document read" mean?
>
> in my case for some reason,mysql binlog couldn't keep such long time(about 2 days). however, our ops can get binlog file offline,so i think is any possibility that scaling surpport read target file, not only mysql slave client sync read binlog?
@dmbj Sorry, we don't support that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] Lucas-307 commented on issue #7223: does scaling support resume ?
Posted by GitBox <gi...@apache.org>.
Lucas-307 commented on issue #7223:
URL: https://github.com/apache/shardingsphere/issues/7223#issuecomment-688754327
@dmbj , Scaling support resume from break-point, which depends on zookeeper service. You should setup an zookeeper service and add config.
${scaling_home}/conf/server.yaml
```yaml
name: scalingjob
registryCenter:
type: zookeeper
serverLists: localhost:2181 # your zk host
props:
```
restart scaling service.
Focus on your exceptions: `Could not find first log file name in binary log index file`, which means binlog has been removed before execute increase job.
So you should turn up binlog expire days.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] kimmking closed issue #7223: does scaling support resume ?
Posted by GitBox <gi...@apache.org>.
kimmking closed issue #7223:
URL: https://github.com/apache/shardingsphere/issues/7223
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] dmbj commented on issue #7223: does scaling support resume ?
Posted by GitBox <gi...@apache.org>.
dmbj commented on issue #7223:
URL: https://github.com/apache/shardingsphere/issues/7223#issuecomment-693782949
> > > @dmbj Sorry, I can't understand, what is "file document read" mean?
> >
> >
> > in my case for some reason,mysql binlog couldn't keep such long time(about 2 days). however, our ops can get binlog file offline,so i think is any possibility that scaling surpport read target file, not only mysql slave client sync read binlog?
>
> @dmbj Sorry, we don't support that.
thank you all the same
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] dmbj edited a comment on issue #7223: does scaling support resume ?
Posted by GitBox <gi...@apache.org>.
dmbj edited a comment on issue #7223:
URL: https://github.com/apache/shardingsphere/issues/7223#issuecomment-691757293
> @dmbj , Scaling support resume from break-point, which depends on zookeeper service. You should setup an zookeeper service and add config.
>
> ${scaling_home}/conf/server.yaml
>
> ```yaml
> name: scalingjob
> registryCenter:
> type: zookeeper
> serverLists: localhost:2181 # your zk host
> props:
> ```
>
> restart scaling service.
>
> Focus on your exceptions: `Could not find first log file name in binary log index file`, which means binlog has been removed before execute increase job.
>
> So you should turn up binlog expire days.
appreciate to answer the question! In my case,as you said binlog has been removed is the exact reason cause the exception,I contact ops found their has a strategy to ensure disk space,But their will not intent to change that.So i need to think another solution to solve my problem:
can scaling surpport file document read to solve increase data sync?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org