You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brpc.apache.org by GitBox <gi...@apache.org> on 2022/03/30 03:59:30 UTC
[GitHub] [incubator-brpc] dboyzhong opened a new issue #1731: brpc https client 偶现ssl 证书错误导致https请求失败
dboyzhong opened a new issue #1731:
URL: https://github.com/apache/incubator-brpc/issues/1731
**Describe the bug (描述bug)**
使用https 方式连接Http服务,偶现ssl 证书错误,日志如下:
W0325 19:39:20.771101 204547 src/brpc/socket.cpp:1899] Fail to read from ssl_fd=463: error:0606506D:digital envelope routines:EVP_DecryptFinal_ex:wrong final block length
W0325 19:39:20.771117 204547 src/brpc/input_messenger.cpp:205] sso read count: -1socket desc: Socket{id=8589936411 fd=463 addr=10.24.2.62:443:17742} (0x0x7f923b7178c0): SSL related operation failed
**To Reproduce (复现方法)**
偶发,不易复现
**Expected behavior (期望行为)**
**Versions (各种版本)**
OS: CentOS7
Compiler: G++4.8
brpc: 0.9.6 & 1.0.0
protobuf:
**Additional context/screenshots (更多上下文/截图)**
原因分析:
1. 代码中加log(iobuf.cpp)发现,https请求的数据实际已经接收完整,SSL_Read触发2次:第一次返回全部http数据,第二次返回-1,错误为:wrong final block length。
2. 
2.1 从 图中看,在1693行读成功后,循环继续读,引二次读错误,在1706行直接返回-1了,如果返回第一次读成功的字节数:1707行,最终的http请求仍会成功。
代码如下:
`ssize_t IOPortal::append_from_SSL_channel(
SSL* ssl, int* ssl_error, size_t max_count) {
size_t nr = 0;
do {
if (!_block) {
_block = iobuf::acquire_tls_block();
if (BAIDU_UNLIKELY(!_block)) {
errno = ENOMEM;
*ssl_error = SSL_ERROR_SYSCALL;
return -1;
}
}
const size_t read_len = std::min(_block->left_space(), max_count - nr);
const int rc = SSL_read(ssl, _block->data + _block->size, read_len);
*ssl_error = SSL_get_error(ssl, rc);
if (rc > 0) {
LOG(ERROR) << "append_from_SSL_channel ssl data len: " << rc << ": " << std::hex << this;
const IOBuf::BlockRef r = { (uint32_t)_block->size, (uint32_t)rc, _block };
_push_back_ref(r);
_block->size += rc;
if (_block->full()) {
Block* const saved_next = _block->portal_next;
_block->dec_ref(); // _block may be deleted
_block = saved_next;
}
nr += rc;
} else {
if (rc < 0) {
LOG(ERROR) << "append_from_SSL_channel ssl error: " << *ssl_error << "errno: " << errno << ": " << std::hex << this ;
if (*ssl_error == SSL_ERROR_WANT_READ
|| (*ssl_error == SSL_ERROR_SYSCALL
&& BIO_fd_non_fatal_error(errno) == 1)) {
// Non fatal error, tell caller to read again
*ssl_error = SSL_ERROR_WANT_READ;
} else {
// Other errors are fatal
LOG(ERROR) << " append_from_SSL_channel fatal error: " << *ssl_error << std::hex << this;
return rc;
//return (nr > 0 ? nr : rc);
}
}
return (nr > 0 ? nr : rc);
}
} while (nr < max_count);
return nr;
}`
2.2 与http 请求不同的是,在iobuf.cpp中,http使用readv只读一次,就返回上层,而不是类似https循环读至EOF或错误再返回上层。所以上http请求上层可以获得正常数据:见:ssize_t IOPortal::pappend_from_file_descriptor()实现。
3. 解决方案:
3.1 IOPortal::append_from_SSL_channel在SSL_read 返回错误后,应判断当前已读数据数量,如果已读数据>0,则返回数据个数:

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] dboyzhong commented on issue #1731: brpc https client 偶现ssl 证书错误导致https请求失败
Posted by GitBox <gi...@apache.org>.
dboyzhong commented on issue #1731:
URL: https://github.com/apache/incubator-brpc/issues/1731#issuecomment-1082759673
> 问题
好的,我正跟进wrong final block length 问题根因,这个问题还有一个逻辑上的问题请教:
1. SSL_Read循环读没问题,当第1次读成功,比如读到100字节数据,而循环再读第二次时,SSL_Read返回-1,此时append_from_SSL_channel是应该返回100?还是 -1?因为毕竟读出来了100字节数据,返回-1是不是不太合适?(如果对端发送完这100字节后,对端发送RST分节,本端socket会返回-1)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] dboyzhong commented on issue #1731: brpc https client 偶现ssl 证书错误导致https请求失败
Posted by GitBox <gi...@apache.org>.
dboyzhong commented on issue #1731:
URL: https://github.com/apache/incubator-brpc/issues/1731#issuecomment-1082809895
> > 1. SSL_Read循环读没问题,当第1次读成功,比如读到100字节数据,而循环再读第二次时,SSL_Read返回-1,此时append_from_SSL_channel是应该返回100?还是 -1?因为毕竟读出来了100字节数据,返回-1是不是不太合适?(如果对端发送完这100字节后,对端发送RST分节,本端socket会返回-1)
>
> 这种情况就要看具体的错误原因了,你看原来代码里是区分的,如果是non fatal error,就返回已经读的字节数,并设置ssl_error。如果是fatal error,就直接返回rc。所以需要分析下当前问题是否属于non fatal error,如果是,可以扩展一下non fatal error的判断条件
好的。多谢指教,谢谢
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] wwbmmm commented on issue #1731: brpc https client 偶现ssl 证书错误导致https请求失败
Posted by GitBox <gi...@apache.org>.
wwbmmm commented on issue #1731:
URL: https://github.com/apache/incubator-brpc/issues/1731#issuecomment-1082783044
> 1. SSL_Read循环读没问题,当第1次读成功,比如读到100字节数据,而循环再读第二次时,SSL_Read返回-1,此时append_from_SSL_channel是应该返回100?还是 -1?因为毕竟读出来了100字节数据,返回-1是不是不太合适?(如果对端发送完这100字节后,对端发送RST分节,本端socket会返回-1)
这种情况就要看具体的错误原因了,你看原来代码里是区分的,如果是non fatal error,就返回已经读的字节数,并设置ssl_error。如果是fatal error,就直接返回rc。所以需要分析下当前问题是否属于non fatal error,如果是,可以扩展一下non fatal error的判断条件
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] wwbmmm commented on issue #1731: brpc https client 偶现ssl 证书错误导致https请求失败
Posted by GitBox <gi...@apache.org>.
wwbmmm commented on issue #1731:
URL: https://github.com/apache/incubator-brpc/issues/1731#issuecomment-1082680858
pappend_from_file_descriptor里只readv一次,是因为事先根据max_count分配了足够的内存空间,所以可以一次性把缓存区中的数据读出。
而SSL没有提供readv机制,所以在未读够max_count字节时,还是需要循环读取直到读不到数据或读取错误。
这里问题在于读取错误的时候是否要返回已读取的数据,注释里提到 `Other errors are fatal`,在出现fatal错误的时候,如果返回成功,是否会掩盖问题?是不是可以分析一下`error:0606506D:digital envelope routines:EVP_DecryptFinal_ex:wrong final block length`这个错误的原因,看能否从根本上解决问题
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org