You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@teaclave.apache.org by Claus M <no...@github.com.INVALID> on 2020/11/19 17:33:46 UTC
[apache/incubator-teaclave-sgx-sdk] Encrypted websockets in SGX
(#291)
Hi!
I am using the embedded websockets crate to connect to a websocket endpoint ([echo.websocket.org](echo.websocket.org)) in a similar style as the tls client example - but with the full tls session inside an encalve (see the code below). However, it doesn't seem to work with encryption and the server seems to respond with an EOF (i.e. a 0 byte read) after sending the websocket handshake. (It works without encryption without issues)
The components I am using is [embedded websockets](https://github.com/ninjasource/embedded-websocket), and ports of [rustls](https://github.com/mesalock-linux/rustls), [mio](https://github.com/mesalock-linux/mio-sgx) and [webpki](https://github.com/mesalock-linux/webpki). Here is the code:
~~~rust
#[no_mangle]
pub extern "C" fn start_thread(fd: c_int, hostname: *const c_char, cert: *const c_char) -> sgx_status_t {
let std_stream = std::net::TcpStream::new(fd).unwrap();
let name_cast = unsafe { CStr::from_ptr(hostname).to_str() };
let name = name_cast.unwrap();
let hostname_ref = webpki::DNSNameRef::try_from_ascii_str(name).unwrap();
let cert_cast = unsafe { CStr::from_ptr(cert).to_str() };
let cert = cert_cast.unwrap();
let cfg = make_config(cert);
let address = name;
println!("Connecting to: {} ({})", address, name);
let mut socket = TcpStream::from_stream(std_stream).unwrap();
let mut stream = TlsClient::new(socket, hostname_ref, cfg.clone());
let mut buffer1: [u8; 4000] = [0; 4000];
let mut buffer2: [u8; 4000] = [0; 4000];
let mut ws_client = ws::WebSocketClient::new_client(ws::EmptyRng::new());
// initiate a websocket opening handshake
let websocket_options = WebSocketOptions {
path: "/",
host: address,
origin: "",
sub_protocols: None,
additional_headers: None,
};
let mut state = ConnectionState::Connecting;
let (len, web_socket_key) = ws_client.client_connect(&websocket_options, &mut buffer1).unwrap();
println!("Sending opening handshake: {} bytes", len);
stream.write_all(&buffer1[..len]).unwrap();
let s = std::str::from_utf8(&buffer1[..len]).unwrap();
println!("-> {}", s);
let t = thread::spawn(move || {
let mut poll = mio::Poll::new().unwrap();
let mut events = mio::Events::with_capacity(32);
stream.register(&mut poll);
loop {
poll.poll(&mut events, None).unwrap();
for ev in events.iter() {
if let Some(is_readable) = stream.ready(&mut poll, &ev) {
state = match state {
ConnectionState::Connecting if is_readable => {
println!("Connecting::READ");
...
},
ConnectionState::Data if is_readable => {
println!("Data::READ");
...
ConnectionState::Data
},
_ => {
println!("OTHER");
ConnectionState::Data
}
}
}
}
}
});
sgx_status_t::SGX_SUCCESS
}
~~~
Relevant part of TlsClient: (this is from the rustls example)
~~~rust
impl TlsClient {
pub fn ready(&mut self,
poll: &mut mio::Poll,
ev: &mio::event::Event) -> Option<bool> {
assert_eq!(ev.token(), CLIENT);
let mut interest = Some(false);
if ev.readiness().is_readable() {
println!("# TLSClient Read interest");
self.do_read();
interest = Some(true);
}
if ev.readiness().is_writable() {
println!("# TLSClient write interest");
self.do_write();
}
if self.is_closed() {
println!("Connection closed");
return None;
}
self.reregister(poll);
interest
}
}
// in do_read() the "closing" is set, as indicated by the println
pub fn do_read(&mut self) {
...
// If we're ready but there's no data: EOF.
if rc.unwrap() == 0 {
println!("EOF");
self.closing = true;
self.clean_closure = true;
return;
}
...
~~~
Running this code above on an Azure confidential compute machine inside the 18.04 docker container generates this output:
~~~shell
root@2fd3b8e3f502:~/sgx/samplecode/hello-websocket-rust/bin# ./app
[+] Init Enclave Successful 2!
Connecting to: echo.websocket.org (echo.websocket.org)
Sending opening handshake: 193 bytes
-> GET / HTTP/1.1
Host: echo.websocket.org
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: AAAAAAAAAAAAAAAAAAAAAA==
Origin:
Sec-WebSocket-Protocol:
Sec-WebSocket-Version: 13
# TLSClient write interest
OTHER
# TLSClient Read interest
EOF
Connection closed
^C
~~~
As I said, it works perfectly fine without encryption - for which I also don't have to use mio, but a raw std::TcpStream - so I think this may be where the issue is located. Does anyone else have a pointer as to where to go from here to try and debug further?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-teaclave-sgx-sdk/issues/291
Re: [apache/incubator-teaclave-sgx-sdk] Encrypted websockets in SGX
(#291)
Posted by Yu Ding <no...@github.com.INVALID>.
This is the payload i use:
```
00000000: 4745 5420 2f20 4854 5450 2f31 2e31 0d0a GET / HTTP/1.1..
00000010: 486f 7374 3a20 6563 686f 2e77 6562 736f Host: echo.webso
00000020: 636b 6574 2e6f 7267 0d0a 5570 6772 6164 cket.org..Upgrad
00000030: 653a 2077 6562 736f 636b 6574 0d0a 436f e: websocket..Co
00000040: 6e6e 6563 7469 6f6e 3a20 5570 6772 6164 nnection: Upgrad
00000050: 650d 0a53 6563 2d57 6562 536f 636b 6574 e..Sec-WebSocket
00000060: 2d4b 6579 3a20 4141 4141 4141 4141 4141 -Key: AAAAAAAAAA
00000070: 4141 4141 4141 4141 4141 4141 3d3d 0d0a AAAAAAAAAAAA==..
00000080: 4f72 6967 696e 3a20 0d0a 5365 632d 5765 Origin: ..Sec-We
00000090: 6253 6f63 6b65 742d 5072 6f74 6f63 6f6c bSocket-Protocol
000000a0: 3a20 0d0a 5365 632d 5765 6253 6f63 6b65 : ..Sec-WebSocke
000000b0: 742d 5665 7273 696f 6e3a 2031 330d 0a0d t-Version: 13...
000000c0: 0a
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-teaclave-sgx-sdk/issues/291#issuecomment-745627275
Re: [apache/incubator-teaclave-sgx-sdk] Encrypted websockets in SGX
(#291)
Posted by Claus M <no...@github.com.INVALID>.
Hey @dingelish - thanks for responding quickly! Here's a sample that I could reproduce the error with.
https://github.com/celaus/websocket-sgx-sample
Let me know if there is anything I can help out with.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-teaclave-sgx-sdk/issues/291#issuecomment-731413666
Re: [apache/incubator-teaclave-sgx-sdk] Encrypted websockets in SGX
(#291)
Posted by Claus M <no...@github.com.INVALID>.
Hi @dingelish - any news? :)
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-teaclave-sgx-sdk/issues/291#issuecomment-745391111
Re: [apache/incubator-teaclave-sgx-sdk] Encrypted websockets in SGX
(#291)
Posted by Claus M <no...@github.com.INVALID>.
Closed #291.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-teaclave-sgx-sdk/issues/291#event-4608077047
Re: [apache/incubator-teaclave-sgx-sdk] Encrypted websockets in SGX
(#291)
Posted by Yu Ding <no...@github.com.INVALID>.
@celaus wireshark guided me to this answer 😂
https://github.com/ctz/rustls/issues/381
echo.websocket.org only supports `TLS_RSA_WITH_AES_128_CBC_SHA` which is not supported by `rustls`. shall we try another wss echo server?
```
nmap --script ssl-enum-ciphers -p 443 echo.websocket.org
Starting Nmap 7.91 ( https://nmap.org ) at 2020-12-30 14:11 PST
Nmap scan report for echo.websocket.org (174.129.224.73)
Host is up (0.080s latency).
rDNS record for 174.129.224.73: ec2-174-129-224-73.compute-1.amazonaws.com
PORT STATE SERVICE
443/tcp open https
| ssl-enum-ciphers:
| TLSv1.2:
| ciphers:
| TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
| compressors:
| NULL
| cipher preference: indeterminate
| cipher preference error: Too few ciphers supported
| warnings:
| Forward Secrecy not supported by any cipher
|_ least strength: A
Nmap done: 1 IP address (1 host up) scanned in 5.13 seconds
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-teaclave-sgx-sdk/issues/291#issuecomment-752774468
Re: [apache/incubator-teaclave-sgx-sdk] Encrypted websockets in SGX
(#291)
Posted by Claus M <no...@github.com.INVALID>.
Hi! So your command connects ok for me, the server closes the connection because of a lack of other messages I assume:
~~~
$ openssl s_client -connect echo.websocket.org:443 -showcerts
CONNECTED(00000005)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = websocket.org
verify return:1
---
Certificate chain
...
---
No client certificate CA names sent
---
SSL handshake has read 2735 bytes and written 538 bytes
---
New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : AES128-SHA
Session-ID: 5FD9F135D37824CCF498B252867CBC59E26872F08961B5EB2E12A36065AC5E9F
Session-ID-ctx:
Master-Key: FC17A81B8B422F29CB4CA0CA34F5EA84C15A051F5136DEC7AE4765615184FA0F4353813D2F8CD4A1CE4E4071AF6508B9
Start Time: 1608119990
Timeout : 7200 (sec)
Verify return code: 0 (ok)
---
closed
~~~
the closed appears only after several seconds of doing nothing. A proper websocket client can be found on www.websocket.org (with chrome or another new browser with websocket support) and then it keeps the connection open.
The payload connects just fine without encryption too, so I don't think that should be the issue (for that, use the socket as a TCP stream as-is and connect on port 80)
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-teaclave-sgx-sdk/issues/291#issuecomment-746189534
Re: [apache/incubator-teaclave-sgx-sdk] Encrypted websockets in SGX
(#291)
Posted by Claus M <no...@github.com.INVALID>.
Hey, thanks for looking into this 😃
The echo server was just a test to get it to work at all, I'll try to reproduce with my actual web server and then come back to you. I checked the ciphers and they _should_ be supported, but I'll need a few days to reproduce. I'll respond here then.
Cheers!
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-teaclave-sgx-sdk/issues/291#issuecomment-756941231
Re: [apache/incubator-teaclave-sgx-sdk] Encrypted websockets in SGX
(#291)
Posted by Yu Ding <no...@github.com.INVALID>.
@celaus i just played with echo.websocket.org:443 a little. it's closing my connection. could you please advise? thanks!
```
$ openssl s_client -connect echo.websocket.org:443 -showcerts
... many outputs and input on stdin...
GET / HTTP/1.1
Host: echo.websocket.org
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: AAAAAAAAAAAAAAAAAAAAAA==
Origin:
Sec-WebSocket-Protocol:
Sec-WebSocket-Version: 13
closed
```
seems server closed the connection after i sent the http request.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-teaclave-sgx-sdk/issues/291#issuecomment-745609006
Re: [apache/incubator-teaclave-sgx-sdk] Encrypted websockets in SGX
(#291)
Posted by Yu Ding <no...@github.com.INVALID>.
Hi @celaus , thanks for the report! I suspicious this is a bug in either this SDK or the ported mio. Could I have a reproducible example? Thanks!
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-teaclave-sgx-sdk/issues/291#issuecomment-730650989
Re: [apache/incubator-teaclave-sgx-sdk] Encrypted websockets in SGX
(#291)
Posted by Yu Ding <no...@github.com.INVALID>.
@celaus haven't got time into this due to high on call pressure ... i'll try to solve it this weekend! sorry for the delay!
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-teaclave-sgx-sdk/issues/291#issuecomment-745467386
Re: [apache/incubator-teaclave-sgx-sdk] Encrypted websockets in SGX
(#291)
Posted by Yu Ding <no...@github.com.INVALID>.
@celaus Hi there, i reproduced the error in non-sgx environment:
https://github.com/dingelish/embedded-websocket
```
cargo r --example client
```
could you please advise how to make my fork's client example work? if it works, i'll get it work in SGX enclave :-) thanks!
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-teaclave-sgx-sdk/issues/291#issuecomment-751311886
Re: [apache/incubator-teaclave-sgx-sdk] Encrypted websockets in SGX
(#291)
Posted by Yu Ding <no...@github.com.INVALID>.
> Hi! Thank you for your support. I can confirm that your sample seems to reproduce the error.
>
is it a bug of `echo.websocket.org`? or misconfiguration of `rustls`?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-teaclave-sgx-sdk/issues/291#issuecomment-752735078
Re: [apache/incubator-teaclave-sgx-sdk] Encrypted websockets in SGX
(#291)
Posted by Claus M <no...@github.com.INVALID>.
Hi! Thank you for your support. I can confirm that your sample seems to reproduce the error.
~~~
13:38:31 /tmp{} 🌊 git clone https://github.com/dingelish/embedded-websocket
Cloning into 'embedded-websocket'...
remote: Enumerating objects: 21, done.
remote: Counting objects: 100% (21/21), done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 244 (delta 9), reused 19 (delta 8), pack-reused 223
Receiving objects: 100% (244/244), 97.18 KiB | 667.00 KiB/s, done.
Resolving deltas: 100% (138/138), done.
13:38:33 /tmp{} 🌊 cd embedded-websocket/
13:38:36 /t/embedded-websocket{master} 🌊 git ^C
13:38:42 /t/embedded-websocket{master} 🌊 cargo run --example client
Updating crates.io index
Downloaded generic-array v0.12.3
Downloaded generic-array v0.13.2
Downloaded as-slice v0.1.4
Downloaded heapless v0.5.6
Downloaded rustls v0.19.0
Downloaded hash32 v0.1.1
Downloaded webpki v0.21.4
Downloaded webpki-roots v0.21.0
Downloaded ring v0.16.19
Downloaded 9 crates (5.8 MB) in 1.89s (largest was `ring` at 5.2 MB)
Compiling typenum v1.12.0
Compiling cfg-if v0.1.10
Compiling cc v1.0.66
Compiling libc v0.2.81
Compiling getrandom v0.1.15
Compiling version_check v0.9.2
Compiling spin v0.5.2
Compiling untrusted v0.7.1
Compiling byteorder v1.3.4
Compiling httparse v1.3.4
Compiling log v0.4.11
Compiling stable_deref_trait v1.2.0
Compiling heapless v0.5.6
Compiling ppv-lite86 v0.2.10
Compiling base64 v0.13.0
Compiling sha1 v0.6.0
Compiling generic-array v0.14.4
Compiling ring v0.16.19
Compiling hash32 v0.1.1
Compiling rand_core v0.5.1
Compiling generic-array v0.12.3
Compiling generic-array v0.13.2
Compiling rand_chacha v0.2.2
Compiling as-slice v0.1.4
Compiling rand v0.7.3
Compiling embedded-websocket v0.3.2 (/private/tmp/embedded-websocket)
Compiling webpki v0.21.4
Compiling sct v0.6.0
Compiling webpki-roots v0.21.0
Compiling rustls v0.19.0
Finished dev [unoptimized + debuginfo] target(s) in 32.24s
Running `target/debug/examples/client`
Connecting to: echo.websocket.org:443
Connected.
Sending opening handshake: 214 bytes
Error: Io(Kind(UnexpectedEof))
~~~
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-teaclave-sgx-sdk/issues/291#issuecomment-751699669
Re: [apache/incubator-teaclave-sgx-sdk] Encrypted websockets in SGX
(#291)
Posted by Yu Ding <no...@github.com.INVALID>.
Hi @celaus I'll be working on this during thanksgiving. sorry for the delay!
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-teaclave-sgx-sdk/issues/291#issuecomment-733407580