Appearance
aowlspt_net.h
Source: abi/aowlspt_net.h — 2881 lines, 71 file-scope functions.
What this header owns
Reproduced verbatim from the header's own banner comment — these notes are frequently the only written record of why the subsystem is shaped the way it is.
text
aowlspt_net.h — sockets and zlib for the backend server.
nimony's `nativesocket` is a 26-line stub and `winlean` has no socket
surface at all, so the syscalls live here, like every other place this repo
has to reach something nimony cannot. Everything above the byte framing —
parsing, routing, the database, the mod API, and every byte that goes back
on the wire — is nimony.
The zlib part is not optional decoration. Tarkov's client sends and expects
**zlib-framed bodies**: a request body arrives deflated and a response must
be deflated too. A server that answers in plain JSON gets a client that
silently fails to parse it, and `curl` against such a server appears to work
while the game does not — which is exactly the trap `tools/aowlprobe.nim`
exists to document, and why it frames its own bodies rather than shelling out
to `curl`. (This named `tools/SptProbe`, which has never existed in this
repo; the C# tool in `tools/SptReflect` is a metadata dumper and speaks no
protocol at all.)
------------------------------------------------------------------------
Why this is a poller and a request pool rather than a pool of accept
threads
------------------------------------------------------------------------
It used to be a fixed pool of threads that each called `accept` on the
shared listening socket and then stayed with the accepted connection until
the client let go of it. That is the simplest thing that works, and it has
one property that cannot be fixed by tuning it: **the pool size is the
connection limit**. Keep-alive means a connection lives for a session, so a
client that opens sixteen sockets and says nothing on any of them owns the
server. `fuzzwire` demonstrated it with twelve: every other client waited
ten seconds. Bounded receive deadlines cut that to five and raising the pool
from eight to sixteen doubled the number of sockets it took, but both are
the same trade — a bigger number for a bounded attacker, and the attacker
picks the number.
What removes it rather than raising it is decoupling *having a connection*
from *having a thread*:
* One **poller** thread owns the listening socket and every connection
that is not currently being served. It waits in `WSAPoll`, accepts, and
reads whatever has arrived into that connection's buffer. It never runs
a route and never blocks on one socket.
* A connection becomes interesting only when a **complete request** is
buffered — headers and the whole declared body. Then it is pushed onto a
ready queue and one of the **request workers** picks it up, calls into
nimony to answer it, and gives the connection back to the poller.
So a connection that stalls, dribbles, or opens and says nothing never
reaches a worker at all. It costs one socket, one `AowlConn` (about a
hundred bytes; it was 80 before the websocket fields went in, and that is a
hand count off the struct rather than a `sizeof` anybody has printed) and
whatever it has actually managed to send — nothing is allocated
for a body it merely *claims* it will send. The limit is
`AOWL_NET_MAX_CONNS` sockets and `AOWL_NET_MAX_BUFFERED` bytes of buffered
request, not a thread count.
------------------------------------------------------------------------
WSAPoll, not IOCP — and what that costs
------------------------------------------------------------------------
IOCP is the other way to write this and is the better one for a server with
thousands of concurrent sockets spread over many cores. This is not that
server. It listens on loopback, for one game, and the realistic connection
count is single digits with a hostile one in the low hundreds.
What IOCP would cost here:
* Every read becomes a posted `WSARecv` with an `OVERLAPPED` and a
completion picked up somewhere else, which means the "have I got a whole
request yet" state machine stops being a loop you can read top to bottom
and becomes a set of callbacks with a lifetime problem attached to each
buffer. The bug class that follows — a completion arriving for a
connection that has just been closed — is exactly the kind this file
must not have, because a crash here is the whole game server.
* Cancellation and shutdown get harder: `CancelIoEx` plus draining the
port, rather than "close the listening socket and set a flag".
What WSAPoll costs, honestly:
* The poll set is rebuilt and scanned linearly every wakeup, so the poller
is O(connections) per tick. At `AOWL_NET_MAX_CONNS` = 1024 that is a
1024-entry array scan on a thread that is otherwise asleep; it is
nothing next to inflating a request.
* There is one poller, so reads are serialised. Reads are a `memcpy` out
of the kernel; the work that is not — inflate, route, deflate — is on
the worker pool, which is where the parallelism was already.
* `WSAPoll` has a documented wart: it never reports `POLLOUT` for a socket
with a connection in progress. Nothing here polls for writability on a
connecting socket — the only `POLLWRNORM` wait is in
`aowl_net_send_within`, reached from `aowl_net_send` and from
`aowl_ws_send`, and in both cases on an already-established connection
— so it does not bite.
------------------------------------------------------------------------
Why the byte framing is in C when everything else is nimony
------------------------------------------------------------------------
The poller has to answer one question to do its job: *how many bytes are one
request*. That is the read path, which is what this file is for, and doing
it here keeps the buffer and the loop that fills it in one place.
It is deliberately the **only** HTTP knowledge in this file, and it is
advisory. `aowlspt_nim_handle` re-parses the same bytes and is the authority
on what the answer is; every case where the two could disagree — a second,
disagreeing `Content-Length`, a header block that never terminates, a
declared body over the limit — is a case nimony answers and **closes the
connection on**. So a framing disagreement can never desynchronise a
kept-alive stream: there is no next request to get wrong. No status line,
header or body is written from C.
------------------------------------------------------------------------
The websocket, and why the split falls where it does
------------------------------------------------------------------------
The game opens `/client/notifier/getwebsocket/<session>` and expects a
websocket it can be *pushed* to: new mail, an insurance return, a flea offer
sold. That used to be impossible here for a structural reason rather than a
protocol one — a held connection took one of a fixed pool of accept threads,
so four players idling in the menu owned the server — and the poller removed
exactly that. A connection nobody is answering now costs a socket and its
buffer. So a held websocket is a thing this shape can afford, and
`mods/tarkov` no longer has to fake it with a poll *for a client that
upgrades*. The poll route is still registered and still the fallback: a
client that has not upgraded, has not finished logging in, or has just lost
its connection gets `ErrNotFound` from `notifyPush` and is drained through
`/client/notifier/getwebsocket` instead. See `mods/tarkov/emu/notify.nim`,
which tries the socket first and falls back on purpose.
**The trap is the completeness rule above.** A connection reaches a worker
when a *complete request* is buffered, and a websocket is precisely the
connection on which that never happens again: after the handshake the bytes
on the wire are frames, not requests, and `aowl_conn_complete` would sit at
false forever while the header scan ran to `AOWL_HEAD_CAP` and the body
deadline expired underneath it. Getting that wrong does not fail loudly — it
rebuilds the starvation this file was rewritten to remove, one socket at a
time. So a websocket connection is taken *out* of the request state machine
at the moment it is adopted (`c->ws`), is never pushed onto the ready queue
again, never consults `aowl_conn_complete`, and gets a deadline that treats
silence as health rather than as a stall.
What is in C, and what is not:
* **Frame boundaries are here**, for the same reason request boundaries
are: the poller has to answer "how many bytes are one frame" to know
whether it has one, and the buffer and the loop that fills it are here.
With it come the read-path refusals, because each of them is a decision
about whether to *keep reading* — a payload length over
`AOWL_WS_MAX_FRAME` is refused off the length field, before a byte is
reserved for it, and an unmasked client frame is refused outright: RFC
6455 requires a client to mask, and a server that tolerates its absence
is a server that has stopped checking.
* **The handshake is not here.** It is `Sec-WebSocket-Key` + the
well-known GUID + SHA-1 + base64, and nimony ships both `std/sha1` and
`std/base64`; writing them again in C would be a second implementation
of two solved things in the language this repo uses least.
* **No frame is composed here either**, which is the same rule the rest of
this file already keeps: every byte the server puts on a socket is
written by nimony. The poller detects a ping, a close or a violation and
calls `aowlspt_nim_ws_control`; nimony builds the pong, the close or the
refusal and hands it back to `aowl_ws_send`, which owns only the lock and
the socket.
`aowl_ws_send` exists because a push comes from a worker (a mod answering a
request) or a timer, while the reads come from the poller, and two threads
writing frames onto one socket interleave into a stream neither side can
parse. One critical section covers every websocket write and the close of a
websocket connection, so a socket is never closed out from under a send in
flight.
**The poller does not wait on it, on any path.** Not on the close, where it
has always used `TryEnterCriticalSection` and deferred to its next wakeup,
and not on the write, where it used to. That is a rule about which thread is
asking rather than about which call it makes, so it is enforced where the
thread is known — `aowl_ws_send` compares the calling thread against the
poller's — and not at the call sites, which are several frames inside nimony
composing a pong. A worker still waits, up to `AOWL_WS_SEND_MS` per stall, because a
worker is a thread whose job is to wait for one client.
What that rule is worth is visible in what it forbids. This lock is one lock
for every websocket, and a worker holds it across a `send` to a peer that
may not be reading — so a blocking poller was one non-reading notifier
client away from stopping the *server*: no accepts, no reads on any
connection, no deadlines, until that push gave up. And it does not give up
at `AOWL_WS_SEND_MS`: that deadline restarts on every byte the peer accepts,
so a client dribbling one byte at a time holds it open indefinitely. A held
connection stalling the whole server is precisely the failure the poller was
written to remove, rebuilt one lock lower down.
A frame the poller cannot write immediately is queued on the connection
(`AOWL_WS_PEND_CAP`) and goes out at the next wakeup, `AOWL_WS_RETRY_MS`
later; a close it cannot make is retried on the same clock. Nothing is
dropped for being unlucky with a lock.Constants
AOWLSPT_NET_HAOWL_BODY_CAPAOWL_BODY_MSAOWL_CONN_CHUNKAOWL_HEAD_CAPAOWL_HEAD_MSAOWL_HOT_GRACE_MSAOWL_HOT_READSAOWL_IDLE_MSAOWL_KEEPALIVE_MAXAOWL_LEN_BADAOWL_NET_MAX_BUFFEREDAOWL_NET_MAX_CONNSAOWL_NET_WORKERSAOWL_SEND_DEADLINE_MSAOWL_SESSION_ID_MAXAOWL_SESSION_SLOTSAOWL_SESSION_SPINSAOWL_SESSION_WAIT_MSAOWL_WS_CLOSE_PROTOCOLAOWL_WS_CLOSE_TOO_BIGAOWL_WS_IDLE_MSAOWL_WS_MAX_FRAMEAOWL_WS_MAX_MESSAGEAOWL_WS_OUT_CAPAOWL_WS_PEND_CAPAOWL_WS_RETRY_MSAOWL_WS_SEND_MS
Functions
| Signature | Line |
|---|---|
int32_t aowl_net_startup(void) | 285 |
int32_t aowl_net_last_error(void) | 290 |
uint64_t aowl_net_bind(int32_t port) | 317 |
int32_t aowl_net_listen_on(uint64_t sock, int32_t backlog) | 394 |
uint64_t aowl_net_listen(int32_t port, int32_t backlog) | 403 |
uint64_t aowl_net_accept(uint64_t server) | 415 |
int32_t aowl_net_recv(uint64_t sock, void* buf, int32_t len) | 420 |
int32_t aowl_net_send_within(uint64_t sock, const void* buf, int32_t len, int32_t withinMs) | 434 |
int32_t aowl_net_send(uint64_t sock, const void* buf, int32_t len) | 477 |
void aowl_net_close(uint64_t sock) | 481 |
void aowl_net_shutdown_recv(uint64_t sock) | 485 |
uint32_t aowl_session_hash(const char* p, int32_t len) | 558 |
int32_t aowl_session_slot(const char* id, int32_t len) | 573 |
int32_t aowl_session_lock(const void* idp, int32_t len) | 627 |
void aowl_session_unlock(int32_t slot) | 645 |
void aowl_ssl_for_sock(SOCKET s) | 919 |
void aowl_net_set_nonblocking(SOCKET s) | 956 |
int aowl_wake_open(void) | 961 |
void aowl_wake_poller(void) | 983 |
void aowl_wake_drain(void) | 988 |
void aowl_ws_free_msg(AowlConn* c) | 999 |
void aowl_ws_free_out(AowlConn* c) | 1013 |
void aowl_conn_free_buf(AowlConn* c) | 1023 |
int32_t aowl_conn_take(SOCKET s) | 1034 |
void aowl_conn_drop(int32_t idx) | 1072 |
int aowl_conn_reserve(AowlConn* c, int32_t want) | 1095 |
int32_t aowl_ci_match(const char* p, int32_t n, const char* lit) | 1125 |
int32_t aowl_scan_length(const char* head, int32_t n) | 1156 |
int aowl_scan_upgrade(const char* head, int32_t n) | 1204 |
void aowl_frame(AowlConn* c) | 1230 |
int aowl_conn_complete(AowlConn* c) | 1270 |
int32_t aowl_conn_want(AowlConn* c) | 1277 |
int aowl_conn_read(AowlConn* c) | 1289 |
ULONGLONG aowl_conn_deadline(AowlConn* c, ULONGLONG now) | 1315 |
int64_t aowl_ws_adopt(uint64_t sock) | 1368 |
int32_t aowl_ws_write_nowait(SOCKET s, const char* p, int32_t len) | 1433 |
int32_t aowl_ws_defer(AowlConn* c, const char* p, int32_t len) | 1449 |
int aowl_ws_flush_pending(AowlConn* c) | 1461 |
int aowl_ws_enqueue_out(AowlConn* c, const char* p, int32_t len) | 1483 |
int aowl_ws_flush_out(AowlConn* c) | 1513 |
int32_t aowl_ws_send(int64_t ticket, const void* buf, int32_t len) | 1560 |
int aowl_ws_pump(AowlConn* c) | 1639 |
void aowl_push_ready(int32_t idx) | 1867 |
int32_t aowl_pop_ready(void) | 1874 |
void aowl_push_handback(int32_t idx) | 1886 |
int aowl_conn_read_hot(AowlConn* c) | 1912 |
int aowl_conn_final(void) | 1967 |
int aowl_serve_one(AowlConn* c) | 1973 |
void aowl_ws_drop_locked(int32_t idx) | 2026 |
int aowl_ws_drop(int32_t idx) | 2055 |
void aowl_ws_drop_blocking(int32_t idx) | 2061 |
DWORD WINAPI aowl_net_worker(LPVOID param) | 2066 |
void aowl_watch_remove(int32_t at) | 2184 |
void aowl_net_accept_from(SOCKET listener, int useTls, ULONGLONG now) | 2196 |
DWORD WINAPI aowl_net_poller(LPVOID param) | 2237 |
int32_t aowl_net_reserve(int32_t port) | 2507 |
int32_t aowl_net_listen_plain(int32_t port) | 2528 |
int32_t aowl_net_port_holder(int32_t port, char* out, int32_t outLen) | 2565 |
int32_t aowl_net_serve_impl(int32_t port, int32_t workers) | 2637 |
int32_t aowl_net_serve(int32_t port, int32_t workers) | 2709 |
int32_t aowl_net_serve_tls(int32_t port, int32_t workers, const char* certPath, const char* keyPath) | 2720 |
int32_t aowl_net_tls_ready(void) | 2734 |
char aowl_net_tls_error(void) | 2735 |
int32_t aowl_net_tls_error_len(void) | 2736 |
int32_t aowl_net_tls_gencert(const char* openssl, const char* certPath, const char* keyPath) | 2740 |
void aowl_net_stop(void) | 2745 |
int32_t aowl_net_running(void) | 2780 |
int32_t aowl_zlib_bound(int32_t srcLen) | 2788 |
int32_t aowl_zlib_deflate(const void* src, int32_t srcLen, void* dst, int32_t dstCap) | 2827 |
int32_t aowl_zlib_inflate(const void* src, int32_t srcLen, void* dst, int32_t dstCap) | 2860 |
int32_t aowl_zlib_looks_framed(const void* p, int32_t len) | 2873 |