This article was last updated on October 25, 2023, 4 months ago. The content may be out of date.
To improve efficiency, caddy tries to reuse sockets between configuration reloads. For Windows systems caddy has to wrap the underlying socket and set deadlines to terminate old use of the socket. For unix systems caddy applies the
SO_REUSEPORT parameter when creating the socket. For counting the number of uses of the socket, caddy also wraps these sockets.
For a long time, caddy has a buggy reuse implementation for packet type sockets. It will never terminate old uses of the socket. The problem is discovered in caddy-l4.
quic-go has a number of optimizations for the
net.PacketConn it uses to create the
http3.QUICEarlyListener. It supports
*net.UnixConn. Since caddy wraps them to reuse the sockets or for statics purposes, we have to implement some of the methods required to gain these optimizations. It became unwieldy after the introduction of GSO. Not only does the packet connection need to implement
SyscallConn method, it also needs to implement
net.Conn interface because the library quic-go uses lazily type asserts it to one even none of the related methods are required. It is very messy.
For some reason, caddy needs a
net.PacketConn when creating the
http3.QUICEarlyListener. The problem manifests when the socket reuse bug is fixed. Now all packet connections will correctly terminate when the configuration is reloaded. Because
http3.QUICEarlyListener still refers to the already “closed”
net.PacketConn, it will fail to work.
Even if we managed to replace the underlying socket used by
http3.QUICEarlyListener, unless it’s on Windows platform, socket optimizations will not be redone on the new socket. Tested when specifying
SO_REUSEPORT and then changing read and write buffer sizes on a linux system.
http3.QUICEarlyListener just stops reading from and writing to the socket, the socket is left as is.
Instead of separating
http3.QUICEarlyListener, these two belong together.
http3.QUICEarlyListener depends on
net.PacketConn to serve http3. Since quic-go won’t close the underlying socket, we’ll need to do it ourselves.
quic-go has a very good reason why
net.PacketConn is not closed when shutting down the server. Standard library http server uses tcp. Tcp listening sockets will create new sockets whenever
Accept succeeds. Closing these sockets won’t affect the listening socket and vice versa.
quic-go uses datagram sockets. This type of socket doesn’t have the notion of
Accept, and all the message exchanges use this socket. quic-go is responsible for maintaining the state machine and distributing messages to relevant handlers.
There is another problem. Caddy wraps the
net.PacketConn. This type can’t be optimized by quic-go directly. Drawing inspiration from standard library’s handling of
http.ResponseWriter interface discovery, we can implement the
Unwrap interface on the wrapped socket to return the underlying socket, and then it can be optimized.
Caddy’s network listening functions allow the easy reuse of the underlying socket. They can also be configured to change the unix socket permissions.
When creating a new
http3.QUICEarlyListener, caddy goes through the following steps:
http3.QUICEarlyListener is an interface that can be used to serve http3,
quic-go.EarlyListener is a concrete type provided by quic-go. Caddy wraps this type to reuse it.
We have to introduce a new method though. Since
ListenQUIC is marked as experimental, there is no guarantee that this method will work across caddy versions. Deprecating the method is acceptable.
When caddy http3 was fixed in c9b5e7f, it didn’t close the
net.PacketConn when the quic listener is destroyed. It didn’t expose the underlying socket for performance optimization either.