Socket API

The Aegis socket layer bridges userspace networking syscalls to the kernel’s TCP/UDP/Unix transport implementations. It provides three address families (AF_INET TCP, AF_INET UDP, AF_UNIX), an epoll I/O multiplexer, and VFS integration so sockets can be used with read(), write(), and close().

v1 maturity notice: The socket layer is the boundary between untrusted userspace and the kernel’s network stack. All code here is C, operating on user-supplied addresses and buffer pointers (with SMAP enforcement via copy_from_user/copy_to_user). As v1 software, this code has not been audited and likely contains exploitable bugs at the syscall boundary – privilege escalation via crafted sockaddr structures, fd table corruption through race conditions, or information leaks through uninitialized buffer contents. These are expected realities for a from-scratch OS at this stage, not hypothetical concerns. Contributions are welcome – file issues or propose changes at exec/aegis.

AF_INET Socket Table (socket.c)

Socket Structure

#define SOCK_TABLE_SIZE  AEGIS_SOCK_TABLE_SIZE   /* 128, from limits.h */
#define SOCK_NONE        0xFFFFFFFFU

typedef struct {
    sock_state_t  state;
    /* Reference count: one per fd pointing at this slot. dup/dup2/fcntl
     * F_DUPFD and fork each fire sock_vfs_dup → refcount_inc;
     * sock_vfs_close → refcount_dec_and_test and only tears down the slot
     * on the last close. Without this, dup-then-close / fork-then-close
     * double-freed the slot (cross-connection UAF). */
    refcount_t    refcount;
    uint8_t       type;          /* SOCK_TYPE_STREAM (1) or SOCK_TYPE_DGRAM (2) */
    uint8_t       nonblocking;
    ip4_addr_t    local_ip;
    uint16_t      local_port;
    ip4_addr_t    remote_ip;
    uint16_t      remote_port;
    uint32_t      tcp_conn_id;   /* index into tcp_conn table; SOCK_NONE if none */
    /* accept queue: ring of completed tcp_conn_id values */
    uint32_t      accept_queue[8];
    uint8_t       accept_head, accept_tail;
    /* UDP receive ring */
    udp_rx_slot_t udp_rx[UDP_RX_SLOTS];
    uint8_t       udp_rx_head, udp_rx_tail;
    /* epoll back-reference */
    uint32_t      epoll_id;      /* SOCK_NONE = not watched */
    uint64_t      epoll_events;
    /* options */
    uint8_t       reuseaddr;
    uint8_t       broadcast;
    uint32_t      rcvtimeo_ticks;
    uint32_t      sndtimeo_ticks;
    /* Wake queue for ALL waiters on this fd — blocking accept/connect/recv
     * (via wait_event) AND sys_poll / sys_epoll_wait. The old single-slot
     * waiter_task pointer has been replaced by this waitq so that multiple
     * tasks blocked on the same socket are all woken correctly. */
    waitq_t       poll_waiters;
} sock_t;

Socket States

typedef enum {
    SOCK_FREE,         /* slot available */
    SOCK_CREATED,      /* allocated, not bound */
    SOCK_BOUND,        /* bound to local address */
    SOCK_LISTENING,    /* listening for connections (TCP) */
    SOCK_CONNECTING,   /* connect() in progress (TCP) */
    SOCK_CONNECTED,    /* established (TCP) */
    SOCK_CLOSED        /* connection terminated */
} sock_state_t;

Socket Lifecycle

socket()    → sock_alloc() → SOCK_CREATED
bind()      → SOCK_BOUND
listen()    → SOCK_LISTENING         [TCP server]
accept()    → new socket in SOCK_CONNECTED
connect()   → SOCK_CONNECTING → SOCK_CONNECTED  [TCP client]
close()     → sock_vfs_close() → sock_free()

VFS Integration

Each AF_INET socket is backed by a VFS file descriptor through vfs_ops_t:

static const vfs_ops_t s_sock_ops = {
    .read    = sock_vfs_read,
    .write   = sock_vfs_write,
    .close   = sock_vfs_close,
    .readdir = NULL,
    .dup     = sock_vfs_dup,
    .stat    = sock_vfs_stat,
    .poll    = NULL,
};

TCP Read (Blocking)

The TCP read path implements signal-interruptible blocking semantics via wait_event_interruptible:

static int sock_vfs_read(void *priv, void *buf, uint64_t off, uint64_t len)
{
    for (;;) {
        int avail = tcp_conn_recv(s->tcp_conn_id, NULL, 0);  /* peek */
        if (avail > 0)
            return tcp_conn_recv(s->tcp_conn_id, buf, want);

        /* Check for EOF (FIN received) */
        tcp_conn_t *tc = tcp_conn_get(s->tcp_conn_id);
        if (!tc || tc->state == TCP_CLOSE_WAIT ||
            tc->state == TCP_CLOSED || tc->state == TCP_TIME_WAIT)
            return 0;  /* EOF */

        if (s->nonblocking)
            return -EAGAIN;

        int rc = 0;
        /* interruptible so Ctrl-C against a parked recv delivers */
        wait_event_interruptible(&s->poll_waiters,
            tcp_conn_recv(s->tcp_conn_id, NULL, 0) > 0 || ..., rc);
        if (rc == -EINTR)
            return -EINTR;
    }
}

Blocking uses wait_event_interruptible on the socket’s poll_waiters waitq rather than a single waiter_task pointer. This correctly wakes all blocked tasks (not just one) and delivers EINTR when a signal arrives while blocked — allowing Ctrl-C to interrupt a stalled recv(). The old waiter_task single-slot pointer has been removed.

TCP Write (SMAP-Safe)

The write path bounces through a 1460-byte kernel staging buffer to avoid SMAP faults when passing userspace pointers to tcp_conn_send():

uint8_t s_sndbuf[1460];
while (sent < len) {
    uint32_t chunk = len - sent;
    if (chunk > 1460) chunk = 1460;
    copy_from_user(s_sndbuf, buf + sent, chunk);
    int n = tcp_conn_send(s->tcp_conn_id, s_sndbuf, chunk);
    if (n <= 0) return sent > 0 ? sent : -32;  /* EPIPE */
    sent += n;
}

UDP Receive Ring

UDP datagrams are buffered in an 8-slot ring within each socket:

#define UDP_RX_SLOTS  8
#define UDP_RX_MAXBUF 1500

typedef struct {
    uint8_t    data[UDP_RX_MAXBUF];
    uint16_t   len;
    ip4_addr_t src_ip;
    uint16_t   src_port;
    uint8_t    in_use;
} udp_rx_slot_t;

The udp_rx() handler in udp.c writes datagrams into this ring. The recvfrom syscall reads from it.

Sockaddr Layout

The kernel uses a k_sockaddr_in_t matching the musl libc struct sockaddr_in layout:

typedef struct {
    uint16_t sin_family;    /* AF_INET */
    uint16_t sin_port;      /* network byte order */
    uint32_t sin_addr;      /* network byte order */
    uint8_t  sin_zero[8];
} k_sockaddr_in_t;

_Static_assert(sizeof(k_sockaddr_in_t) == 16, "...");

UDP / musl-ABI Compatibility

Several musl libc behaviors require specific kernel-side handling that was added in v1.2.1:

1. SOCK_CLOEXEC / SOCK_NONBLOCK in type argument: musl folds these flags into the type argument of socket() (e.g., SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK). The kernel extracts the base type with type & SOCK_TYPE_MASK (which masks out both flag bits) and handles nonblocking separately. Without this, musl’s DNS resolver received EINVAL and all DNS lookups silently failed.

2. Wildcard bind(port=0) → ephemeral port assignment: bind() with port 0 assigns an ephemeral source port rather than returning an error. musl’s resolver calls bind(0.0.0.0:0) expecting auto-assignment before sending DNS queries.

3. Implicit ephemeral bind on sendto(): An unbound UDP socket that calls sendto() without a prior bind() gets an ephemeral port auto-assigned (udp_ensure_local_port()). This is the Linux behavior musl’s resolver depends on as a fallback.

4. Signal-interruptible blocking syscalls: accept(), connect(), recv(), recvfrom(), and sendto() all use wait_event_interruptible and return EINTR when a signal arrives while blocked. Before v1.2.1, these calls were uninterruptible — Ctrl-C had no effect against a stalled network operation and the process appeared to hang.

AF_UNIX Domain Sockets (unix_socket.c)

AF_UNIX sockets provide local IPC with connected, bidirectional, byte-stream semantics. They support fd passing (SCM_RIGHTS), peer credential retrieval, and VFS-backed file descriptors.

Socket Structure

#define UNIX_SOCK_MAX    AEGIS_UNIX_SOCK_MAX   /* 64, from limits.h */
#define UNIX_PATH_MAX    108
#define UNIX_BUF_SIZE    4096   /* MUST be a power of two — see ring_used() */

typedef struct {
    uint8_t        in_use;
    unix_state_t   state;
    uint8_t        nonblocking;
    char           path[UNIX_PATH_MAX];
    /* Ring buffer -- this socket's TX direction (peer reads from it) */
    uint8_t       *ring;           /* kva-allocated page */
    uint16_t       ring_head;      /* write position */
    uint16_t       ring_tail;      /* read position */
    /* Peer link */
    uint32_t       peer_id;
    /* Accept queue (listening sockets) */
    uint32_t       accept_queue[8];
    uint8_t        accept_head, accept_tail;
    /* Peer credentials */
    uint32_t       peer_pid, peer_uid, peer_gid;
    /* fd passing staging area */
    unix_passed_fd_t passed_fds[UNIX_PASSED_FD_MAX];  /* 16 slots */
    uint8_t          passed_fd_count;
    /* Refcount for dup/fork */
    refcount_t     refcount;
    /* Wake queue for ALL waiters (blocking read/write via wait_event,
     * and sys_poll/sys_epoll_wait). Replaces the old single waiter_task. */
    waitq_t        poll_waiters;
} unix_sock_t;

States

typedef enum {
    UNIX_FREE, UNIX_CREATED, UNIX_BOUND, UNIX_LISTENING,
    UNIX_CONNECTING, UNIX_CONNECTED, UNIX_CLOSED
} unix_state_t;

Name Table

Bound sockets are registered in a static name table mapping paths to socket IDs:

#define UNIX_NAME_MAX 32
typedef struct {
    char     path[UNIX_PATH_MAX];
    uint32_t sock_id;
    uint8_t  in_use;
} unix_name_t;

• name_register(): Registers a path, returns -EADDRINUSE if already bound
• name_unregister(): Removes the binding (called on socket close)
• name_lookup(): Returns the socket ID for a path, or UNIX_NONE

Connection Flow

Client                          Server
------                          ------
socket(AF_UNIX, STREAM, 0)      socket(AF_UNIX, STREAM, 0)
                                bind("/run/service.sock")
                                listen()
connect("/run/service.sock")    
  |                             accept() [blocks]
  +-- name_lookup → listener_id
  +-- Allocate server-side socket
  +-- Allocate ring buffers (2 pages)
  +-- Cross-link peer_ids
  +-- Enqueue in listener accept queue
  +-- Wake listener
  |                               |
  v                               v
UNIX_CONNECTED                  UNIX_CONNECTED (new fd)

Ring Buffer Design

Each connected Unix socket has its own TX ring buffer (a single kva_alloc_pages(1) page, 4096 usable bytes). The peer reads from this ring. The buffer size MUST be a power of two — ring_used() below uses (head - tail) & (UNIX_BUF_SIZE - 1) which only computes a true modulo for power-of-two sizes. Pre-1.0.1 the size was 4056, which silently miscounted: 8 & 4055 == 0, so a ring with 8 bytes in it looked empty and Lumen would freeze on bare metal. Fixed to 4096 in 1.0.1; do not change.

Socket A                    Socket B
+---------+                +---------+
| ring_a  | --- read ----> | peer    |
| (A's TX)|                | reads   |
+---------+                +---------+
                           | ring_b  | --- read ----> Socket A reads
                           | (B's TX)|
                           +---------+

• Write: A writes to its own ring at ring_head
• Read: A reads from peer’s ring at peer’s ring_tail

Ring buffer functions:

static uint16_t ring_used(unix_sock_t *s) {
    return (uint16_t)((s->ring_head - s->ring_tail) & (UNIX_BUF_SIZE - 1));
}
static uint16_t ring_free(unix_sock_t *s) {
    return (uint16_t)(UNIX_BUF_SIZE - 1 - ring_used(s));
}

Close and Ring Lifetime

Ring buffer ownership is carefully managed across close:

• When socket A closes but peer B is still alive: A’s ring is not freed because B still reads from it. The ring pointer remains valid even after in_use=0.
• When B subsequently closes: B frees both its own ring and A’s orphaned ring.
• This prevents use-after-free while allowing the peer to drain remaining buffered data after the sender closes.

Peer Credentials

Credentials are captured at connect() time:

/* Client credentials → server-side socket */
s_unix[server_id].peer_pid = proc->pid;
s_unix[server_id].peer_uid = proc->uid;
s_unix[server_id].peer_gid = proc->gid;

/* Server credentials → client socket (filled at accept time) */
client->peer_pid = accepting_proc->pid;

Retrieved via unix_sock_peercred():

int unix_sock_peercred(uint32_t id, uint32_t *pid, uint32_t *uid, uint32_t *gid);

fd Passing (SCM_RIGHTS)

Unix sockets support passing file descriptors between processes:

Staging (sender side):

int unix_sock_stage_fds(uint32_t peer_id, unix_passed_fd_t *fds, uint8_t count);

Copies VFS ops/priv/flags into the peer’s staging area (up to 16 fds).

Receiving (receiver side):

int unix_sock_recv_fds(uint32_t id, int *fd_out, int max_fds);

Installs staged fds into the receiving process’s fd table. Any fds that cannot be installed (fd table full) are closed via their ops->close callback.

On socket close, any unreceived staged fds are also cleaned up.

VFS Integration

AF_UNIX sockets use SMAP-safe VFS ops that bounce through a 1024-byte kernel buffer:

static int unix_vfs_read(void *priv, void *buf, uint64_t off, uint64_t len) {
    uint8_t kbuf[1024];
    int n = unix_sock_read(id, kbuf, want);
    if (n > 0) copy_to_user(buf, kbuf, n);
    return n;
}

epoll (epoll.c)

Design

The epoll implementation provides scalable I/O event notification compatible with the Linux epoll API. It supports both level-triggered and edge-triggered modes.

Structures

#define EPOLL_MAX_INSTANCES  8
#define EPOLL_MAX_WATCHES    64

typedef struct {
    uint32_t fd;
    uint32_t events;        /* interest mask (incl. EPOLLET/EPOLLONESHOT) */
    uint64_t data;
    uint8_t  in_use;
    uint8_t  oneshot_disabled;   /* EPOLLONESHOT: disarmed after first delivery */
} epoll_watch_t;

typedef struct {
    uint8_t        in_use;
    /* Reference count for dup/fork safety (same UAF class as sock_t). */
    refcount_t     refcount;
    epoll_watch_t  watches[EPOLL_MAX_WATCHES];
    uint8_t        nwatches;
} epoll_fd_t;

Event Flags

#define EPOLLIN   0x00000001U   /* readable */
#define EPOLLOUT  0x00000004U   /* writable */
#define EPOLLERR  0x00000008U   /* error */
#define EPOLLHUP  0x00000010U   /* hangup */
#define EPOLLET   0x80000000U   /* edge-triggered */

epoll_ctl

int epoll_ctl_impl(uint32_t epoll_id, int op, int fd, k_epoll_event_t *ev);

epoll_notify

epoll_notify() is a no-op stub retained for source compatibility. The notification path is now fully waitq-driven: each watched fd’s producer (TCP rx, pipe write, etc.) calls waitq_wake_all on its own poll_waiters, which wakes any registered epoll_wait caller directly. The old epoll_notify → ready-list → wake-waiter chain has been replaced.

epoll_wait

int epoll_wait_impl(uint32_t epoll_id, uint64_t events_uptr,
                    int maxevents, uint32_t timeout_ticks);

The implementation:

1. Readiness sweep: For each watched fd, query its readiness via the VFS poll op or the per-fd waitq dispatch (fd_get_waitq). This handles heterogeneous fd types (sockets, pipes, console, PTY).
2. Return immediately if any events are ready. Copy to userspace via copy_to_user().
3. Block (if no events and timeout > 0): Register a stack-allocated waitq_entry_t on each watched fd’s waitq (obtained via fd_get_waitq), plus g_timer_waitq if a deadline is set. Registrations happen before the final readiness re-check, eliminating the lost-wakeup race. Then call sched_block().
4. On wakeup: unregister all waitq entries, re-sweep readiness, return results.
5. Signal interruptibility: A pending signal between registration and sched_block() causes an EINTR return after all waitq entries are cleaned up (no leaks — entries are stack-allocated).
6. EPOLLET / EPOLLONESHOT: Edge-triggered watches are not re-armed after delivery; EPOLLONESHOT watches set oneshot_disabled until EPOLL_CTL_MOD re-arms them.

epoll Event Structure

typedef struct __attribute__((packed)) {
    uint32_t events;
    uint64_t data;    /* user data (epoll_data_t union) */
} k_epoll_event_t;

_Static_assert(sizeof(k_epoll_event_t) == 12, "matches Linux ABI");

Socket API Summary

Table Size Limits

Security Considerations

The socket layer is a critical attack surface – it processes attacker-controlled data (sockaddr structures, buffer lengths, fd numbers) in kernel context. Key v1 concerns:

• AF_UNIX fd passing: unix_sock_stage_fds() copies VFS ops pointers between processes. A bug in this path could allow arbitrary kernel function pointer injection.
• copy_from_user / copy_to_user: SMAP enforcement prevents direct kernel access to userspace memory, but the bounce-buffer pattern (1024 bytes for Unix, 1460 bytes for TCP) means buffer overflows in the staging buffers are a risk if length validation is incorrect.
• Socket table as shared mutable state: 64-slot sock_t array accessed under sock_lock from both syscall and ISR context. Lock ordering bugs or missed lock acquisitions could corrupt socket state.
• Capability checks on socket operations: socket() (for AF_INET) requires CAP_KIND_NET_SOCKET. AF_UNIX socket creation requires CAP_KIND_IPC. Both checks are enforced at the syscall boundary in sys_socket.c. A process without the relevant capability receives EPERM. Network address binding and interface configuration additionally require CAP_KIND_NET_ADMIN.

The long-term plan includes migrating the socket layer to Rust as part of the broader kernel C-to-Rust migration. The capability system (kernel/cap/lib.rs) is the first kernel component already written in Rust, establishing the FFI patterns that will be extended to networking.

Related Documentation

• Network Stack Overview – architecture and packet flow
• TCP/IP Implementation – protocol internals
• Device Drivers – NIC driver details
• Syscall Interface – syscall numbers and signatures