Shared Memory Transport

Zero-copy inter-process communication for participants on the same host.

Overview

The SHM transport bypasses the network stack entirely, using memory-mapped segments and futex/WaitOnAddress synchronization for sub-microsecond latency.

Performance targets:

• Writer push: < 200 ns
• Reader poll: < 100 ns
• End-to-end (with wake): < 1 us

Platform Support

Platform	Segment API	Synchronization	Status
Linux	shm_open / mmap	futex(FUTEX_WAIT/WAKE)	Production
Windows 8+	CreateFileMappingW / MapViewOfFile	WaitOnAddress / WakeByAddressAll	Production
macOS	N/A	N/A	Not supported (use UDP)

How It Works

Process A (Writer)              Shared Memory              Process B (Reader)
     |                     +-------------------+                |
     |--- push(data) ----->| Ring Buffer       |                |
     |                     | [slot0][slot1]... |<--- poll() ----|
     |--- futex_wake() --->| Control Block     |--- wake ------>|
     |                     +-------------------+                |

1. Writer allocates a shared memory segment (/hdds_d{domain}_w{guid})
2. Writer pushes data into a lock-free ring buffer using atomic operations
3. Writer wakes readers via futex (Linux) or WakeByAddressAll (Windows)
4. Reader discovers the segment via SEDP user_data (shm=1;host_id=...)
5. Reader opens the segment and polls the ring buffer

Configuration

SHM Policy

use hdds::{Participant, TransportMode};
use hdds::transport::shm::ShmPolicy;

// Prefer SHM when available, fallback to UDP (default)
let p = Participant::builder("app")
    .shm_policy(ShmPolicy::Prefer)
    .build()?;

// Require SHM -- fail if not on same host
let p = Participant::builder("local_only")
    .shm_policy(ShmPolicy::Require)
    .build()?;

// Disable SHM -- always use UDP
let p = Participant::builder("udp_only")
    .shm_policy(ShmPolicy::Disable)
    .build()?;

Policy	Description
Prefer	Use SHM when conditions are met, fallback to UDP (default)
Require	Fail if SHM conditions are not met
Disable	Never use SHM, always use network transport

Requirements for SHM

SHM is automatically selected when all conditions are met:

1. Both participants are on the same host (matching host_id)
2. Both endpoints use BestEffort QoS (SHM does not support retransmission)
3. Remote participant advertises SHM capability in SEDP user_data
4. SHM policy is Prefer or Require (not Disable)

When any condition is not met, HDDS falls back to UDP transparently.

C API

#include <hdds.h>

HddsParticipantConfig* cfg = hdds_config_create("my_app");
hdds_config_set_domain_id(cfg, 0);

// Set SHM policy
hdds_config_set_shm_policy(cfg, HDDS_SHM_PREFER);

// Combine with transport preference
hdds_config_set_transport_preference(cfg, HDDS_TRANSPORT_PREF_SHM_PREFERRED);

HddsParticipant* p = hdds_config_build(cfg);

C Enums

typedef enum {
    HDDS_SHM_PREFER  = 0,   // Prefer SHM, fallback to UDP
    HDDS_SHM_REQUIRE = 1,   // Require SHM, fail if unavailable
    HDDS_SHM_DISABLE = 2,   // Disable SHM, always use network
} HddsShmPolicy;

Monitoring

SHM Metrics

use hdds::transport::shm_global_metrics;

let metrics = shm_global_metrics();
println!("SHM writes: {}", metrics.writes);
println!("SHM reads: {}", metrics.reads);
println!("Fallback to UDP: {}", metrics.fallback_udp);
println!("Overruns: {}", metrics.overruns);

hdds-shm-viewer

Inspect live SHM segments:

# Linux: scan /dev/shm for HDDS segments
hdds-shm-viewer

# Windows: inspect a specific segment
hdds-shm-viewer --segment hdds_d0_w0102030405060708090a0b0c0d0e0f10

# JSON output
hdds-shm-viewer --json

# Watch mode (refresh every second)
hdds-shm-viewer --watch

Architecture Details

Segment Naming

Platform	Format	Example
Linux	/hdds_d{domain}_w{guid_hex}	/hdds_d0_w0102030405060708090a0b0c0d0e0f10
Windows	Local\hdds_d{domain}_w{guid_hex}	Local\hdds_d0_w0102030405060708090a0b0c0d0e0f10

Ring Buffer Layout

+------------------+  offset 0
| ShmControl       |  64 bytes (cache-line aligned)
|   head: AtomicU64|  -- writer's current position
|   capacity: u32  |  -- number of slots (power of 2)
|   magic: u32     |  -- 0xDD5F0001
+------------------+  offset 64
| ShmSlot[0]       |  4160 bytes each
|   seq: AtomicU64 |  -- sequence number (odd = writing)
|   len: AtomicU32 |  -- payload length
|   payload: [u8]  |  -- 4096 bytes
+------------------+
| ShmSlot[1]       |
| ...              |
| ShmSlot[N-1]     |
+------------------+

Synchronization

• Writes: Release ordering ensures payload is visible before sequence commit
• Reads: Acquire ordering ensures sequence is read before payload access
• ABA prevention: Sequence numbers are monotonically increasing, never reused
• Torn read detection: Odd sequence = write in progress, reader retries

Troubleshooting

SHM Not Being Used

Check that both participants advertise SHM capability:

# Look for shm=1 in discovery dump
hdds-discovery-dump --domain 0 | grep shm

Common causes:

• Different hosts (check host_id match)
• One endpoint uses Reliable QoS
• SHM policy set to Disable

Segment Cleanup (Linux)

Stale segments from crashed processes can accumulate in /dev/shm:

# List HDDS segments
ls /dev/shm/hdds_*

# Clean up stale segments for domain 0
hdds-shm-viewer --cleanup --domain 0

On Windows, the OS automatically cleans up segments when all handles are closed.

Limitations

Limitation	Description
Same host only	SHM requires participants on the same machine
BestEffort only	SHM does not support reliable QoS (no retransmission)
No macOS	macOS is not supported (use UDP)
4 KB payload	Each ring buffer slot holds up to 4096 bytes
Stale segments	Crashed processes may leave segments in /dev/shm (Linux)

Next Steps

• TCP Transport -- Reliable transport for remote peers
• QUIC Transport -- Modern NAT-friendly alternative
• Advanced Transport Features -- DSCP, filtering, TSN