Common Issues

Quick solutions to frequently encountered HDDS problems.

Discovery Issues

No Participants Found

Symptom: Participants don't see each other, no matched endpoints.

Diagnosis:

# Check SPDP traffic
tcpdump -i any -n udp port 7400

# Enable discovery logging
export RUST_LOG=hdds::discovery=debug

Solutions:

Cause	Solution
Different domain ID	Use same ROS_DOMAIN_ID or domain parameter
Firewall blocking	Open UDP 7400-7500
Multicast disabled	Enable multicast or use static discovery
Wrong interface	Set HDDS_INTERFACE=eth0

// Force specific interface
let config = DomainParticipantConfig::default()
    .interface("192.168.1.0/24");

Endpoints Not Matching

Symptom: Participants discovered but no writer/reader matches.

Diagnosis:

# Check endpoint details
export RUST_LOG=hdds::discovery=trace

Solutions:

Cause	Solution
Topic name mismatch	Check spelling, case sensitivity
Type name mismatch	Use identical IDL definitions
QoS incompatible	Writer reliability ≥ Reader
Partition mismatch	Same partition or empty

// Verify QoS compatibility
let writer_qos = DataWriterQos::default()
    .reliability(Reliability::Reliable);  // Compatible with Reliable readers

let reader_qos = DataReaderQos::default()
    .reliability(Reliability::BestEffort);  // Incompatible!

Slow Discovery

Symptom: Takes several seconds to discover participants.

Solutions:

let config = DomainParticipantConfig::default()
    // Faster initial announcements
    .initial_announcement_period(Duration::from_millis(50))
    .initial_announcement_count(10)
    // Shorter lease
    .lease_duration(Duration::from_secs(10));

For known peers:

let config = DomainParticipantConfig::default()
    .initial_peers(vec!["192.168.1.100:7400".parse()?]);

Communication Issues

Messages Not Received

Symptom: Writer succeeds but reader receives nothing.

Diagnosis:

# Check if data reaches network
tcpdump -i any -n udp port 7410

# Check reader status
reader.subscription_matched_status();

Solutions:

Cause	Solution
No match	Check discovery issues
History full	Increase history_depth
Deadline missed	Check timing requirements
Network loss	Enable reliability

Data Loss / Gaps

Symptom: Some messages missing, sequence gaps in received data.

Solutions:

// Use reliable delivery
let qos = DataWriterQos::default()
    .reliability(Reliability::Reliable {
        max_blocking_time: Duration::from_secs(1),
    })
    .history(History::KeepLast { depth: 100 });

// Matching reader
let qos = DataReaderQos::default()
    .reliability(Reliability::Reliable)
    .history(History::KeepLast { depth: 100 });

Writer Blocks / Timeout

Symptom: write() blocks or returns timeout error.

Causes:

1. Reliable delivery, reader not acknowledging
2. History cache full
3. Network congestion

Solutions:

// Increase history depth
let qos = DataWriterQos::default()
    .history(History::KeepLast { depth: 1000 })
    .reliability(Reliability::Reliable {
        max_blocking_time: Duration::from_millis(100),
    });

// Or use best effort for high-rate data
let qos = DataWriterQos::default()
    .reliability(Reliability::BestEffort);

Late Joiner Misses Data

Symptom: Subscriber that starts late doesn't receive historical data.

Solution: Use TransientLocal durability:

// Writer
let qos = DataWriterQos::default()
    .durability(Durability::TransientLocal)
    .history(History::KeepLast { depth: 10 });

// Reader
let qos = DataReaderQos::default()
    .durability(Durability::TransientLocal);

Type Issues

Type Mismatch Error

Symptom: TypeConsistency check failed error.

Solutions:

1. Regenerate types from same IDL:

hdds-gen -l rust my_types.idl

2. Use compatible extensibility:

@appendable  // Or @mutable
struct SensorData {
    uint32 sensor_id;
    float value;
};

3. Enable type coercion:

let qos = DataReaderQos::default()
    .type_consistency(TypeConsistency::AllowTypeCoercion);

Serialization Error

Symptom: Serialization failed or Invalid CDR data.

Causes:

• Incompatible types between writer and reader
• Corrupted network data
• Endianness mismatch

Solutions:

# Check wire format
export RUST_LOG=hdds::serialization=debug

# Verify type hash matches
hdds-gen -l rust --show-type-hash my_types.idl

Memory Issues

Out of Memory

Symptom: Process crashes with OOM or OutOfResources error.

Diagnosis:

# Monitor memory usage
watch -n 1 'ps -o rss,vsz,pid,cmd -p $(pgrep my_app)'

Solutions:

// Limit resource usage
let qos = DataReaderQos::default()
    .resource_limits(ResourceLimits {
        max_samples: 1000,
        max_instances: 100,
        max_samples_per_instance: 10,
    })
    .history(History::KeepLast { depth: 10 });

Memory Leak

Symptom: Gradual memory growth over time.

Common causes:

1. History grows unbounded (KeepAll)
2. Instances not disposed
3. Listeners holding references

Solutions:

// Use KeepLast instead of KeepAll
let qos = DataReaderQos::default()
    .history(History::KeepLast { depth: 100 });

// Dispose instances when done
writer.dispose(&sample, handle)?;

// Unregister instances
writer.unregister_instance(&sample, handle)?;

QoS Issues

Incompatible QoS

Symptom: InconsistentQosPolicy error or no match.

Rules:

Policy	Compatibility
Reliability	Writer ≥ Reader
Durability	Writer ≥ Reader
Deadline	Writer ≤ Reader
Ownership	Writer = Reader

Example fix:

// Reliable writer can match both reliable and best-effort readers
let writer_qos = DataWriterQos::default()
    .reliability(Reliability::Reliable);

// But best-effort writer cannot match reliable reader
let reader_qos = DataReaderQos::default()
    .reliability(Reliability::Reliable);  // Won't match best-effort writer

Deadline Missed

Symptom: OfferedDeadlineMissed or RequestedDeadlineMissed callbacks.

Solutions:

// Increase deadline period
let qos = DataWriterQos::default()
    .deadline(Duration::from_millis(200));  // Was 100ms

// Or remove deadline constraint
let qos = DataWriterQos::default();  // No deadline

Liveliness Lost

Symptom: LivelinessLost callback, reader thinks writer is gone.

Solutions:

// Use automatic liveliness
let qos = DataWriterQos::default()
    .liveliness(Liveliness::Automatic {
        lease_duration: Duration::from_secs(10),
    });

// For manual liveliness, assert regularly
loop {
    writer.assert_liveliness()?;
    sleep(Duration::from_millis(500));
}

Network Issues

Multicast Not Working

Symptom: Same-host works, cross-host fails.

Diagnosis:

# Test multicast
ping -c 3 239.255.0.1

# Check route
ip route show | grep multicast

Solutions:

# Add multicast route
sudo ip route add 239.0.0.0/8 dev eth0

# Or use unicast discovery
export HDDS_MULTICAST_DISABLE=1

Firewall Blocking

Symptom: No network traffic seen.

Solutions:

# Open discovery ports
sudo ufw allow 7400:7500/udp

# Or with iptables
sudo iptables -A INPUT -p udp --dport 7400:7500 -j ACCEPT

Build Issues

hdds-gen Not Found

# Install from cargo
cargo install hdds-gen

# Or build from source
cargo build --release -p hdds-gen
export PATH=$PATH:./target/release

Rust Version Too Old

Symptom: Compilation errors about missing features.

Solution:

# Update Rust
rustup update stable

# Check version (need 1.75+)
rustc --version

Missing OpenSSL

Symptom: Cannot find -lssl.

Solution:

# Debian/Ubuntu
sudo apt install libssl-dev

# Fedora
sudo dnf install openssl-devel

# macOS
brew install openssl
export OPENSSL_DIR=$(brew --prefix openssl)

Runtime Issues

Panic: Already Borrowed

Symptom: RefCell borrow panic in callback.

Solution: Don't hold references across callbacks:

// Bad: holding reference during callback
let data = some_data.borrow();
reader.take();  // May panic

// Good: drop before callback
{
    let data = some_data.borrow();
    // use data
}  // dropped here
reader.take();  // Safe

Thread Panic

Symptom: thread panicked message.

Diagnosis:

export RUST_BACKTRACE=1
./my_app

Error Codes Reference

Error	Meaning	Solution
AlreadyDeleted	Entity was deleted	Check lifecycle
BadParameter	Invalid argument	Check inputs
ImmutablePolicy	Can't change QoS	Set before enable
InconsistentPolicy	QoS conflict	Check QoS rules
NotEnabled	Entity not enabled	Call enable()
OutOfResources	Memory/limit hit	Increase limits
PreconditionNotMet	Invalid state	Check entity state
Timeout	Operation timed out	Increase timeout
Unsupported	Feature not available	Check feature flags

Next Steps

• Debug Guide - Detailed debugging
• Performance Issues - Performance problems
• Error Codes - Complete error reference