- Home
- Insights
Distributed systems
Reliable distributed communication systems
Publish/subscribe middleware such as DDS and Zenoh makes distribution easy to start and hard to get right. Data models, QoS and failure behaviour need to be designed, not defaulted.
- Published
- Reading time
- 4 min read
Data-centric middleware has changed how distributed embedded systems are built. With DDS implementations such as Eclipse Cyclone DDS, or with Eclipse Zenoh, nodes discover each other automatically, publish typed data and receive it wherever it is needed. A first prototype can be running in an afternoon.
The difficulty arrives later. Under real load, on real networks, with nodes that restart and links that degrade, the system’s behaviour is determined by dozens of decisions that were never consciously made. This article covers the ones that matter most.
The data model is the architecture
In a publish/subscribe system, components do not call each other; they share data. The topics, types and keys therefore are the interfaces of the system, and they deserve the same care as an API.
A few principles hold up well in practice:
- Model state and events differently. A topic that represents the current state of something (a sensor’s latest reading, an actuator’s set-point) behaves differently from a topic that carries discrete events (a command, an alarm). Mixing them leads to QoS settings that are right for neither.
- Use keys deliberately. In DDS, a key turns a topic into a set of independent instances: one per vehicle, per device, per channel. Keyed topics allow per-instance history and lifecycle; without keys, every update overwrites every other.
- Plan for evolution. Types will change. Decide early how: extensible types and optional members, versioned topic names, or both. Changing a type that dozens of nodes depend on without such a plan requires a coordinated big-bang update.
Quality of service is a specification
DDS exposes a rich set of quality-of-service policies; Zenoh offers its own controls over reliability, congestion and priority. Defaults are chosen to make examples work, not to make your system correct. Each data flow should have an explicit, reviewed profile.
For DDS the most consequential policies are:
| Policy | Question it answers |
|---|---|
| Reliability | Must every sample arrive, or is the latest value sufficient? |
| Durability | Should a late-joining subscriber receive data published earlier? |
| History | How many samples per instance are kept, and what happens to the rest? |
| Deadline | How often must new data arrive before its absence is an error? |
| Liveliness | How quickly must the loss of a publisher be detected? |
| Resource limits | What bounds memory use when subscribers fall behind? |
Combinations matter as much as individual settings. Reliable delivery with unbounded history to a slow subscriber is a memory leak with extra steps. Best-effort delivery of commands that must not be lost is a latent defect. Writing the profile down per flow, with its rationale, surfaces these mismatches during design rather than during field trials.
Design for the network you will actually have
Laboratory networks are fast, quiet and reliable. Deployed networks are not. Multicast may be unavailable or filtered, Wi-Fi and cellular links lose packets in bursts, and bandwidth must be shared with traffic you do not control.
Practical consequences:
- Discovery must be configured for the deployment: static peers or a discovery service where multicast is unavailable, and sensible limits on what is advertised where.
- Partitioning of the data space (DDS domains and partitions, Zenoh key expressions) limits how far each piece of data travels, which matters for both bandwidth and security.
- Bridging between local real-time networks and wider-area links is often best done at an explicit gateway, where rates can be reduced, data filtered and failures isolated. This is a natural role for Zenoh in front of a DDS-based subsystem.
Make failure behaviour observable and tested
Every distributed system will experience node restarts, network partitions and overload. The question is whether its behaviour in those conditions is specified and verified.
Useful questions for each critical data flow:
- What does the subscriber do when data stops arriving? Is the deadline monitored, and is there a defined safe reaction?
- What happens when a publisher restarts? Does state recover through durability, or must it be re-sent?
- How does the system behave when one subscriber is much slower than the others?
- Which middleware status events are logged and monitored in production?
These questions translate directly into integration tests: kill a process, impair a link with packet loss and latency, saturate a topic. Run them automatically, not once before a milestone.
Summary
Data-centric middleware moves complexity from code into configuration and data design. That is a good trade, provided the data model is treated as architecture, QoS as specification, and failure behaviour as something to be tested rather than assumed.