Matrix Service Design

This page proposes a service architecture for the Drupal side of the Matrix integration. It is a proposal, not a decision, and it is deliberately framed as a delta on the existing matrix_api modules rather than a greenfield design — most of the pieces already exist in some form. See Matrix for module status and Matrix Integration Topology & Gaps for the architecture review this design responds to.

Throughout, services are marked exists (already in the matrix_api modules), restructure (exists, but needs reshaping), or new.

Service topology

flowchart LR
  synapse[("Synapse<br/>homeserver")]
  drupal[("Drupal<br/>entities")]

  subgraph ingress["Ingress — Matrix to Drupal"]
    direction LR
    recv["Transaction Receiver<br/>auth · dedup · enqueue · 200"]
    ledger["Transaction<br/>Ledger"]
    iq[("Ingress<br/>Queue")]
    iworker["Ingress Worker<br/>cron + Drush daemon"]
    norm["Event<br/>Normalizer"]
    router["Event Router<br/>relevance filter"]
    apply["Inbound<br/>Applicator"]
    recv --> iq --> iworker --> norm --> router --> apply
    recv --- ledger
  end

  subgraph egress["Egress — Drupal to Matrix"]
    direction LR
    osync["Outbound<br/>Sync"]
    eq[("Egress<br/>Queue")]
    eworker["Egress<br/>Worker"]
    rate["Rate<br/>Limiter"]
    breaker["Circuit<br/>Breaker"]
    client["Matrix<br/>Client"]
    osync --> eq --> eworker --> rate --> breaker --> client
  end

  subgraph shared["Mapping — consulted by both directions"]
    direction LR
    emap["Event Map"]
    imap["Identity Map"]
    rres["Room Resolver"]
    guard["Loop Guard"]
  end

  subgraph policy["Provisioning and Policy"]
    direction LR
    prov["Room<br/>Provisioner"]
    ghost["Ghost<br/>Manager"]
    power["Power Level<br/>Policy"]
  end

  telemetry["Telemetry<br/>queue depth · lag"]

  synapse ==>|"AS push"| recv
  apply ==> drupal
  drupal ==>|"entity hooks"| osync
  client ==> synapse

  router -.-> shared
  eworker -.-> shared
  policy --> client
  iq -.-> telemetry
  eq -.-> telemetry

  classDef newsvc stroke-dasharray: 5 5;
  class iq,iworker,eq,eworker,rate,breaker,guard,rres,prov,ghost,telemetry newsvc;

Dashed borders mark services that do not exist today; solid borders are services that exist or are a restructure of something that does. Thick arrows are the hot paths; dotted arrows are consultation and observation.

Deployment boundaries

A recurring question about this design is where the queue manager lives. The answer: the message broker is the only non-Drupal component. Every service above is Drupal-side PHP. What differs is which process runs it — and the important split is that the workers are long-running CLI processes, not web requests.

flowchart TB
  subgraph external["External infrastructure"]
    direction LR
    synapse["Synapse homeserver<br/>(own host, own Postgres)"]
    broker["RabbitMQ broker<br/>(already running at drupal.org)"]
  end

  subgraph web["Drupal web tier — PHP-FPM, inside the HTTP request"]
    recv["Transaction Receiver"]
  end

  subgraph workers["Drupal worker tier — long-running CLI processes"]
    iworker["Ingress Worker<br/>(Drush daemon under systemd)"]
    eworker["Egress Worker<br/>(Drush daemon under systemd)"]
    cron["Cron<br/>(safety net drain)"]
  end

  db[("Drupal database<br/>entities + event/identity maps")]

  synapse -->|"AS transaction push"| recv
  recv -->|"publish + await confirm"| broker
  broker -->|"consume"| iworker
  broker -->|"consume"| eworker
  cron -.->|"backup drain"| broker
  iworker --> db
  eworker -->|"Matrix Client"| synapse
  db -->|"entity hooks publish"| broker

Three consequences worth stating plainly:

• The web tier does almost nothing. The Transaction Receiver authenticates the request, claims the transaction ID, publishes to the broker, and returns 200. It performs no entity loads and no Matrix API calls.
• The workers are separate processes with their own lifecycle. They are Drush commands run as systemd services, scaled and restarted independently of the web tier, with cron as a safety net rather than the primary drain. Cron alone is not sufficient at chat volume — it runs on a schedule with a time limit.
• Nothing new needs to be operated. Synapse and RabbitMQ both already exist; this design adds Drupal code and systemd units, not infrastructure.

Services

Ingress — Matrix to Drupal

Service	Status	Drupal primitive	Responsibility
Transaction Receiver	restructure	Controller + Queue	Authenticate hs_token, validate shape, claim the transaction, enqueue, return 200. Does zero entity work.
Transaction Ledger	restructure	Dedicated table with a unique index	claim(txnId) as an atomic insert-or-false; garbageCollect() on cron.
Ingress Worker	new	QueueWorker plugin + Drush daemon	Drains the ingress queue.
Event Normalizer	exists	Value objects	Raw array to a typed, immutable event object; absorbs spec-version variance.
Event Router	restructure	Symfony EventDispatcher	Cheap relevance filter — right type? tracked room? known thread root? Short-circuits irrelevant traffic before any entity load.
Inbound Applicator	exists	Entity API + DB transaction	Mutates Drupal. Idempotent via the event map.

Mapping — consulted by both directions

Service	Status	Responsibility
Event Map	exists	The matrix_event_map repository. Needs a genuine unique index on event_id — it is the loop-prevention substrate.
Identity Map	exists	Drupal user to Matrix user resolution, both directions.
Room Resolver	new	Entity to room/space resolution via the polymorphic attachment, including Group to Space. Replaces the per-site room lookup hook.
Loop Guard	new	An explicit, testable service rather than scattered conditionals. Two checks: did we originate this event, and is the sender one of our own ghost users? Echo loops are the classic bridge failure mode.

Egress — Drupal to Matrix

Service	Status	Responsibility
Outbound Sync	exists, needs decoupling	Translates an entity change into a Matrix intent and enqueues it. Currently calls the client inline.
Egress Worker	new	Drains the egress queue.
Matrix Client	exists	Client-Server API wrapper; application-service auth with user impersonation; honours the server’s retry hints.
Rate Limiter	new	Token bucket, per-user and global. Synapse rate-limits aggressively; at community scale this is not optional.
Circuit Breaker	new	Homeserver unavailable means fail fast and leave items queued, with a half-open probe to resume.

Provisioning, policy, and cross-cutting

Service	Status	Responsibility
Room Provisioner	new	Create and upgrade rooms and spaces from room-type templates; handle replacement; idempotent.
Ghost Manager	new	Ensure application-service ghost users exist for Drupal accounts.
Power Level Policy	partially exists	Map Drupal group roles to Matrix power levels on membership change.
Telemetry	new	Queue depth and lag. Operating this at community scale without it is not realistic — “is the queue draining fast enough” has to be answerable from a dashboard.

Queue store

RabbitMQ, not Redis — and drupal.org already runs it. drupal/rabbitmq is a real dependency of the drupal.org site codebase, and five queues are already routed to it through Drupal’s per-queue backend override:

$settings['queue_service_drupalorg_security_issue_webhook_queue_worker'] = 'queue.rabbitmq.default';
$settings['queue_service_drupalorg_project_activity_webhook_queue_worker'] = 'queue.rabbitmq.default';
$settings['queue_service_drupalorg_contribution_activity_webhook_queue_worker'] = 'queue.rabbitmq.default';
$settings['queue_service_drupalorg_issue_forks_queue_worker'] = 'queue.rabbitmq.default';
$settings['queue_service_drupalorg_doc_edit_queue_worker'] = 'queue.rabbitmq.default';

Those existing queues are structurally the same problem this design faces: absorb an external event quickly, process it later. Adding Matrix means two more lines, not new infrastructure:

$settings['queue_service_matrix_ingress'] = 'queue.rabbitmq.default';
$settings['queue_service_matrix_egress'] = 'queue.rabbitmq.default';

Drupal’s QueueFactory checks queue_service_<queue_name> per named queue, so each queue selects its own backend with no code change.

Without those lines the default applies, which is Drupal\Core\Queue\DatabaseQueue — the queue table in the main site database. That is the wrong place for chat-volume traffic: every claim is a sorted select plus a lease update against a single hot table, competing with page-serving queries on the same database.

One operational detail inherited from the existing broker configuration: the connection sets a short read/write timeout, so a slow or unreachable broker surfaces as an exception inside the receiver. The receiver must therefore return a 5xx and let Synapse retry — never return 200 and drop the transaction.

Durability

This is the constraint that shapes the whole ingress path.

Once the receiver returns 200 for an application-service transaction, Synapse considers those events delivered and will never resend them. From that moment the ingress queue is the sole custodian of data that exists nowhere else reachable. Losing the queue means losing those events permanently.

That imposes three requirements together — all three, or the guarantee does not hold:

1. A durable queue.
2. Persistent message delivery mode.
3. Publisher confirms.

And it imposes an ordering rule: publish, await the broker’s confirm, then return 200. Acknowledging Synapse before the broker has confirmed the publish opens a window in which an event is silently and permanently lost.

This is also the argument against Redis for the ingress queue specifically: under its common default configuration it is not durable enough to be the sole custodian of unrecoverable data.

Design decisions

Two queues, not one. Ingress and egress have independent failure modes. Sharing a queue means a homeserver outage parks failing egress items at the head and blocks inbound comment creation — head-of-line blocking for a reason unrelated to inbound.

Egress being asynchronous matters more than ingress. A synchronous outbound call means Matrix downtime breaks content editing on drupal.org: a node save blocking on an HTTP round-trip to a homeserver that is not answering. That is a worse failure than delayed inbound sync.

The application-service endpoint stays O(1). Authenticate, claim, publish, return. This is what lets it survive bursts, and it is the concrete fix for the write-behind gap.

Per-room causal ordering is a real hazard. Matrix events form a causally ordered graph. Parallelising the drain naively allows a thread reply to be applied before the root it hangs from exists. Two viable strategies: partition by room so a worker claims a room rather than an item, preserving per-room order while parallelising across rooms; or defer and retry, using Drupal’s DelayedRequeueException to postpone an item whose root is missing without failing it or blocking others. The first is cleaner. Core’s Queue API provides neither for free, so this needs deciding before the worker is written.

For the egress side, Drupal core already provides the idiomatic circuit-breaker integration point: throwing SuspendQueueException from a queue worker tells cron to abandon that queue for the run without marking items failed.

What this closes, and what stays open

Closed by this design: write-behind in both directions; event_id uniqueness, enforced in the event map; reaping of expired transaction-dedup records; and backpressure, via the rate limiter and circuit breaker.

Still open:

• Cross-entity room ID uniqueness — room identifiers must be unique across both the room and space entity types, which Drupal cannot enforce with a database constraint. It needs an application-level validator in the Room Provisioner.
• Homeserver capacity planning — worker topology and database tuning for a given concurrent-user target. An operations question this service layer cannot answer.
• Whether provisioning needs its own queue. Room and space provisioning shares the homeserver dependency with egress, so it could ride the same queue — but provisioning stuck behind a large message backlog means “create this group’s room” waits on chat traffic, which is the wrong priority order. Either a third queue or a priority lane; worth deciding explicitly rather than defaulting.