Design a Scalable Notification System

The problem and scope

Sending a notification seems straightforward: an event happens, and the server sends a message to a user’s phone or inbox.

In reality, three hard problems quickly emerge at scale:

• Critical alerts getting stuck behind marketing blasts: If the marketing team sends a 500,000-user promotional email at the exact moment someone is trying to log in, a single-file system will make the login code wait behind half a million discount flyers. A 30-second delay on an OTP means an abandoned login.
• Third-party delivery services slowing down or crashing: Your system doesn’t deliver messages directly to phones; it hands them off to external partners like Apple (APNs), Google (FCM), Twilio, or SendGrid. If an external partner slows down or experiences an outage, your servers can freeze waiting for a response, eventually crashing and losing unsent messages.
• Notification spam and buzzing lock screens: If five people like your photo in three seconds, your phone shouldn’t buzz five times. The system must group rapid-fire actions into a single summary, respect time zones so marketing doesn’t wake people at 3:00 AM, and drop alerts that are no longer relevant.

To solve this, the architecture is built around three rules: urgent alerts get a dedicated fast lane, messages are saved safely to storage before contacting external vendors, and delivery providers can be swapped or added without rewriting core logic.

System boundary and non-goals

• In Scope: The distributed pipeline between internal service triggers and third-party vendor handoff—including durable multi-AZ ingress, isolated priority queuing, template hydration, user preferences/quiet hours, burst debouncing, egress Token Bucket rate limiting, dual-tier retries, and click/receipt telemetry.
• Out of Scope (Non-Goals): Upstream domain trigger logic (deciding when an alert is needed), copywriting/creative design, payload end-to-end encryption, and telecommunication carrier networks.

---

Functional requirements

Core platform delivery operations

1. Accept and dispatch notifications across channels. The system supports push (iOS via APNS, Android via FCM, and desktop via Web Push), SMS (via Twilio or equivalent), and email (via SendGrid or equivalent). The channel set is pluggable.
2. Persist every accepted notification before dispatch. Once the system accepts a notification, it immediately saves it to durable storage before attempting delivery. If an external vendor (like Apple or Twilio) goes offline, the notification remains safely on disk for retry rather than being lost. Tracking database records are updated in the background so they never slow down urgent sends.
3. Route by priority to isolated processing lanes. High-priority messages (OTPs, security alerts) route to dedicated topics separate from medium (transactional) and low (promotional) traffic with independent worker pools. A promotional surge must not starve critical OTPs.
4. Retry failed deliveries via dual-tier mechanics. Transient failures retry via seconds-scale in-worker backoff for high-priority OTPs (before channel fallback) and Kafka delay topics for medium/low traffic; permanently undeliverable messages move to a Dead Letter Queue (DLQ) and the durable audit log.
5. Support click callbacks. Every notification payload includes a callback URL. When the user taps the notification, the client fires a webhook to the tracking service, recording the click event against the notification ID.
6. Group related notifications. Multiple rapid events of the same type for the same user (e.g., five likes in three seconds) collapse into a single summary notification rather than five separate alerts.
7. Enforce user preferences and opt-outs. Before dispatch, the system checks per-user, per-channel opt-in status and drops messages the user has declined.
8. Rate-limit per user and per tenant. The system enforces maximum notification frequency per user per channel and per-client API rate limits to prevent a noisy internal service from degrading the platform.
9. Support scheduled and delayed delivery. A notification can specify a future delivery time. A scheduler polls for due messages and enqueues them.
10. Track delivery lifecycle. Each notification transitions through accepted → queued → dispatched → delivered → clicked (or failed), with each state change recorded for audit.

---

Non-functional requirements and assumptions

Operational benchmarks and availability boundaries

The following operational metrics establish target performance baselines for a multi-channel notification platform handling millions of daily events across transactional and promotional workloads.

Property	Target	Rationale
Daily volume	10M push, 1M SMS, 5M email (~16M total)	Reference-design assumption based on a mid-scale consumer platform.
Sustained throughput	~185 notifications/second average	16M / 86,400s. Not a raw QPS challenge.
Peak throughput	~3,300/s during marketing bursts (5–10 min)	Promotional campaigns create short, intense surges.
Critical message latency	OTP handed to provider within 5s of acceptance (p99)	Platform handoff SLA; last-mile delivery depends on carrier networks and OS push sockets.
Transactional message latency	p95 < 30 seconds	Order confirmations, payment receipts.
Availability	99.9% for the ingestion API	A down ingestion path silently drops business-critical messages.
Durability	Zero loss of accepted notifications	Replicated multi-AZ Kafka ingestion before acknowledging callers.
Consistency	At-least-once delivery	Retries across third-party boundaries can double-send; messages must be designed to be idempotent.
Idempotency	Deduplicate by (tenant_id, event_id) with type-scoped TTL	OTP replay window ~5–10 min; campaign replay window ~24 hr.
Security	Internal API locked by AppKey/AppSecret	Prevent external spam injection into the notification pipeline.

---

The mental model and architecture

Four primitives that govern the design

1. Persist before dispatch (Zero-loss ingestion):

   • Decoupled origin (Transactional Outbox): When a business event occurs (e.g., an order is placed), the upstream service saves an “outbox” record in its own database alongside the order. An async worker forwards this to the notification gateway. This guarantees the alert is never lost, without letting notification delays or failures impact the core business transaction.
   • In-flight deduplication: The gateway tracks unique event IDs in a fast in-memory cache (Redis). If a duplicate request arrives while the first is still being processed, the gateway recognizes it and ignores it so the user never gets two identical messages.
   • Confirm only after safe storage: The gateway writes the notification to a distributed message queue (Kafka) replicated across multiple servers. It only returns 202 Accepted once the message is safely saved to disk. If writing fails, the gateway clears the event ID from cache so the caller can retry immediately.

2. Physical priority isolation (Zero starvation):

   • A single shared queue inevitably starves high-priority traffic during marketing surges. Architectural priority requires physical topic segregation over software priority flags.
   • The platform routes validated notifications to three dedicated Kafka topics: notifications.high/med/low.
   • Each lane runs independent consumer groups and isolated worker pools, guaranteeing promotional backlogs never delay OTP workers.

3. Egress rate shaping over vendor sockets:

   • Internal workers consume tens of thousands of messages per second, but external providers enforce rigid account quotas (e.g., Twilio 100 SMS/second, SendGrid dedicated IP warmup limits, APNs stream caps).
   • Unthrottled dispatches trigger HTTP 429 cascades or account suspensions. Dispatch workers enforce client-side Token Bucket rate limiters matched to vendor contracts, safely buffering bursts inside Kafka channel topics rather than overwhelming third-party sockets.

4. Bifurcated mobile payload contracts:

   • Notification Messages (title, body): Intercepted directly by the mobile OS system tray when backgrounded or killed; renders UI banners without waking app code. User taps launch the app and pass the payload.
   • Data Messages (custom key-value pairs): Delivered silently to the client background handler (onMessageReceived on Android) for background sync, cache pre-fetching, and badge updates without alerting the user.
   • Collapse & Offline Delivery: Payloads specify a Time-To-Live (TTL) and Collapse Key (apns-collapse-id for iOS, collapse_key / tag for Android). If a device is offline during bursts, cloud gateways deliver only the latest update upon reconnection; on-screen banners update in place rather than stacking alerts.

High-level architecture: The four-tier pipeline

The architecture decouples notification acceptance from third-party delivery across four tiers via a Two-Stage Topic Pipeline in Kafka:

1. Ingestion & Durable Ingress: Stateless API validates schemas, derives priority and expires_at strictly server-side from event_type and tenant policy (caller-supplied priority flags are ignored), acquires the Redis pending lease, and writes to Stage 1 Priority Topics (acks=all). High-priority traffic uses round-robin partitioning (null key) to maximize consumer thread saturation and eliminate head-of-line blocking. Bulk campaigns enter via an asynchronous Campaign Fan-Out Worker that chunks recipient batches directly into notifications.low in throttled increments to prevent gateway resource exhaustion.
2. Priority Queues & Processing Core: Consumes from Stage 1 topics by priority. Evaluates user channel opt-ins, quiet hours, and Redis debounce grouping before hydrating compact template variables into localized payloads.
3. Channel Queues & Egress Dispatch Workers: Routes hydrated tasks to Stage 2 Channel Topics segmented by channel and priority (channel.push/sms/email.high/med/low). Specialized workers apply the Adapter Pattern to translate payloads into vendor wire schemas, meter dispatches via Token Buckets to protect provider quotas (e.g., Twilio 100 SMS/s), and prune dead device tokens on provider feedback. Priority segregation ensures promotional push floods never head-of-line block OTP dispatches at the vendor socket.
4. Feedback, Delayed Retries & DLQ: Provider receipts and click beacons flow into an immutable Cassandra audit log. High-priority OTP failures execute fast in-worker retries before immediate SMS fallback; medium/low failures route through Kafka delay topics (retry.1m/5m/15m/1h), with exhausted (>4 attempts) or expired records archived to the DLQ and Notification DB.

End-to-end system topology

The topology is structured into a clean forward dispatch pipeline (the sub-5s hot path) and an out-of-band telemetry and recovery loop. The hot path flows strictly top-to-bottom, ensuring zero head-of-line blocking from asynchronous feedback or database writes.

End-to-end system topology Expand diagram

×

• The Forward Hot Path (Sub-5s Dispatch):

  • Durable Ingestion: Ingestion returns 202 Accepted only after persisting to Kafka (RF=3, acks=all, min.insync.replicas=2), keeping the hot path off the relational database while guaranteeing multi-zone durability.
  • Two-Stage Priority Isolation: Stage 1 queues by urgency (notifications.high/med/low). Stage 2 isolates channel adapters (channel.<type>.<priority>) with dedicated, over-provisioned worker pools so marketing surges never starve critical OTPs.
  • Egress Rate Shaping: Dispatch workers use token bucket limiters matched to vendor contracts (e.g., Twilio 100 SMS/s), buffering spikes in Kafka rather than triggering provider HTTP 429 cascades.

• The Out-of-Band Loop (Recovery & Telemetry):

  • Dual-Tier Retries: Transient OTP failures retry immediately in-worker before falling back to SMS; non-critical traffic cycles through Kafka delay topics. Stale messages drop via expires_at, and exhausted retries move to the DLQ.
  • Asynchronous Telemetry: Delivery webhooks, device push ACKs, and click callbacks feed Tracking Svc, updating Notification DB status and appending immutable audit records to Cassandra without contending with active dispatches.

---

Components and contracts

Service responsibility matrix

The following contract matrix outlines domain ownership, contracts, and failure containment across the architecture.

Component	Owns	Receives	Returns / Emits	Failure behavior
Notification Gateway	Auth, schema validation, deterministic notification_id, Redis dedup claim	HTTPS requests from internal callers	Returns 202 Accepted + notification_id; publishes to Kafka Priority Topics (acks=all)	Returns HTTP 429 on abusive bursts; diverts authorized bulk surges to delayed intake; HTTP 400 on bad schema.
Notification DB (PostgreSQL / MySQL)	Lifecycle status, user preferences, device tokens, scheduled triggers	Async status events; scheduled registrations	Eventual status queries; scheduled triggers	Active tokens and preferences are cached in Redis with short TTLs so database failovers never stall in-flight OTP routing.
Priority Queues (Kafka)	SLA-isolated message buffering (notifications.high/med/low)	Events from Gateway and Fan-Out worker	Messages to Processor consumer groups	Replicated topics (RF=3, acks=all, min.insync.replicas=2) guarantee zero message loss.
Notification Processor	Resolves device tokens, fans out multi-device tasks, evaluates preferences, debounces bursts, hydrates templates	Messages from Priority Queues	Dispatches hydrated tasks to Stage 2 Channel Queues	Thread pools scaled by priority lag; diverts rate-limited bulk traffic to delayed topics.
Channel Queues (Kafka)	Channel- and priority-segregated buffering (channel.<channel>.<priority>)	Hydrated payloads from Processor	Tasks to Channel Workers	Physical separation prevents provider latency swings from cross-blocking lanes.
Rate Limiter & Debounce Cache	Tenant quotas, frequency caps, debounce buffers (Redis)	Key lookups and increments	Pass/drop flags; consolidated batch payloads	In-memory execution (<1ms); sliding window counters and Sorted Set timers.
Preference Gateway	User channel opt-ins, quiet hours	Preference queries from Processor	Channel routing set; quiet-hour status	Drops opted-out alerts; parks non-critical quiet-hour messages in Redis.
Template Registry	Storage of templates (HTML, plain text, push JSON)	Template IDs and dynamic variables	Hydrated, localized message strings	Falls back to default locale on missing translation dictionary.
Channel Workers (Adapter Pattern)	Translates internal payloads into 3rd-party vendor-specific payloads (e.g., APNs HTTP/2, Twilio SMS, etc.), binds notification_id into vendor callback parameters (StatusCallback), shapes egress traffic via token buckets, and prunes dead device tokens	Hydrated tasks from Channel Queues	3rd-party provider API calls; delivery status events to Kafka	Fast in-worker retry (single 500ms backoff) for High lane before SMS fallback; throttles on provider quotas.
External Providers	Push, SMS, Email delivery (APNs, FCM, Twilio, SendGrid)	Payloads from Channel Workers	Delivery receipts to Callback Service	Decoupled via async workers; subject to egress rate limiters.
Tracking & Callback Svc	Ingestion of delivery receipts and recipient taps	Webhook callbacks and client tap beacons	Updates Notification DB; streams to Cassandra	Idempotent by (notification_id, action); redirects browser clicks to deep links.
Cassandra Audit Trail	Immutable historical audit log	Events from Tracking Service and Workers	Audit records in Cassandra	Passive consumer; lag does not block live notification dispatch.
Dual-Tier Retry Engine & DLQ	Transient failure retries and dead-letter parking	Failure events from Channel Workers	High lane: fast in-worker retry + SMS fallback; Med/Low: Kafka delay topics	Moves exhausted attempts (>4) or expired alerts to DLQ and updates DB status.

The Transactional Outbox: where it belongs in the architecture

A common architectural trap is placing an outbox database inside the notification gateway itself. In a scalable design, domain services and the notification platform have strictly separated responsibilities:

• Upstream Domain Services (Owns the Outbox): Services like Order or Billing commit their business record and an outbox event in a single local ACID transaction. A log-based Change Data Capture (CDC) tailer relays these events to the notification gateway, eliminating dual-write loss without letting notification latency affect the business flow.
• The Notification Gateway (Stateless Ingress): The gateway does not write to a database outbox. Putting a relational database in front of Kafka creates an ingestion bottleneck and polling lag for critical OTPs. Instead, the gateway verifies the payload and streams directly to Kafka (RF=3, acks=all, min.insync.replicas=2) for sub-20ms multi-AZ durability before returning 202 Accepted.

Transactional outbox pattern Expand diagram

×

---

Technology choices

Message broker: Kafka for priority topics

Pressure: High write bursts, zero-loss persistence, event replay for debugging, and hard worker pool isolation across priority lanes.

Option	Strengths	Trade-offs
Kafka (chosen)	High-throughput partitioned streaming, durable replay, sub-millisecond produce latency.	Operational complexity; lacks native per-message delays (requires delay topics).
AWS SQS	Serverless, zero maintenance, native 15-minute message delays.	No event replay; higher per-message latency than persistent streaming connections.
RabbitMQ	Native message priorities, flexible exchange routing.	Complex partition replay; requires quorum queues to match Kafka-level durability.

Decision: Kafka serves as the backbone across three physical topics (notifications.high/med/low). Dedicated topics provide physical consumer group isolation so promotional blasts cannot starve OTP workers. (At extreme scale, isolate the High lane onto dedicated broker clusters to eliminate disk and network I/O contention from bulk campaigns).

Metadata and lifecycle state: Relational database (PostgreSQL or MySQL)

Pressure: User device tokens, opt-in preferences, templates, and scheduled delivery timers require indexed relational queries, while clients need a queryable status endpoint.

Decision: A relational database serves strictly as the domain metadata and status store. It manages registered hardware tokens, preferences, templates, and scheduled timers. It is never on the hot ingestion path: Kafka acts as the high-speed write buffer, while background consumers update database delivery statuses (ACCEPTED → DISPATCHED → DELIVERED) asynchronously in batches.

Rate limiting, deduplication, and egress shaping: Redis

Pressure: Ingress rate limiting, deduplication checks, and outbound token buckets execute on every notification and must complete in sub-milliseconds to protect the sub-5-second hot path.

Decision: Redis is chosen for high-throughput, in-memory atomic operations across three hot-path concerns:

• Rate limiting: Sliding-window counters enforce per-user frequency caps and per-tenant ingress limits.
• In-flight deduplication: Event-scoped idempotency keys (SET ... NX EX) catch duplicate retries at both the ingress gateway and worker dispatch boundaries before duplicate messages can be published or sent.
• Egress rate shaping: Token bucket counters meter outbound traffic to match third-party provider quotas (e.g., Twilio 100 SMS/s) with atomic decrements.

Audit trail: Cassandra

Pressure: Delivery lifecycle events are append-only, high-volume writes that must not contend with transactional logs.

Decision: A Cassandra cluster serves as the append-only audit trail. A passive consumer tails dispatch Kafka topics and writes immutable records, completely isolating analytical writes from the primary delivery path.

Template engine: In-process library

Pressure: Passing fully rendered HTML for millions of queued notifications wastes network bandwidth.

Decision: Queued messages carry compact event variables and a template_id. The in-process template engine hydrates the payload and applies localization immediately before vendor dispatch.

---

How the system works

The end-to-end dispatch flow

This sequence traces a single OTP notification from ingestion through delivery, highlighting queue decoupling, policy enforcement, template hydration, and provider egress token checks.

End-to-end dispatch flow Expand diagram

×

• The Auth Service receives 202 Accepted within single-digit to low tens of milliseconds — it never waits for FCM or a relational database write.
• The Gateway claims a pending deterministic notification_id in Redis before publishing and finalizes it upon Kafka confirmation, preventing concurrent double-enqueueing while ensuring caller retries receive the original ID.
• The OTP is classified as High priority and published to notifications.high with acks=all, bypassing any promotional backlog in notifications.low.
• The Processor resolves active device tokens from Redis cache (backed by the relational database) and fans out one dispatch task per registered device into channel.push.high.
• Dedicated worker pools drain channel.push.high with bounded in-flight concurrency, ensuring urgent alerts are never head-of-line blocked behind bulk promotional pushes in channel.push.low.
• The worker verifies provider egress rate limits (token bucket) before dispatch, protecting vendor socket quotas.

Priority routing to distinct topics

The priority classifier runs during the Validation stage, mapping event_type to a priority level using a configurable rules table. The following allocation matrix defines the routing rules, target Kafka topics, and latency SLAs across priority tiers.

Priority Tier	Target Kafka Topic	Example Event Types	SLA & Consumer Allocation
High	notifications.high	otp_login, password_reset, fraud_alert, wire_transfer	Provider handoff < 5 seconds; dedicated over-provisioned worker pool.
Medium	notifications.medium	order_confirmed, payment_receipt, shipment_out_for_delivery	Provider handoff < 30 seconds; auto-scaled worker pool based on lag.
Low	notifications.low	flash_sale, weekly_newsletter, price_drop_digest	Best-effort (minutes to hours); throttled during peak hours.

Each priority topic operates with dedicated consumer groups and independently scaled worker pools. The High lane is deliberately over-provisioned: even if marketing events flood notifications.low with 100,000 messages during a flash sale, the High lane’s consumer threads remain idle and instantly process arriving OTPs without head-of-line blocking.

Dual-tier retry architecture: Kafka delay topics and DLQ

When third-party providers fail (HTTP 503s, 429 rate limits, or socket timeouts), retries are bifurcated by priority.

Instead of traditional database retry tables—where polling WHERE next_retry_at <= NOW() creates connection spikes, table locks, and multi-hour recovery lag after outages—the platform offloads transient retries to Kafka delay topics. Because every message in a bucket shares an identical delay, partition heads remain strictly chronological, allowing workers to drain retries at streaming scale without touching the database.

• High-Priority (Fast In-Worker Retry + Fallback): OTPs expire quickly and cannot wait in delayed queues. Workers enforce a strict 1.5s provider socket timeout, allow a single 500ms jittered retry, and immediately trigger secondary channel fallback (Push → SMS), completing vendor handoff within ~3.7 seconds.
• Medium & Low-Priority (Kafka Delay Topics): Non-critical messages route through bucketed delay topics (notifications.retry.1m/5m/15m/1h), safely absorbing massive provider backlogs without database contention.
• Dead-Letter Queue & Expiration: Before any dispatch attempt, workers verify expires_at > now(), silently dropping stale alerts. Exhausted retries (>4 attempts) publish to notifications.dlq and record DEAD_LETTER in Notification DB for operator alerts and replay scripts.

High-priority latency budget under failure

The following latency budget maps worst-case execution during a primary provider outage, demonstrating how the sub-5-second SLA is preserved:

Pipeline Step	Nominal Latency	Worst-Case Failure Latency	Failure Handling Behavior
Ingestion & Ingress Dedup (Gateway)	10–20ms	50ms	Fallback to at-least-once ingestion if Redis slow
Stage 1 Queue Buffering (Kafka)	5–15ms	50ms	Partition leader failover / producer backoff
Notification Processor & Prefs	10–30ms	60ms	In-memory Redis cache with fallback
Stage 2 Channel Queue (Kafka)	5–15ms	50ms	Consumer pickup on dedicated high lane
Push Dispatch Attempt 1 (FCM/APNs)	150–400ms	1,500ms	Socket timeout triggers fast retry
Fast In-Worker Retry (Push)	—	500ms	Single retry; failure triggers immediate fallback
Channel Fallback Dispatch (SMS)	300–800ms	1,500ms	Handoff to secondary provider (Twilio)
Total Provider Handoff	~500–800ms	~3,710ms (< 5s SLA)	Preserves SLA even during primary push outage

Notification lifecycle state machine Expand diagram

×

Click and engagement callback tracking

Tracking mechanics diverge fundamentally between browser links and native mobile applications:

• Email & SMS (HTTP 302 Redirect): Outbound links are wrapped with a tracking endpoint (GET /v1/callbacks/:id?redirect=<url>). When clicked, the service logs the event to Cassandra and immediately issues an HTTP 302 redirect to the destination.
• Mobile Push (Deep-Link First + Async Beacon): Tapping a banner must never block on network redirects. The native OS extracts deep-link parameters directly from the payload dictionary to render the target screen instantaneously. The client SDK then fires an out-of-band telemetry beacon (POST /v1/callbacks/telemetry) to Cassandra without introducing UI latency.

Mobile push click telemetry Expand diagram

×

Notification grouping and debounce

When high-velocity interactions occur (e.g., multiple likes on a photo within seconds), sending individual push notifications causes alert fatigue. The platform collapses rapid bursts using a two-tier strategy:

Notification grouping and debounce Expand diagram

×

• Backend Aggregation (Redis Sorted Sets):
  • Open window: The initial event opens a short window (e.g., 3s), storing actor IDs in a Redis hash and registering maturity (now() + 3s) in a Sorted Set (debounce:active_windows).
  • Coalesce: Subsequent events within the window increment counters and append actors without triggering dispatches.
  • Flush: Background workers poll mature windows (ZRANGEBYSCORE), atomically extract the batch via a Lua script, and publish a single summary alert (“Alice and 4 others liked your photo”) to Stage 2 Kafka topics.
• Client-Side Banner Collapsing: Push payloads include an OS collapse key (apns-collapse-id for iOS, tag for Android). If alerts reach the device close together, the mobile OS updates the existing banner in place rather than stacking cards on the lock screen.

Data model

The data model decouples global user identity from transient hardware device tokens, and isolates delivery attempt records to maintain an immutable audit trail.

Data model Expand diagram

×

• NOTIFICATION represents the logical accepted event with unique constraint (tenant_id, event_id). DELIVERY_ATTEMPT tracks each individual physical dispatch per device and channel.
• NOTIFICATION.expires_at prevents delivering stale alerts; workers drop any task where expires_at < now().
• NOTIFICATION.retry_count and next_retry_at are asynchronous audit fields written by status consumers, not polled for queuing. Active retries flow strictly through Kafka delay topics.
• DEVICE.device_token is ephemeral. A token refresh webhook updates this field when APNS or FCM rotates the token.
• USER_PREFERENCE stores granular opt-in by channel and event type. The preference gateway checks this before dispatch.
• NOTIFICATION.scheduled_time supports delayed delivery. The Scheduler queries rows where scheduled_time <= NOW() and status = 'ACCEPTED'.

API contract

The gateway exposes an asynchronous REST interface authenticated via service credentials (AppKey/AppSecret) and protected by idempotent event tracking:

Method & Route	Purpose	Key Contract	Response
POST /v1/notifications/send	Ingest transactional or priority notification	event_id, user_id, event_type, variables, optional redirect_url	202 Accepted (notification_id, status)
POST /v1/campaigns	Ingest bulk marketing campaign	segment_id, template_id, scheduled_time, target channels	201 Created (campaign_id, status)
GET /v1/notifications/:id/status	Query delivery lifecycle state	notification_id	200 OK (eventually consistent)
GET /v1/callbacks/:id	Web/SMS redirect tracker	notification_id, query param redirect	302 Found (redirects to destination)
POST /v1/callbacks/telemetry	Native mobile push tap beacon	notification_id, action, device_id	204 No Content
PUT /v1/devices/token	Register or update push token	user_id, platform, device_token	200 OK

Ingestion schema (POST /v1/notifications/send)

Security rule: Priority is derived strictly server-side from event_type and tenant policy. Caller-supplied priority parameters are ignored to prevent marketing jobs from hijacking high-priority OTP lanes. expires_at defaults automatically per event type (e.g., 10 minutes for OTPs).

{
  "event_id": "evt-a1b2c3d4",
  "user_id": "usr-9876",
  "event_type": "otp_login",
  "variables": {
    "otp_code": "482901"
  },
  "redirect_url": "myapp://login"
}

Response (202 Accepted):

{
  "notification_id": "ntf-xyz-789",
  "event_id": "evt-a1b2c3d4",
  "status": "ACCEPTED",
  "created_at": "2026-11-01T12:00:00Z"
}

Bulk campaign fan-out and off-hours traffic shaping

When internal marketing teams launch promotional campaigns targeting millions of users, an unconstrained blast risks exhausting vendor rate limits or triggering checkout rushes. The system enforces two operational safety controls:

1. The Campaign Fan-Out Engine: Marketing services register a single campaign via POST /v1/campaigns specifying segment_id, template_id, and scheduled_time. Dedicated Fan-Out Workers query the user database using keyset-paginated cursor batches (e.g., 5,000 users per chunk), generate lightweight notification records, and enqueue them directly into notifications.low in throttled increments.
2. Off-Hours Traffic Shaping: Bulk promotional jobs are scheduled to run exclusively during regional off-peak hours (e.g., 10:00 AM to 4:00 PM local user time), preventing large batches from contending with peak evening transactional commerce traffic.

---

The hardest design choices

Deduplication TTL scoping: OTPs vs. bulk campaigns

The pressure: Deduplication TTLs must match the caller’s plausible retry window without exhausting Redis memory:

• OTPs (10-minute TTL): Matches user authentication timeout. Keeping keys longer wastes memory on transient codes that can never be reused.
• Bulk Campaigns (24-hour TTL): A crashed fan-out worker can restart hours later and replay an entire batch of 200,000 recipients.
• Decision: Keys are scoped by tenant and event: dedup:ingress:{tenant_id}:{event_id} with type-specific TTLs. On duplicate hits, the gateway returns the original notification_id (202 Accepted) without re-enqueueing.

Rate-limit breaches: Outright drops vs. delayed queue routing

The pressure: When an internal service or campaign breaches its rate quota, returning 429 Too Many Requests creates failed business events and caller retry storms. Conversely, admitting bursts into real-time queues breaches the sub-5-second OTP SLA.

Decision: The platform applies dual-policy enforcement based on traffic origin:

• Abusive or unauthenticated traffic: Rejected immediately with HTTP 429 Too Many Requests.
• Authorized internal bursts: Diverted into a bounded notifications.delayed topic with an explicit expiration. A throttled consumer drains this topic during off-peak windows, guaranteeing eventual delivery without degrading transactional lanes.

---

Design deep dives

Quiet hours, timezone awareness, and digest modes

A marketing push notification at 3:00 AM in the user’s timezone degrades retention. Non-transactional alerts evaluate the recipient’s local timezone, parking overnight notifications in Redis Sorted Sets before releasing them as a consolidated morning digest.

Quiet hours and timezone routing Expand diagram

×

• The USER_PREFERENCE table stores user quiet hours and delivery preferences in a flexible JSON settings column.
• Non-transactional notifications that arrive during quiet hours are parked in a Redis sorted set with the score set to the user’s next morning delivery time. Transactional messages (OTPs, security alerts) bypass quiet hours.
• When quiet hours end, the scheduler releases parked messages. If multiple messages accumulated, the digest mode merges them into a single summary notification.

Channel fallback and geo-routing

When primary push delivery fails or times out, the system avoids blind retries. The fallback decision tree balances delivery SLAs against telecom costs:

Channel fallback decision tree Expand diagram

×

• Sequential Fallback vs. Cost Containment:
  • Critical OTPs: Use sequential fallback (Push → SMS after timeout) to guarantee delivery within 5 seconds. Simultaneous multi-channel broadcast is reserved strictly for account security emergencies.
  • Promotional alerts: Push failures are logged and dropped—never falling back to SMS due to prohibitive carrier costs ($0.01–$0.05/SMS).
• Dead Token Invalidation Loop: Workers deactivate device tokens (DEVICE.is_active = FALSE) only on explicit unregistration responses (APNs 410 Unregistered, FCM 404 UNREGISTERED). This prevents future marketing blasts from wasting millions of calls on dead tokens. Schema bugs (400 InvalidArgument) trigger alerts without deactivating devices.
• Geo-Routing for Regional Push: Where global push gateways are restricted (e.g., Google FCM in mainland China), the adapter routes notifications to regional OEM push gateways (e.g., Xiaomi, Huawei) based on device metadata.

Outbound provider egress rate limiting and traffic shaping

Downstream egress shaping prevents third-party API throttling (HTTP 429) and account suspension by metering dispatch calls to vendor contracts. Channel Workers implement this via three distributed coordination patterns:

• Isolated Priority Reservoirs: Egress quotas are partitioned across priority lanes (e.g., reserving 20 SMS/s for channel.sms.high out of a 100 SMS/s Twilio quota). Marketing surges in channel.sms.low can buffer safely in Kafka, but can never starve instantaneous OTP dispatches.
• Batched Token Leases (Redis Lua): Rather than having hundreds of distributed worker pods execute per-message round-trips to Redis, workers claim token leases in blocks (e.g., 10 tokens for a 1-second window) via an atomic Lua script, draining them locally to eliminate network latency.
• Non-Blocking Partition Polling: Calling consumer.pause() when vendor quotas are exhausted causes max.poll.interval.ms timeouts, rebalance storms, and blocks unrelated tenants on the same partition. Instead, workers maintain active polling and divert quota-exceeded tasks to delay topics (channel.sms.throttled) or an in-memory delayed ring buffer.

Asynchronous DB updates via event streams

A peak burst of 3,300 notifications/s generates 6,000–10,000 lifecycle events/s as messages transition across states (QUEUED → SENT → DELIVERED or FAILED). Writing these synchronously back to the primary database would instantly exhaust connection pools and lock table rows.

Channel workers instead emit delivery status events to a dedicated notification-status-events Kafka topic. A dedicated batch consumer reads these events in micro-batches (500 records or 100ms windows) and executes bulk UPSERTs into PostgreSQL (for current lifecycle status) and bulk appends into Cassandra (for the immutable audit trail). This completely decouples worker dispatch throughput from database write latency.

---

Scale and capacity

Back-of-the-envelope calculations

Assumptions: 10M push + 1M SMS + 5M email = 16M notifications/day. Average payload size 500 bytes. Notification DB row ~200 bytes. Retention: 90 days for active notifications, indefinite for audit trail. The following capacity matrix outlines daily event volumes, ingress throughput, and storage growth across the platform.

Dimension	Arithmetic	Result
Sustained write throughput	16M / 86,400s	~185/s
Peak write throughput	Marketing burst at ~18× sustained	~3,300/s for 5–10 minutes
Kafka ingress bandwidth	3,300/s × 500 B	~1.65 MB/s (trivial for Kafka)
Notification DB storage (90 days)	16M × 200 B × 90	~288 GB
Audit trail growth (monthly)	16M × 300 B × 30	~144 GB/month (Cassandra handles easily)
Redis dedup key memory	16M keys × ~50 B average	~800 MB (within a single Redis node)
Device token table	50M devices × 200 B	~10 GB

Limiting resource: This system is not bottlenecked by raw QPS. The architectural challenge is traffic isolation — preventing a promotional burst from starving OTPs — and provider resilience — surviving a Twilio outage without losing accepted notifications.

---

Failure modes and recovery

Failure mitigation matrix

The following operational playbook outlines automated mitigation pathways for component failures across the notification ingestion, queuing, and delivery lifecycle.

Failure	Impact	Mitigation	Residual risk
Notification DB primary crash	Status queries and scheduled job checks temporarily pause.	Ingestion continues uninterrupted via Kafka. Automated failover promotes standby replica (< 30s). Active preferences and device tokens are cached in Redis with short TTLs so in-flight OTP dispatches continue without stalling. Async status batch consumer buffers in Kafka until DB is back.	Brief delay in dashboard status updates and scheduled releases.
Kafka broker failure	Partitions on failed broker temporarily unavailable.	Multi-broker replication (RF=3, acks=all, min.insync.replicas=2, unclean.leader.election.enable=false). Partition leader failover completes within seconds; producer retries with backoff.	Transient sub-second publication latency spike.
Redis failure	Rate limiting and fast dedup checks unavailable.	Fall through: accept notifications into Kafka to preserve availability. Redis Cluster with multi-zone replicas for HA. During downtime, workers operate in at-least-once mode without in-memory dedup.	Brief window of potential duplicate queue processing under concurrent client retries; downstream messages must be idempotent.
Third-party provider outage (Twilio down)	SMS delivery fails for all messages.	High-priority messages execute a single fast in-worker retry (500ms) with a 1.5s socket timeout and immediately fall back to secondary channels (e.g., Push/Email). Medium and Low messages retry via Kafka delay topics (retry.1m, retry.5m, retry.15m, retry.1h). DLQ and DB status update for permanently exhausted messages.	SMS delayed until provider recovers or fallback succeeds.
Provider quota throttling (HTTP 429)	Provider returns rate limit errors under burst.	Channel workers enforce token bucket traffic shaping before calling APIs. Excess load buffers safely in Stage 2 Kafka channel topics without dropping.	Increased delivery latency for low-priority bulk campaigns during peak surges.
Worker pool exhaustion	Messages accumulate in Kafka; consumer lag grows.	Auto-scaling worker pools based on consumer lag metrics (bounded by Kafka partition counts). Channel workers employ asynchronous, non-blocking HTTP/2 clients (e.g., Netty) with bounded in-flight concurrency, committing Kafka offsets only after provider handoff completes.	Transient delivery delay for medium/low priority messages proportional to burst size.

SLA-centric observability

Aggregate metrics (CPU, total queue lag) are misleading: marketing blasts cause false-alarm alert storms, while third-party socket freezes in the high lane cause silent OTP outages. Production monitoring relies on two principles:

• Lane-Isolated SLA Attainment: Alerts and autoscaling bind strictly to rolling handoff percentiles per lane (High: p99 ≤ 5s; Medium: p95 ≤ 30s; Low: drain rate). A marketing backlog never pages on-call if OTP lanes meet their SLA.
• Hop-by-Hop Attribution: Payloads carry timestamps (t_ingest, t_enqueue, t_dispatch, t_vendor_ack). Telemetry instantly distinguishes internal worker starvation (t_dispatch - t_enqueue) from downstream vendor degradation (t_vendor_ack - t_dispatch).

What changes at 10× scale

At 160M notifications/day (~1,850/s sustained, ~33,000/s peak):

1. User-Based Database Sharding: Shard relational metadata by user_id to co-locate preferences, device tokens, and rate limits on a single shard, eliminating distributed transactions and cross-shard fan-out joins.
2. Round-Robin Partitioning for OTPs: Publish notifications.high with a null key (round-robin) to distribute load uniformly across all partitions and saturate 100% of consumer threads without head-of-line blocking. Keying by user_id is reserved strictly for stateful workflows requiring causal ordering (seq_num).
3. HTTP/2 Connection Multiplexing: Reuse long-lived HTTP/2 multiplexed connections to APNs and FCM HTTP v1, eliminating per-request TCP/TLS handshakes at 33,000 requests/s.
4. Multi-Region Active-Active: Deploy stateless gateways and dispatch workers across multiple geographical regions for disaster recovery and proximity to regional cloud endpoints.

---

Key takeaways

Core design takeawaysReview the core architectural paths before moving on to the quiz.Show summaryHide summary

Scalable Notification Platform

Review the core architectural paths before moving on to the quiz.

The Problem & ScopeStarvation prevention, vendor fault boundaries, and system limits4

Critical Alerts2

Marketing blast delays critical OTP authentication

OTP delay > 30s causes high drop-off / abandoned logins

Third-Party Delivery Services2

External partners (APNs, FCM, Twilio) can slow down or crash

Provider outages freeze servers and drop unsent messages

Notification Spam1

Rapid events (5 likes in 3s) create alert fatigue / buzzing

System Boundary2

In Scope: Multi-AZ ingress, priority queues, rate-limiting, retries, telemetry

Out of Scope: Upstream trigger logic, creative design, carrier networks

Requirements and AssumptionsScale benchmarks, delivery SLAs, and core platform capabilities2

Functional Requirements10

Accept & dispatch across channels (Push, SMS, Email)

Persist every accepted notification before dispatch

Route by priority to isolated processing lanes

Dual-Tier Retries: In-worker backoff + Kafka delay topics

Support click callbacks via webhook telemetry

Group related events (e.g., 5 likes in 3s) into summary

Enforce user preferences, quiet hours & opt-outs

Rate limit per user and tenant (Token Bucket)

Scheduled & delayed delivery via Redis ZSET scheduler

Track lifecycle state transitions (accepted to clicked)

Non-Functional Requirements9

Daily Volume: 16M (10M push, 1M SMS, 5M email)

Sustained Throughput: ~185 notifications/second average

Peak Throughput: ~3,300/s during marketing bursts

Critical Latency: OTP delivered to provider < 5s (p99)

Transactional Latency: p95 < 30 seconds

Availability: 99.9% for Ingestion API

Durability: Zero loss of accepted notifications

Consistency: At-least-once delivery

Idempotency: Deduplicate by (tenant_id, event_id)

The Mental Model & PipelineFour core primitives and the end-to-end multi-tier pipeline2

Four Design Primitives4

Persist before dispatch: Zero-loss durable multi-AZ ingress

Physical priority isolation: Dedicated topics prevent OTP starvation

Egress rate shaping: Token bucket matches vendor socket quotas

Bifurcated mobile payloads: OS system tray vs silent data sync

Four-Tier Pipeline4

Ingestion Tier: Fast dedup, validation, & multi-AZ Kafka publication

Prioritization Tier: notifications.high / medium / low topics

Processing Tier: Preferences, templates, quiet hours, debounce

Dispatch Tier: Token bucket rate limits, provider adapters, DLQ

Core Components and Deep DiveStateless ingress, dedicated topic lanes, processing engine, and workers5

Ingestion Gateway1

Stateless API layer: Redis dedup + multi-AZ Kafka publication

Notification DB1

Metadata & state tracking: PostgreSQL primary + read replicas

Buffer Queues1

Physical topic lanes: High (OTP), Medium, Low (Marketing)

Template & Channel Processor Engine1

Preferences, templates, quiet hours, debouncing

Channel Workers1

Provider adapters, token bucket rate limits, dead-letter DLQ

Data Storage & PatternsHybrid polyglot persistence, access patterns, and cost efficiency4

Storage Choice3

Cassandra/ScyllaDB: Immutable append-only audit trail (write-heavy)

PostgreSQL/MySQL: Relational metadata (devices, templates, prefs)

Redis: Ephemeral dedup, token bucket leases, debounce & quiet hours

Access Patterns1

Transactional: Single-key point lookups by user_id / device_token

Audit & Lifecycle Tracking1

Kafka consumer bulk writes to Cassandra audit logs

Scale & Cost1

Bounded Redis footprint (~800 MB); DB 90-day retention (~288 GB)

Critical Real-time and EgressFault handling, dual-tier retries, channel fallback, and traffic shaping4

Failure Scenarios3

Ingestion continues during Notification DB failure via Kafka

Redis failure: fall through to at-least-once Kafka ingestion

Twilio/APNs outage: fast in-worker retry and fallback

Dual Retries and Queues1

Temporary failure: Fast in-worker retry vs delay topics

Channel Fallback1

Tiered fallback: Push -> SMS / Email for critical OTPs

Egress Rate Shaping1

Token bucket per vendor socket prevents 429 throttling

Failure, Scale and Recovery10x growth evolution, partition keying strategies, and multi-region topology1

10x Scaling Decisions4

Relational DB: Sharded by user_id to co-locate prefs & tokens

Kafka: Round-robin (null key) for notifications.high OTPs

HTTP/2 Multiplexing: Long-lived sockets to APNs and FCM

Multi-Region Active-Active: Low-latency local ingress & failover

---

Architecture recap challenge

Notification platform architecture challenge

Can you design a notification platform that never drops a critical alert?

Ten critical scenarios across transactional outbox ingress, priority isolation, egress rate limiting, debouncing, and failure recovery. Test your architectural instincts.

1 / 10

Question 1 of 10

An upstream Order service needs to trigger a notification without risking dual-write data loss or slowing down critical OTPs at the Notification Gateway. Where does the Transactional Outbox reside, and when does the Gateway return 202 Accepted?

AInside the Notification Gateway's relational database, returning 202 Accepted immediately after a local SQL commit.BIn the upstream domain database relayed via CDC, while the stateless Gateway returns 202 Accepted only after publishing to Kafka with acks=all.CInside Redis using an in-memory queue before asynchronous flushing to MySQL on the hot ingestion path.DDirectly inside Cassandra's append-only audit log before routing tasks to RabbitMQ quorum queues.