The problem is, I also want to automate it. I’ve been building a personal automation system that ties together Obsidian, AI agents, and Apple Reminders. I needed to create reminders, assign them to sections, set up recurrence rules, and have it all sync to everyone’s phones. From the terminal. Programmatically. And increasingly, through AI agents using MCP servers, Claude Code skills, and OpenClaw.

The existing CLI tools (remindctl, reminders-cli) use Apple’s EventKit API. They handle basic operations fine. But they can’t create sections, assign reminders to sections, or reliably sync everything to iCloud. I figured someone had solved this already. Nobody had. So I built remi.

It started with sections, but the problems went deeper

Sections are the organizational feature Apple added to Reminders in macOS 13. You group reminders under headings like “Weekly,” “Monthly,” “Urgent.” It’s how I organize my household routine - 40+ recurring tasks across 11 sections.

EventKit, Apple’s public API, has zero awareness of sections. No property, no method, no query. This is why every existing CLI tool ignores them.

But sections were just where the problems started. I also found:

• Recurrence rules don’t persist when added after creation. You have to add them before the first save() call, or they silently disappear.

• Date-only reminders get a “00:00” time if you set hour/minute to zero. You have to omit time fields entirely.

• Subtask relationships aren’t exposed through EventKit. The ZPARENTREMINDER column exists in SQLite but Apple never surfaced it publicly.

• iCloud sync just doesn’t happen unless you understand how Apple’s sync daemon decides what to push.

Each of these individually is annoying. Together, they mean no existing tool can reliably manage Apple Reminders with the full feature set people actually use.

Three layers deep into Apple’s stack

Building remi required going through three separate systems, each more undocumented than the last:

Layer 1: EventKit handles the basics. Creating reminders, setting due dates, recurrence rules, marking things complete. This is the safe, supported layer. All the other CLI tools stop here.

Layer 2: ReminderKit is Apple’s private framework at /System/Library/PrivateFrameworks/ReminderKit.framework. It exposes section CRUD through Core Data. Since it goes through Core Data properly, section changes sync to iCloud automatically. You access it at runtime via NSClassFromString. It could break with any macOS update. But it’s the only way to create sections that actually sync.

Layer 3: SQLite + CRDT token maps is where things got genuinely interesting. Section membership (assigning a reminder to a section) isn’t exposed by EventKit OR ReminderKit. The only way to do it is to write directly to Apple’s Reminders SQLite database. And getting those writes to sync to iCloud required reverse-engineering Apple’s sync daemon.

How remindd actually works

Apple’s remindd daemon manages iCloud sync for Reminders. I expected it to use NSPersistentCloudKitContainer, Apple’s standard Core Data + CloudKit integration. It doesn’t.

remindd runs a custom CloudKit sync engine with CRDT-style vector clocks for field-level conflict resolution. Each syncable field on a list record has an entry in a “resolution token map” stored in the database:

 {
   “map”: {
     “membershipsOfRemindersInSectionsChecksum”: {
       “counter”: 5,
       “modificationTime”: 796647639.739,
       “replicaID”: “C8116DE3-F9C5-4C94-B3FF-1A10D5184298”
     }
   }
 }

When remindd prepares a CloudKit push, it compares each field’s counter against the last-synced state. Fields with incremented counters get included. Fields without changes get skipped.

Simply writing to the SQLite database does nothing for sync. The counter doesn’t change, so remindd doesn’t know anything changed. Your data sits there locally, forever.

Three bugs that took days each to find

Bug 1: Token map entries need to be nested inside a “map” key with a replicaID. I initially wrote the membership counter at the top level of the JSON. remindd ignored it completely. Apple nests all token map entries inside a "map" key, and each entry needs a UUID called replicaID. Neither of these is documented anywhere. I found it by comparing my database entries against a list where sections were syncing correctly via the Reminders UI.

Bug 2: The SQLite connection must be closed before triggering sync. After writing membership data, I trigger a sync cycle by making a trivial EventKit edit. But if the database connection is still open, the WAL (Write-Ahead Log) hasn’t been checkpointed, and remindd reads stale data. My changes were being written correctly but remindd was reading the pre-change state. Adding a sqlite3_close() followed by a 500ms delay before the sync trigger fixed it.

Bug 3: Editing a reminder doesn’t trigger a list-level push. This was the hardest. The standard technique for triggering sync is to toggle a trailing space on a reminder’s notes field. This triggers remindd to push the reminder record. But membership data lives on the list record, not the reminder record. remindd wasn’t checking the list’s token map when only a reminder changed.

The fix: create and immediately delete a temporary reminder. This forces remindd to update reminderIDsMergeableOrdering on the list record, which triggers a full list-level CloudKit push that includes the membership data.

Each of these took 1-2 days to diagnose. The symptoms were always the same: everything looked correct locally, but changes never appeared on other devices. No error messages. No crash. Just silence.

The result

remi wraps all of this behind a simple CLI:

 remi create-section “Groceries” “Produce”
 remi add “Groceries” “Bananas” --section “Produce”
 remi move “Groceries” “Bananas” --to-section “Dairy”
 remi today
 remi overdue

Underneath: EventKit, ReminderKit, SQLite, SHA-512 checksums, CRDT vector clocks, WAL checkpointing, and temporary reminder sync triggers. All syncing to every Apple device via iCloud.

It also supports natural language dates (--due "next tuesday"), recurrence rules (--repeat "every 2 weeks"), fuzzy name matching (remi list shopping finds “Groceries / Shopping List”), JSON output for automation, and an MCP server for AI agent integration (remi --mcp).

Why I’m sharing this

I built remi because I needed it for my family’s shared reminders. But I think other people will want this too.

If you’re using AI agents (Claude, OpenClaw, or any MCP-compatible tool) and want them to manage your reminders properly, including sections, there hasn’t been a way to do that. remi ships as an MCP server, a Claude Code plugin, a skills.sh skill, and an OpenClaw skill. Install it once and your agents can manage your reminders.

If you’re a developer who lives in the terminal and wants a proper Reminders CLI, the existing tools are good at what they do but they stop at EventKit’s boundaries. remi goes further.

And if you’re building your own tools that interact with Apple Reminders, the internals document will save you days. The token map structure, the sync trigger mechanism, the ReminderKit bridge pattern - none of this is documented by Apple.

remi is on GitHub. Install with brew tap mattheworiordan/tap && brew install remi. I’d love feedback.

remi is available via Homebrew, npm, Claude Code plugin, skills.sh, OpenClaw, and the MCP Registry.

Today: Pricing models that reward customer success instead of penalizing it.

Why pricing models matter

When we started Ably, we had a clear motive: deliver the best value, fairest pricing, and maximum transparency so businesses could succeed with our platform. That’s why we started the business in the first place.

Your pricing model either aligns with customer success or creates friction. It either rewards growth or penalizes it. And in realtime infrastructure, where usage patterns can vary dramatically, this matters more than almost any other SaaS category.

Three approaches to pricing

The industry has evolved three primary pricing models for realtime platforms. Let’s compare them directly.

Package-based Pricing (used by Pusher)

How it works: Fixed tiers that bundle multiple dimensions together (connections + messages + channels).

Example: Startup plan ($49/month) = 1M messages/day + 500 concurrent connections [1]

What happens when limits are exceeded: According to Pusher’s own documentation, when you hit message limits, you cannot publish any more events and get a 403 “over quota” error. When you hit connection limits, new connections are rejected with “over quota” errors. Resolution: Must upgrade to next plan tier. [2]

The problem: If you exceed ANY single dimension, you must upgrade your ENTIRE package—even if you’re nowhere near limits on other metrics.

Real customer experience (from Trustpilot review):

“Pusher doubled our plan because we had an increased traffic for 3 days in April” and “now they refuse to decrease our plan back to what it was, so we’re paying double for no reason” [3]

Pusher themselves acknowledge this problem in their blog post “Making it easier to outgrow usage limits”:

“Hitting your hard limits can be a frustrating experience when it happens unexpectedly” [4]

They built emergency features (3x overage allowance, automatic upgrades) specifically because customers were complaining about service disruptions.

MAU-based pricing (PubNub)

How it works: “Simple” Monthly Active User pricing that appears straightforward on the surface.

The marketing: “Simple, Transparent Pricing... based on one simple metric: Monthly Active Users. No hidden fees, no surprises.” [5]

The reality: Transaction-based metering underneath with 60+ transaction types, 2KB billing buckets, and fan-out multipliers. [6]

This trap means customers either:

• Overpay because they’re not using their per user quota of transactions [7]

• End up surprised by the “hidden” overage charges for transactions they were told they wouldn’t be charged for [8]

Real developer experience (Jayson Lindsley’s case study):

• Told PubNub upfront he needed 3,000 active users

• PubNub response: “Bundled transactions should be enough”

• Result: Nearly $3,000 in unexpected overage charges

• Nearly half from single operation (getMembers()) failing and retrying

• Conclusion: “GetStream pays for itself - even at a higher base price”

• He migrated away [8]

Consumption-based pricing (Ably, AWS, Stripe)

How it works: Pay for the resources you actually consume, measured in units that reflect value delivered.

At Ably: 3 dimensions:

• Messages: The core value - the data moving between devices/people, which is what users come to Ably for.

• Connection minutes: The amount of time devices are connected.

• Channel minutes: The amount of time channels, used as logical streams of data, are active.

Why these dimensions: ~90% of our consumption revenue comes from messages because that’s where the value lies. If you’re delivering a game, every user move is a message. If it’s sports, every score update is a message. If it’s education, every interaction is a message. The value is in the messages, so that’s the primary billing dimension. In the same way that bandwidth is the primary billing dimension for CDNs.

No hidden complexity:

• No transaction-type classifications (every update is simply a message)

• No bundling that forces over-provisioning

• Decoupled metrics you can optimize costs independently

AWS validates this model: The most successful cloud company in the world doesn’t bundle compute, bandwidth, storage, and observability into a single EC2 package. They charge separately for each resource because that’s what aligns with customer value and enables cost optimization.

Ably’s transparent approach

Full disclosure: We also offer MAU pricing. Some customers prefer to budget that way.

But here’s what we can tell you with certainty: Almost every customer, once they understand both models, adopts consumption-based pricing because it’s superior. We don’t force that choice—customers decide. But we’re transparent about why most of our customers choose consumption-based.

The reason is simple mathematics:

With MAU, you’re either:

• Scenario 1: Quiet month → Transaction quotas unused → You overpaid for capacity

• Scenario 2: Successful day → Monthly user count climbs → Surprisingly high bill because that one successful day cost you a month of usage

• Scenario 3: Busy users → Transaction quota exceeded → Your budgeted MAU cost goes out the window as you pay overages for additional transactions

You can never match perfectly → You’re always in Scenario 1, 2 or 3

With consumption pricing: Quiet day = low cost. Busy day = pay for that day. Costs track actual usage patterns.

Customer optionality: We’re not here to tell customers they’re wrong about how they want to buy. But we ensure anyone who chooses MAU can switch to consumption-based at any point. And in practice, they do—because once you understand your consumption patterns, paying for what you use versus what you might use is always more efficient.

Transparency Makes Better Decisions

We’re not claiming our model is perfect for everyone. We’re claiming it’s transparent.

You can:

• See exactly what you’re paying for

• Optimize costs per dimension independently

• Predict costs based on actual usage patterns

• Never get penalized for successful days or events

That’s not impossible. That’s just fair business.

Next Post

Coming next week: Something you wouldn’t expect from a company in our position.

Putting our money where our mouth is on transparency and competition.

See you then.

Matthew O’Riordan
Founder & CEO, Ably

References

[1] Pusher Channels Pricing

[2] Pusher Documentation: What happens when I hit my Channels plan limits

[3] Pusher Customer Review - Trustpilot

[4] Pusher Blog: Making it easier to outgrow usage limits in your Channels plan

[5] PubNub Pricing Page

[6] PubNub Transaction Classification (60+ types, 2KB buckets)

[7] The true cost of MAU

[8] Jayson Lindsley: PubNub Chat Case Study ($3K overages)

[9] Ably Blog: Why Consumption-Based Pricing

First, we talked about maintaining zero impact during a significant AWS outage. All while PubNub experienced 10+ hours of degraded service.

Then, we covered scaling WebSockets to billions of connections globally. Highlighting the achievements of Socket.io, Pusher, and PubNub.

In this post: Another impossible thing by Ably. Guaranteeing stream integrity in a globally distributed system.

What is stream integrity?

When developers build realtime applications, they need confidence that:

1. Messages arrive in order (sequencing matters)

2. Messages arrive exactly once (no duplicates)

3. Messages don’t get lost (even during network interruptions)

These aren’t “nice to have” features. They’re fundamental for building reliable realtime applications.

Examples where ordering matters:

• Chat: Message threading, reactions to specific messages

• Streaming AI: OpenAI, Anthropic, Gemini APIs send token deltas—out-of-order = garbled responses

• Collaborative apps: Delta updates without CRDTs require strict ordering to prevent divergence

• Financial systems: Price updates, transaction sequences

• Gaming: Command sequences determine state

The developer shouldn’t have to think about this. Just like TCP/IP guarantees packet ordering, a realtime platform should guarantee message ordering so you focus on building features, not reliability infrastructure.

Standing on the shoulders of the early pioneers

When we started building Ably, we learned from the realtime platforms that came before us:

Socket.io (by Guillermo Rauch, now CEO of Vercel): Pioneered elegant WebSocket APIs that made realtime accessible to mainstream developers.

Pusher: Proved that managed platforms could eliminate DevOps burden, scale WebSocket infrastructure and do this delightfully for developers.

PubNub: Demonstrated ambition for global-scale realtime messaging across many points of presence.

Each achieved something remarkable. But each also made architectural trade-offs that limit their ability to provide stream integrity guarantees.

Why they can’t guarantee ordering

Socket.io: clustering introduces race conditions

Socket.io is a brilliant open-source library designed for self-hosting. Not as a managed platform.

The constraint: Beyond ~10K-30K connections per instance, you need to cluster multiple servers using Redis. When messages flow through Redis Pub/Sub across instances, race conditions emerge

Result: Socket.io has no ordering guarantees.

From creator Guillermo Rauch (Software Engineering Daily):

“I suffered from that quite a bit with Socket.io... there was only so much I could do to transfer the best practices of how to scale it at a massive concurrency. I could write a readme, but that’s as far as open source can typically go.”

Provides: Elegant WebSocket API, flexibility, open source.
Doesn’t provide: Ordering guarantees at scale, managed clustering, geographic distribution.

Pusher: parallel processing sacrifices ordering

Pusher uses native WebSocket protocol (unlike PubNub) and provides managed infrastructure - both positive.

The constraint: Each app lives in a single cluster. Within that cluster, messages are processed in parallel for throughput.

From Pusher’s documentation (“Why Don’t Channels Events Arrive In Order?”):

“Messages are processed by many machines in parallel on our backend.”

Result: Pusher has no ordering guarantee.

Their recommended workaround: Add sequence numbers, buffer messages, sort client-side, handle gaps. You’ve built a reliability layer on top of the platform.

Provides: Native WebSockets, managed infrastructure, zero operations.
Doesn’t provide: Ordering guarantees, multi-region per app, connection state recovery.

PubNub: Stateless HTTP can’t guarantee sequence

As covered in my previous post, Scaling Websockets to Billions of Connections Globally, PubNub uses HTTP long-polling (280-310s timeouts) instead of WebSockets.

From PubNub’s support docs (“Does PubNub guarantee the order of messages?”):

“No. If your application requires strict ordering, we recommend you add a sequence field to the message payload and order by it.”

The reason: HTTP long-polling is stateless. Each message is an independent HTTP request. Concurrent requests can complete in different orders.

On delivery: “PubNub is not a guaranteed message delivery service.”

On recovery: “The default message queue size is 100 messages. Publishing over 100 messages in the window of the subscribe reconnect time inevitably results in older messages overflowing the queue and getting discarded.”

Translation: Disconnect during high volume = permanent message loss.

Result: PubNub has no ordering guarantee

Provides: Global distribution (15+ points of presence), managed infrastructure.
Doesn’t provide: Ordering guarantees, WebSocket protocol, reliable delivery beyond 100-message queue.

The Pattern: Architectural trade-offs

Each platform made conscious design choices:

These aren’t failures. They’re architectural decisions that made sense for different goals. But they create a gap: What if you need global distribution AND stream integrity guarantees?

How Ably achieves both

When we built Ably, we asked: Could we provide stream integrity guarantees in a globally distributed system? Not single-region. Not single-server. But across many globally distributed datacenters with sub-40ms global latency?

The answer: Yes. But it required building distributed infrastructure from the ground up with stream integrity as a first-class requirement.

Our Guarantees

1. Message Ordering

Messages from any publisher over a single connection are delivered to all subscribers in the same order.

How it works: Each message gets a unique serial number at the time of publication. These serial numbers ensure subscribers see messages from that publisher in the correct sequence. Even across regions.

We achieve this through WebSocket’s persistent connection property plus sequential message processing. And we do it while maintaining 6.5ms message delivery latency within a region- proving that ordering guarantees don’t require sacrificing performance.

Read more: Data Integrity in Ably Pub/Sub

2. Exactly-once delivery

Once we acknowledge receipt of a message, we guarantee it will be delivered exactly once. No duplicates.

How it works: Idempotent publishing with unique message IDs, plus clients resume with a serial number. That serial number tells us exactly where the client left off in the stream, so even if you lose connection, you resume from precisely where you left off - guaranteeing you never get duplicates.

We chose this model (client tracking position via serial number) instead of server tracking client position because otherwise you can’t provide guaranteed ordering in a distributed system.

Read more: Achieving Exactly-Once Delivery

3. Connection state recovery

If a client disconnects briefly (within 2 minutes), we automatically resume with preserved state and zero message loss.

For longer disconnections, our History API extends beyond that window, allowing clients to catch up on any duration of missed messages.

No manual re-subscription. No message gap detection. No custom recovery logic.

These guarantees are in our SLA. They’re quantified in our Four Pillars of Dependability. They’re not marketing. They’re engineering commitments.

The global ordering nuance

An important technical point: While it’s technically possible to enforce a single global ordering of all messages across all publishers, doing so would fundamentally compromise the low-latency, fault-tolerant distributed system that realtime applications require.

Example: Publisher A in the US and Publisher B in Asia both send messages simultaneously. If we enforced global ordering, messages from nearby regions would be held back awaiting messages from distant regions, adding significant latency to all messages globally. Instead, a subscriber in Europe receives each message with minimal latency, with the tradeoff being that network physics determines which arrives first, not some artificial global sequence.

What we DO guarantee: Every subscriber sees messages from each publisher (using a transport that supports ordering, such as WebSockets) in the same order. This is “causal consistency per client” and it’s what actually matters for real-world applications.

Why this solves real problems: Developers need confidence that each participant’s messages maintain order. Chat from one user stays sequential. Token deltas from one AI stream arrive in sequence. Price updates from one exchange remain ordered.

The alternative (single global coordination point) would:

• Add latency to every message globally

• Create a single point of coordination and congestion

• Sacrifice geographic distribution benefits

• Introduce a critical failure point

• Not solve any real-world use case we’ve encountered

Our approach: Guarantee ordering from each publisher while optimizing for low latency through regional independence.

Read more: Chat Architecture for Reliable Message Ordering

Making the impossible possible at global scale

Providing these guarantees in a single datacenter is achievable, but hard. Providing them across 11 globally distributed datacenters with active-active replication? That takes engineering at its finest — and exactly the kind of challenge we thrive on.

As mentioned in my post about the AWS outage, we operate many datacenters globally including two in US-East alone.

The engineering required to make this work includes:

• Persistent WebSocket connections

• Regional message sequencing with serial numbers

• Cross-region replication with ordering preservation

• Connection state management across failover

• Idempotency tracking across distributed nodes

• 2-minute recovery windows with queued messages

This is not impossible. It’s just crazy difficult and expensive engineering work that we chose to do.

Developers shouldn’t have to implement reliability infrastructure. They should assume these basics - even though they’re extremely difficult - are in place.

Just like TCP/IP lets you operate at a higher level, realtime infrastructure should let you focus on features, not sequence numbers and deduplication.

What’s coming Wednesday

Today we talked about stream integrity guarantees that most platforms can’t provide.

Wednesday’s topic: Pricing that rewards success, not penalizes it (impossible? No)

MAU models and bundled pricing ultimately result in billing surprises or inefficient spending. We’ll talk through why transaction-based transparency matters and how we approach pricing that aligns with customer success.

See you Wednesday.

Matthew O’Riordan
Founder & CEO, Ably

Verify Everything

I’m making bold claims. Don’t trust me verify the evidence:

1. First Gen Realtime Solutions Documentation:

• PubNub: “Does not guarantee message order”

• Pusher: “Messages processed by many machines in parallel”

• Socket.io: Clustering documentation showing manual Redis + sticky sessions required

2. Ably Documentation:

• Four Pillars of Dependability - Performance, Integrity, Reliability, Availability

• Data Integrity in Ably Pub/Sub - Technical implementation details

• Achieving Exactly-Once Delivery

• Message Durability and QoS

• Ably platform architecture

The evidence is public. The data is verifiable. The architectural differences are documented.

Today: another thing our competitors probably thought was impossible.

Scaling WebSockets globally to billions of connections daily.

Standing on the Shoulders of Giants

When we started building Ably in 2016, we didn’t start from scratch. We studied the platforms that came before us, learned from their innovations, and understood their trade-offs.

Three pioneers shaped our thinking:

1. Socket.io: The WebSocket Pioneer

Guillermo Rauch (now CEO of Vercel) created Socket.io and showed the world that WebSockets could be accessible to mainstream developers. Before Socket.io, WebSockets were complex, low-level, and hard to use. After Socket.io, millions of developers could add realtime features with elegant, event-driven APIs.

Socket.io pioneered:

• Native WebSocket protocol with automatic fallbacks

• Event-based messaging that just made sense

• Open source accessibility (free forever)

• Excellent developer experience

What we learned: WebSocket can have beautiful APIs. Developers don’t want to think about protocols—they want to think about features.

2. Pusher: The Managed Platform

Pusher took Socket.io’s WebSocket approach and made it a managed service. No servers to run. No Redis to configure. No DevOps burden. Just beautiful APIs and infrastructure that worked.

Pusher proved:

• Developers would pay for zero-operations platforms

• Developer experience matters as much as features

• WebSocket-based managed services were viable businesses

What we learned: The future isn’t self-hosting libraries—it’s managed platforms that eliminate operational complexity.

3. PubNub: The Global Scale Ambition

PubNub demonstrated that globally distributed pub/sub infrastructure could work at scale. With 15+ points of presence worldwide, they showed ambition for true internet-scale realtime messaging.

What we learned: Developers building global applications need infrastructure that’s already distributed globally. Single-region solutions create latency problems.

The Trade-Offs Each Made

Each platform achieved something remarkable. But each also made architectural trade-offs that created limitations:

Socket.io: Elegance Without Scale Operations

Socket.io is a library, not a managed platform. You host it yourself.

The trade-off:

• ✅ Strength: Free, open source, complete control

• ❌ Limitation: Full DevOps burden, manual clustering, single-region (self-hosted)

Connection limits: Production deployments typically hit 10,000-30,000 concurrent connections per Node.js instance before performance degrades. Beyond this, you need to manually cluster using Redis and sticky sessions.

Guillermo Rauch’s own acknowledgment (from Software Engineering Daily podcast):

“I suffered from that quite a bit with Socket.io, which was one of my open source projects because there was only so much I could do to like kind of transfer the best practices of how to scale it at a massive concurrency. I could write a readme, but that’s, that’s as far as open source can typically go.”

Real-world evidence:

• Trello initially used Socket.io but encountered “problems with scaling up to more than 10K simultaneous client connections” and eventually built a custom WebSocket implementation handling “nearly 400,000” connections

• Disney+ Hotstar evaluated Socket.io but selected MQTT instead for their social feed infrastructure

The constraint: Socket.io wasn’t designed for internet scale. It excels at small-to-medium deployments where you want flexibility and control. At massive scale, the operational complexity becomes overwhelming.

Pusher: WebSockets, But Single-Region

Pusher took the right approach with WebSockets and managed infrastructure. But each Pusher app lives in a single cluster—one datacenter location.

Available clusters: 9 regional options (N. Virginia, Ohio, Oregon, Ireland, Singapore, Mumbai, Tokyo, Sydney, São Paulo)

The trade-off:

• ✅ Strength: Simple architecture, WebSocket protocol, zero operations

• ❌ Limitation: Single-region per app means higher latency for global users, no automatic geographic failover

From Pusher’s documentation on why they chose single-region:

“Consistency is important to Pusher, and they are not happy with the compromises needed to replicate data across clusters, such as inaccurate ordering of messages, dropped messages, or increased latency.”

This is a valid architectural choice—single-region simplifies operations and guarantees consistency. But it creates trade-offs:

• Users far from your selected cluster experience higher latency

• If your datacenter goes down, your entire app is affected

• No automatic failover between clusters—you must implement manual failover logic

The constraint: Pusher optimized for simplicity over geographic distribution. Great for regional applications. Problematic for global ones.

PubNub: Global Distribution, But Not WebSockets

PubNub demonstrated global distribution with 15+ points of presence (exact locations not publicly specified). But here’s where it gets interesting.

PubNub’s own documentation reveals the protocol reality:

Their blog claims (WebSockets vs REST article):

“At PubNub we have taken WebSockets to extreme scale and reliability, being able to service millions of devices across the globe and billions of messages each day being sent across the wire.”

Their technical guide says (WebSockets guide):

“WebSocket shortcomings can be difficult to manage if you are not an expert in building real-time systems.”

“PubNub takes a protocol-agnostic stance, but in our current operations, we have found that long polling is the best bet for most use cases.”

Their support documentation confirms (Subscribe Long Poll Mechanism):

• Uses HTTP long-polling, not WebSockets

• Default timeout: 310 seconds

• Presence events use “~280s server pings”

The trade-off:

• ✅ Strength: Global distribution, multi-region architecture

• ❌ Limitation: HTTP long-polling instead of WebSockets (higher latency, more overhead, stateless connections)

Why HTTP long-polling matters:

• Higher latency (repeated connection setup/teardown vs persistent WebSocket)

• Higher overhead (HTTP headers on every request)

• Stateless (harder to provide ordering guarantees—more on this Monday)

• More bandwidth usage

• Worse mobile battery life

You can verify this yourself: Open browser DevTools → Network tab. Connect to a PubNub demo. You’ll see HTTP GET requests, not WebSocket frames.

The constraint: PubNub achieved global distribution but traded away WebSocket’s technical benefits.

Platform Comparison Visualised

Why This Combination Is Hard

None of these platforms could deliver all four simultaneously:

1. ✅ Native WebSocket protocol (not HTTP long-polling)

2. ✅ Globally distributed architecture (not single-region)

3. ✅ Billions of connections at scale

4. ✅ Reliability guarantees (ordering, delivery, recovery)

Why? Because combining these is genuinely difficult engineering:

Challenge 1: WebSocket State Management Across Regions

• WebSockets are stateful, persistent connections

• Distributing state across global datacenters while maintaining consistency is complex

• Most platforms chose either WebSockets OR global distribution, not both

Challenge 2: Ordering Guarantees in Distributed Systems

• The CAP theorem means trade-offs between consistency, availability, and partition tolerance

• Most platforms prioritize availability over strong ordering

• Providing ordering guarantees at global scale requires careful architectural decisions (more on this Monday)

Challenge 3: Operations at Scale

• Socket.io: Operations burden falls on users (manual clustering, Redis, monitoring)

• Pusher: Operations managed, but single-region limits scale

• PubNub: Global operations, but HTTP’s stateless nature limits capabilities

What Ably Built

At Ably, we asked: Could we achieve all four?

Could we combine:

• Socket.io’s WebSocket elegance

• Pusher’s zero-operations experience

• PubNub’s global distribution ambitions

• Plus reliability guarantees none of them provide

Turns out: yes. But it required building distributed infrastructure from the ground up with all four as first-class requirements.

The Architecture

Multi-region active-active: 11 globally distributed core routing datacenters

• Each region operates independently

• Regions communicate peer-to-peer (no central hub)

• Failure in one region doesn’t cascade to others

• 700+ edge acceleration points globally

Native WebSocket protocol with intelligent fallbacks:

• WebSocket specification implemented in all SDKs

• Automatic fallback to Comet/long-polling only when WebSocket unavailable

• Binary protocol support (MessagePack) for efficiency

• Open source SDKs so developers can verify implementation

Automatic clustering and failover:

• Unlike Socket.io’s manual Redis clustering, Ably’s clustering is built-in

• Unlike Pusher’s single-region design, Ably automatically routes to nearest healthy datacenter

• Connection state preserved during failover

• 2-minute connection recovery window with automatic message recovery

Reliability guarantees:

• Message ordering for messages from a publisher over a single connection

• Exactly-once delivery with idempotency

• Automatic connection state recovery

• (More on ordering guarantees in Monday’s post)

The Scale We’re Running

Ably by the numbers (current platform stats):

• 2 trillion realtime API transactions monthly

• 1.5 billion daily WebSocket connections

• 30 billion monthly connections

• 2.4 billion devices reached monthly

• 37ms median roundtrip latency globally

• 6.7ms average edge message delivery latency

• 11 globally distributed regions

• 700+ edge acceleration PoPs

• 99.999% uptime SLA (achieved 100% uptime for 5+ consecutive years)

For context:

• Ably handles ~50,000x more connections than Socket.io’s per-instance limit (1.5B daily vs 30K per instance)

• Delivers this across 11 datacenters vs single-region deployments

• With sub-40ms global latency vs routing all traffic to one location

The Engineering Philosophy

Guillermo Rauch was right: You can’t transfer “massive concurrency” best practices through documentation alone.

That’s why we built a platform, not a library.

Our philosophy: Do the hard infrastructure work so our customers don’t have to.

Socket.io requires customers to:

• Manage servers, Redis, load balancers

• Configure sticky sessions and clustering

• Implement monitoring and failover

• Handle scaling, security, and operations

Ably handles all of this:

• Zero servers to manage

• Zero Redis to configure

• Zero load balancers to tune

• Automatic scaling, security, and operations

This isn’t about Socket.io being wrong—it’s about different tools solving different problems. Socket.io excels at providing free, flexible WebSocket capabilities for small-to-medium scale. Ably provides internet-scale infrastructure with enterprise guarantees.

The validation isn’t just our engineering—it’s that customers from all three platforms have migrated to Ably when they outgrew their original solution’s architectural constraints. Common themes: operational complexity at scale, geographic distribution requirements for global user bases, and the need for reliability guarantees their previous platform’s architecture couldn’t provide.

What’s Coming Monday

Today’s post focused on achieving WebSockets at global scale—combining protocol benefits with geographic distribution and operational simplicity.

But WebSockets alone don’t guarantee reliability.

Monday: How Ably’s architecture enables guarantees they can’t provide—and why this matters for financial systems, gaming, collaboration, chat and any application where message sequence matters.

See you Monday.

Matthew O’Riordan
Founder & CEO, Ably

Verify Everything

I’m making bold claims. Don’t trust me—verify the evidence:

1. Socket.io Connection Limits:

• Production reports: “many people have issues getting above 30k connections”

• Trello case study: Problems scaling beyond 10K

• Socket.io clustering docs: Manual Redis + sticky sessions required

2. Pusher Single-Region Architecture:

• Cluster configuration: “When you create a Channels app, you can choose which cluster it exists in”

• No automatic failover: “Channels has no built in failover between clusters”

3. PubNub HTTP Long-Polling:

• WebSocket guide: “long polling is the best bet for most use cases”

• Subscribe documentation: 310-second timeout

• Presence events: “~280s server pings”

• Verify yourself: Open DevTools, test their demo

4. Ably Architecture:

• Platform architecture: Multi-region design

• WebSocket specification: Protocol implementation

• Open source SDKs: Verify implementation yourself

• Network map: 11 regions + 700+ edge points

The evidence is public. The data is verifiable. The architectural differences are documented.

The results were stark:
Ably: Operational throughout. Zero customer impact. Zero downtime.
PubNub: 10+ hours of degraded service across two incidents.

I expected we’d share technical post-mortems and move on. What I didn’t expect was PubNub’s CEO, Todd Greene, to publicly question our results, suggesting they were “impossible” and that we’d “solved that pesky speed-of-light limitation“

Those comments became the ultimate validation of years of architectural investment.

What Todd Called “Impossible”

Todd’s comment implied that maintaining operational status during a regional AWS failure defies the laws of physics. Let me explain what he’s calling “impossible” - it’s actually well-thought-through infrastructure design.

The core principle: Regional redundancy.

Ably operates TWO datacenters within the US-East region:

• AWS us-east-1 (Northern Virginia)

• AWS us-east-2 (Ohio)

The network latency between these regions from major East Coast cities:

• New York: 5-10ms

• Boston: 8-12ms

• Washington DC: 7-10ms

• Atlanta: 10-12ms

When us-east-1 became unavailable, we failed over to us-east-2. Our monitoring showed a worst-case median latency increase of 12ms (anyone can easily verify this is possible and does not require any light speed violations 🤭).

That’s basic geography, not bending physics.

Our blog post on how Ably’s multi-region architecture held up during the US-East AWS outage has the full technical details including performance graphs.

What Happened at PubNub

According to PubNub’s public status page, their experience was very different:

Incident 1: October 20, 07:06-23:48 UTC (16 hours 42 minutes)
“Elevated latencies and errors for multiple services in US-west and US East”

Incident 2: October 20th, 2025 at 07:06 UTC (2 hours 1 minutes)
“Increased latency and errors observed in US-West”

Total: Over 16 hours of documented degraded service.

Their blog post about the incident explained why: “The physical distance to other regions was going to increase latencies.”

What this reveals: PubNub doesn’t appear to have multi-datacenter redundancy within US-East. When us-east-1 failed, they had to route traffic across the country or globe - hence the multi-hour latency issues.

This Isn’t Infrastructure 101

I want to be clear: building truly resilient global infrastructure is hard. This isn’t “infrastructure 101” - it’s the result of 10 years of deliberate architectural choices and significant investment.

What we have built at Ably:

Multi-region architecture with no single points of failure:

• Each region operates independently

• Regions communicate peer-to-peer (no central hub)

• Failure in one region doesn’t cascade to others

• 700+ edge locations globally via CloudFront

Connection state recovery:

• 2-minute window to seamlessly resume after disconnections

• Messages queued on servers, not just client-side

• Zero message loss during network interruptions

• Automatic reconnection without application code

SDK resilience:

• Automatic failover to healthy datacenters

• Multiple fallback hosts tried in sequence

• Connection state preserved during transitions

• Transparent to end users

Ably engineering philosophy: “Do the hard infrastructure work so our customers don’t have to”

This philosophy and investment is clearly evident in our Four Pillars of Dependability.

The Broader Pattern

The AWS outage wasn’t an anomaly. Looking at the last year of public incident data (October 2024 - October 2025):

Ably (status.ably.com):

• 5 customer impacting incidents

• 531 minutes of degraded service

PubNub (status.pubnub.com):

• 26 customer impacting incidents (4.2x more than Ably)

• 6,045 minutes of degraded service (10.3x more than Ably)

• Pattern of Presence service failures

Ably is not claiming perfection, but we are demonstrating that architectural choices have real consequences for reliability.

(Source: Reliability By The Numbers: Ably vs Pusher vs PubNub incident analysis)

Why This Matters to Customers

This is exactly why customers migrate from PubNub to Ably.

To date, our enterprise customers, who migrated from PubNub to Ably in search of a better service, have generated a whopping $7.5m of revenue.

We’re not aware of any enterprise customers who’ve migrated in the opposite direction.

These customers consistently cite similar reasons: They needed reliability guarantees they could actually depend on.

One recent enterprise customer (we can’t name them - PubNub threatened legal action when they tried to publish a case study) reported:

• 65% reduction in operating costs

• 95% reduction in waiting room issues related to realtime

• 100% elimination of ghost-check-in times

• 99% reduction in app-closing errors

These metrics aren’t about marketing claims. They’re about real-world performance under real-world conditions - including infrastructure failures.

To Developers: Verify Everything

I’m making bold claims. Don’t trust me - verify the evidence:

1. Status Pages

• See Ably status and PubNub us-east-1 AWS incident

• See raw data and analysis for for 3 years worth of Ably, Pusher and Pubnub incidents

2. Run Your Own Latency Test:

• Ping from US locations to us-east-1 and us-east-2 and see the latency differences

• Here’s some data and a handy script you can run yourself to reproduce this

3. Review Architecture Documentation:

• Ably Four Pillars: https://ably.com/four-pillars-of-dependability

• Ably platform architecture: https://ably.com/docs/platform-architecture

• PubNub architecture: .. tumble weed .. (I genuinely cannot find this information)

Transparency Makes Everyone Better

The more transparent we are about how we design systems and how they perform under real conditions, the more we can all learn from each other. There’s plenty of space for multiple strong companies in the realtime infrastructure market.

The AWS outage provided a rare, public test of every platform’s architecture. The results are documented on both our status pages.

Ably maintained operational status. PubNub experienced 10+ hours of degradation. That’s not impossible. That’s the difference between architectural choices made years ago.

What’s Coming Next 👊

This incident reminded me that we should talk more about the “impossible” things Ably achieves.

Over the next week, I’m publishing a series on things our competitors probably think are impossible:

Friday: Scaling native WebSockets to billions of connections daily

• Why we chose the hard path instead of easier alternatives

• What this enables for customers

• The engineering investment required

Monday: Guaranteeing message ordering and exactly-once delivery in globally distributed systems

• Why most platforms can’t offer these guarantees

• How our architecture makes it possible

• What this means for building reliable applications

Wednesday: Transparent pricing that doesn’t penalize your most successful customers

• Why transaction complexity creates billing surprises

• How consumption-based pricing aligns with customer success

• The difference between simple marketing and simple reality

Each post will show what’s actually possible when you do the hard engineering work.

See you on Friday!

Matt
Technical co-founder and CEO of Ably

So I pulled the data. All of it. Public. Verifiable. Objective.

I analysed 24 months of incident data from public status pages across Ably, PubNub, and Pusher - 133 incidents total, which is available in this Google Sheet.

The uptime vs. experience gap

When platforms claim “99.999% uptime,” they’re referring to SLA definitions that measure consecutive minutes of complete unavailability, excluding scheduled maintenance, brief errors, and degraded performance.

SLA uptime ≠ customer experience.

This analysis doesn’t calculate SLA uptime - that’s not measurable from status pages. Instead, it measures what customers actually experienced: every incident where status pages documented errors, failures, or material latency increases, regardless of SLA implications.

That’s the metric that matters. Here’s what the data shows:

Last 12 Months

Ably demonstrated the strongest performance with 4 major incidents (avg 90 min duration) and 531 customer degraded minutes over 12 months.

PubNub had 5.0x more major incidents (20 vs 4), with an average major incident duration of 136 minutes. Customer degraded minutes totaled 6045 (11.4x more than Ably).

Pusher experienced 1.0x more major incidents (4 vs 4), with notably longer average durations at 227 minutes. Customer degraded minutes reached 5132 (9.7x more than Ably).

Last 24 Months

Ably demonstrated the strongest performance with 7 major incidents (avg 180 min duration) and 1711 customer degraded minutes over 24 months.

PubNub had 3.7x more major incidents (26 vs 7), with an average major incident duration of 148 minutes. Customer degraded minutes totalled 7903 (4.6x more than Ably).

Pusher experienced 2.4x more major incidents (17 vs 7), with notably longer average durations at 314 minutes. Customer degraded minutes reached 9661 (5.6x more than Ably).

Detailed Analysis (24 Months)

Over a 24-month period, 102 incidents were analysed across Ably, Pubnub, Pusher.

Key Findings (Customer-Impacting Degraded Service):

• Ably: 16 customer impact incidents, 1711 total degraded minutes (28.5 hours)

• PubNub: 35 customer impact incidents, 7903 total degraded minutes (131.7 hours)

• Pusher: 24 customer impact incidents, 9661 total degraded minutes (161.0 hours)

Major Incident Summary:

• Ably: 7 major incidents, average duration 180 minutes

• PubNub: 26 major incidents, average duration 148 minutes

• Pusher: 17 major incidents, average duration 314 minutes

Overall Statistics

Total Incidents by Platform

PubNub shows 74% of incidents classified as major compared to Ably’s 29%, indicating a higher proportion of severe customer-impacting issues.

Customer-Impacting Degraded Service Time

Pusher’s average incident duration of 314 minutes is significantly longer, contributing to 5.6x more total degraded service time compared to Ably.

Major Incident Duration Summary

PubNub demonstrates the fastest average resolution time at 148 minutes, indicating more effective incident response and remediation processes.

Comparative Analysis

Monthly Averages

On a monthly basis, PubNub averages 4.6x more degraded service minutes than Ably, representing a consistent pattern of higher customer impact over the analysis period.

Methodology

Data Source

• Platform: Statuspage.io public incident APIs

• Period: 24 months (October 2023 - October 2025)

• Incidents: 102 incidents analysed (133 total collected)

Classification

• LLM-Based: Claude Sonnet analysed incident descriptions, timelines, and component impacts as data classification by vendors was not reliable

• Categories: Major, Minor, Control Plane / Non-Customer Impact

• Customer Impact: Separate tracking of customer-visible degradation

• Timestamp Corrections: LLM corrected inaccurate timestamps by reading natural language in incident messages

Definitions

Incident Severity:

• Major: Service failures, customer-visible errors, or extended periods of material latency increases

• Minor: Brief degraded performance, short-duration issues, or transient errors

• Control Plane / Non-Customer Impact: Internal/control plane issues OR incidents with no customer impact (dashboards, stats, documentation, billing, or internal issues handled gracefully)

Customer Impact:

• Customer-Visible Errors: ANY errors that customers could see, even if brief

• Material Latency: Significant slowdown mentioned in incident for extended periods

• No Customer Impact: Control-plane issues, or internal issues handled gracefully (retries, failover) with no visible errors

Key Metrics:

• Customer Degraded Service Time: Total minutes where customers experienced errors OR material latency

• Non-Degraded Service Availability: 100% - (customer impact minutes / total minutes in period)

• Major Incident Duration: Average length of major incidents, indicating how quickly issues are resolved

This report was generated from live data on 2025-10-29

Round size

The round sizes have unsurprisingly grown significantly, with some tapering off in 2022. Pre-seed rounds in 2016 ranged from $200k to $500k, whereas they now range from $1–$2m. Series A have grown from $3m to $12m in 2016 to $6m and $18m in 2022 showing no sign of drop off in spite of the macro economic situation.

Valuation

Valuations have materially increased across the board, with some signs this is rebounding this year. Pre-seed valuations in 2022 are up to $12m from $3m in 2016, and Series B valuations are up to $270m last year up from $100m in 2016.

Comparing this to public SaaS multiples, it’s clear that Series B valuations more closely track the markets, whereas Seed and Series A valuations seem to be bump upward regardless of the market.

Data I would love to see is how each annual cohort performed for investors i.e. what returns did investors get based on which year they invested in and the valuations at the time. Unfortunately I don’t have access to this data.

ARR

The ARR expectations for seed companies has not shifted much, up from $0 to $0.6m to now $0 to $1m. Series A has shown significant change with the expectation in 2016 being $1.2m to $3m, in 2021 being $0.5m to $4m, and this year being $0.5m to $2.5m showing that Series A companies can certainly raise with less ARR today. Series B mins have dropped too from $4.2m to $3m.

Capital Efficiency

2022 is the year where every VC suddenly started talking about capital efficiency in every meeting with their portfolios. It’s not surprising given access to capital is no longer cheap given the wider economic shift in the markets and the “silent” VC crash. Predictably this year Christopher Janz added capital efficiency to the SaaS Napkin as a new metric you now need to think about at Series A (3.5x) and Series B (2.5x). If your investors have not asked you about capital efficiency yet, they must be living under a rock. If you don’t know what it is:

The Capital Efficiency Ratio is the ratio of how much a company has spent growing revenue and how much they’re receiving in return. It is the broadest measure of company effectiveness in generating ARR

What next?

I honestly don’t know what’s coming next, I am not sure anyone really knows either. However, what I can clearly observe is that interest rates continue to climb resulting in downward pressure on equity multiples, and all startup and growth businesses around me are tightening their belts and extending their runways in anticipation of a bumpy fundraising ride ahead. If you’re looking to fundraise now but have the option to delay, my advice would be to hold off until there is more stability in the market and access to money eases.

I’m Matthew O’Riordan and CEO and Co-Founder of Ably Realtime. Ably is a Series B developer-first business backed by Insight Partners, Dawn Capital, MMC Ventures amongst others, and our customers are using our realtime APIs to power live and collaborative experiences for more than 300m people each month.

Ably is a serverless WebSocket PaaS operating at the edge. There’s no infrastructure to provision or manage, just an evolving suite of SDKs and APIs that give you the freedom and flexibility to power live and collaborative experiences with a few lines of code. Our mathematically modeled system design provides a globally-distributed edge network that brings users closer to your app, unique data ordering and delivery guarantees to ensure a seamless end-user experience, a legitimate 99.999% uptime SLA underpinned by fault tolerant infrastructure, and instant elasticity to enable effortless scale.
Find out more about Ably.

Today, I am proud to say the Capybara Screenshot gem has finally reached a stable version release of 1.0.0, and supports RSpec, Cucumber, Spinach, MiniTest and TestUnit. And impressively, according to RubyGems, it has now had more than 550,000 downloads to date, equivalent to roughly 1% Rails’ downloads.

If those numbers are at all accurate, and this gem is helping even 1% of the Rails community, then I am a happy man for contributing something back.

View the changelog for the 1.0.0 release.

In order to understand the significance of the expected volume of 10k SSL requests per second, a single Amazon EC2 small instance can terminate approximately 25 2048bit SSL connections per second. If we were to terminate these connections ourselves using small instances, then we’d need at least 400 small instances running simultaneously with an even spread of traffic through a DNS load balancer to achieve this. Personally, I don’t like the idea of managing 400 CPUs (equivalent of a small instance) at each of our locations.

Hence we turned to ELB and started running our tests to see if it would scale as needed and deliver. One thing to bear in mind with ELB is that it does not officially support WebSockets, so in order to use ELB you have to configure it to balance using TCP sockets on port 80 and 443, which works fine except that the source IP is not available. I am told Amazon are working on a solution, but at present none exists, and it’s not such a big problem for us anyway.

With the help of Spencer at Amazon, who was incredibly helpful throughout the entire process (which lasted more than 2 weeks), we managed to get to a point where we knew the tests were accurate, any bottle necks that we found were indeed ELB and not caused by other factors, and we could replicate our tests easily. Spencer set up CloudFormation templates (awesome stuff) so that we could spin up the servers and the load testing clients in the same configuration each time, and I worked on building a simple WebSocket server, a WebSocket testing client, and a controller app (based on Bees With Machine Guns) so that we could spin up any number of load testing clients we needed, and start a progressive and realistic attack on the load balancers. On Github you can find my core repositories for the WebSocket load testing client and server, and the modified Bees With Machine Guns that does not rely on SSH connections as I found these to be unreliable when you spin up more than 10 load testing machines (we went up to 80 c1.medium).

In summary, the results were very encouraging, but certainly not perfect. In my original post I alleged that ELB was flat lining at around 2k requests per second, unfortunately this was wrong, and was caused by me due to the way the testing clients were designed. Once I made the controller more reliable, made the load testing client not wait for connections to close, and replicated more realistic traffic growth, then when I ran the tests myself in my EC2 cloud I easily got to 10k requests per second, and so did Spencer (Amazon) with his tests using CloudFormation and the libraries described above. What we found is that ELB does seem to have unlimited scaling capabilities, however it doesn’t necessarily scale as fast as you want it to all the time. If you look at the graphs below, you will see plenty of dips where we would expect ELB to scale up horizontally to cope with more demand, yet it doesn’t until some time passess. Eventually ELB does catch up, but it’s worth bearing in mind that if you get spikes in traffic, latency will increase and potentially some connections may drop as ELB plays catch up.

My tests using ELB and EC2 showing expected versus actual in terms of new SSL connections per second

Amazon’s (Spencer) test on ELB where we expect a smooth bicubic curve

So to summarise:

• ELB is pretty damn awesome and scales incredibly (pretty much limitlessly it seems)

• ELB has its problems and is not perfect. It’s designed to scale as necessary, but your traffic growth may exceed ELB’s natural scale-out algorithms meaning you will experience latency or dropped connections for short periods.

• We’re going to be using ELB as we believe it’s the best solution out there by a long stretch (we considered hardware load balancing & SSL termination, we considered Rackspace, GoGrid, and a bunch of others).

• Amazon have been amazingly helpful throughout this exercise, way beyond what I could have expected.

Update: Since I made this post, Amazon have in fact been in touch and have been extremely helpful looking into this issue and running numerous tests to ensure ELB is performing as it should. In short, many of the tests in this post are wrong.

Please go to Part 2: How elastic are Amazon Elastic Load Balancers (ELB)? Very.

Recently I’ve been working on a project that will feasibly need to respond to 10,000+ SSL web socket connection requests per second, and I’ve been racking my brain as to figure out what type of infrastructure I would need to cope with that.

I did some initial tests and determined that a single CPU instance on EC2 (with 1 EC2 compute power) can cope with a measly 25 SSL handshakes per second. Based on my 10k target, that means I would need 400 CPU instances on EC2 just to deal with handshakes. I soon learn that the certificate key size is paramount, and dropping from the normal 2048 bit key down to 1024 bit improves performance 5 fold, so that now means I only need 80 CPU instances. Still way too many instances, and to be honest, not a piece of architecture I particularly want to look after either as I will need to build a system that can auto-scale on demand.

So I started experimenting with Amazon’s Elastic Load Balancers on the premise that they are elastic. I ran some quick apache bench tests against an ELB server and unsurprisingly got 25 SSL handshakes per second throughput with a 2048 bit key. So I can deduce that an ELB when it starts out is simply a single CPU instance on EC2. I then tried to run apache bench for 30 minutes to see how ELB scales, but unfortunately quickly discovered that that type of test is quite contrived, and ELB scales out dynamically using DNS. So a single load test client would only ever hit one ELB instance, and as such, ELB may not scale as only one IP is hitting it and even if it did, the client would never be able to benefit from that scaling.

So I went ahead and built a load testing farm of daemons that distribute their “attack” on the ELB from loads of different IP addresses (I spun up 40+ micro instances on EC2), and will cleverly keep querying DNS to see when ELB scales and distribute traffic to the new IP addresses that ELB responds to for its public DNS name.

And my findings were interesting, and disappointing. It seems that ELB does in fact scale, but for some reason I am hitting a hard limit for some reason or another. Perhaps Amazon have set a hard limit for some reason, perhaps I need to have something enabled on my account, or perhaps ELB is just not as elastic as it’s claimed to be. I’ll only find out if Amazon get in touch with me to explain ;)

Here is a graph demonstrating the attempted requests per second from my load testing daemons that scaled up request from 50 per second in minute one, up to 6,000 requests per second in the 120th minute. As you can see, other than a few more small upward blips, ELB pretty much flattens out at around 2,000 requests per second with a 1024 bit key and ELB spread across 3 zones (and thus 3 instances).

If anyone has any suggestions or feedback on how to make ELB scale as it should, please do get in touch. I’ve posted my findings into the Amazon forums here and here, but I have little hope that Amazon themselves will respond (see below, Amazon did get in touch and have been brilliant!).

In the mean time, it looks like I’ll be finding another solution.

Update: Since I made this post, Amazon have in fact been in touch and have been extremely helpful looking into this issue and running numerous tests to ensure ELB is performing as it should. In short, many of the tests in this post are wrong.

Please go to Part 2: How elastic are Amazon Elastic Load Balancers (ELB)? Very.