Apple’s Sept 9th livestream got a lot of press, both good (great products), and bad: broken livestream, site downtime, and so on. We’ll leave the products to the press, but let’s dissect (some) of the perf problems…

Running the livestream page through WPT shows that even without the video stream the landing page weighs in at 5,790KB - not completely outrageous (we’ve all seen worse), but pretty hefty. Let’s break it down.

348 image requests accounting for ~3.7MB! That’s a lot of image requests. Digging deeper, most of them appear to come from the curated feed of tweets and images. More specifically the page fetches feed.json from www.apple.com origin, which provides the text and image URLs. At this point, we arrive at our first perf fail:

• The JSON file is 531.7KB, and its served uncompressed! Applying gzip to the file would reduce it to 57KB - that’s 90% savings.
• To make matters worse, the maxage is set to <10 seconds! On one hand, this is understandable, you want to expire old content and get new updates to your users. But, this also significantly reduces the effectiveness of edge caching - apple.com is served via Akamai.

Moving on. The fetched JSON file contains 290+ entries, most of which contain some text and a set of image references documenting highlights from the keynote. The site immediately dispatches all of the requests… at once. No on demand loading, just give me all of it, please, and now.

But, 290 entries and 348 image requests? It still doesn’t add up. Turns out, the site is fetching not one, but two image assets: a “lores” preview and a full replacement image. Also, the site also appears to adapt to screen DPR, which indicates that the ~5MB total page weight indicated by WPT is not the highest amount either, since we could have downloaded higher res photos. Yes, I’m looking at you, you lucky “Retina screen” users - you got the 10MB+ experience! Those image bytes sure add up quickly.

To be fair, the “lores” previews are scaled down and well compressed. That said, something tells me most of them were useless since the visitor never sees the majority of them - even if you tried, you can’t scroll fast enough to see all the previews. Hi, Jank. That said, now things take a turn for the worse: the max-age on these image assets is also <10s. 

Why? No idea. All of them are served via images.apple.com, which is also fronted by Akamai, but once again, a short TTL really doesn’t help with caching, which means there were a lot of requests hitting the Apple origin servers. Those poor Apache servers powering Apple’s site must have been working really, really hard. I’m not surprised the site was experiencing intermittent outages.

Oh, and speaking of load on origin servers… Remember feed.json? Every 10 seconds the page makes a polling request to the server to fetch the latest version. Combine that with a really short maxage TTL and missing gzip compression, and you’ve just created a self-inflicted DDoS.

So, lessons learned? Well, there’s a bunch:

1. Compress your JSON. No really, gzip helps.
2. On demand image loading strategy would have significantly reduced the number of requests for latecomers to the stream.
3. Your images are likely static, cache them with a long TTL!
4. The “lores” previews seems to have done more harm than good.
5. Providing a push feed instead of polling, or a “list of recent updates since X timestamp” functionality would have significantly reduced the amount of data in feed.json

Perf fail happens to the best of us.

P.S. The actual media stream was delivered via a different origin (also via Akamai). Did the above site perf problems affect the streaming? Maybe, hard to say. Perf fail is often additive in bizarre and interesting ways.