These days building an MVP is easy, launching it is a challenge but if you succeed, usually your site will stay up (since it’s such a basic version). The problem really starts when you start growing your site, and your site’s traffic starts growing with it.
That’s what happened with Nomad List. In half a year it grew to over 500,000 pageviews per month, with over 100 million assets served per month. That’s about 200 assets per page load, mostly pictures of the cities). My tiny Linode NGINX server has been pretty good at handling it, but it hasn’t made the site particularly fast.
The thing is, when your site is going well, you’re now competing with all the other big sites. And they have people dedicated to making everything fast. And the more mainstream you go with your site, the share of “mega fans” of your site that are willing to wait for the site to load will decrease. You’ll be seeing “normal” people visit your site, you know the ones coming from Facebook etc., and if your site isn’t loaded in a few seconds, they’ll call it “slow”.
I mean, honestly, I do the same. So a few weeks ago, I was done with it and stuff had to speed up.
Testing
I used Pingdom to test load times. And always picked San Jose, CA as a base server. My Linode is located in London, so I felt that’d be a good test. I must say Pingdom is not incredibly reliable, so I’d take all of this data with a big grain of salt. Load times do vary somewhat even without changing anything.
We continue.
Basic caching
I already implemented basic caching months ago. Why? Well, the city database had become so big with 550 cities and 65 attributes per city (like apartment rent cost, hotel cost etc.), that it was now over 35,000 data points. That’s fine, but there was calculations done on those and they were done on every page load. I know, stupid. It worked in the beginning with 50 cities and 500 data points, not with 35,000. Doh.
I made this:
php -f /srv/http/nomadlist.com/app/index.php — ‘currency=usd’ — ‘units=metric’ > /srv/http/nomadlist.com/cache/index-usd-metric.html
And built a script to generate static pages for each combination of currency (EUR, USD, GBP etc.) and unit system (metric or imperial). A cron job runs it every few hours or so. I know, tedious, but it works for now until I deploy a new version of the site that does this in a better way.
Caching brought down the load time from a crazy 29.89s (which users never saw, as I implemented caching way earlier) to 3.90s.
Conclusion: Caching processing in a page speeds up everything drastically (well doh)
SPDY
With SPDY enabled, it actually pushed the load time from 3.90s (on SSL) to 3.97s. Not really sure why, maybe Pingdom doesn’t support/measure SPDY? For browsers supporting SPDY, it definitely sped it up noticeable making it much more snappy. Especially on subsequent loads. That means once you’re already on a site, the page loads from then on are much faster. SPDY keeps a live connection between the user and server, so that means it doesn’t have to re-connect on every page load (like HTTP/HTTPS).
Conclusion: SPDY doesn’t change speed significantly according to Pingdom, but does make it snappier on browsers that support it
CloudFront by AWS
@levelsio Why don't you load the static files from a CDN (Cloudfront with a custom origin). The CDN is location aware.
— Pieter Bogaerts (@PizzaPete) May 8, 2015
@PizzaPete on Twitter recommended me to set up CloudFront. It’s Amazon AWS’s CDN service. A CDN is pretty much a lot of servers around the world that mirror your content, so that when a user requests the file (like a photo of a city on Nomad List), it gets sent from the closest location to the user (e.g. a user in Amsterdam gets the image from London, a user in Shanghai gets the image from Tokyo).
Setting this up was quite easy.
I simply made a script that after scaling my images, would put them up on S3 with S3cmd:
s3cmd put /srv/http/nomadlist.com/public/assets/cities/500px/amsterdam-netherlands.jpg s3://nomadlistcdnbucket/assets/cities/500px/amsterdam-netherlands.jpg
Then I went into AWS panel and set up a CloudFront distribution that connected to my S3 bucket.
This means that instead of loading the images from https://nomadlist.com/assets/cities/500px/amsterdam-netherlands.jpg, it would now load them from https://d39d3kdh3l.cloudfront.net/assets/cities/500px/amsterdam-netherlands.jpg.
So did it speed things up? Let’s see.
CloudFront doesn’t support SPDY, only SSL. But with my own server’s SPDY enabled and CloudFront over SSL, load time went from 3.97s to 3.86s. Again, a very tiny difference. Without SPDY and just SSL it decreased to 3.82s. Over HTTP, CloudFront’s speed up was the biggest, from 5.40s to 3.64s. That’s almost 2 seconds!
Conclusion: CloudFront seem to improve load times over HTTP, SSL and SPDY with a very strong effect ovr HTTP
PageSpeed
@levelsio pagespeed module is pretty awesome too
— KJ Prince (@KJ_Prince) May 1, 2015
Another tip was PageSpeed, recommended to me by KJ Prince. I think I’m pretty late to the party as Google seems to be deprecating it. But I wanted to try it any way.
It decreased load times on SSL from 3.86s to 3.82s. Over SPDY it actually increased load times from 3.97s to 4.06s! Again, over HTTP the effect was strongest: from 5.40s to 4.13s load times. Almost 1 second won.
Conclusion: PageSpeed seem to improve load times over HTTP and SSL, not over SPDY, with a very strong effect over HTTP
PageSpeed + CloudFront
Let’s combine the two and see what happens.
With both enabled, on SPDY we get 3.95s. Without both enabled, SPDY was actually faster! Over SSL we get our fastest time of 3.79s. And over HTTP, we go from 5.40s to our fastest time of 3.56s. Again almost 2 seconds won.
Conclusion: PageSpeed and CloudFront combined seem to improve load times over HTTP and SSL, not over SPDY.
SSL vs. SPDY on Chrome
Since Pingdom doesn’t give us realistic results for SPDY, let’s test it with Chrome on OSX. I tested it from Bangkok, Thailand on a solid connection. Since my server is in London, that’s 9,526 km far. Pingdom’s connection from San Jose to London was about equal at 8,627 km distance.
Conclusion: On Chrome, SPDY seems to improve load times vs SSL, from 5.24s to 3.58s (-1.66s) or a 31% speed increase.
Results
| Load time | Change | % | |||||
|---|---|---|---|---|---|---|---|
| Pingdom | Cache | HTTP | CloudFront | PageSpeed | 3.56s | -1.84s | +34% |
| Pingdom | Cache | HTTP | CloudFront | 3.64s | -1.76s | +32% | |
| Pingdom | Cache | HTTP | PageSpeed | 4.13s | -1.27s | +23% | |
| Pingdom | Cache | HTTP | 5.40s | ||||
| Pingdom | |||||||
| Pingdom | Cache | SSL | CloudFront SSL | PageSpeed | 3.79s | -0.10s | +2% |
| Pingdom | Cache | SSL | CloudFront SSL | 3.82s | -0.08s | +2% | ||
| Pingdom | Cache | SSL | 3.90s | ||||
| Pingdom | |||||||
| Pingdom | Cache | SPDY | CloudFront SSL | 3.86s | -0.11s | +3% | |
| Pingdom | Cache | SPDY | CloudFront SSL | PageSpeed | 3.95s | -0.02s | +0.5% |
| Pingdom | Cache | SPDY | 3.97s | ||||
| Pingdom | Cache | SPDY | PageSpeed | 4.06s | +0.09s | ||
| Pingdom | |||||||
| Pingdom | No cache | SPDY | 28.45s | ||||
| Pingdom | No cache | SSL | 29.72s | ||||
| Pingdom | No cache | HTTP | 29.89s | ||||
| Chrome | |||||||
| Chrome | Cache | SPDY | CloudFront SSL | PageSpeed | 3.58s | -1.66s | +31% |
| Chrome | Cache | SSL | CloudFront SSL | PageSpeed | 5.24s |
Conclusion
The best result we got is CloudFront + PageSpeed over HTTP with a speed increase of 34% (-1.84s), the second best was from SSL to SPDY with CloudFront + PageSpeed enabled on both with 31% (-1.66s).
The odd result here that there’s no real difference between using only SPDY (3.97s) or only SSL (3.90s) or using SPDY + CloudFront SSL + PageSpeed (3.95s) or using SSL + CloudFront SSL + PageSpeed (3.97s). That might be explained by that SSL slows down all connections, and SPDY only speeds up connections if everything is on one server. If you use a CDN like CloudFront you add more servers to the load times, which the client has to connect with. On SPDY the first connection actually takes a lot of time. From then on it’s fast. So for some sites, it might be faster to NOT use a CDN if they use SPDY.
PageSpeed’s results were negligible.
I managed to speed up the site on HTTP with 34% and on SPDY with 31%. I will keep CloudFront’s CDN enabled for the mere fact that the CDN still might help speed up connections in remote parts of the world. That’s a guess though.
P.S. I'm on Twitter too if you'd like to follow more of my stories. And I wrote a book called MAKE about building startups without funding. See a list of my stories or contact me. To get an alert when I write a new blog post, you can subscribe below: