A Step by Step Guide to setting up an AutoScaling Private WebPageTest instance

If you have any interest in website performance optimisation, then you have undoubtebly heard of WebPageTest. Being able to test your websites from all over the world, on every major browser, on different operating systems, and even on physical mobile devices, is the greatest ever addition to a web performance engineer’s toolbox.

One small shelf of Pat Meenan's epic WebPageTest device lab

The sheer scale of WebPageTest, with test agents literally global (even in China!), of course means that queues for the popular locations can get quite long – not great when you’re in the middle of a performance debug session and need answers FAST!

Also since these test agents query your website from the public internet they won’t be able to hit internal systems – for example pre-production or QA, or even just a corporate intranet that isn’t accessible outside of a certain network.

In this article I’ll show you how to set up your very own private instance of WebPageTest in Amazon AWS with autoscaling test agents to keep costs down

Once you have this you can optionally extend the setup, for example:

  • Always-on test agent(s) – get your test result faster
  • Your own test agent within the China firewall – get realistic test results from the perspective of a user behind the Great Firewall
  • Automation and scripting – schedule regular tests
  • Reporting and visualisation – graph your tests to find trends

We’ll get on to those topics later, but first up let’s focus on building a solid WebPageTest foundation!

Getting familiar with AWS

You don’t need to be an AWS guru to get this all running. AWS can be confusing, however this setup only requires a few clicks throughout the AWS console. Make sure you have your own AWS account ready to go.

There are 2 or 3 main areas we will need to use in AWS for setting up our private WebPageTest instance:

  1. A user with permissions to create new EC2 instances which the WebPageTest server uses to spin up test agents, and optionally S3 permissions to archive the tests.
  2. A place to archive test results, freeing up space on the WebPageTest server itself (optional, but highly recommended – and cheap).
  3. A VM instance on which to host your WebPageTest server which orchestrates the whole process.

Let’s start off with –

1. Create a WebPageTest User

IAM is the AWS Identity and Access Management (aka IAM – "I am"…) area; we need to create a new programmatic user for the WebPageTest server to run under.

Log in to your AWS console and head over to IAM by finding it in the Services menu:

AWS IAM

From there, click Add user, enter a name, and select Programmatic Access:

AWS IAM - creating wpt user

Now we need to Set permissions. Select Attach Existing Policies and search for and select "AmazonEC2FullAccess" (so it can start test agents)…

AWS IAM - setting permissions for EC2

… and search for and select "AmazonS3FullAccess" (to archive tests into S3).

AWS IAM - setting permissions for S3

The tests can easily fill up even a large EC2 volume quickly, so archiving to S3 is strongly recommended – S3 is super cheap, EBS volumes are not.

Archiving tests to S3 won’t change how WebPageTest looks and feels; it will fetch the zipped test from S3 on demand incredibly quickly.

Copy the AWS Access Key ID and Secret Access Key for this new user somewhere as we’ll need them in a moment.

AWS IAM - user created

2. Configure Test Storage in S3

EBS volumes (the hard drive on a EC2 instance) will fill up quickly and, although expanding them is possible, it isn’t easy and it can’t be reversed. S3 is the super cheap, almost limitless, alternative.

Head over to the S3 area of your AWS console – again, you can find this under the Services menu:

AWS S3

Give your bucket a unique name – your bucket name cannot already exist in the same region, even in another user’s AWS account – and set the region where you’ll be creating your WebPageTest server:

AWS S3 - create a WPT bucket

With that done, now you’re ready to create the server!

3. Setting up the WebPageTest server on EC2

Now let’s create the actual WebPageTest server itself; this is quite a long set of steps so get that coffee ready!

Head over to the EC2 area within the AWS console and select Launch Instance:

AWS EC2

Search for "webpagetest" and select Community AMIs:

AWS EC2 - choose a WPT AMI

Select the top webpagetest-server result.

Choose instance size: t2.micro is ok to start with as you can always scale up if necessary, and t2.micro is currently free:

AWS EC2 - set AMI size

This is an important step: instead of launching the server and logging in to edit settings, we can actually define the settings in "user data" which is passed in to the instance at start-up.

If you prefer, you can connect to your WebPageTest instance after launching and configure these settings directly in the file /var /www /webpagetest /www /settings/ settings.ini; you can copy the sample and edit it.

Tap on Configure instance details, then Advanced, and paste in user data similar to this, filling in the blanks:

# User to start/stop agents and save tests to s3
ec2_key=<the IAM user access key ID>
ec2_secret=<the IAM user secret access key>

# Set the API key to use when programmatically enqueuing tests
# Can't think of one? check out http://new-guid.com/
api_key=<choose a super secret api key>

# show user timing marks in waterfalls - very handy
waterfall_show_user_timing=1

# better images for screenshots - you're using S3, right?
# so you have the storage!
iq=75
pngss=1

# archiving to s3
archive_s3_server=s3.amazonaws.com
archive_s3_key=<the IAM user key>
archive_s3_secret=<the IAM user secret>
archive_s3_bucket=<the WPT archive bucket>

# number of days to keep tests locally before archiving
archive_days=1

# run archive script hourly automatically as agents poll for work
cron_archive=1

The full list of options that can be set are over on the github repo for WebPageTest, as the settings.ini.sample file; as mentioned earlier, you could skip the user data step and set these options in a ini file to be created in /var /www /webpagetest /www /settings /settings.ini

AWS EC2 - set user data

Now we need to make sure we can access the server over port 80; for this you need to select Configure security group, then add rule and choose HTTP:

AWS EC2 - set security groups

Alright! Ready to rock! Let’s hit Review and Launch and after you’ve selected or created a keypair (used to log in to the instance), and after it’s finished initialising we’ll be given a URL, where it says Public DNS:

AWS EC2 - starting up

Head over to that URL and you should see the familiar WebPageTest homepage:

AWS EC2 - WPT server running, submitting a test

Try it out – pop in a URL, select a location, and submit the test. If you then check the EC2 area of your AWS account, after a moment you’ll notice a new instance starting up called "WebPagetest Agent":

AWS EC2 - WPT test agent being created

A new test agent can take a few minutes to actually connect and start testing – sometimes up to 10 minutes – but once connected it’ll automatically pick up the enqueued test and run it:

AWS EC2 - submitted test being picked up

That was easy, right? The server will update itself from the WebPageTest github repo regularly, as will the test agents. Your tests will be automatically archived to, and retrieved from, S3.

AWS EC2 - test running

Your private WebPageTest foundation has been laid! We will build on this over a few more articles.

BONUS) Scaling Up and Staying There

By default, only one test agent per 100 tests will be created per region. One of the reasons I wanted a private WebPageTest instance was my impatience at queueing up to find out how slow my sites are. If you can throw money at the problem, then change this in your user data:

# This will create one new test agent for every 5 tests queued
# *up to the location max*:
EC2.ScaleFactor=5

# The default max per location is 1, so you need to override it per
# location to enable scale out:
EC2.us-east-1.max=10
EC2.us-west-1.max=20
EC2.eu-west-1.max=15
...
# etc

If there are no more tests queued which that agent can pick up for an hour (or however long you want; an hour is the default) then it will be terminated. This means that you’ll have to wait for the agent to be recreated next time, but it also means you’re not paying for an EC2 instance that you’re not using; these agents should be c4.large, so they’re not free tier eligible.

To avoid this you can alter the user data (you’ll need to stop the WebPageTest server instance before you can edit it) to add in a line for each location you want to keep an agent always available, e.g.:

 EC2.us-east-1.min=1
 EC2.us-west-1.min=1
 EC2.eu-west-1.min=1

It will now scale down to a minimum of one test agent for each of the specified regions.

Debugging

Wondering why your test isn’t starting? Getting impatient? Check /install to make sure you have green everywhere, except at the bottom where no test agents will exist (since they’re spun up on demand):

AWS debugging - install check

You can also check /getTesters.php?f=html to see what is connected:

AWS debugging - no test agents connected

The test agents are only created on demand, so let’s ensure there’s a test actually registered at /getLocations.php?f=html:

AWS debugging - confirming a test has been submitted for a location

You can also log in and check what’s happening on the server:

ssh -i "<your keypair>.pem" [email protected]<the url of your instance>

e.g.

ssh -i "webpagetest.pem" [email protected]

Now let’s check nginx for requests to getwork.php – this is the URL that test agents poll to pick up the next test in their queue:

tail -f /var/log/nginx/access.log | grep getwork

e.g.

[email protected]:~$ tail -f /var/log/nginx/access.log | grep getwork
127.0.0.1 - - [25/Apr/2019:20:42:01 +0000] "GET /work/getwork.php HTTP/1.1" 200 5 "-" "Wget/1.15 (linux-gnu)"
127.0.0.1 - - [25/Apr/2019:20:43:01 +0000] "GET /work/getwork.php HTTP/1.1" 200 5 "-" "Wget/1.15 (linux-gnu)"
127.0.0.1 - - [25/Apr/2019:20:44:01 +0000] "GET /work/getwork.php HTTP/1.1" 200 5 "-" "Wget/1.15 (linux-gnu)"
127.0.0.1 - - [25/Apr/2019:20:45:02 +0000] "GET /work/getwork.php HTTP/1.1" 200 5 "-" "Wget/1.15 (linux-gnu)"

hmm… not a lot happening there. That’s the server pinging itself for some reason; you can tell this since the requests are from "127.0.0.1". Usually after a few more minutes you’ll see:

172.31.20.24 - - [25/Apr/2019:20:45:37 +0000] "GET /work/getwork.php?f=json&shards=1&reboot=1&location=eu-west-1&pc=EC2AMAZ-CO5OM1I&key=4d446cb0d60d76dced79ffa39cf3c1e953db594b&ec2=i-0323a543b58e9ba00&ec2zone=eu-west-1a&version=190221.200223&screenwidth=1920&screenheight=1200&freedisk=7.464&upminutes=9 HTTP/1.1" 200 282 "-" "python-requests/2.21.0"
127.0.0.1 - - [25/Apr/2019:20:46:01 +0000] "GET /work/getwork.php HTTP/1.1" 200 5 "-" "Wget/1.15 (linux-gnu)"
172.31.20.24 - - [25/Apr/2019:20:46:39 +0000] "GET /work/getwork.php?f=json&shards=1&reboot=1&location=eu-west-1_IE11&pc=EC2AMAZ-CO5OM1I&key=4d446cb0d60d76dced79ffa39cf3c1e953db594b&ec2=i-0323a543b58e9ba00&ec2zone=eu-west-1a&version=190221.200223&screenwidth=1920&screenheight=1200&freedisk=7.404&upminutes=10 HTTP/1.1" 200 31 "-" "python-requests/2.21.0"
172.31.20.24 - - [25/Apr/2019:20:46:39 +0000] "GET /work/getwork.php?f=json&shards=1&reboot=1&location=eu-west-1&pc=EC2AMAZ-CO5OM1I&key=4d446cb0d60d76dced79ffa39cf3c1e953db594b&ec2=i-0323a543b58e9ba00&ec2zone=eu-west-1a&version=190221.200223&screenwidth=1920&screenheight=1200&freedisk=7.404&upminutes=10 HTTP/1.1" 200 284 "-" "python-requests/2.21.0"

Notice the timestamps show there was nothing going on for over 3 minutes after I had started to get impatient; AWS EC2 test agents can take a while to wake up and connect, so bear this in mind.

AWS debugging - test agent connected

Interesting point: notice the querystring parameters in the request; the test agent informs the server a lot about itself, even including available disc space.

Things to check:

  • User Data – did you set the correct IAM details for EC2? If not, then the WebPageTest server will not be able to spin up the agents
  • IAM – did you give EC2 permissions to the IAM user you created? If not, same issue as above.
  • WPT logs – you can check the logs for issues in the WebPageTest logs, which can be found in /var/www/webpagetest/www/log/ (not /logs/, as this is where the submitted tests are logged, not errors).
  • nginx logs – you can check if the agents are able to connect to your the WebPageTest server at all.

Summary

Hopefully you followed along and successfully set up your own private WebPageTest instance, and can now queue up all the tests you like!

If you have issues, then hit me up on twitter or head over to the WebPageTest forums – seriously, given how many times the same question must be asked in there, the gang are exceptionally patient and helpful.

Good luck!

Lazy Loading Images? Don’t Rely On JavaScript!

So much of the internet is now made up of pages containing loads of images; just visit your favourite shopping site and scroll through a product listing page for an example of this.

As you can probably imagine, bringing in all of these images when the page loads can add unnecessary bloat, causing the user to download lots of data they may not see. It can also make the page slow to interact with, due to the page layout constantly changing as new images load in, causing the browser to reprocess the page.

One popular method to deal with this is to “Lazy Load” the images; that is, to only load the images just before the user will need to see them.

If this technique is applied to the “above the fold” content – i.e., the first average viewport-sized section of the page – then the user can get a significantly faster first view experience.

So everyone should always do this, right?

Before we get on to that, let’s look at how this is usually achieved. It’s so easy to find a suitable jQuery plugin or angularjs module that a simple install command later and you’re almost done; just add a new attribute to image tags or JavaScript method to process the images you want to delay loading for.

So surely this is a no-brainer?

Continue reading

I’ll Be Speaking At OSCON EU

OSCON

I’m lucky enough to have been allowed to speak at OSCON EU this year, with – as per usual – the awesome Dean “Wrote The Book On Web Performance” Hume (that’s his legal full name, thanks to him having actually written a book on web performance).

OSCON – the Open Source Conference – “celebrates, defines, and demonstrates the best that open source has to offer.” From small businesses to the enterprise, open source is the first choice for engineers around the world, and OSCON is the place to celebrate that.

The workshop we’ll be presenting is Automating Web Performance – first thing on Wednesday morning.

As regular readers may notice, I do like my web performance optimisation – in fact, I’ve spoken about it once or twice.

What’s different this year is that .Net is finally open source, so, as long time .net-ers, we felt it was time to spread the .Net love amongst the open source community! I’m really excited for this conference – a different focus (I’m used to almost everyone at the conference talking about a similar thing to me – i.e., web performance optimisation – and the line up at OSCON is exceptionally diverse), a different location (the wonderful city of Amsterdam) and a different format for us (a 90 minute workshop instead of a 40 minute presentation).

We’ll be talking about tech that, although not specific to .Net, can be applied to such web projects – and to pretty much any other tech stack too – in order to reap the benefits of automated web performance optimisation.

We’ll go through automating the optimisation of images, css, javascript, and html, as well as introducing WebP images, critical css, unused css, and ultimately automating the continual testing of these optimisations.

It’s going to be a great start to the third day of the conference; if you’re attending, and you’re looking for something fun to start your last day with, then come and sit in with us – you won’t regret it!

If you’re not already attending and I’ve managed to convince you how wrong you are, then perhaps you’d also like to get 25% discount off of your ticket? How does that sound? And a cookie? Just use the code SPEAKER25 when you buy your ticket for that discount, and come find me at the conference for that cookie. *

(* cookie may not exist; the cookie is a lie)