The fantastic WebPageTest, free to use and public, has been available to set up your own private instances for many years; I wrote this up a while back, and scripted a Terraform version to make this as easy and automated as possible.
For AWS it was just a case of creating an EC2 instance (other installation options are available) with a predefined WPT server AMI (amazon machine image), add in a few configuration options and boom – your very own, autoscaling, globally distributed, website performance testing solution! New test agents would spin up automatically in other AWS regions, all based on WebPageTest Agent AMIs.
In 2020 WebPageTest was bought by Catchpoint and we finally saw improvements being made, pull requests being closed, and the WebPageTest UI getting a huge update; things were looking great for WebPageTest enthusiasts! If you havent heard of Catchpoint before, they are a company who are all about global network and web application monitoring, so a good match for WebPageTest.
Unfortunately, however, this resulted in the handy WebPageTest server EC2 AMIs no longer existing. If you want your own private installation you now need to build your own WebPageTest server from a base OS. It can be a bit tricky, though it gives you greater understanding of how it works under the hood, so hopefully you’ll feel more confident extending your installation in future.
In this article, I’ll show you how to create a WebPageTest private instance on AWS from scratch (no AMI), create your own private WebPageTest agents using the latest and greatest version of WebPageTest, and wire it all up.
A few days ago these workmen were using cutting machinery dangerously close to my broadband cables:
Shortly after this picture was taken – bang! No internet! They cut the cables while doing their work!
Two adults working from home on back to back video calls, one high-schooler also on video classes, and one primary-schooler with streaming classes – all suddenly disconnected from the world!
That afternoon we huddled around the kitchen table, mobile phones on with hotspots enabled to get through the rest of the day – but this wouldn’t work for regular use.
The broadband company said it wouldn’t be fixed for weeks due to how badly everything was damaged; the pavement would have to come up! I had to think of a pragmatic solution.
In this article I’ll go through the steps I took to completely swap my home broadband for a Raspberry Pi and a spare mobile phone – and show the results!
In my last article I walked through the steps to get an Android phone connected to a private WebPageTest server as a web page test agent.
In this post I’ll list out the actual hardware I used in order to get this all working!
Running detailed website performance tests is often necessary to understand how a website is experienced by an end user in order to identify opportunities for improvements.
WebPageTest.org gives us the ability to run these tests from all over the world – the public instance even gives us access to real devices, so we can check how a site works across different browsers on different versions of different operating systems on different real devices!
In my previous articles I explained how to easily set up your very own private, autoscaling, WebPageTest server. This private instance creates test agents in AWS, dotted around AWS regions, which can emulate a mobile browser; this uses the device emulation in Chrome to throttle network, CPU, memory, etc and change the available screen size.
While this mobile emulation is simple to set up and use, sometimes an emulator isn’t enough; device-specific edge cases, operating system limitations, and performance on a real device may need to be validated to get confidence that everything works as expected in the real world.
In this article I’ll show you how to set up an Android phone as your own WebPageTest agent to connect to your private WebPageTest server, controlled by a Raspberry Pi!
A while back I showed how to use the Microsoft Speaker Recognition APIs in the simplest way I could think of; using a web page to record audio and call the various APIs to set up and test Speaker Verification and Speaker Identification.
Honestly, the hardest part of this by far was getting the audio recorded in the correct format for the APIs to accept! I hacked the wonderful recorderjs from Matt Diamond to set the correct bitrate, channels, etc, and eventually got there through trial and error (and squinting at the minified source code of the Microsoft demo page)!
In the run up to //Build this year, there have been a lot of changes in the Microsoft AI space.
One of these changes managed to break my existing Speaker Recognition API applications (it’s still in Preview, so don’t be surprised!) by moving Speaker Recognition under the Speech Service project, slightly changing the APIs and their endpoints, and adding new functionality (exciting!)
In this article I’ll show the same web page implementation, but use the updated 2020 Speaker Recognition APIs, and introduce the new Verification API that doesn’t rely on a predefined list of passphrases.
In my recent article Eco Worriers: Saving the Planet, One Unoptimized Website at a Time for the fantastic annual Perf Planet advent calendar, I mentioned how I created a Private WebPageTest setup to use my own custom test agents, which were configured to use an extra Lighthouse plugin (The Green Web Foundation’s "greenhouse").
In this article I’ll show how to create custom WebPageTest agents, and how to configure your Private WebPageTest instance to use these instead of the default test agents.
Hopefully you’ve already had a chance to play around with the amazing WebPageTest during your website performance testing adventure so far.
In case not, I have a few articles you might like to browse, to help you get up to speed using this fantastic, free, open source, website performance testing tool.
It has a website interface and also an API, which I went through in the previous article.
In this article I’ll show you how to use the incredible webpagetest-api nodejs package to make the orchestration and automation of your WebPageTest setup even easier!
WebPageTest is incredible. It allows us to visit a web page, enter a few values and then produce performance results from any destination around the world. Best of all, you can do this in many different possible browser configurations; even on many different real devices.
If you’re doing this a lot, then using that simple web form can become the bottleneck to rapidly iterating on your web performance improvements.
In this article I’ll show you how to easily execute your web performance tests in a simple, repeatable, automated way using the WebPageTest API.
Update: November 2021
This article is out of date; there are no longer WebPageTest Server AMIs, so you need to install WPT on a base OS. There is an updated article – non-terraform – here WebPageTest Private Instance: 2021 Edition; I’m working on an updated Terraform article
In a previous article I went through the steps needed to create your own private, autoscaling, WebPageTest setup in Amazon AWS. It wasn’t particularly complicated, but it was quite manual; I don’t like pointing and clicking in a GUI since I can’t easily put it in version control and run it again and again on demand.
Fortunately, whatever you create within AWS can be described using a language called CloudFormation which allows you to define your infrastructure as code.
Unfortunately it’s not easy to understand (in my opinion!) and I could never quite get my head around it, which annoyed me no end.
In this article I’ll show you how to use Terraform to define your private autoscaling WebPageTest setup in easily understandable infrastructure as code, enabling an effortless and reproducable web performance testing setup, which you can then fearlessly edit and improve!
Update: November 2021
This article is out of date; there are no longer WebPageTest Server AMIs, so you need to install WPT on a base OS. The updated article is here WebPageTest Private Instance: 2021 Edition
If you have any interest in website performance optimisation, then you have undoubtedly heard of WebPageTest. Being able to test your websites from all over the world, on every major browser, on different operating systems, and even on physical mobile devices, is the greatest ever addition to a web performance engineer’s toolbox.
The sheer scale of WebPageTest, with test agents literally global (even in China!), of course means that queues for the popular locations can get quite long – not great when you’re in the middle of a performance debug session and need answers FAST!
Also since these test agents query your website from the public internet they won’t be able to hit internal systems – for example pre-production or QA, or even just a corporate intranet that isn’t accessible outside of a certain network.
In this article I’ll show you how to set up your very own private instance of WebPageTest in Amazon AWS with autoscaling test agents to keep costs down