Automating WebPageTest using the nodejs webpagetest-api package

WebPageTest-api NodeJS

Hopefully you’ve already had a chance to play around with the amazing WebPageTest during your website performance testing adventure so far.

In case not, I have a few articles you might like to browse, to help you get up to speed using this fantastic, free, open source, website performance testing tool.

It has a website interface and also an API, which I went through in the previous article.

In this article I’ll show you how to use the incredible webpagetest-api nodejs package to make the orchestration and automation of your WebPageTest setup even easier!

Continue reading

Automating WebPageTest via the WebPageTest API

webpagetest robots

WebPageTest is incredible. It allows us to visit a web page, enter a few values and then produce performance results from any destination around the world. Best of all, you can do this in many different possible browser configurations; even on many different real devices.

If you’re doing this a lot, then using that simple web form can become the bottleneck to rapidly iterating on your web performance improvements.

In this article I’ll show you how to easily execute your web performance tests in a simple, repeatable, automated way using the WebPageTest API.

Continue reading

Smart TV 101 : Part #3 – Deploying to TV

I’m committing to doing 12 months of “101”s; posts and projects themed at beginning something new (or reasonably new) to me. January was all about node development awesomeness. February is all about Smart TV apps.

Deploying to a TV

Now that we’ve got a basic Smart TV app this post will investigate how to get that app on to the TV itself.

Packaging using the IDE

During the initial installation of the IDE you will have been asked to installed Apache; this is what it’s all been leading up to! You actually just need a web server on your home network somewhere; doesn’t have to be apache, doesn’t have to be on your developer pc.

Prerequisites

Make sure you’ve configured your Server settings within the IDE preferences:
samsung-packaging-server-prefs
samsung-packaging-server-prefs-root

The packaging process will drop a zip file into a Widget/ subdirectory of this directory.

Initiating package creation

As for actually creating the package, if you’re using the Eclipse IDE then you’re spoiled for choice: highlight the project in your project explorer and then either

  1. Click the Samsung App Packaging button
    samsung-app-packaging-1
  2. Click the Samsung Smart TV SDK menu, then click App Packaging
    samsung-smart-sdk-packaging-menu
  3. Right click the project in project explorer, Samsung Smart TV SDK, Packaging
    samsung-sdk-packaging-context-menu

Whichever you do you’ll end up with the same results:
samsung-packaging-dialog
samsung-packaging-confirmation

Results

Assuming you’ve set up the server settings in your preferences then you’ll end up with:

  1. a zip file placed within the SDK installation’s Package/ directory
    samsung-package-sdk-dir
  2. the same zip file placed in a Widget/ subdirectory on your configured server
    samsung-package-widget-dir
  3. a new (or updated) widgetlist.xml file in the root of your configured server’s directory

    samsung-widgetlist-xml

Make sure that you can browse to this file and that Apache is running by opening a browser and putting in http://<your development pc’s IP>/widgetlist.xml

Anatomy of a package and a widgetlist

So what is a package made of? Looking at the image above for the the zip file that’s created you’ll see that it looks almost identical to the contents of your application within the workspace:
samsung-app-workspace

So essentially the packaging step is zipping up your project directory, putting it into a specified web server subdirectory, and updating an XML file. Obviously, you shouldn’t need an IDE or SDK to do this sort of thing and I’ll be getting on to this development & deployment process without using Eclipse or installing Apache in a later post.

Deploying!!

Now that we have a package it’s time to load it on to your Smart TV. For this post I’ll be talking about deploying from the development pc via your home network, and in a later post will be talking about loading in packages externally.

TV setup

Make sure your TV is connected to your network and that your development pc’s Windows Firewall is off (or at least configured to allow local network traffic).

  • Turn on the TV
  • Go to your app hub/Smart Hub
  • Press the Login button
  • Create an account using the username “develop” and set a password

developer account

After you’ve successfully created the develop user you need to

  • Open the Settings menu
  • Open the new Development sub menu
  • Choose Setting server IP and enter the IP of your development PC
  • Choose User application synchronisation to check the apps that are listed in widgetlist.xml and install (or update) them all

download dev app

You should now find your application on the App Hub screen with a little red “user” banner over it; select it to run it, just like any other app.

asos-app-running-on-smart-tv-1

Node.js 101: Wrap up

Year of 101s, Part 1 – Node January

Summary – What was it all about?

I set out to spend January learning some node development fundementals.

Part #1 – Intro

I started with a basic intro to using node – a Hello World – which covered what node.js is, how to create the most basic of all programs, and mentioned some of the development environments.

Part #2 – Serving web content

Second was creating a very simple node web server, which covered using nodemon to develop your node app, the concept of exports, basic request routing, and serving various content types.

Part #3 – A basic API

Next was a simple API implementation, where I proxy calls to the Asos API, return a remapped subset of the data returned, reworked the routing creating basic search functionality and a detail page, and touched on being able to pass in command line arguements.

Part #4 – Basic deployment and hosting with Appharbor, Azure, and Heroku

Possibly the most interesting and fun post for me to work on involved deploying the node code on to three cloud hosting solutions where I discovered the oddities each provider has, various solutions to the problems this raises, as well as some debugging cleverness (nice work, Heroku!). The simplicity of a git-remote-push-deploy process is incredible, and really makes quick application development and hosting even more enjoyable!

Part #5 – Packages

Another interesting one was getting to play with node packages, the node package manager (npm), the express web framework, jade templating engine, and stylus css pre-processor, and deploying node apps with packages to cloud hosting.

Part #6 – Web-based development

The final part covered the fantastic Cloud9IDE, including a (very) basic intro to github, and how Cloud9 can still be used in developing and deploying directly to Azure, Appharbor, or Heroku.

What did I get out of it?

I really got into githubbing and OSSing, and really had to try hard to not over stretch myself as I had starting forking repos to try and make a few tweaks to things whilst working on the node month.

It has been extremely inspiring and has opened up so many other random tangents for me to explore in other projects at some other time. Very motivating stuff.

I’ve now got a month of half decent blog posts – I had only intended to do a total of 4 posts but including this one I’ve done 7, since I kept adding more information as it turned up and needed to split a few posts into two.

Also I’ve learned a bit about blogging; trying to do posts well in advance allowed me to build up the details once I’d discovered more whilst working on subsequent posts. For example, how Appharbor and Azure initially track master – but can be configured to track different branches. Also, debugging with Heroku only came up whilst working with packages in Heroku.

Link list

Node tutorials and references

Setting up a node development environment on Windows
Node Beginner – a great article, and I’ve also bought the associated eBooks.
nodejs.org – the official node site, the only place to go for reference

Understanding Javascript better

Execution in The Kingdom of Nouns
Object Orientation and Inheritance in Javascript

Appharbor

Appharbor and git

Heroku

Heroku toolbelt download and reference
node on Heroku

Azure

Checkout what Azure can do!

February – coming up, Samsung Smart TV App Development!

Yeah, seriously. How random is that?.. 🙂

Node.js 101 : Part #3 – A Basic API

Following on from my recent post about doing something this year, I’m committing to doing 12 months of “101”s; posts and projects themed at begining something new (or reasonably new) to me.

January is all about node, and I started with a basic intro, then cracked open a basic web server with content-type manipulation and basic routing.

Building and calling an API in node

Now on to the meat of this month; building a basic RESTful API. I don’t plan on writing the underlying business logic myself, so will just wrap an existing API in order to further demonstrate the routing, content type usage, and proxying calls to another server.

For this post I’ll be using the Asos API for querying the Asos database of clothes and returning the data necessary to build other basic applications on; intially a web site, but later on various apps on various devices.

The Underlying API: Asos.com

Asos, the online fashion “destination”, had an API open for developers to mess aorund with for a short period and as one of the first people to get involved I managed to snap up an api key. This will give me the ability to query the product catalogue and do basic functions such as adding products to a basket.

Asos
asos-1

Asos API
asos-api-1

An example request takes the format:

http://api1.asos.com/product/{productId}/{locale}/{currency}?api_key={apiKey}

and an example response is:

{
   "BasePrice":35.0,
   "Brand":"ASOS",
   "Colour":null,
   "CurrentPrice":"£35.00",
   "InStock":true,
   "IsInSet":false,
   "PreviousPrice":"",
   "PriceType":"Full",
   "ProductId":1703489,
   "ProductImageUrls":[
      "http://images.asos.com/inv/media/9/8/4/3/1703489/red/image1xxl.jpg",
      "http://images.asos.com/inv/media/9/8/4/3/1703489/image2xxl.jpg",
      "http://images.asos.com/inv/media/9/8/4/3/1703489/image3xxl.jpg",
      "http://images.asos.com/inv/media/9/8/4/3/1703489/image4xxl.jpg"
   ],
   "RRP":"",
   "Size":null,
   "Sku":"101050",
   "Title":"ASOS Fringe Sleeve Mesh Crop",
   "AdditionalInfo":"100% Polyester\n\n\n\n\n\nSIZE &amp; FIT \n\nModel wears: UK 8/ EU 36/ US 4\n\n\n\nSize UK 8/ EU 36/ US 4 side neck to hem measures: 46cm/18in",
   "AssociatedProducts":[{
         "BasePrice":35.0,
         "Brand":"ASOS",
         "Colour":null,
         "CurrentPrice":"£35.00",
         "InStock":false,
         "IsInSet":false,
         "PreviousPrice":"",
         "PriceType":"Full",
         "ProductId":1645550,
         "ProductImageUrls":[
            "http://images.asos.com/inv/media/0/5/5/5/1645550/black/image1l.jpg"
         ],
         "RRP":"",
         "Size":null,
         "Sku":null,
         "Title":"ASOS Panel Mesh Body Contour Top",
         "ProductType":"Recommendations"
      }],
   "CareInfo":"Machine wash according to instructions on care label",
   "Description":"Fringed crop top, featuring a reinforced boat neckline, raglan style slashed sleeves with tasselled fringe trim, and a cropped length, in a sheer finish.",
   "Variants":[
      {
         "BasePrice":35.00,
         "Brand":null,
         "Colour":"Beige",
         "CurrentPrice":"£35.00",
         "InStock":true,
         "IsInSet":false,
         "PreviousPrice":"",
         "PriceType":"Full",
         "ProductId":1716611,
         "ProductImageUrls":[
            "http://images.asos.com//inv/media/9/8/4/3/1703489/beige/image1xxl.jpg"
         ],
         "RRP":"",
         "Size":"UK 6",
         "Sku":null,
         "Title":null
      }]
}

For the purposes of this post all I want to do is wrap a couple of the slightly confusing and overly complex Asos API calls with some really basic, more RESTy, ones.

To do this I’m going to initially create a new module called:

proxy.js

var http = require('http');

function getRemoteData(host, requestPath, callback){
    var options = {
        host: host,
        port: 80,
        path: requestPath
    };
 
    var buffer = '';
    var request = http.get(options, function(result){
        result.setEncoding('utf8');
        result.on('data', function(chunk){
            buffer += chunk;
        });
        result.on('end', function(){
            callback(buffer);
        });
    });
    request.on('error', function(e){console.log('error from proxy call: ' + e.message)});
    request.end();
};

exports.getRemoteData=getRemoteData;

As you can see, all this does is make an HTTP GET call to a remote server, passing the “options” object.

Using the “on” event wiring up notation, I’ve just appended the chunks of data returned from the GET call to a variable, which is then passed to the referenced callback function.

Now I’ll wire this up:
requestHandlers.js:

var proxy = require('./proxy');

function products(response) {
console.log("Request handler 'products' was called");

  var host = 'api1.asos.com';
  var requestPath = '/productlisting/search/jeans/1/PriceAscending/en_API/GBP?api_key={snipped api key}';
  
  response.writeHead(200, {"Content-Type": "application/json"});

  proxy.getRemoteData(host, requestPath, function(json){
  response.write(json);
    response.end();
  });  
}

exports.products = products;

I’m removing the previously entered hello, goodbye, and favicon routes for brevity. Notice the reference to the proxy module at the top as well as the new handler itself.

The URL used above executes a product search for the term “jeans”.

Wire it all up:
server.js:

var http = require("http"),
    url = require("url");

function start(route, handle, port) {
  function onRequest(request, response) {
    var pathname = url.parse(request.url).pathname;
    route(handle, pathname, response);
  }

http.createServer(onRequest).listen(port);
  console.log("Server has started listening on port " + port);
}

exports.start = start;

app.js

var server = require("./server"),
  router = require("./route"),
  requestHandlers = require("./requestHandlers");

var handle = {}
handle["/products"] = requestHandlers.products

var port = process.env.PORT || 3000;
server.start(router.route, handle, port);

Kick off

nodemon app.js

If you were to have an API key and had put it in the URL above, you’d see something like:

asos-products-1

Right. Ok. That’s a lot of data. Just for now I’d like to make it easier to view, so I’ll limit what is returned and also just write out a basic HTML page.

requestHandlers.js:

var proxy = require('./proxy');

function products(response) {
console.log("Request handler 'products' was called");

  var host = 'api1.asos.com';
  var requestPath = '/productlisting/search/jeans/1/PriceAscending/en_API/GBP?api_key={snipped api key}';
  response.writeHead(200, {"Content-Type": "text/html"});

  proxy.getRemoteData(host, requestPath, function(json){
    var data = JSON.parse(json);

  var html = "<h1>Asos Search for JEANS</h1>";
  response.write(html);

    for(var i=0; i<data.ItemCount; i++) {
      if (data.Listings[i] != null){
      response.write("<li>"
       + data.Listings[i].Title + "<br /><img src='" 
       + data.Listings[i].ProductImageUrl + "' /></li>");
      }
    }

    response.end();
  });  
}

exports.products = products;

Given that the Asos Api returns valid JSON I can just parse it and then access the structure of that JSON; in this case the ItemCount & Listings at the top level and Title & ProductImageUrl within Listings.

This will now display something like:
asos-products-2

(Really? A beanie is the first result in the search for “jeans”? Anyway…)

Actually searching

Next we’ll just make the request actually execute a search with the value passed in to our own API, using the format “/products/{search term}”

Firstly I’ll edit the router to take the primary route handler from the first part of the URL (e.g “http://localhost:3000/products/jeans”) and pass the full path into the router for further use.

router.js:

function route(handle, pathname, response) {
  var root = pathname.split('/')[1];

  if (typeof handle[root] === 'function') {
    handle[root](response, pathname);
  } else {
    console.log("No request handler found for " + pathname);
    response.writeHead(404, {"Content-Type": "text/plain"});
    response.write("404 Not found");
    response.end();
  }
}

exports.route = route;

Next change the request handler to pick out the next section from the url e.g. “http://localhost:3000/products/jeans

requestHandlers.js:

var proxy = require('./proxy');

function products(response) {
console.log("Request handler 'products' was called");

  var search = path.split('/')[2];
  var host = 'api1.asos.com';
  var requestPath = '/productlisting/search/' + search + '/1/PriceAscending/en_API/GBP?api_key={snipped api key}';
  
  response.writeHead(200, {"Content-Type": "text/html"});

  proxy.getRemoteData(host, requestPath, function(json){
    var data = JSON.parse(json);

  var html = "<h1>Asos Search for " + search + "</h1>";
  response.write(html);

    for(var i=0; i<data.ItemCount; i++) {
      if (data.Listings[i] != null){
      response.write("<li>"
       + data.Listings[i].Title + "<br /><img src='" 
       + data.Listings[i].ProductImageUrl + "' /></li>");
      }
    }

    response.end();
  });  
}

exports.products = products;

One last tweak to the initialisation file to remove a leading slash which isn’t needed now that we’re splitting the url to match instead of using the full url path:

app.js:

var server = require("./server"),
  router = require("./router"),
  requestHandlers = require("./requestHandlers");

var handle = {}
handle["products"] = requestHandlers.products;

var port = process.env.PORT || 3000;
server.start(router.route, handle, port);

We now have basic search capabilities:
asos-products-search-1

Now let’s get a basic product detail page working. For this I should need to just add a new request handler and wire it up.

requestHandlers.js:

var proxy = require('./proxy');

function products(response, path) {
console.log("Request handler 'products' was called");

  var search = path.split('/')[2];
  var host = 'api1.asos.com';
  var requestPath = '/productlisting/search/' + search + '/1/PriceAscending/en_API/GBP?api_key={snipped api key}';
  
  response.writeHead(200, {"Content-Type": "text/html"});

  proxy.getRemoteData(host, requestPath, function(json){
    var data = JSON.parse(json);

  var html = "<h1>Asos Search for " + search + "</h1>";
  response.write(html);

    for(var i=0; i<data.ItemCount; i++) {
      if (data.Listings[i] != null){
      response.write("<li><a href='/product/" + data.Listings[i].ProductId + "'>"
       + data.Listings[i].Title + "<br /><img src='" 
       + data.Listings[i].ProductImageUrl + "' /></a></li>");
      }
    }

    response.end();
  });  
}

function product(response, path) {
  console.log("Request handler 'product' was called for " + path);

  var productId = path.split('/')[2];
  var host = 'api1.asos.com';
  var requestPath = '/product/' + productId + '/en_API/GBP?api_key={snipped api key}';

  response.writeHead(200, {"Content-Type": "text/html"});
  proxy.getRemoteData(host, requestPath, function(json){
  var data = JSON.parse(json);

    var html = "<h1>" + data.Title + "</h1>"
    + "<img src='" + data.ProductImageUrls[0].replace('xxl','xl') + "' />"
    response.write(html);
    response.end();
  });  
}
exports.products = products;
exports.product = product;

As well as the new handler I’ve also added a link from the listing page to the detail page, just for FUN.

app.js:

var server = require("./server"),
  router = require("./router"),
  requestHandlers = require("./requestHandlers");

var handle = {}
handle["products"] = requestHandlers.products;
handle["product"] = requestHandlers.product;

var port = process.env.PORT || 3000;
server.start(router.route, handle, port);

asos-product-1

Back to JSON

Ok, so that’s a very basic website wrapped around an API. Since I plan to use this wrapper as a basic API itself I’m going to revert it to returning JSON and simplify the data structure for my needs.

requestHandlers.js:

var proxy = require('./proxy');

function products(response, path) {
console.log("Request handler 'products' was called");

  var search = path.split('/')[2];
  var host = 'api1.asos.com';
  var requestPath = '/productlisting/search/' + search + '/1/PriceAscending/en_API/GBP?api_key={stripped api key}';
  
  response.writeHead(200, {"Content-Type": "application/json"});

  proxy.getRemoteData(host, requestPath, function(json){
    var data = JSON.parse(json);

  var newJson = {
    category: data.Description,
    products: []
  };

  data.Listings.forEach(function(listing){
      newJson.products.push({
        id: listing.ProductId,
        title: listing.Title,
        price: listing.CurrentPrice,
        image: listing.ProductImageUrl[0]
      })
    });

    response.write(JSON.stringify(newJson));
    response.end();
  });  
}

function product(response, path) {
  console.log("Request handler 'product' was called for " + path);

  var productId = path.split('/')[2];
  var host = 'api1.asos.com';
  var requestPath = '/product/' + productId + '/en_API/GBP?api_key={snipped api key}';

  response.writeHead(200, {"Content-Type": "application/json"});
  proxy.getRemoteData(host, requestPath, function(json){
  var data = JSON.parse(json);

    var newJson = {
      id: data.ProductId,
      title: data.Title,
      price: data.CurrentPrice,
      available: data.InStock,
      image: data.ProductImageUrls[0]
    };

    response.write(JSON.stringify(newJson));
    response.end();
  });  
}
exports.products = products;
exports.product = product;

Which ends up looking like:
asos-json-1

That’ll do me for now, even though it would be nice to abstract the mapping out somewhere else. Out of scope for me at the moment though.

Once last thing for this post:

Passing in command line arguments

Throughout this post I’ve been diligently snipping out my API key before pasting the code in. There are many approaches to dev/qa/staging/production configuration management (some as basic as a text file, some a bit more complex) which would handle this sort of thing but for my immediate requirements I will just pass the API key in as a command line argument.

To handle this I need to edit the initialisation code in order to pick up any args passed, and documented on the nodejs.org site:

app.js:

var server = require("./server"),
  router = require("./router"),
  requestHandlers = require("./requestHandlers");

var handle = {}
handle["products"] = requestHandlers.products;
handle["product"] = requestHandlers.product;

var apiKey = process.argv[2];
var port = process.env.PORT || 3000;
server.start(router.route, handle, port, apiKey);

Now just pass that value around the rest of the system:

server.js:

var http = require("http"),
  url = require("url");

function start(route, handle, port, apiKey) {
  function onRequest(request, response) {
    var pathname = url.parse(request.url).pathname;
    route(handle, pathname, response, apiKey);
  }

  http.createServer(onRequest).listen(port);
  console.log("Server has started listening on port " + port);
}

exports.start = start;

route.js:

function route(handle, pathname, response, apiKey) {
  var root = pathname.split('/')[1];

  if (typeof handle[root] === 'function') {
    handle[root](response, pathname, apiKey);
  } else {
    console.log("No request handler found for " + pathname + " (" + root+ ")");
    response.writeHead(404, {"Content-Type": "text/plain"});
    response.write("404 Not found");
    response.end();
  }
}

exports.route = route;

requestHandlers.js:

var proxy = require('./proxy');

function products(response, path, apiKey) {
console.log("Request handler 'products' was called");

  var search = path.split('/')[2];
  var host = 'api1.asos.com';
  var requestPath = '/productlisting/search/' + search + '/1/PriceAscending/en_API/GBP?api_key=' + apiKey;
  
  response.writeHead(200, {"Content-Type": "application/json"});

  proxy.getRemoteData(host, requestPath, function(json){
    var data = JSON.parse(json);

  var newJson = {
    category: data.Description,
    products: []
  };

  data.Listings.forEach(function(listing){
      newJson.products.push({
        id: listing.ProductId,
        title: listing.Title,
        price: listing.CurrentPrice,
        image: listing.ProductImageUrl[0]
      })
    });

    response.write(JSON.stringify(newJson));
    response.end();
  });  
}

function product(response, path, apiKey) {
  console.log("Request handler 'product' was called for " + path);

  var productId = path.split('/')[2];
  var host = 'api1.asos.com';
  var requestPath = '/product/' + productId + '/en_API/GBP?api_key=' + apiKey;

  response.writeHead(200, {"Content-Type": "application/json"});
  proxy.getRemoteData(host, requestPath, function(json){
  var data = JSON.parse(json);

    var newJson = {
      id: data.ProductId,
      title: data.Title,
      price: data.CurrentPrice,
      available: data.InStock,
      image: data.ProductImageUrls[0]
    };

    response.write(JSON.stringify(newJson));
    response.end();
  });  
}
exports.products = products;
exports.product = product;

Then to pass in the api key just change the nodemon call to

nodemon app.js myApIK3y

The files for this post can be found over on github

Coming up

The next post this month will cover some nice deployment & hosting options for node!