Node.js 101: Wrap up

Year of 101s, Part 1 – Node January

Summary – What was it all about?

I set out to spend January learning some node development fundementals.

Part #1 – Intro

I started with a basic intro to using node – a Hello World – which covered what node.js is, how to create the most basic of all programs, and mentioned some of the development environments.

Part #2 – Serving web content

Second was creating a very simple node web server, which covered using nodemon to develop your node app, the concept of exports, basic request routing, and serving various content types.

Part #3 – A basic API

Next was a simple API implementation, where I proxy calls to the Asos API, return a remapped subset of the data returned, reworked the routing creating basic search functionality and a detail page, and touched on being able to pass in command line arguements.

Part #4 – Basic deployment and hosting with Appharbor, Azure, and Heroku

Possibly the most interesting and fun post for me to work on involved deploying the node code on to three cloud hosting solutions where I discovered the oddities each provider has, various solutions to the problems this raises, as well as some debugging cleverness (nice work, Heroku!). The simplicity of a git-remote-push-deploy process is incredible, and really makes quick application development and hosting even more enjoyable!

Part #5 – Packages

Another interesting one was getting to play with node packages, the node package manager (npm), the express web framework, jade templating engine, and stylus css pre-processor, and deploying node apps with packages to cloud hosting.

Part #6 – Web-based development

The final part covered the fantastic Cloud9IDE, including a (very) basic intro to github, and how Cloud9 can still be used in developing and deploying directly to Azure, Appharbor, or Heroku.

What did I get out of it?

I really got into githubbing and OSSing, and really had to try hard to not over stretch myself as I had starting forking repos to try and make a few tweaks to things whilst working on the node month.

It has been extremely inspiring and has opened up so many other random tangents for me to explore in other projects at some other time. Very motivating stuff.

I’ve now got a month of half decent blog posts – I had only intended to do a total of 4 posts but including this one I’ve done 7, since I kept adding more information as it turned up and needed to split a few posts into two.

Also I’ve learned a bit about blogging; trying to do posts well in advance allowed me to build up the details once I’d discovered more whilst working on subsequent posts. For example, how Appharbor and Azure initially track master – but can be configured to track different branches. Also, debugging with Heroku only came up whilst working with packages in Heroku.

Link list

Node tutorials and references

Setting up a node development environment on Windows
Node Beginner – a great article, and I’ve also bought the associated eBooks.
nodejs.org – the official node site, the only place to go for reference

Understanding Javascript better

Execution in The Kingdom of Nouns
Object Orientation and Inheritance in Javascript

Appharbor

Appharbor and git

Heroku

Heroku toolbelt download and reference
node on Heroku

Azure

Checkout what Azure can do!

February – coming up, Samsung Smart TV App Development!

Yeah, seriously. How random is that?.. 🙂

Node.js 101 : Part #3 – A Basic API

Following on from my recent post about doing something this year, I’m committing to doing 12 months of “101”s; posts and projects themed at begining something new (or reasonably new) to me.

January is all about node, and I started with a basic intro, then cracked open a basic web server with content-type manipulation and basic routing.

Building and calling an API in node

Now on to the meat of this month; building a basic RESTful API. I don’t plan on writing the underlying business logic myself, so will just wrap an existing API in order to further demonstrate the routing, content type usage, and proxying calls to another server.

For this post I’ll be using the Asos API for querying the Asos database of clothes and returning the data necessary to build other basic applications on; intially a web site, but later on various apps on various devices.

The Underlying API: Asos.com

Asos, the online fashion “destination”, had an API open for developers to mess aorund with for a short period and as one of the first people to get involved I managed to snap up an api key. This will give me the ability to query the product catalogue and do basic functions such as adding products to a basket.

Asos
asos-1

Asos API
asos-api-1

An example request takes the format:

http://api1.asos.com/product/{productId}/{locale}/{currency}?api_key={apiKey}

and an example response is:

{
   "BasePrice":35.0,
   "Brand":"ASOS",
   "Colour":null,
   "CurrentPrice":"£35.00",
   "InStock":true,
   "IsInSet":false,
   "PreviousPrice":"",
   "PriceType":"Full",
   "ProductId":1703489,
   "ProductImageUrls":[
      "http://images.asos.com/inv/media/9/8/4/3/1703489/red/image1xxl.jpg",
      "http://images.asos.com/inv/media/9/8/4/3/1703489/image2xxl.jpg",
      "http://images.asos.com/inv/media/9/8/4/3/1703489/image3xxl.jpg",
      "http://images.asos.com/inv/media/9/8/4/3/1703489/image4xxl.jpg"
   ],
   "RRP":"",
   "Size":null,
   "Sku":"101050",
   "Title":"ASOS Fringe Sleeve Mesh Crop",
   "AdditionalInfo":"100% Polyester\n\n\n\n\n\nSIZE & FIT \n\nModel wears: UK 8/ EU 36/ US 4\n\n\n\nSize UK 8/ EU 36/ US 4 side neck to hem measures: 46cm/18in",
   "AssociatedProducts":[{
         "BasePrice":35.0,
         "Brand":"ASOS",
         "Colour":null,
         "CurrentPrice":"£35.00",
         "InStock":false,
         "IsInSet":false,
         "PreviousPrice":"",
         "PriceType":"Full",
         "ProductId":1645550,
         "ProductImageUrls":[
            "http://images.asos.com/inv/media/0/5/5/5/1645550/black/image1l.jpg"
         ],
         "RRP":"",
         "Size":null,
         "Sku":null,
         "Title":"ASOS Panel Mesh Body Contour Top",
         "ProductType":"Recommendations"
      }],
   "CareInfo":"Machine wash according to instructions on care label",
   "Description":"Fringed crop top, featuring a reinforced boat neckline, raglan style slashed sleeves with tasselled fringe trim, and a cropped length, in a sheer finish.",
   "Variants":[
      {
         "BasePrice":35.00,
         "Brand":null,
         "Colour":"Beige",
         "CurrentPrice":"£35.00",
         "InStock":true,
         "IsInSet":false,
         "PreviousPrice":"",
         "PriceType":"Full",
         "ProductId":1716611,
         "ProductImageUrls":[
            "http://images.asos.com//inv/media/9/8/4/3/1703489/beige/image1xxl.jpg"
         ],
         "RRP":"",
         "Size":"UK 6",
         "Sku":null,
         "Title":null
      }]
}

For the purposes of this post all I want to do is wrap a couple of the slightly confusing and overly complex Asos API calls with some really basic, more RESTy, ones.

To do this I’m going to initially create a new module called:

proxy.js

var http = require('http');

function getRemoteData(host, requestPath, callback){
    var options = {
        host: host,
        port: 80,
        path: requestPath
    };
 
    var buffer = '';
    var request = http.get(options, function(result){
        result.setEncoding('utf8');
        result.on('data', function(chunk){
            buffer += chunk;
        });
        result.on('end', function(){
            callback(buffer);
        });
    });
    request.on('error', function(e){console.log('error from proxy call: ' + e.message)});
    request.end();
};

exports.getRemoteData=getRemoteData;

As you can see, all this does is make an HTTP GET call to a remote server, passing the “options” object.

Using the “on” event wiring up notation, I’ve just appended the chunks of data returned from the GET call to a variable, which is then passed to the referenced callback function.

Now I’ll wire this up:
requestHandlers.js:

var proxy = require('./proxy');

function products(response) {
console.log("Request handler 'products' was called");

  var host = 'api1.asos.com';
  var requestPath = '/productlisting/search/jeans/1/PriceAscending/en_API/GBP?api_key={snipped api key}';
  
  response.writeHead(200, {"Content-Type": "application/json"});

  proxy.getRemoteData(host, requestPath, function(json){
  response.write(json);
    response.end();
  });  
}

exports.products = products;

I’m removing the previously entered hello, goodbye, and favicon routes for brevity. Notice the reference to the proxy module at the top as well as the new handler itself.

The URL used above executes a product search for the term “jeans”.

Wire it all up:
server.js:

var http = require("http"),
    url = require("url");

function start(route, handle, port) {
  function onRequest(request, response) {
    var pathname = url.parse(request.url).pathname;
    route(handle, pathname, response);
  }

http.createServer(onRequest).listen(port);
  console.log("Server has started listening on port " + port);
}

exports.start = start;

app.js

var server = require("./server"),
  router = require("./route"),
  requestHandlers = require("./requestHandlers");

var handle = {}
handle["/products"] = requestHandlers.products

var port = process.env.PORT || 3000;
server.start(router.route, handle, port);

Kick off

nodemon app.js

If you were to have an API key and had put it in the URL above, you’d see something like:

asos-products-1

Right. Ok. That’s a lot of data. Just for now I’d like to make it easier to view, so I’ll limit what is returned and also just write out a basic HTML page.

requestHandlers.js:

var proxy = require('./proxy');

function products(response) {
console.log("Request handler 'products' was called");

  var host = 'api1.asos.com';
  var requestPath = '/productlisting/search/jeans/1/PriceAscending/en_API/GBP?api_key={snipped api key}';
  response.writeHead(200, {"Content-Type": "text/html"});

  proxy.getRemoteData(host, requestPath, function(json){
    var data = JSON.parse(json);

  var html = "<h1>Asos Search for JEANS</h1>";
  response.write(html);

    for(var i=0; i<data.ItemCount; i++) {
      if (data.Listings[i] != null){
      response.write("<li>"
       + data.Listings[i].Title + "<br /><img src='" 
       + data.Listings[i].ProductImageUrl + "' /></li>");
      }
    }

    response.end();
  });  
}

exports.products = products;

Given that the Asos Api returns valid JSON I can just parse it and then access the structure of that JSON; in this case the ItemCount & Listings at the top level and Title & ProductImageUrl within Listings.

This will now display something like:
asos-products-2

(Really? A beanie is the first result in the search for “jeans”? Anyway…)

Actually searching

Next we’ll just make the request actually execute a search with the value passed in to our own API, using the format “/products/{search term}”

Firstly I’ll edit the router to take the primary route handler from the first part of the URL (e.g “http://localhost:3000/products/jeans”) and pass the full path into the router for further use.

router.js:

function route(handle, pathname, response) {
  var root = pathname.split('/')[1];

  if (typeof handle[root] === 'function') {
    handle[root](response, pathname);
  } else {
    console.log("No request handler found for " + pathname);
    response.writeHead(404, {"Content-Type": "text/plain"});
    response.write("404 Not found");
    response.end();
  }
}

exports.route = route;

Next change the request handler to pick out the next section from the url e.g. “http://localhost:3000/products/jeans

requestHandlers.js:

var proxy = require('./proxy');

function products(response) {
console.log("Request handler 'products' was called");

  var search = path.split('/')[2];
  var host = 'api1.asos.com';
  var requestPath = '/productlisting/search/' + search + '/1/PriceAscending/en_API/GBP?api_key={snipped api key}';
  
  response.writeHead(200, {"Content-Type": "text/html"});

  proxy.getRemoteData(host, requestPath, function(json){
    var data = JSON.parse(json);

  var html = "<h1>Asos Search for " + search + "</h1>";
  response.write(html);

    for(var i=0; i<data.ItemCount; i++) {
      if (data.Listings[i] != null){
      response.write("<li>"
       + data.Listings[i].Title + "<br /><img src='" 
       + data.Listings[i].ProductImageUrl + "' /></li>");
      }
    }

    response.end();
  });  
}

exports.products = products;

One last tweak to the initialisation file to remove a leading slash which isn’t needed now that we’re splitting the url to match instead of using the full url path:

app.js:

var server = require("./server"),
  router = require("./router"),
  requestHandlers = require("./requestHandlers");

var handle = {}
handle["products"] = requestHandlers.products;

var port = process.env.PORT || 3000;
server.start(router.route, handle, port);

We now have basic search capabilities:
asos-products-search-1

Now let’s get a basic product detail page working. For this I should need to just add a new request handler and wire it up.

requestHandlers.js:

var proxy = require('./proxy');

function products(response, path) {
console.log("Request handler 'products' was called");

  var search = path.split('/')[2];
  var host = 'api1.asos.com';
  var requestPath = '/productlisting/search/' + search + '/1/PriceAscending/en_API/GBP?api_key={snipped api key}';
  
  response.writeHead(200, {"Content-Type": "text/html"});

  proxy.getRemoteData(host, requestPath, function(json){
    var data = JSON.parse(json);

  var html = "<h1>Asos Search for " + search + "</h1>";
  response.write(html);

    for(var i=0; i<data.ItemCount; i++) {
      if (data.Listings[i] != null){
      response.write("<li><a href='/product/" + data.Listings[i].ProductId + "'>"
       + data.Listings[i].Title + "<br /><img src='" 
       + data.Listings[i].ProductImageUrl + "' /></a></li>");
      }
    }

    response.end();
  });  
}

function product(response, path) {
  console.log("Request handler 'product' was called for " + path);

  var productId = path.split('/')[2];
  var host = 'api1.asos.com';
  var requestPath = '/product/' + productId + '/en_API/GBP?api_key={snipped api key}';

  response.writeHead(200, {"Content-Type": "text/html"});
  proxy.getRemoteData(host, requestPath, function(json){
  var data = JSON.parse(json);

    var html = "<h1>" + data.Title + "</h1>"
    + "<img src='" + data.ProductImageUrls[0].replace('xxl','xl') + "' />"
    response.write(html);
    response.end();
  });  
}
exports.products = products;
exports.product = product;

As well as the new handler I’ve also added a link from the listing page to the detail page, just for FUN.

app.js:

var server = require("./server"),
  router = require("./router"),
  requestHandlers = require("./requestHandlers");

var handle = {}
handle["products"] = requestHandlers.products;
handle["product"] = requestHandlers.product;

var port = process.env.PORT || 3000;
server.start(router.route, handle, port);

asos-product-1

Back to JSON

Ok, so that’s a very basic website wrapped around an API. Since I plan to use this wrapper as a basic API itself I’m going to revert it to returning JSON and simplify the data structure for my needs.

requestHandlers.js:

var proxy = require('./proxy');

function products(response, path) {
console.log("Request handler 'products' was called");

  var search = path.split('/')[2];
  var host = 'api1.asos.com';
  var requestPath = '/productlisting/search/' + search + '/1/PriceAscending/en_API/GBP?api_key={stripped api key}';
  
  response.writeHead(200, {"Content-Type": "application/json"});

  proxy.getRemoteData(host, requestPath, function(json){
    var data = JSON.parse(json);

  var newJson = {
    category: data.Description,
    products: []
  };

  data.Listings.forEach(function(listing){
      newJson.products.push({
        id: listing.ProductId,
        title: listing.Title,
        price: listing.CurrentPrice,
        image: listing.ProductImageUrl[0]
      })
    });

    response.write(JSON.stringify(newJson));
    response.end();
  });  
}

function product(response, path) {
  console.log("Request handler 'product' was called for " + path);

  var productId = path.split('/')[2];
  var host = 'api1.asos.com';
  var requestPath = '/product/' + productId + '/en_API/GBP?api_key={snipped api key}';

  response.writeHead(200, {"Content-Type": "application/json"});
  proxy.getRemoteData(host, requestPath, function(json){
  var data = JSON.parse(json);

    var newJson = {
      id: data.ProductId,
      title: data.Title,
      price: data.CurrentPrice,
      available: data.InStock,
      image: data.ProductImageUrls[0]
    };

    response.write(JSON.stringify(newJson));
    response.end();
  });  
}
exports.products = products;
exports.product = product;

Which ends up looking like:
asos-json-1

That’ll do me for now, even though it would be nice to abstract the mapping out somewhere else. Out of scope for me at the moment though.

Once last thing for this post:

Passing in command line arguments

Throughout this post I’ve been diligently snipping out my API key before pasting the code in. There are many approaches to dev/qa/staging/production configuration management (some as basic as a text file, some a bit more complex) which would handle this sort of thing but for my immediate requirements I will just pass the API key in as a command line argument.

To handle this I need to edit the initialisation code in order to pick up any args passed, and documented on the nodejs.org site:

app.js:

var server = require("./server"),
  router = require("./router"),
  requestHandlers = require("./requestHandlers");

var handle = {}
handle["products"] = requestHandlers.products;
handle["product"] = requestHandlers.product;

var apiKey = process.argv[2];
var port = process.env.PORT || 3000;
server.start(router.route, handle, port, apiKey);

Now just pass that value around the rest of the system:

server.js:

var http = require("http"),
  url = require("url");

function start(route, handle, port, apiKey) {
  function onRequest(request, response) {
    var pathname = url.parse(request.url).pathname;
    route(handle, pathname, response, apiKey);
  }

  http.createServer(onRequest).listen(port);
  console.log("Server has started listening on port " + port);
}

exports.start = start;

route.js:

function route(handle, pathname, response, apiKey) {
  var root = pathname.split('/')[1];

  if (typeof handle[root] === 'function') {
    handle[root](response, pathname, apiKey);
  } else {
    console.log("No request handler found for " + pathname + " (" + root+ ")");
    response.writeHead(404, {"Content-Type": "text/plain"});
    response.write("404 Not found");
    response.end();
  }
}

exports.route = route;

requestHandlers.js:

var proxy = require('./proxy');

function products(response, path, apiKey) {
console.log("Request handler 'products' was called");

  var search = path.split('/')[2];
  var host = 'api1.asos.com';
  var requestPath = '/productlisting/search/' + search + '/1/PriceAscending/en_API/GBP?api_key=' + apiKey;
  
  response.writeHead(200, {"Content-Type": "application/json"});

  proxy.getRemoteData(host, requestPath, function(json){
    var data = JSON.parse(json);

  var newJson = {
    category: data.Description,
    products: []
  };

  data.Listings.forEach(function(listing){
      newJson.products.push({
        id: listing.ProductId,
        title: listing.Title,
        price: listing.CurrentPrice,
        image: listing.ProductImageUrl[0]
      })
    });

    response.write(JSON.stringify(newJson));
    response.end();
  });  
}

function product(response, path, apiKey) {
  console.log("Request handler 'product' was called for " + path);

  var productId = path.split('/')[2];
  var host = 'api1.asos.com';
  var requestPath = '/product/' + productId + '/en_API/GBP?api_key=' + apiKey;

  response.writeHead(200, {"Content-Type": "application/json"});
  proxy.getRemoteData(host, requestPath, function(json){
  var data = JSON.parse(json);

    var newJson = {
      id: data.ProductId,
      title: data.Title,
      price: data.CurrentPrice,
      available: data.InStock,
      image: data.ProductImageUrls[0]
    };

    response.write(JSON.stringify(newJson));
    response.end();
  });  
}
exports.products = products;
exports.product = product;

Then to pass in the api key just change the nodemon call to

nodemon app.js myApIK3y

The files for this post can be found over on github

Coming up

The next post this month will cover some nice deployment & hosting options for node!