Generate a Blob Storage Web Site using RazorEngine

Last episode I introduced the concept of utilising RazorEngine and RazorMachine to generate html files from cshtml Razor view files and json data files, without needing a hosted ASP.Net MVC website.

We ended up with a teeny ikkle console app which could reference a few directories and spew out some resulting html.

This post will build on that concept and use Azure Blob Storage with a worker role.

File System

We already have a basic RazorEngine implementation via the RenderHtmlPage class, and that class uses some basic dependency injection/poor man’s IoC to abstract the functionality to pass in the Razor view, the json data, and the output location.

IContentRepository

The preview implementation of the IContentRepository interface merely read from the filesystem:

namespace CreateFlatFileWebsiteFromRazor
{
    internal class FileSystemContentRepository : IContentRepository
    {
        private readonly string _rootDirectory;
        private const string Extension = ".cshtml";

        public FileSystemContentRepository(string rootDirectory)
        {
            _rootDirectory = rootDirectory;
        }

        public string GetContent(string id)
        {
            var result = 
                File.ReadAllText(string.Format("{0}/{1}{2}", _rootDirectory, id, Extension));
            return result;
        }
    }
}

IDataRepository

A similar story for the IDataRepository file system implementation:

namespace CreateFlatFileWebsiteFromRazor
{
    internal class FileSystemDataRepository : IDataRepository
    {
        private readonly string _rootDirectory;
        private const string Extension = ".json";

        public FileSystemDataRepository(string rootDirectory)
        {
            _rootDirectory = rootDirectory;
        }

        public string GetData(string id)
        {
            var results = 
                File.ReadAllText(string.Format("{0}/{1}{2}", _rootDirectory, id, Extension));
            return results;
        }
    }
}

IUploader

Likewise for the file system implemention of IUploader:

namespace CreateFlatFileWebsiteFromRazor
{
    internal class FileSystemUploader : IUploader
    {
        private readonly string _rootDirectory;
        private const string Extension = ".html";

        public FileSystemUploader(string rootDirectory)
        {
            _rootDirectory = rootDirectory;
        }

        public void SaveContentToLocation(string content, string location)
        {
            File.WriteAllText(
                string.Format("{0}/{1}{2}", _rootDirectory, location, Extension), content);
        }
    }
}

All pretty simple stuff.

Blob Storage

All I’m doing here is changing those implementations to use blob storage instead. In order to do this it’s worth having a class to wrap up the common functions such as getting references to your storage account. I’ve given mine the ingenious title of BlobUtil:

class BlobUtil
{
    public BlobUtil(string cloudConnectionString)
    {
        _cloudConnectionString = cloudConnectionString;
    }

    private readonly string _cloudConnectionString;

    public void SaveToLocation(string content, string path, string filename)
    {
        var cloudBlobContainer = GetCloudBlobContainer(path);
        var blob = cloudBlobContainer.GetBlockBlobReference(filename);
        blob.Properties.ContentType = "text/html";

        using (var ms = new MemoryStream(Encoding.UTF8.GetBytes(content)))
        {
            blob.UploadFromStream(ms);
        }
    }

    public string ReadFromLocation(string path, string filename)
    {
        var blob = GetBlobReference(path, filename);

        string text;
        using (var memoryStream = new MemoryStream())
        {
            blob.DownloadToStream(memoryStream);
            text = Encoding.UTF8.GetString(memoryStream.ToArray());
        }
        return text;
    }

    private CloudBlockBlob GetBlobReference(string path, string filename)
    {
        var cloudBlobContainer = GetCloudBlobContainer(path);
        var blob = cloudBlobContainer.GetBlockBlobReference(filename);
        return blob;
    }

    private CloudBlobContainer GetCloudBlobContainer(string path){
        var account = CloudStorageAccount.Parse(_cloudConnectionString);
        var cloudBlobClient = account.CreateCloudBlobClient();
        var cloudBlobContainer = cloudBlobClient.GetContainerReference(path);
        return cloudBlobContainer;
    }
}

This means that the blob implementations can be just as simple.

IContentRepository – Blob Style

Just connect to the configured storage account, and read form the specified location to get the Razor view:

class BlobStorageContentRepository : IContentRepository
{
    private readonly BlobUtil _blobUtil;
    private readonly string _contentRoot;

    public BlobStorageContentRepository(string connectionString, string contentRoot)
    {
        _blobUtil = new BlobUtil(connectionString);
        _contentRoot = contentRoot;
    }

    public string GetContent(string id)
    {
        return _blobUtil.ReadFromLocation(_contentRoot, id + ".cshtml");
    }
}

IDataRepository – Blob style

Pretty much the same as above, except with a different “file” extension. Blobs don’t need file extensions, but I’m just reusing the same files from before.

public class BlobStorageDataRespository : IDataRepository
{
    private readonly BlobUtil _blobUtil;
    private readonly string _dataRoot;

    public BlobStorageDataRespository(string connectionString, string dataRoot)
    {
        _blobUtil = new BlobUtil(connectionString);
        _dataRoot = dataRoot;
    }

    public string GetData(string id)
    {
        return _blobUtil.ReadFromLocation(_dataRoot, id + ".json");
    }
}

IUploader – Blob style

The equivalent for saving it is similar:

class BlobStorageUploader : IUploader
{
    private readonly BlobUtil _blobUtil;
    private readonly string _outputRoot;

    public BlobStorageUploader(string cloudConnectionString , string outputRoot)
    {
        _blobUtil = new BlobUtil(cloudConnectionString);
        _outputRoot = outputRoot;
    }
    public void SaveContentToLocation(string content, string location)
    {
        _blobUtil.SaveToLocation(content, _outputRoot, location + ".html");
    }
}

Worker Role

And tying this together is a basic worker role which looks all but identical to the console app:

public override void Run()
{
    var cloudConnectionString = 
        CloudConfigurationManager.GetSetting("Microsoft.Storage.ConnectionString");

    IContentRepository content = 
        new BlobStorageContentRepository(cloudConnectionString, "content");

    IDataRepository data = 
        new BlobStorageDataRespository(cloudConnectionString, "data");

    IUploader uploader = 
        new BlobStorageUploader(cloudConnectionString, "output");

    var productIds = new[] { "1", "2", "3", "4", "5" };
    var renderer = new RenderHtmlPage(content, data);

    foreach (var productId in productIds)
    {
        var result = renderer.BuildContentResult("product", productId);
        uploader.SaveContentToLocation(result, productId);
    }
}

The Point?

By setting the output container to be public, the html files can be browsed to directly; we’ve just created an auto-generated flat file website. You could have the repository implementations access the local file system and the console app access blob storage; generate the html locally but store it remotely where it can be served from directly!

Given that we’ve already created the RazorEngine logic, the implementations of content location are bound to be simple. Swapping file system for blob storage is a snap. Check out the example code over on github

Next up

There’s a few more stages in this master plan, and following those I’ll swap some stuff out to extend this some more.

Generate a Flat File Web Site using RazorEngine and RazorMachine

Razor and ASP.Net

As a normal person you’d probably be happy with how Razor template files are used within MVC; there’s a nice convention for where they live – they’ll be in a Views folder within your project most likely – and you refer to them either by name or sometimes just by convention – what’s that? You have an ActionResult method called “Index”? I’ll go fetch the “Index” view from the folders I normally expect the cshtml files to live in for ya then.

The way this works is fantastic; development can steam ahead without the pain and confusion of all of the possible ways you could do it wrong when choosing webforms and .aspx files.

Of course, the MS implementation of an MVC framework in itself is a wonderful thing; all but enforcing the separation of concerns that is just so easy to ignore in webforms.

Razor outside of ASP.Net

But what about when you want to dynamically generate html without a process being hosted as a website? One big use case for this is email generation; sure, you could host an MVC web API and have the content generation process constantly call it, but that seems a little inefficient.

RazorEngine and RazorMachine

There are a few solutions to this; you can actually hand roll your own (I might get onto that in a future post) or you can try out some reasonably well known open source solutions like RazorEngine:

A templating engine built on Microsoft’s Razor parsing engine, RazorEngine allows you to use Razor syntax to build dynamic templates:

string template = "Hello @Model.Name, welcome to RazorEngine!";
string result = Razor.Parse(template, new { Name = "World" });

and RazorMachine:

RazorMachine is a robust and easy to use .Net Razor v2/v3 template engine. The master branch uses Razor v3. This implementation supports layouts (masterpages) and a _viewStart construct, just like MVC does support these features. The RazorEngine works independently from MVC. It only needs the System.Web.Razor reference. It almost works exacly like Asp.Net MVC

var rm = new RazorMachine();
var result = 
   rm.Execute("Hello @Model.FirstName @Model.LastName", new {FirstName="John", LastName="Smith"});

There’s a short stackoverflow answer comparing them (and RazorTemplates, another similar OSS solution) too.

Getting stuck in

Create a new project and use some nuget awesomeness

Install-Package razormachine

Then, if you don’t already, add references to

system.web.helpers // for json.decode
microsoft.csharp // for dynamic types

If you want to debug this functionality via a console app running from VisualStudio, you may need to uncheck “enable visual studio hosting process” in Project -> Properties -> Debug

If you want to run this outside of Visual Studio, you can just run the compiled exe (bin/debug) as admin.

If you’re using a test runner then you might be fine as is. I can’t actually remember the issue I was having as I now can’t recreate it, but I think it might have been around using dynamic models and Json decoding.

RazorEngine

This is the core bit of functionality for a basic use case for RazorEngine:

var model = Json.Decode("{\"Description\":\"Hello World\"}");
var template = "<div class=\"helloworld\">@Model.Description</div>";
const string layout = "<html><body>@RenderBody()</body></html>";

template = string.Format("{0}{1}", "@{Layout=\"_layout\";}", template);

using (var service = new TemplateService())
{
    service.GetTemplate(layout, null, "_layout");
    service.GetTemplate(template, model, "template");

    var result = service.Parse(template, model, null, "page");

    Console.Write(result);
    Console.ReadKey();
}

Your output should be:

<html><body><div class="helloworld">Hello World</div></body></html>

Pretty easy, right?

RazorMachine

Here’s the equivalent in RazorMachine

var model = Json.Decode("{\"Description\":\"Hello World\"}");
var template = "<div class=\"helloworld\">@Model.Description</div>";
const string layout = "<html><body>@RenderBody()</body></html>";

var rm = new RazorMachine();
rm.RegisterTemplate("~/shared/_layout.cshtml", layout);

var renderedContent = 
    rm.ExecuteContent(string.Format("{0}{1}", "@{Layout=\"_layout\";}", template), model);
var result = renderedContent.Result;

Console.Write(result);
Console.ReadKey();

Again, same output:

<html><body><div class="helloworld">Hello World</div></body></html>

Notice that in both of them you have to lie about there being a layout file existing somewhere; in RazorEngine you give it a name:

template = string.Format("{0}{1}", "@{Layout=\"_layout\";}", template);

then refer to that name when adding the template:

service.GetTemplate(layout, null, "_layout");

In RazorMachine you register the template as a dummy virtual file:

rm.RegisterTemplate("~/shared/_layout.cshtml", layout);

then refer back to it as you would normally do within ASP.Net MVC when executing the content:

var renderedContent = 
      rm.ExecuteContent(string.Format("{0}{1}", "@{Layout=\"_layout\";}", template), model);

Differences

I’ve found it easier to process sub templates (such as @Include) within RazorEngine, as I just recursively scan a file for that keyword and add the corresponding template to the service, e.g. look at the ProcessContent and ProcessSubContent methods below:

public class RenderHtmlPage
{
    private readonly IContentRepository _contentRepository;
    private readonly IDataRepository _dataRepository;

    public RenderHtmlPage(IContentRepository contentRepository, 
                          IDataRepository dataRepository)
    {
        _contentRepository = contentRepository;
        _dataRepository = dataRepository;
    }

    public string BuildContentResult(string page, string id)
    {
        using (var service = new TemplateService())
        {
            // get the top level razor template, e.g. "product"
            // equivalent of "product.cshtml"
            var content = GetContent(page);
            var data = GetData(id);

            ProcessContent(content, service, data);
            var result = service.Parse(content, data, null, page);

            return result;
        }
    }

    private void ProcessContent(string content, 
                                TemplateService service, 
                                dynamic model)
    {
        // does the string passed in reference a Layout at the start?
        const string layoutPattern = @"@\{Layout = ""([a-zA-Z]*)"";\}";

        // does the string passed in reference an Include anywhere?
        const string includePattern = @"@Include\(""([a-zA-Z]*)""\)";

        // recursively process the Layout
        foreach (Match match in Regex.Matches(content, layoutPattern, 
                                                RegexOptions.IgnoreCase))
        {
            ProcessSubContent(service, match, model);
        }

        // recursively process the @Includes
        foreach (Match match in Regex.Matches(content, includePattern, 
                                                RegexOptions.IgnoreCase))
        {
            ProcessSubContent(service, match, model);
        }
    }

    private void ProcessSubContent(TemplateService service, 
                                    Match match, 
                                    dynamic model)
    {
        var subName = match.Groups[1].Value; // got an include/layout match?
        var subContent = GetContent(subName); // go get that template then
        ProcessContent(subContent, service, model); // recursively process it

        service.GetTemplate(subContent, model, subName); // add it to the service
    }

    private string GetContent(string templateToLoad)
    {
        // hit the filesystem, db, API, etc to retrieve the razor template
        return _contentRepository.GetContent(templateToLoad);
    }

    private dynamic GetData(string dataToLoad)
    {
        // hit the filesystem, db, API, etc to return some Json data as the model
        return Json.Decode(_dataRepository.GetData(dataToLoad));
    }
}

Why is this useful?

I’m not going to go into the details of either RazorMachine or RazorEngine; there’s plenty of documentation up on their respective websites already. I’ve used @Includes in the examples above due to its simplicity; the libraries have differing support for things like @Html.Partial and also can be extended.

Unfortunately, the html helpers (like @Html.Partial) need to have an HttpContext and run inside of ASP.Net MVC; which is what I’m trying to avoid for now.

If you pull down my initial teeny solution from github and look at the tests you’ll notice the content of the template, layout, and model are either strings or coming from the filesystem; not related to the structure of the project or files in the project or anything like that.

This means we can deploy a rendering process that returns rendered html based on strings being passed to it. Let’s play with this concept a bit more.

Flat File Web Page Generation

Say you wanted to “host” a website directly within a CDN/cache, thus avoiding the hosting via the normal route of servers and related infrastructure. Sure, writing flat html in a text editor is a solution, but what if you wanted to still be able to structure your pages into common modules, write C# code to manage the logic to dynamically combine them, and use Razor syntax and views for defining the front end?

This next section plays on this concept a bit more; we’ll write a small app that accesses a couple of directories – one for Razor files, one for data files – and generates a flat website into a third directory.

I will then expand on this concept over a series of posts, to make something more realistic and potentially useful.

Command Line & FileSystem FTW

I’ve created another repo up on github for this section, but cutting to the chase – here is the guts of demo console app:

const string workingRoot = "../../files";
IContentRepository content = 
    new FileSystemContentRepository(workingRoot + "/content");

IDataRepository data = 
    new FileSystemDataRepository(workingRoot + "/data");

IUploader uploader = 
    new FileSystemUploader(workingRoot + "/output");

var productIds = new[] {"1", "2", "3", "4", "5"};
var renderer = new RenderHtmlPage(content, data);

foreach (var productId in  productIds)
{
    var result = renderer.BuildContentResult("product", productId);
    uploader.SaveContentToLocation(result, productId);
}

The various FileSystemXX implementations either just read or write files from/to the file system. Natch.

So what we’ve got here is an implementation of the RazorEngine methods I pasted in above wrapped in a RenderHtmlPage class, being called for a number of “productIds”; these happen to exist as json files on disc, e.g. “1.json”.

Each file is being combined with whatever Razor templates are listed in the product cshtml file and its referenced @Includes. The resulting html is then saved back to the file system.

So with these views in files/content:
razorengine-flat-file-website-views

And these json files in files/data:
razorengine-flat-file-website-jsondata

We get these html files generated in files/output:
razorengine-flat-file-website-htmloutput

Hopefully you can see where this is leading; we can keep Views in one place, get the model data from somewhere else, and have the extremely generic rendering logic in another place.

The Theory

With this initial version we could take an existing ASP.Net MVC website (assuming it didn’t use any html helpers in the views..) and process it offline with a known dataset to create a readonly version of the website, ready to serve from a filesystem.

Next Up

I’ll take this concept and run with it across various implementations, gradually ending up on something that might even be useful!