Can We Code It?

Thursday, 16 April 2015

Use Media Type Formatters to make your Web API Controllers more testable

One of the benefits of moving to ASP.NET MVC from classic ASP.NET is the improved Separation of Concerns that can be achieved:

Controllers handle the retrieval and manipulation of the appropriate data models and the selection of the appropriate View to display the data.
Views handle the display of the data.

But why should you care about separation of concerns?

The most obvious benefit is that it makes your application more testable. You can easily create unit tests that create an instance of your controller class and call the action methods on it without needing to create or mock a HttpContext object.

The MVC framework includes many other interfaces and plug-points that can be used to ensure that Controllers are kept as free as possible from the "concern" of how incoming data is parsed from the HTTP request and the results of executing the controller method are written to the outgoing HTTP Response. Some other time I'll write a blog about some of these....

...but today(!) I want to talk about a feature that the Web API has for ensuring better separation of concerns: Media Type Formatters. Web API is ASP.NET MVC's techie twin brother who has been specially adapted with bionic body parts that allow him to implement web services in really neat ways. In ASP.NET 6 Web API are MVC will be somehow merged into one, but for now they are two separate beasts.

What is a Media Type Formatter? I first encountered them when I created a web service method to add a document to a backing store:

        [HttpPut]
        public HttpResponseMessage AddDocument(string id,
            [FromBody] Stream content)
        {
            // Add document to backing store
        }

Looks sensible enough, but the first time I tried to call it I got a HTTP 415 Response: "Unsupported Media Type".

This is because Web API is designed to protect you from having to handle the incoming raw Stream. The Web API has a collection of Media Type Formatters which is stored in

System.Web.Http.GlobalConfiguration.Configuration.Formatters

These are a bit like Model Binders in MVC - when a Web API method argument needs to be populated from an incoming request body Web API calls the MediaTypeFormatter.CanReadType(Type) method on each Media Type Formatter in turn, passing the type of the method argument, until it finds one that returns true. In my case none of the four built in Media Type Formatters were capable of deserialising a Stream to.... a Stream.

(NOTE: If you are reading this and are only interested in how you can make a Web API method that handles PUT requests accept content of any MIME Type then you may want to skip straight to my next post; if you're interested in how Media Type Formatters work for incoming content then keep reading this post before you go on to the next one.)

No problem, because it's easy to create a Media Type Formatter of your own! Below is a basic Media Type Formatter that will allow you to have a method argument of type Stream:

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net.Http;
using System.Net.Http.Formatting;
using System.Net.Http.Headers;
using System.Threading.Tasks;
using System.Web;

namespace WebApplication5
{
    public class StreamMediaTypeFormatter : MediaTypeFormatter
    {
        public StreamMediaTypeFormatter()
        {
            this.SupportedMediaTypes.Add(new MediaTypeHeaderValue("application/octet-stream"));
        }

        public override bool CanReadType(Type type)
        {
            return typeof(Stream) == type;
        }

        public override bool CanWriteType(Type type)
        {
            return false;
        }

        public override Task<object> ReadFromStreamAsync(Type type, 
            Stream readStream, HttpContent content, IFormatterLogger formatterLogger)
        {
            return Task.FromResult((object)readStream);
        }

        public override Task<object> ReadFromStreamAsync(Type type, Stream readStream, 
            HttpContent content, IFormatterLogger formatterLogger, System.Threading.CancellationToken cancellationToken)
        {
            return ReadFromStreamAsync(type, readStream, content, formatterLogger);
        }
    }
}

(The CanWriteType method is there because Media Type Formatters are also used to serialise return types to the HTTP output stream. In this post I'm only interested in deserialisation, but you may want to create a custom Media Type Formatter to serialise content too).

To tell the Web API to use our Media Type Formatter we need to add it to collection of formatters during application start up, like this:

    public class WebApiApplication : System.Web.HttpApplication
    {
        protected void Application_Start()
        {
            AreaRegistration.RegisterAllAreas();
            GlobalConfiguration.Configure(WebApiConfig.Register);
            FilterConfig.RegisterGlobalFilters(GlobalFilters.Filters);
            RouteConfig.RegisterRoutes(RouteTable.Routes);
            BundleConfig.RegisterBundles(BundleTable.Bundles);

            GlobalConfiguration.Configuration.Formatters.Insert(0, new StreamMediaTypeFormatter());
        }
    }

Note that we are inserting our new formatter at position 0 in the collection which means that it will take precedence over the existing formatters in the collection.

This is a good start but what if we would like our Web API to have access to some other information about the incoming Stream such as its length and the language of its contents? These can be read from the Content-Length and Content-Language headers on the incoming request. Now it would be possible to simply access the Request property from within our AddDocument method, like this:

        [HttpPut]
        public HttpResponseMessage AddDocument(string id, [FromBody]Stream value)
        {
            long contentLength = Request.Content.Headers.ContentLength.Value;
            string contentLanguage = Request.Content.Headers.ContentLanguage.FirstOrDefault();
            // Add document to backing store
        }

The problem with this is that it starts to break our principle of Separation of Concerns (or at least gnaw away at it): now the Controller needs to know something about the structure of the incoming HTTP Request and how to retrieve information from it. And this makes the Controller less testable; if we want to call this method from a unit test we would need to initialise the Request property of the Controller before calling it which is a little fiddly.

We could use IValueProviders to bind these header values to separate arguments on the Controller method (see http://www.asp.net/web-api/overview/formats-and-model-binding/parameter-binding-in-aspnet-web-api) but as these two values relate to the content of the Stream itself it seems neater to me to define a new type that encapsulates all the information about the content that we are interested in, StreamContentSource, and modify the Media Type Formatter to create an instance of that type. The type definition is:

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Web;

namespace WebApplication5
{
    public class StreamContentSource : IContent
    {
        private Stream stream;

        public StreamContentSource(Stream stream, string contentLanguage, long contentLength)
        {
            this.stream = stream;
            this.ContentLength = contentLength;
            this.ContentLanguage = contentLanguage;
        }

        public Stream GetContentAsStream()
        {
            return stream;
        }

        public long ContentLength
        {
            get;
            private set;
        }

        public string ContentLanguage
        {
            get;
            private set;
        }
    }
}

The Media Type Formatter now looks like this:

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net.Http;
using System.Net.Http.Formatting;
using System.Net.Http.Headers;
using System.Threading.Tasks;
using System.Web;

namespace WebApplication5
{
    public class StreamMediaTypeFormatter : MediaTypeFormatter
    {
        public StreamMediaTypeFormatter()
        {
            this.SupportedMediaTypes.Add(new MediaTypeHeaderValue("application/octet-stream"));
        }

        public override bool CanReadType(Type type)
        {
            return typeof(StreamContentSource) == type;
        }

        public override bool CanWriteType(Type type)
        {
            return false;
        }

        public override Task<object> ReadFromStreamAsync(Type type, 
            Stream readStream, HttpContent content, IFormatterLogger formatterLogger)
        {
            long contentLength = content.Headers.ContentLength.Value;
            string contentLanguage = content.Headers.ContentLanguage.FirstOrDefault();
            return Task.FromResult((object)new StreamContentSource(readStream, contentLanguage, contentLength));
        }

        public override Task<object> ReadFromStreamAsync(Type type, Stream readStream, 
            HttpContent content, IFormatterLogger formatterLogger, System.Threading.CancellationToken cancellationToken)
        {
            return ReadFromStreamAsync(type, readStream, content, formatterLogger);
        }
    }
}

And we modify our Controller method to accept an argument of type SteamContentSource.

        [HttpPut]
        public HttpResponseMessage AddDocument(string id, [FromBody]StreamContentSource value)
        {
            // Add document to backing store
            return new HttpResponseMessage(HttpStatusCode.Created);
        }

This gives us proper separation of concerns and makes our Controller method extremely testable.

UPDATE:
There are still some problems with this: our Media Type Formatter will only be invoked when the Content Type of the incoming request is application/octet-stream. If this is not good for you then you need to read my next post.

photo credit: Whatever! via photopin (license)

Wednesday, 15 April 2015

Handling Binary Content Using PowerShell

Yesterday I was testing a PDF IFilter on my development machine, and I needed a quick way to get a PDF file into a VARBINARY(MAX) column. There are probably ways to do this using SSIS and BCP but as I only needed to do it as a one-off I reached for my favourite tool to do miscellaneous tasks: PowerShell, the Swiss Army chainsaw of Windows Maintenance tasks.

The problem:

I have a 35KB PDF that I want to import into a SQL table.

I can append the content to the column using the SQL .WRITE statement, like this:

UPDATE [Documents] SET [Document] .WRITE(0x1267A2B3, NULL, 0) WHERE NumId = 3

In order to do that I need to get the bytes of the file into a 0xA63410..... format.

The solution:

PS C:\Users\Alex\Documents> (Get-Item .\BlankAnnualReturn.pdf).OpenRead() | %{$buffer = [System.Linq.Enumerable]::ToArray( [System.Linq.Enumerable]::Repeat( [Convert]::ToByte(0), 8040));$bytesRead = $_.Read($buffer, 0, $buffer.Length); while
($bytesRead -gt 0) {$output = (New-Object System.Text.StringBuilder).Append("0x"); for($i = 0; $i -lt $bytesRead; $i = $
i + 1){[void] $output.AppendFormat("{0:x2}",$buffer[$i])};$output.ToString();$bytesRead = $_.Read($buffer,0,$buffer.Length);}$_.Close();}

That gives me the content of .\BlankAnnualReturn.pdf in 8040 byte blocks (the optimum size for the SQL .WRITE statement) in 0x... format.

I'll just point out my favourite bits of this:

The easiest way to create an empty byte array in PowerShell is using the Linq Repeat method. In C# this would be:
```
byte[] buffer = Enumerable.Repeat((byte)0, 8040).ToArray();
```
But in PowerShell we don't have access to the compiler tricks that make extension methods work, so we have to call these methods as if they were normal static methods:

$buffer = [System.Linq.Enumerable]::ToArray([System.Linq.Enumerable]::Repeat([Convert]::ToByte(0), 8040));
Formatting a byte as two digit hex.
Dead simple this bit, just:

[void] $output.AppendFormat("{0:x2}",$buffer[$i]);

The cast to [void] is needed to prevent the StringBuilder from being echoed to the console after each append.

Single line PowerShell rocks.

photo credit: chainsaw man in the mountains via photopin (license)

Thursday, 1 May 2014

Storing DateTimeOffset values in Azure Table Storage

I recently stumbled upon a weakness of Azure Table Storage: there is no native support for DateTimes with timezone information (a .NET DateTimeOffset datatype). If I’d read all the documentation thoroughly beforehand I’d have known this (see http://msdn.microsoft.com/library/azure/jj553018.aspx) but like most of us I didn’t read all the documentation… Embarrassed smile

And, to be fair, I did have some reason for thinking that Table Storage would support DateTimeOffset values. Below is an example of a very basic ITableEntity class that can be read from/written to Table Storage using the Azure SDK methods in the
Microsoft.WindowsAzure.Storage.Table namespace:

public class ExampleEntity : ITableEntity
{
    public void ReadEntity(IDictionary<string, EntityProperty> properties, OperationContext operationContext)
    {
        ExampleDateWithTimeZone = properties.GetDateTimeOffset("ExampleDateWithTimeZone").GetValueOrDefault();
    }
 
    public IDictionary<string, EntityProperty> WriteEntity(OperationContext operationContext)
    {
        Dictionary<string, EntityProperty> properties = new Dictionary<string, EntityProperty>();
        properties.Add("ExampleDateWithTimeZone", EntityProperty.GeneratePropertyForDateTimeOffset(ExampleDateWithTimeZone));
        return properties;
    }
 
    public DateTimeOffset ExampleDateWithTimeZone { get; set;}
 
    public string ETag { get; set;}
 
    public string PartitionKey{ get; set;}
 
    public string RowKey{ get; set;}
 
    public DateTimeOffset Timestamp{ get; set;}
}

Look at lines 5 and 11: there are GetDateTimeOffset and GeneratePropertyForDateTimeOffset extension methods to read and write DateTimeOffset values, but no equivalent methods that read and write DateTime values. I saw this and thought “Great! It only supports dates with timezone information, which makes sense when the datacentre will normally be in a different timezone from the users”.

But unfortunately not…

When you do use those tempting helper methods for DateTimeOffSets what happens is:

When you write the value, Azure converts it to GMT, so

24/04/2014 14:31 +02:00

becomes

24/04/2014 12:31 +00:00
When you read it back you get the GMT value.

I didn’t notice this for a while because until the clocks changed in spring all my work on this project had taken place in GMT (one of the perils of developing in the U.K; you’re generally pretty careful about date formats, but not so careful about timezones). I did have some unit tests that used DateTimeOffset values with timezones but they were passing because according to DateTimeOffset.Equals():

24/04/2014 14:31 +02:00 == 24/04/2014 12:31 +00:00

(I can’t quite decide whether that’s a good thing or not)

So what did I do?

I still needed to store DateTimeOffset values in Azure Table Storage, so I converted them to strings. Not that revolutionary, but works quite nicely. In case I ever needed to sort them as DateTimeOffsets I used the format string “yyyyMMddHHmmssfffffffzzz”, so

01 May 2014 23:11:19 +01:00

becomes:

201405012312508966200+01:00

It turned out to be a fairly easy change to introduce because all my ReadEntity and WriteEntity implementations made heavy use of the same set of Extension Methods that look roughly like this:

public static void AddPropertyIfNotNull(this Dictionary<string, EntityProperty> properties, string propertyName, int? propertyValue)
{
    if (propertyValue != null)
    {
        properties.Add(propertyName, EntityProperty.GeneratePropertyForInt(propertyValue));
    }
}
 
public static int? GetInt32(this IDictionary<string, EntityProperty> properties, string propertyName)
{
    return getValue(properties, propertyName, ep => ep.Int32Value, () => null);
} 

private static T getValue<T>(IDictionary<string, EntityProperty> properties, string propertyName, Func<EntityProperty, T> valueAccessor, Func<T> nullValue)
{
    if (properties.ContainsKey(propertyName))
    {
        return valueAccessor(properties[propertyName]);
    }
    else
    {
        return nullValue();
    }
}

I have AddPropertyIfNotNull and GetXXX methods for every primitive type that I need, including DateTimeOffset. So all I had to do was change the extension methods for DateTimeOffset to look like this:

private const string DATETIMEOFFSET_FORMAT = "yyyyMMddHHmmssfffffffzzz";

public static DateTimeOffset? GetDateTimeOffset(this IDictionary<string, EntityProperty> properties, string propertyName)
{
    string valueAsString = getValue(properties, propertyName, ep => ep.StringValue, () => null);
    if (valueAsString == null)
    {
        return null;
    }
    else
    {
        return DateTimeOffset.ParseExact(valueAsString, DATETIMEOFFSET_FORMAT,  CultureInfo.DefaultThreadCurrentUICulture);
    }
}
 
public static void AddPropertyIfNotNull(this Dictionary<string, EntityProperty> properties, string propertyName, DateTimeOffset? propertyValue)
{
    if (propertyValue != null)
    {
        properties.Add(propertyName,  EntityProperty.GeneratePropertyForString(propertyValue.Value.ToString(DATETIMEOFFSET_FORMAT)));
    }
}

And the rest of the application carried on working as normal.

And if Azure Table Storage starts supporting DateTimeOffset natively before I go live all I have to do is switch it back…

Monday, 3 February 2014

Customising Backbone’s Sync Module

I’ve started using the Backbone MVC JavaScript framework recently, and have been pleasantly surprised by how easy it is to customise bits of the framework when I need some additional functionality. by
Backbone communicates with backend web services using the Backbone.sync function; this function examine the model object being synchronised to determine whether the operation is a “create”, “read”, “update” or “delete” and then uses jQuery.ajax to perform a HTTP POST, GET, PUT or DELETE on the backend web service. The change I want to make is to use PROPFIND and PROPPATCH instead of GET and PUT. I may go into the reasons for this change more in subsequent posts, but for now just trust me (please) that there is a reason why I want to do this.
One of the reasons that I chose Backbone was this section from the “Extending Backbone” section in the documentation:

Many JavaScript libraries are meant to be insular and self-enclosed, where you interact with them by calling their public API, but never peek inside at the guts. Backbone.js is not that kind of library.
Because it serves as a foundation for your application, you're meant to extend and enhance it in the ways you see fit

So here goes! The Backbone.sync function is function (method, model, options) where method = “create” | “read” | “update” | “delete”. Looking at the source of the function, the first line of the function does this:

var type = methodMap[method];
and the definition of methodMap is:

var methodMap = {     'create': 'POST',     'update': 'PUT',     'patch': 'PATCH',     'delete': 'DELETE',     'read':   'GET' };

So to use PROPFIND and PROPPATCH instead of GET and PUT I should be able to simply create a different MyApp.methodMap hash (the original variable is private to the anonymous function that defines the Backbone namespace and functions), create a new MyApp.sync function which is an exact copy of Backbone.sync but referencing my new MyApp.methodMap hash and replace Backbone.sync with MyApp.sync. Something like this:

MyApp.methodMap = {'create': 'POST', 'update': 'PROPPATCH', 'patch':'PATCH', 'delete': 'DELETE', 'read':'PROPFIND'};
MyApp.sync = function (method, model, options) {      var type = MyApp.methodMap[method];
     // all the rest of Backbone.sync
}; Backbone.sync = MyApp.sync;
But it doesn’t work. Monitoring the HTTP traffic using Fiddler confirms that the PROPFIND verb is being sent correctly when I call Model.fetch() but my model isn’t actually being populated with the data returned. Looking further into the original Backbone.sync function, on about the 42nd line it decides whether to process the returned data based upon the HTTP verb that it’s sending:

// Don't process data on a non-GET request. if (params.type !== 'GET' && !options.emulateJSON) { params.processData = false; }

I just have to replace the ‘GET’ with a ‘PROPFIND’ and my model objects are populated correctly.
So to recap, all I had to do to make Backbone use PROPFIND and PROPPATCH instead of GET and PUT was to:

Create a new MyApp.sync function and MyApp.methodMap hash that were exact copies of the originals.
Modify the new MyApp.sync function to reference MyApp.methodMap.
Modify MyApp.methodMap to return PROPFIND for read and PROPPATCH for update.
Modify MyApp.sync to process returned data for PROPFIND instead of GET.

You may want to make quite different changes to Backbone.sync, but I hope you are inspired by reading this to have a go at it because it isn’t difficult.

Tuesday, 21 January 2014

Get some Backbone!

I’ve started looking at MVC frameworks for javascript recently. For anyone else attempting this I would recommend the excellent TodoMVC site which contains the same simple application (a TODO list) coded using many different MVC frameworks. If you’ve got enough time on your hands, I can see it would be a great idea to look in detail through all the implementations and pick the one that has the best combination of good technical features, flexibility and ongoing development.
But I didn’t have that much time so I had to pick one Disappointed smile

.
I chose Backbone because it has:

Many live sites.
Regular releases.
Fairly lightweight approach (describes itself as “a library not a framework”): can be used in a variety of ways rather than prescribing just one way that you should use it. This is particularly important to me given that there is no consensus yet on how to “do” MVC in javascript, so I want a framework that is flexible enough if I change my mind half way though my project.
Open architecture: you are positively encouraged to make changes to the library if you don’t like the way part of it works. Obviously javascript’s dynamic nature makes it a lot easier to do this than I’m used to, being a .NET statically typed man by training.

That’s all for now, I will let you know about my success (or otherwise) with Backbone in future posts.