Blog

Topological sorting in .net

Sat, 12 Jan 2013 19:37:03 GMT

Every now and then I need to topologically sort some data. Usually I end up just writing a trivial implementation of the algorithm. I have done this countless times, but my last problem had a new twitch: there might be nodes with circular references in my graph. So this time the problem was hard enough to warrant some googling, and when I found an algorithm (Tarjan's) and coded it up, I decided to make it generic enough to solve all my topological sort problems for all future. Presumably I am not the only one who occasionally needs topological sorting, so it might be useful for a general public. Enjoy (BSD license)!

public static class TopologicalSorter {
	// Tarjan's algorithm: http://en.wikipedia.org/wiki/Tarjan%27s_strongly_connected_components_algorithm
	private class Algorithm<T> {
		private readonly ILookup<T, T> _edges;
		private readonly IEqualityComparer<T> _comparer;

		private Dictionary<T, int> _indices;
		private Dictionary<T, int> _lowlink;
		private List<IList<T>> _result;
		private Stack<T> _s;
		private int _index;

		public Algorithm(ILookup<T, T> edges, IEqualityComparer<T> comparer) {
			_edges = edges;
			_comparer = comparer ?? EqualityComparer<T>.Default;
		}

		public List<IList<T>> MainLoop(IEnumerable<T> vertices) {
			_result = new List<IList<T>>();
			_indices = new Dictionary<T, int>(_comparer);
			_lowlink = new Dictionary<T, int>(_comparer);
			_s = new Stack<T>();
			_index = 0;
			foreach (var v in vertices) {
				if (!_indices.ContainsKey(v)) {
					StrongConnect(v);
				}
			}
			return _result;
		}

		private void StrongConnect(T v) {
			_indices[v] = _index;
			_lowlink[v] = _index;
			_index++;
			_s.Push(v);

			foreach (var w in _edges[v]) {
				if (!_indices.ContainsKey(w)) {
					StrongConnect(w);
					_lowlink[v] = Math.Min(_lowlink[v], _lowlink[w]);
				}
				else if (_s.Contains(w)) {
					_lowlink[v] = Math.Min(_lowlink[v], _indices[w]);
				}
			}

			if (_lowlink[v] == _indices[v]) {
				var scc = new List<T>();
				T w;
				do {
					w = _s.Pop();
					scc.Add(w);
				} while (!_comparer.Equals(v, w));
				_result.Add(scc);
			}
		}
    }

	public static IList<IList<T>> FindAndTopologicallySortStronglyConnectedComponents<T>(IEnumerable<T> source, IEnumerable<Tuple<T, T>> edges, IEqualityComparer<T> comparer = null) {
		var result = new Algorithm<T>(edges.ToLookup(e => e.Item1, e => e.Item2), comparer).MainLoop(source);
		return result;
	}

	public static IList<IList<TSource>> FindAndTopologicallySortStronglyConnectedComponents<TSource, TVertex>(IEnumerable<TSource> source, Func<TSource, TVertex> getVertex, IEnumerable<Tuple<TVertex, TVertex>> edges, IEqualityComparer<TVertex> comparer = null) {
		var backref = source.ToDictionary(getVertex, comparer ?? EqualityComparer<TVertex>.Default);
		return FindAndTopologicallySortStronglyConnectedComponents(backref.Keys, edges, comparer).Select(l => (IList<TSource>)l.Select(t => backref[t]).ToList()).ToList();
	}

	public static IEnumerable<T> TopologicalSort<T>(IEnumerable<T> source, IEnumerable<Tuple<T, T>> edges, IEqualityComparer<T> comparer = null) {
		var result = FindAndTopologicallySortStronglyConnectedComponents(source, edges, comparer);
		if (result.Any(x => x.Count > 1))
			throw new InvalidOperationException("Cycles in graph");
		return result.Select(x => x[0]);
	}

	public static IEnumerable<TSource> TopologicalSort<TSource, TVertex>(IEnumerable<TSource> source, Func<TSource, TVertex> getVertex, IEnumerable<Tuple<TVertex, TVertex>> edges, IEqualityComparer<TVertex> comparer = null) {
		var backref = source.ToDictionary(getVertex, comparer ?? EqualityComparer<TVertex>.Default);
		return TopologicalSort(backref.Keys, edges, comparer).Select(t => backref[t]);
	}
}

Refactoring the node.js file system browser

Thu, 11 Oct 2012 09:58:00 GMT

In the last post we used the Saltarelle C# to Javascript compiler to create a simple file system browser. Although it worked, the code was quite messy and a clear violation of the SRP since the same method was responsible for both reading the contents of a directory as well as rendering the output. In this post we will refactor the code that does the directory reading into a separate module.

Creating the project

Start with following the instructions in the last post, but this time call the project ModularizedNodeJSFileBrowser.

Now add a new class library called DirectoryEnumerator and install the Saltarelle.NodeJS package into this project as well. Unload and reload, and remove the attributes it complains about when building. Now open Properties\AssemblyInfo.cs and add

[assembly: ModuleName("DirectoryEnumerator")]

to it. This instructs the compiler to add all exported symbols to the exports variable rather than to the global object, as well as it tells any assembly that references this assembly which module it should require() in order to get access to the exported symbols. However, there is no magic that tells node where to find the script, so we set up a post-build task to copy it into the node_modules folder of the application project:

copy "$(TargetDir)$(TargetName).js" "$(ProjectDir)..\ModularizedNodeJSFileBrowser\node_modules"

Now also add a reference from the application project to the class library.

Implementing the directory enumerator

Now it is time for us to actually implement the directory enumerator module. First create a type to hold the returned results by creating a file called DirectoryEntry.cs with the following content:

using System;
using System.Runtime.CompilerServices;

namespace DirectoryEnumerator {
	[Imported]
	[Serializable]
	public class DirectoryEntry {
		public string Name { get; private set; }
		public bool? IsDirectory { get; private set; }
		public int? Size {  get; private set; }
		public Error Error { get; private set; }

		public DirectoryEntry(string name, bool? isDirectory, int? size, Error error) {
		}

		public DirectoryEntry(string name, Error error) {
		}
	}
}

If you used the Visual Studio Add Class command to do this, it will have inserted a reference to System, whcih you have to remove. This is really annoying, but it has to be done every time the Add Class command is used.

The Imported and Serializable attributes on this type will ensure that it will not appear in the generated script and that Javascript object literal syntax, { name: value }, will be used whenever we create an instance (for more information on these and other useful attributes that control the generated script, see the attribute list). The code would have worked perfectly fine without these attributes, but the generated code would have been different.

Now create a file called DirectoryEnumerator.cs with the following content:

using System;
using System.Collections.Generic;
using System.Runtime.CompilerServices;
using System.Threading.Tasks;
using NodeJS.FSModule;

namespace DirectoryEnumerator {
    [IgnoreNamespace]
    public class Enumerator {
        public static async Task<List<DirectoryEntry>> EnumerateDirectory(string physicalPath) {
            string[] files = await FS.ReaddirTask(physicalPath);

            Task<Stats>[] statTasks = files.Map(f => FS.StatTask(physicalPath + "\\" + f));

            try {
                await Task.WhenAll(statTasks);
            }
            catch (AggregateException) {
            }

            var result = new List<DirectoryEntry>();
            for (int i = 0; i < files.Length; i++) {
                var t = statTasks[i];
                if (t.IsFaulted) {
                    result.Add(new DirectoryEntry(files[i], ((JsErrorException)t.Exception.InnerExceptions[0]).Error));
                }
                else {
                    result.Add(new DirectoryEntry(files[i], t.Result.IsDirectory(), t.Result.Size, null));
                }
            }

            return result;
        }
    }
}

The single method in this class will take a physical path as an argument and asynchronously return all items in the directory. The [IgnoreNamespace] is another one of those attributes that control the generated script. If you want to experiment, you can change this to [GlobalMethods] and see what happens to the generated script. The content of the method is very similar to what was covered in the last post.

Implementing the application

Now replace the content of the Program.cs file in the application project with this:

using System;
using System.Collections.Generic;
using DirectoryEnumerator;
using NodeJS.HttpModule;
using NodeJS.PathModule;
using NodeJS.UrlModule;

namespace ModularizedNodeJSFileBrowser {
    class Program {
        static void Main() {
        Http.CreateServer(async (req, res) => {
                try {
                    var url = Url.Parse(req.Url);
                    var path = url.Path;
                    if (path.StartsWith("/"))
                        path = path.Substr(1);
                    if (path.EndsWith("/"))
                        path = path.Substr(0, path.Length - 1);

                    var physicalPath = "C:\\" + string.DecodeUri(Path.Normalize(path));

                    List<DirectoryEntry> content;
                    try {
                        content = await Enumerator.EnumerateDirectory(physicalPath);
                    }
                    catch (Exception ex) {
                        if (ex is AggregateException)
                            ex = ((AggregateException)ex).InnerExceptions[0];
                        res.Write("<html><body>");
                        res.Write("Error reading directory:<br>");
                        res.Write(ex.Message + "<br>");
                        res.Write("</body></html>>");
                        return;
                    }

                    res.Write("<html><body>");

                    if (content.Count == 0) {
                        res.Write("The directory is empty");
                    }
                    else {
                        res.Write("<ul>");
                        if (path != "") {
                            res.Write("<li><a href=\"/" + Path.Normalize(path + Path.Sep + "..") + "\">..</a></li>");
                        }

                        foreach (var entry in content) {
                            res.Write("<li>");
                            if (entry.Error != null) {
                                res.Write(entry + ": " + entry.Error.Message);
                            }
                            else {
                                if (entry.IsDirectory.Value) {
                                    var newPath = "/" + (path != "" ? path + "/" : "") + entry.Name;
                                    res.Write("<a href=\"" + newPath + "\">" + entry.Name + "</a>");
                                }
                                else {
                                    res.Write(entry.Name + ": " + entry.Size + "B");
                                }
                            }
                            res.Write("</li>");
                        }
                        res.Write("</ul>");
                    }
                    res.Write("</body></html>");
                }
                finally {
                    res.End();
                }
            }).Listen(8000);
        }
    }
}

Final Words

This code is available in the Saltarelle Compiler Samples. Of course there are other possible refactorings to do with this code, but those are left as an exercise to the reader.

Writing node.js applications in C#

Thu, 11 Oct 2012 09:57:56 GMT

Node.js has come from nothing to being one of the most watched projects on GitHub incredibly quickly. and I think that by today everyone knows, and has an opinion, it. Personally, I find it interesting and I really like the simplicity it provides. But what I don't like is that I have to code my application in Javascript in order to use it. I have tried it, and being a .net developer used to Visual Studio with Resharper, I was frustrated every single minute because none of the IDEs I tried (and I think I tried all the major Javascript IDEs) was able to give me IntelliSense good enough to help more than it hurt.

If you are like me, you probably wish that you can write a Node.js application in C#. Say, for example, that there were a tool that would allow you to write, in a .cs file in Visual Studio, with (working) IntelliSense at every point, a program like this:

using NodeJS.HttpModule;

class Program {
    static void Main() {
        Http.CreateServer((req, res) => {
            res.Write("Hello, world");
            res.End();
        }).Listen(8000);
    }
}

and translate it into something like:

var http = require('http');
http.createServer(function(req, res) {
    res.write('Hello, world');
    res.end();
}).listen(8000);

I don't know about such zero-overhead tool, but there is a tool which will transform it to

require('mscorlib');
var http = require('http');
////////////////////////////////////////////////////////////////////////////////
// Program
var $$Program = function() {
};
$$Program.$main = function() {
    http.createServer(function(req, res) {
        res.write('Hello, world');
        res.end();
    }).listen(8000);
};
Type.registerClass(null, '$Program', $$Program, Object);
$$Program.$main();

The Saltarelle C# to Javascript compiler now has support for Node.js. You probably also feel that the new C#5.0 keywords async and await would be a good fit for Node.js programs. So do I, and many of the Node.js APIs have counterparts that return a Task, which can be awaited.

A Simple File Browser

To demonstrate the capabilities of Saltarelle with Node.js, here is a tutorial wich creates a fully asynchronous (on the server) file browser by writing the source code in C#.

Creating the project

First we need to create a project for our application. Create a new Visual Studio project of type Console Application and call it SimpleNodeJSFileBrowser. Then open the package manager console and

> Install-Package Saltarelle.NodeJS

Now unload the project and reload it again (we need to do this because of a Visual Studio issue when replacing the standard mscorlib with a custom one), and remove the attributes it complains about when you try to compile. Now, in order to make the project run, and work, when we hit Ctrl+F5 we need to do a few more things:

Download the correct node executable to the project folder.
The Script generated by Saltarelle requires a module called 'mscorlib' to be included. This module is available with the Saltarelle.Runtime package (which was installed as a dependency when we installed Saltarelle.NodeJS). Therefore, create a directory called node_modules in the project directory. Unload the project from the solution explorer and add the following code to the BeforeBuild target:
```
  <Target Name="BeforeBuild">
    <ItemGroup>
      <MscorlibScript Include="$(SolutionDir)packages\**\mscorlib.js"/>
    </ItemGroup>
    <Copy SourceFiles="@(MscorlibScript)" DestinationFolder="$(ProjectDir)node_modules"/>
  </Target>
```
This will ensure that whenever we update the Saltarelle.Runtime package, the correct mscorlib.js file will be copied to our node_modules directory so it can be found when we run our project.
In the project properties, on the Debug tab enter the following values:
- Start action: Start external program "C:\Windows\System32\cmd.exe"
- Command line arguments: /K "C:\<path to your project>\node.exe" SimpleNodeJSFileBrowser.js
The reason to do it like this rather than put node.exe directly in the "Start external program" option is that by doing it in the above way, the console window will prevail in case of an error that terminates the node process. Unfortunately, msbuild properties such as $(ProjectName) do not work in these options.
Test that everything works by pasting the simple program above into the Program.cs file, hit Ctrl+F5 and navigate to http://localhost:8000. The browser should display "Hello, world".

Adding the file browser code

Now that we have a working skeleton, it is time for us to create the actual file browser that we want. This code will use the C# 5.0 features 'async' and 'await', so you need Visual Studio 2012 to compile this (the RC works). Now it is time to actually type in our program, line by line (no, you won't do this because you will use editor features, but this is for explanation; Additionally, the syntax highlighter I use does unfortunately not support indenting a whole block of code so you have to pretend that the indentation is correct:

using System;
using System.Threading.Tasks;
using NodeJS.FSModule;
using NodeJS.HttpModule;
using NodeJS.PathModule;
using NodeJS.UrlModule;

All the different node modules reside in their own namespaces. Adding a using directive does not cause the module to be require()d, that only happens when you use something from a module.

namespace SimpleNodeJSFileBrowser {
    public class Program {
        public static void Main() {
            var server = Http.CreateServer(async (req, res) => {
                try {
                    var url = Url.Parse(req.Url);
                    var path = url.Path;
                    if (path.StartsWith("/"))
                        path = path.Substr(1);
                    if (path.EndsWith("/"))
                        path = path.Substr(0, path.Length - 1);
                    
                    var physicalPath = "C:\\" + string.DecodeUri(Path.Normalize(path.Replace("/", Path.Sep)));
                    
                    res.Write("<html><body>");

This is just some boring code to ensure that the path neither starts nor ends with a slash, and to convert the URL to a physical path. Of course, if you want to browse something that is not C:\, you can change this.

                    string[] files;
                    try {
                        files = await FS.ReaddirTask(physicalPath);
                    }
                    catch (AggregateException ex) {
                        res.Write("Error reading directory:<br>");
                        foreach (var m in ex.InnerExceptions) {
                            res.Write(m.Message + "<br>");
                        }
                        res.Write("</body></html>");
                        return;
                    }

This is where it gets interesting. The FS.ReaddirTask call will do some magic to transform the call to a Node.js "fs.readdir()". Awaiting the task will return either the list of files, or throw an exception if the node call failed (meaning that the callback was invoked with a non-null error argument). If this happens, we just print the error.

                    if (files.Length == 0) {
                        res.Write("The directory is empty");
                        res.Write("</body></html>");
                        return;
                    }
                    
                    Task<Stats>[] statTasks = files.Map(f => FS.StatTask(physicalPath + "\\" + f));
                    try {
                        await Task.WhenAll(statTasks);
                    }
                    catch (AggregateException) {
                    }

This code uses Javascript's Array.map method to start tasks to invoke the node stat() method on each file we previously found. Note that all these tasks will be executed in parallell. Then we wait for all the tasks to finish with an "await Task.WhenAll(...)". This await will throw an AggregateException if any of the involved tasks failed. We ignore this exception because we will check the outcome of each individual task later.

                    res.Write("<ul>");
                    
                    if (path != "") {
                        res.Write("<li><a href=\"/" + Path.Normalize(path + Path.Sep + "..") + "\">..</a></li>");
                    }
                    
                    for (int i = 0; i < files.Length; i++) {
                        res.Write("<li>");
                        if (statTasks[i].IsFaulted) {
                            res.Write(files[i] + ": Error retrieving stats: " + statTasks[i].Exception.InnerExceptions[0].Message);
                        }
                        else {
                            var stats = statTasks[i].Result;
                            if (stats.IsDirectory()) {
                                var newPath = "/" + (path != "" ? path + "/" : "") + files[i];
                                res.Write("<a href=\"" + newPath + "\">" + files[i] + "</a>");
                            }
                            else {
                                res.Write(files[i] + ": " + statTasks[i].Result.Size + "B");
                            }
                        }
                        res.Write("</li>");
                    }
                    
                    res.Write("</ul></body></html>");

Again, this code is rather boring. It just loops over each of the tasks and writes different strings depending on whether the task was successful, whether the item is a directory, etc.

                }
                finally {
                    res.End();
                }
            });
            server.Listen(8000);
        }
    }
}

This finally block will ensure that whenever an exception is thrown, the res.end() method will be invoked. It does not matter whether this is due to normal completion, an unhandled exception, or a failing task. Finally (no pun intended), we tell the server to listen to port 8000, just as you would write in Javascript directly.

A look at the generated code

Warning: This is not for the faint hearted, the generated code can look quite indimidating at first. Bear in mind, though, that we are invoking an asynchronous method, using the result to dispatch a bunch of parallell asynchronous calls and waiting for them all to finish before we process the result. If you did not use Saltarelle, this would get you into the mess of nested callbacks, or using one of the many control flow libraries available, which can also be a pain to debug and step into. I'm not saying the generated code is "better" than the alternatives, but I don't think it's far worse. And the source C# is quite nice.

The generated file is in the project output directory (by default bin\Debug) and called SimpleNodeJSFileBrowser.js. Open it and investigate. I will not go through the entire contents here, but I will make a few comments:

The compiler knows which modules we require and inserts the appropriate calls to import them.
The entire body of our async lambda has been replaced with a nested function called $sm. The only real operations performed by the method is to set the initial state and invoke this inner function.
Whenever there is an await, it is replaced with code like
```
$state = NEW_VALUE;
$t1.continueWith($sm);
$doFinally = false;
return;
```
This code will cause the state machine to continue from its current position as soon as the awaited task finishes. It also ensures that any finally blocks are not run when awaiting a task.
The FS.ReaddirTask (and FS.StatTask) calls are replaced with code like ss.Task.fromNode(fs, 'readdir', physicalPath). This is the magic call that will invoke the given method with a set of arguments, and will add a callback that does the right things.
Code blocks that do not contain any awaits are translated quite literally.

Final Words

This code is available in the Saltarelle Compiler Samples.

The (C# source) code, although it works, is quite messy. In the next post we will refactor it so the actual directory lookup will become a Node.js module.

And, of course, if you prefer nested callbacks over the auto-generated state machine, there is of course metadata for those methods as well, so you can choose the style you prefer.

Forum for Saltarelle Compiler users set up

Fri, 21 Sep 2012 12:34:00 GMT

Due to public request, I have set up a forum dedicated to users of the Saltarelle C# to Javascript compiler. It is found at http://saltarellecompiler.freeforums.org/.

My new favorite MSBuild hack

Fri, 21 Sep 2012 10:28:00 GMT

Every now and then, when building my Visual Studio projects, I run into problems such as auto-generated files not being generated before the compiler is run, or build failing due to locked files. The solution to these problems is often to add

<UseHostCompilerIfAvailable>false</UseHostCompilerIfAvailable>

to the first PropertyGroup in the .csproj file. This is to Visual Studio what "zoom: 1" is to IE6!

Mentioned by Miguel de Icaza

Fri, 07 Sep 2012 13:45:34 GMT

No, this post does not contain anything interesting, it is just intended to boost my ego and tell everyone about it. Just 4 days after officially introducing the Saltarelle compiler, it has been mentioned by Xamarin founder and Mono guru Miguel de Icaza. This project is really getting attention!

Aftermath of the compiler introduction

Tue, 04 Sep 2012 20:13:16 GMT

When I wrote my post introducing the Saltarelle C# to Javascript compiler and submitted it to Hacker News, I was a little worried that it would just get lost in the noise and disappear into the void immediately without many people noticing. One can easily say that this was not the case. It got to second place for a few hours and stayed on the first page for many hours. And this newly started blog has now had more than 7,500 unique visitors! It would be an understatement to say that this exceeded my expectations by quite a bit. And it also resulted in quite a few followers to the project on gitub. Thank you everyone for caring!

It also turned into quite a debate on Hacker News on whether the whole thing is a good or a bad idea. For many projects it is not, if all you want to do is to add some dynamic behaviour to a website, using a compiler like this would be like killing mosquitos with an elephant rifle. For example, the compiler's own website does not actually use it. That being said, for larger projects, it is my strong belief that a tool like Saltarelle (or Script# or GWT or something similar) will help you achieve a working, maintainable, application with less effort. This shouldn't be too controversial, most programming languages are suitable for certain thing but not for other and Javascript is the exception in that people use it for all tasks ranging from adding fade effect to an ad to writing multi-100-kloc single-page applications.

I do feel like, however, that some of the arguments against cross-compiling just stem from lack of knowledge about the concept and deserve to be countered by someone who has actually used the technology in question, so here are my answers to a few of them:

"The very best of luck when it comes to debugging the code when it is running in the browser"

This argument I do not really understand. As an example, we can take this code (from the features page of the compiler home page):

public class C {
	public string Property;
	public List<string> Collection;

	public C() {
		Collection = new List<string>();
	}
}

public class Driver {
	public void Main() {
		var c = new C() { Property = "Property", Collection = { "Value 1", "Value 2", "Value 3" } };
		var sb = new StringBuilder();
		sb.AppendLine("c.Property = " + c.Property);
		sb.AppendLine("c.Collection = " + c.Collection);

		Document.Instance.GetElementById("output").InnerText = sb.ToString();
	}
}

This compiles to the JavaScript

C = function() {
	this.property = null;
	this.collection = null;
	this.collection = new Array();
};

Driver = function() {
};
Driver.prototype = {
	main: function() {
		var $t1 = new C();
		$t1.property = 'Property';
		$t1.collection.add('Value 1');
		$t1.collection.add('Value 2');
		$t1.collection.add('Value 3');
		var c = $t1;
		var sb = new ss.StringBuilder();
		sb.appendLine('c.Property = ' + c.property);
		sb.appendLine('c.Collection = ' + c.collection);
		(document).getElementById('output').innerText = sb.toString();
	}
};

I do not see how this is particularly hard to debug. Of course, if you want to use features that you simply cannot do in JS (yield, goto), the code loses similarity to the C# source, but if this is a concern to you, you can opt not to use those features. Also, usually the translation is even more direct than this, many statements are identical in C# and Javascript, except that the JS version, by default, uses camel-case for all members. Of course, source maps can help, but I honestly don't think it's necessary, and it might also hurt your debugging experience if the problem you are debugging is the result of a leaky abstraction or incorrect metadata in an import library.

"I fear projects like these tend to attract programmers who know C#, Java but don't quite want to learn javascript and they don't even try to do it once they run out of runway in this tool. And, the code becomes undebuggable, bloated on the browser."

So what are you trying to say with this? That people should not be allowed to write code to make a website dynamic without fully understanding the Javascript type system (a concept which is probably about as well understood among the crowd as quantum physics). Is it not just good that everyone can be productive in a web environment? Are you fearing for the average code quality on the web? If so, why? Also, I think the code quality on the web is already low enough that it can hardly get worse. Being able to code web but not have to struggle with Javascript intricacies might even evolve the web; organizations debate whether to do their internal tools as client applications or web apps, with Javascript almost always being an argument against doing it on the web. For the debuggability I touched on that above. And who cares if the code is bloated?

"I don't see how you can really use this without already being very familiar with Javascript"

This argument does have some validity. If you use a tool like Saltarelle, you will be bitten by leaky abstractions. Not very often, perhaps not even in every project, but it will happen. If I had the choice, I would much rather execute the MSIL directly inside the browser, but unfortunately I do not have that choice. As long as you are working in a web environment you need some basic Javascript skills, but with a tool like Saltarelle there are things you don't have to care about, eg. forgetting to use strict instead of loose equality. This means that instead of everyone having to be a Javascript guru, you can have only one such guru per project, or perhaps even only one in your organization.

"My largest criticism for projects like this is that I absolutely abhor running into compiler errors"

(Assuming the OP meant compiler bugs rather than compiler errors)
Running into compiler bugs is always nasty. But this might happen every time you use a compiler. I remember myself having discovered my shared of gcc bugs many years ago, and this is not fun. Does this mean you write all your code in assembler? Saltarelle compiler bugs should be relatively easy to identify, if there is one you can just check the generated script and verify. Hopefully there are not many and those that do exist should be fewer and fewer by the week. You'll just have to ask yourself whether it is likely that you will spend more time tracking down compiler bugs than you gain by being able to use static typechecking and IntelliSense.

Anyway, thanks everyone for reading and showing interest in the project!

Introducing the Saltarelle open source C# to JavaScript compiler.

Mon, 03 Sep 2012 09:31:33 GMT

I have been writing web applications for a really long time. In my early days I remember looking forward to my employer upgrading from IE3 to IE4 so I could use the brand new feature of dynamically adding rows to a table in what was then called DHTML. Then, later, came Remote Scripting, which was Ajax (just 8 years earlier), implemented with IFRAMEs or Java applets. During all these years, it has become apparent to me that Javascript, while excellent for achieving smallish effects like form validation or conditionally showing text, is not well suited for developing large systems. Of course it is possible to create a great large system in Javascript, just as it is possible to build a computer that directs ship fire from gears and disks, but it is my strong belief that it is not the most cost-effective way of doing it if other alternatives are available. Then, in 2007 or so, I found Script#, a project that takes a C# project and generates a Javascript file as output. I have used this extensively over the past few years, and the idea is excellent! It brings all the productivity tools you have in C# to web development. You can use ReSharper refactorings, get (correct) IntelliSense and you won't ever se an "object does not support this property" error message again.

Unfortunately, though, development of Script# has since then all but ceased, with hardly any compiler features being added over the past 3 or so years. When .net 3.5 was new, not having implicitly typed variables or lambda expressions (or quite a lot of other features) wasn't too bad, but today it feels very old. Once again, server-side programming is an order of magnitude less annoying than client-side programming.

But no more. About a month ago I released the Saltarelle compiler. It is intended as an almost drop-in replacement for Script#, but with almost full support for C# 4. And it is also open source! Over the past weeks it has matured to the point that it is now stable and ready for public use. There is a Getting started guide for those new to the concept, and also a guide for migrating from Script#.

I do not think that this is a tool that is suitable for everything but if you are developing a fat client application in HTML (especially, but not only, if your serverside code is .net) it will make your life easier.

Generating typed proxies for your data access layer

Thu, 30 Aug 2012 16:26:39 GMT

In the realm of database development, there are two camps. There are the application developers who love their ORMs and prefer to consider the database to be a simple data store. On the other hand there are the DBAs that consider the database to be the application and everything else to just be an appendix, and force all communication between applications and the database to be through stored procedures and views. I am not entering the argument whether one approach is better than the other, they both have pros and cons, but if you have for some reason decided on the DBA model, you are quite likely going to have a lot of code like this:

using (SqlCommand cmd = CreateCommand()) {
	cmd.CommandText = "dbo.some_proc";
	cmd.CommandType = CommandType.StoredProcedure;
	cmd.CommandTimeout = CommandTimeout;
	SqlParameter p;
	p = cmd.Parameters.Add("@EmployeeID", SqlDbType.Int, 4);
	p.Value = employeeId;
	p = cmd.Parameters.Add("@Title", SqlDbType.NVarChar, 100);
	p.Value = title ?? DBNull.Value;
	p = cmd.Parameters.Add("@HireDate", SqlDbType.DateTime, 8);
	p.Value = (object)hireDate ?? DBNull.Value;
	using (var rdr = cmd.ExecuteReader()) {
		while (rdr.Read()) {
			DoSomething(MapReaderToEntity(rdr));
		}
	}
}

This is an awful lot of code. Additionally, the MapReaderToEntity method will likely contain a lot of magic strings mapping columns to data members (or worse yet, magic numbers relying on column ordinals. Of course, EF and NHibernate both have support for stored procedures for doing specific things like updating an entity, but if I'm going through the complexity of SPs I don't want to go through the complexity of an ORM as well.

To solve this problem for myself, I created a small library based on T4 that I call SqlProxy, which can be found here. The thought is that you add the DatabaseProxy.tt and SqlProxy.cs files to your project. Then whenever you save the DatabaseProxy.tt, it will go to your database and generate typed C# classes based on the information it finds.

How to use the generated code

Usage of the SqlProxy library is very simple:

using (var proxy = new DatabaseProxy("server=.; integrated security=true; initial catalog=AdventureWorks")) {
	int employeeId = proxy.QueryHumanResourcesVEmployee(r => r.FirstName == "Gigi" && r.MiddleName == "N" && r.LastName == "Matthew", null).Single().EmployeeID;
	foreach (var row in proxy.UspGetEmployeeManagers(employeeId)) {
		Console.WriteLine(row.FirstName + " " + row.LastName);
	}
}

This example finds a single matching row in a view, then calls a SP with that result and iterates through the result set returned by the SP. Of course, the result set is not buffered anywhere, it is streamed directly from the underlying datareader directly to the consumer. And, since it is all generated code, IntelliSense is fully functional.

If a stored procedure returns more than one result set, you cannot foreach over it, but you can foreach over the result sets which are named ResultSet1, ResultSet2 and so on. Due to SQLServer and the streaming nature of SqlProxy, you need to enumerate the result sets in order. If the SP returns a result set and also has output variables, you can access the value of the output variables as properties on the result object, but after you do this, all result sets will be closed (again, due to how SQLServer works). If the SP has output parameters but does not return any result sets, the parameters are simple ref parameters to the method.

In addition to generating proxies for stored procedures and tables/view, the DatabaseProxy class also has a few utility methods, in particular:

ExecuteTransacted, which will execute a block of code inside a transaction, and can optionally retry the execution if the process was chosen as a deadlock victim (the famous error 1205),
BeginTransaction, which will return a transaction wrapper, and until this transaction is disposed, all commands issued from this proxy will automatically be enlisted in the transaction. I don't understand why a plain SqlTransaction does not work like this, but it does not.

Customizing the code generation

SqlProxy has a few options you can change to control the generated code. All these options are variable definitions in the DatabaseProxy.cs file. Most of them are self-explanatory, but these ones are worth mentioning:

generateSqlClr: Does magic stuff that makes the generated proxy class being usable in SQLCLR on SQL Server 2005.
schemaNames: Only include objects from these schemas.
customExtractions: This dictionary allows you to specify which arguments to use when executing procedures in order to determine the metadata. The key is the procedure name, the value is a list of parameter values.
namedQueries: This allows you to create named queries which can be used in the same way as stored procedures. In the example, you can write proxy.EmployeesByBirthDate(minDate, maxDate), exactly as if that query were a stored procedure with one result set. Of course, the database is queried to determine the layout of the result sets.

How it works

SQLServer has a lot of metadata information available, you just need to read it. For getting table, view and custom query information, it uses the GetSchemaTable() method of SqlDataReader. For stored procedures, it calls the procedure inside a transaction which is rolled back. If a custom extraction is not specified for a procedure, it also uses SET FMTONLY ON. Note: You need to specify a custom extraction for a procedure (even if all parameters are null) if you use an IF/ELSE to return different result sets, because given the code below:

CREATE PROCEDURE p(@v int) AS BEGIN
    IF @v < 0
        SELECT 1
    ELSE
        SELECT 0
END

the metadata extraction based on SET FMTONLY ON will think that the procedure returns two result sets.

On a side note, in SQLServer 2012, SET FMTONLY ON is replaced with a procedure called sp_describe_first_result_set which doesn't have the above problem but, OTOH, is not usable for SPs that return more than one result set. It is left as an exercise to anyone interested in this code to port it to use the new method instead.

Conclusion

Is this really useful? No doubt, if you're using SPs for communicating with your database. It seems, though, that ORMs are gaining the upper hand in the fight, and I won't recommend you to do one thing or the other, both sides have pros and cons.

Deploying SQLCLR code as part of your build

Thu, 23 Aug 2012 15:31:00 GMT

SQLCLR is a relatively little known and used technology that enables you to run .net code in SQL Server. This gives you the option to do simple things like write a regexp_ismatch function, as well as more complex things such as (like I do) use ANTLR inside SQL Server to transform some DSL into SQL statements to be executed and whose results are returned to the client. I'm not debating (at least not now) whether it is a good or a bad thing; like most technologies it has both pros and cons.

One thing that is, without any question, terrible is its deployment model. From a high level perspective, is sounds neat: you create a .net assembly, annotate some methods with special attributes, compile your assembly, and finally you deploy the compiled assembly to the database. The only catch in this is the last part, deploy the compiled assembly. This involves executing the CREATE ASSEMBLY statement (which, additionally, if you only want to supply your ops team with a single SQL file involves hex-encoding the .dll), then you need to execute CREATE FUNCTION and CREATE PROCEDURE (and other) statements to register the assembly members with the database.

But wait, didn't we annotate the methods with attributes? Yes, we did.

Aren't those annotations used? Yes, and no. No, it is not used in the sense that just because you put a [SqlFunction] attribute on a method, it does not mean that the method is available for use as a SQL function. But, some metadata is not available in the CREATE statments but are instead read from the attributes. And some of the metadata is available in both the attribute and the CREATE statements and if different values are specified strange things might happen.

Visual Studio to the "rescue"

Fortunately, Microsoft realized that this model might be an issue and created the SQLCLR project type, available in Visual Studio Ultra Expensive Edition (actually, Professional can open those projects but doesn't give the choice to create them so you have to manually enter a GUID in a .csproj; I wonder if MS' intention was for this to work). Projects of this type can be deployed to a database using just a right click in the Solution Explorer.

Unfortunately, the deploy functionality seems to be available only from within Visual Studio, not from msbuild. This means that the way Microsoft intended development of this kind of code to be is something close to:

Go along and write your code,
Enter the connection string to your production server in the project property page in VS,
Click the Deploy item in the context menu. Can it be more simple?

Alternatively, the slightly more sophisticated way:

Write your code,
Deploy the code to your development server,
Use SQL Server's Generate Script feature to generate the script for all objects from the assembly.

Actually, this second approach does allow some scripting since at least the last step is scripable albeit cumbersome. But if you want your build server to generate the deployment script from sources, you're out of luck because of step 2.

Additionally, for some godforsaken reason SQLCLR projects can only reference other SQLCLR projects so if you want to share a DTO project between your SQLCLR code and your web front-end, you're completely out of luck with this approach. Also, you can only pick your references from the assemblies currenly installed in SQL Server. You cannot (at least not officially) reference a random assembly on disk and accept that you need to deploy this assembly as well.

In other words, this integration can only be described as Halfway finished crap, not suitable for professional software development.

A working solution

However, SQLCLR should not completely be discarded as a useful technology just because the Visual Studio tooling is so bad. I'm not claiming it's a good fit for most projects (perhaps not even for mine; but changing it now would be a lot of work), but it certainly has its uses. The deployment model, though, is something that has to be resolved, so I did just that. It only took a 200ish-line PowerShell script, which can be found here: https://gist.github.com/3437689. You have to get (the excellent) Mono.Cecil and put the dll into the same file as the powershell script and then you can do

.\GenerateSqlClrAssembly -File "MySqlClrAssembly"

and it will print the deployment script to the pipeline (stdout).

This script contains one unofficial feature: If you specify a name with a dot in an attribute, it will be treated as the multipart identifier schema.object_name instead of the single-part identifier [schema.object_name] (which is what Visual Studio does). If you don't want this behavior, modify the Get-Qualified name function.

The joys of Subversion

Thu, 23 Aug 2012 11:04:00 GMT

I have been using Git exclusively for so long now that I didn't remember the amount of fun you're up to when you see this message:

Running QUnit tests in NUnit

Wed, 22 Aug 2012 20:42:00 GMT

QUnit is a great tool for web development. Admittedly, it is not the most impressive piece of software ever written, but it does its job and does so well. Unfortunately, its job is to run a suite of JavaScript tests and display the outcome to the user which is, albeit helpful, not everything you want when it comes to testing. For it to be really useful, you need to run the tests automatically and how to do this in a .net environment is something that Google provides little help on.

To solve this problem, maybe you have written some custom tool to parse the results? Maybe you use some feature of your CI server? Or perhaps you don't do it at all? Myself, I use the same tool that I use to run all my other tests: NUnit. This means I can user ReSharper to run my tests, no special-casing in the build script, just smooth sailing all the time. And if I use it with this test file:

test('Loose_equality_should_consider_string_and_number_equal', function() {
	ok(1 == '1');
});

test('Strict_equality_should_consider_string_and_number_equal', function() {
	ok(1 === '1');
});

test('Browser_should_not_be_MSIE', function() {
	ok(navigator.userAgent.indexOf('MSIE') == -1, 'User-agent string should not contain MSIE');
});

I get output like this:

And, finally, I don't have to write more C# than this:

public class MyTest : TestBase {
	protected override string PageAddress {
		get { return "QUnitTests/MyTests.html"; }
	}
}

How it is done

The key to all this is the use of the Selenium browser automation framework, together with the [TestCaseSource] NUnit attribute. The code can be found at https://gist.github.com/3428959. To use it, simply create a test that derives from TestBase (just like demonstrated above), and override the PageAddress property to point to the relative URL of any QUnit page you want to run (with this approach you need one class for each page, which I think is the best balance between amount of (strictly speaking unnecessary) C# code to write, and output quality.

The only missing part in this is what to put in the BaseAddress constant in the TestBase class, which needs to be an address where your website is available. This can be as simple as using the address specified in the Web page in the project settings in Visual Studio (although this leaves some to be desired if you want to run on a CI server). For my own purposes, I use an IIS site that my build script sets up automatically, but more on this in a later post (actually I don't even use a constant).

Hello, World

Tue, 21 Aug 2012 16:59:57 GMT

I am very happy to announce that the world is going to become a slightly better place over the following weeks, because I have decided to start blogging. I have been about to do that many times before, but this time I feel the moon is in the right phase. Wish me luck!