'The Official Programmer's Heaven Blog' Blog RSS Feed

Very Quick Guide To DLinq: Part 3 (LIKE, IN, DATEDIFF)

pheaven — Wed, 30 Apr 2008 06:50:28 -0700

Much of the time, it's fairly straightforward to express an SQL query in Linq instead. However, there are some things that aren't quite so clear how to do. In this article, I'll take a look at some of those.

=== LIKE ===
There are a couple of ways to replace LIKE. If you just want to directly use LIKE, then you can do so using the SqlMethods class. To use this, you must add the following using declaration at the top of the file:

using System.Data.Linq.SqlClient;

Then we can use it like this:

var Result = from Book in DB.BookResources
             where SqlMethods.Like(Book.Title, "%java%")
             select Book;

This will create an SQL query that locates all books containing "java" (case-insensitively). Interestingly, you can also do the following:

var Result = from Book in DB.BookResources
             where Book.Title.Contains("java")
             select Book;

And it is equivalent. This is useful and results in the same SQL, but also a tad confusing: you get the case sensitivity (or default lack thereof) configured in the database, even though normally in C# code outside of a Linq query this method is case sensitive.

You can also use the .StartsWith and .EndsWith methods of a string, which can replace some LIKE queries a little more readably. They have the same case quirks, though (under the hood, they all compile down to a LIKE query).

=== IN ===
Our example table has a rating field that allows us to store a book's rating out of ten. We want to get those books scoring between 7 and 10 points (or some other arbitrary List of scores). We can use the List's Contains method inside the Linq query, which will cause it to generate an IN query in the SQL.

var GoodRatings = new List<short>() { 7, 8, 9, 10 };
var Result = from Book in DB.BookResources
             where GoodRatings.Contains(Book.Rating)
             select Book;

Nice and easy.

=== DATEDIFF ===
The SqlMethods class also contains various methods to allow us to express the SQL DATEDIFF built-in. In the following example, we select all books that were added to the database in the last 30 days.

var Result = from Book in DB.BookResources
             where SqlMethods.DateDiffDay(Book.EntryDate, DateTime.Now) <= 30
             select Book;

There are a variety of related methods available, including DateDiffYear, DateDiffHour and so forth, depending what unit of time you want to get the result back in. These compile to an SQL DATEDIFF query.

Note that methods on TimeSpan objects are not translated by Linq, therefore you can not write:

var Result = from Book in DB.BookResources
             where DateTime.Now.Subtract(Book.EntryDate.Value).Days < 30
             select Book;

Trying to do so will give you a NotSupportedException at runtime (unfortunately meaning that you won't find out that this won't work at compile time).

=== If All Else Fails ===
If you are really stuck trying to express a tricky SQL query in Linq, but using Linq generally, then you can still use the DataContext's ExecuteQuery method to send a query directly to the database in SQL.

var Result = DB.ExecuteQuery<BookResource>(
    "SELECT * FROM [BookResources] WHERE Title LIKE '%java%'");

We use a type parameter to specify the type of object that will contain the result of our query, then just specify the query to execute as a string. Note that we could have expressed this one in Linq, it's just an example.

If you want to just get one field, write the type of that field.

var Result = DB.ExecuteQuery<string>(
    "SELECT [Title] FROM [BookResources] WHERE Title LIKE '%java%'");

If you're getting a few fields back, then you will probably need to create a class with properties that have names matching those of the fields that you use in the query. However, there is another variant of ExecuteQuery that takes a Type object as a first parameter, so you can most probably instantiate an anonymous type using new() { ... } with some dummy values to get the member types correct, then call GetType and pass that as the parameter. I'll leave testing this beautiful bit of evil as an exercise for the reader.

=== Final Thoughts ===
Linq works to make working with databases feel close to working with other objects in your program. Thus IN feels very natural and, while they provide a direct way to use LIKE, some methods of the String class are usable too. However, in some cases you have to fall back to using some of the methods in the SqlMethods class, and in an extreme few further cases you might need to drop all the way back to using SQL.

An Introduction to Mock Objects

pheaven — Tue, 15 Apr 2008 14:26:34 -0700

Writing automated tests is a tried and tested way to improve the quality of software. In the initial phase of development, tests help to verify that the code functions correctly. In Test Driven Development, tests are written before the code, so any knowledge about the ins and outs of the implementation won't influence the writing of the tests. After the initial development, as changes are made over time, a comprehensive test suite can quickly point out unintended changes in the behavior of the code, so the bugs can be fixed before the software is shipped. Importantly, the tests are automated, so they are very cheap to run in terms of time.

=== What do we want from a test suite? ===
There are a range of properties that are good to have in a test suite.
* It often allows bugs to be pinpointed to a relatively small area of the code base, saving debugging time.
* It should run quickly enough to make running the suite feel like something developers can afford to do regularly. I've worked on projects where the test suite could take a very long time to run, and the consequence was inevitable: people didn't always run the tests before committing changes. I'm at least as guilty as anyone else on the project for doing this, especially when hacking on my slow old laptop.
* It tests not only individual bits of the software, but also tests them running in combination.
This last point is important. We want to be able to test relatively small units of code in isolation, giving us the ability to pinpoint the problems. However, we also need integration tests. Sure, two classes might work fine on their own and pass their tests, but what happens when you start using them in combination?

The tests for small units of the code are easy enough to write for some things. For example, the tests for the data access layer may not depend on any other bits of the code. However, the class that uses the data access layer to pull data from the database and do a series of complex calculations based on it is harder to test in isolation.

Testing it in combination with the data access layer doesn't tell us right away where the bug lies if a wrong result comes out: in the data access layer, the data in the database or in the calculation code. Moreover, database queries take time, and slow down the running of our tests. And if we want to test the calculations with some edge cases, we have to go and add them to the database, or write code to do so. This is a simple example, but as you build complex software with complex dependencies, it gets more and more important to try and test individual pieces of the software both in isolation and rapidly.

=== Introducing Mock Objects ===
A mock object is one that has the same interface (that is, the same properties and methods) as some real object, but unlike the real object just returns dummy data. It will also validate the data that we pass it as arguments, simulating failure when these arguments would have led to a failure.

When we run tests for a class that would normally use the real object, we can instead supply it with the mock object. This allows us a new level of flexibility in our testing.
* We can modify the mock object during the tests to get it to get it to return a range of different data - valid, invalid, edge cases and so forth - to test how the code calling it handles such situations.
* We can have the mock object simulate failures, such as an inability to connect to a database, to test the failure mode of the code using in.
* We can log exactly what the code using the mock object did (e.g. what methods it called, and with what parameters), which may aid the debugging process.
Additionally, because the mock object is not really doing a great deal of work, but instead is just returning data, our tests may be able to run much more quickly.

Imagine we made a mock object for our data access layer. Since it didn't really have to go off to the database to fetch the data or do the updates, the tests would be a lot faster to run. Additionally, we could return a range of different data and test how the code doing the calculations responds without having to actually modify the database itself. We can also simulate database failure without taking down the database!

=== Mocking Your Language ===
Writing your own classes for mock objects would be tedious. Thankfully, a range of mocking frameworks exist, which help take the boring work out of creating mock objects for you by allowing you to specify the mock objects more declaratively. In a future article, I'll take a look at NMock, a mocking framework for .Net. In the meantime, here are some mock object frameworks for different platforms and languages.
* .Net Languages: NMock
* Java: jMock
* Perl: Test::MockObject
* C++: mockpp
* Ruby: Mocha
* Python: pMock
Happy mocking!

When Linq and ref Parameters Meet

pheaven — Tue, 01 Apr 2008 07:17:31 -0700

It's fair to say that I'm a pretty heavy user of Linq. You'll find uses of it scattered across my code, from the obvious (using DLinq to query a database) to the slightly more exotic (writing queries over collections obtained from classes in System.Reflection). Linq often allows you to express a problem very neatly, resulting in compact, readable code. It also factors out the application of operations and leaves you to worry about the operations themselves, likely decreasing bugs.

=== Fixing A Bug ===
Today I ran across some code that took a parameter, then used it in a Linq query. Omitting the clutter, it looked something like this:

public void Lookup(string URL, ref int ID, ref int Status)
{
    // ...stuff...
    var Result = from D in DB.Datas
                 where D.URL == URL
                 select D;
    // ...more stuff...
}

It then set the couple of ref parameters based upon the results of the query (they really should have been out parameters, and I'll likely do some further refactoring later). However, the code calling this also appeared to be assuming that that the URL parameter would be updated to exactly match the entry in the database. This mattered because the "where" clause would actually match if the case was different, and the calling code wanted to know when this was the case. Clearly, this code couldn't be updating URL; it wasn't a ref parameter. Making it a ref parameter here and in the calling code was the obvious fix, so I did that. The code now looked more like this:

public void Lookup(string ref URL, ref int ID, ref int Status)
{
    // ...stuff...
    var Result = from D in DB.Datas
                 where D.URL == URL
                 select D;
    // ...more stuff, including updating URL...
}

And then I hit compile.

=== The Compile Error ===
Making the above change gave me the following compile error.

Cannot use ref or out parameter 'URL' inside an anonymous
method, lambda expression, or query expression

Or, re-phrased to fit this specific problem, "you can't use ref parameters in a Linq query". My first thought was, "huh, what the...". My second was, "OK, we can easily work around that." But of course, the main question that lingered after the initial "just do something that works so I can fix the bug" was, "but why can't I?"

=== The Workaround ===
The workaround is straightforward enough: you just make a local variable and assign the ref parameter to it.

public void Lookup(string ref URL, ref int ID, ref int Status)
{
    // ...stuff...
    var Lookup_URL = URL;
    var Result = from D in DB.Datas
                 where D.URL == Lookup_URL
                 select D;
    // ...more stuff, including updating URL...
}

This compiles and works just fine. So if that's all you were looking for, you have an answer. But now let's dig into the "why".

=== Introducing ILDASM ===
ILDASM is short for Intermediate Language Disassembler. A .Net executable contains a sequence of low-level instructions encoded in a binary format that we call bytecode, as well as tables of metadata about the classes and methods it contains. ILDASM turns that bytecode into IL, or Intermediate Language: a human readable representation of the binary data.

To get us started, let's just look at disassembling the Hello World program. First, we compile the following C# into an EXE file:

class Program
{
    static void Main(string[] args)
    {
        Console.WriteLine("Hello, world!");
    }
}

And then we use ILDASM to get the Intermediate Language. Stripping away some sections we need not care about, we get:

.class private auto ansi beforefieldinit Program
       extends [mscorlib]System.Object
{
  .method private hidebysig static void  Main(string[] args) cil managed
  {
    .entrypoint
    // Code size       13 (0xd)
    .maxstack  8
    IL_0000:  nop
    IL_0001:  ldstr      "Hello, world!"
    IL_0006:  call       void [mscorlib]System.Console::WriteLine(string)
    IL_000b:  nop
    IL_000c:  ret
  } // end of method Program::Main

  .method public hidebysig specialname rtspecialname
          instance void  .ctor() cil managed
  {
    // Code size       7 (0x7)
    .maxstack  8
    IL_0000:  ldarg.0
    IL_0001:  call       instance void [mscorlib]System.Object::.ctor()
    IL_0006:  ret
  } // end of method Program::.ctor

} // end of class Program

You'll see that there is a class declaration and two method declarations. The first is the Main method that we wrote. We can see that it loads the string "Hello, world!" onto the stack, then calls the the System.Console class' WriteLine method. Note that it finds its argument on the stack. The second method is actually a constructor and was generated for us; all it does is call our superclass' constructor.

As an aside for those of you wandering, "why the two nop (no operation) instructions", the reason for these is that I build this in debug mode. It inserts nop instructions anywhere that you could want to insert a breakpoint, and then at runtime actually patches the bytecode when you attach a debugger to contain "break" rather than "nop" opcodes.

=== The .Net CLR Doesn't Do Anonymous Methods ===
Now we'll write an example with an anonymous method. This code simply creates a list of integer values, then removes all of those that are less than or equal to five. I've highlighted the anonymous method - written with the compact lambda syntax - in red. It takes a single parameter, x, written to the left of the "=>", then checks that it is less than 5, producing a boolean result.

class Program
{
    static void Main(string[] args)
    {
        var TestList = new List<int>() { 1, 3, 8, 9, 10 };
        TestList.RemoveAll(x => x <= 5);
        foreach (var Val in TestList)
            Console.WriteLine(Val.ToString());
    }
}

If you run ILDASM on this program, you get quite a lot of output. I'll spare you the full listing (but feel free to try it) and just show you the parts that matter. The biggest thing you'll notice when going through the listing is that we have a new method.

.method private hidebysig static bool '<Main>b__1'(int32 x)
  cil managed
{
  .custom instance void [mscorlib]System.Runtime.CompilerServices.
    CompilerGeneratedAttribute::.ctor() = ( 01 00 00 00 )
  // Code size       12 (0xc)
  .maxstack  2
  .locals init ([0] bool CS$1$0000)
  IL_0000:  ldarg.0
  IL_0001:  ldc.i4.5
  IL_0002:  cgt
  IL_0004:  ldc.i4.0
  IL_0005:  ceq
  IL_0007:  stloc.0
  IL_0008:  br.s       IL_000a

  IL_000a:  ldloc.0
  IL_000b:  ret
} // end of method Program::'<Main>b__1'

This is actually our anonymous method, but it's not so anonymous once compiled. It's name is "<Main>b__1" - a name that we could never write in C# since it contains angle brackets, and thus has no chance of conflicting with anything else in our program. What's notable is that this is a method of its own, with only the name to hint to us what method it was originally contained within.

There are other interesting things in IL that we could spend a while looking at, including the fact that it makes it a delegate for this method and caches it in a private field to aid performance, however we'll put that aside for now and look at the next issue.

=== The .Net CLR Doesn't Do Lexical Scope ===
In the .Net CLR, methods can't see each other's local variables (unless we call one and pass it as a "ref"). This is normally the case in C# too, until we start writing anonymous methods. Then we can write anonymous methods that refer to variables in the enclosing method. Let's change our program to exhibit this.

class Program
{
    static void Main(string[] args)
    {
        var TestList = new List<int>() { 1, 3, 8, 9, 10 };
        int Maximum = 5;
        TestList.RemoveAll(x => x <= Maximum);
        foreach (var Val in TestList)
            Console.WriteLine(Val.ToString());
    }
}

This produces the same results as the previous version of the program. Now let's look at the disassembly. The first surprise we get is to discover that our Program class now actually contains a new, nested class!

  .class auto ansi sealed nested private beforefieldinit
    '<>c__DisplayClass2'
         extends [mscorlib]System.Object
  {
    .custom instance void [mscorlib]System.Runtime.CompilerServices.
      CompilerGeneratedAttribute::.ctor() = ( 01 00 00 00 )
    .field public int32 Maximum
    .method public hidebysig specialname rtspecialname
            instance void  .ctor() cil managed
    {
      // Code size       7 (0x7)
      .maxstack  8
      IL_0000:  ldarg.0
      IL_0001:  call       instance void [mscorlib]System.Object::.ctor()
      IL_0006:  ret
    } // end of method '<>c__DisplayClass2'::.ctor

    .method public hidebysig instance bool
            '<Main>b__1'(int32 x) cil managed
    {
      // Code size       17 (0x11)
      .maxstack  2
      .locals init ([0] bool CS$1$0000)
      IL_0000:  ldarg.1
      IL_0001:  ldarg.0
      IL_0002:  ldfld      int32 Program/'<>c__DisplayClass2'::Maximum
      IL_0007:  cgt
      IL_0009:  ldc.i4.0
      IL_000a:  ceq
      IL_000c:  stloc.0
      IL_000d:  br.s       IL_000f

      IL_000f:  ldloc.0
      IL_0010:  ret
    } // end of method '<>c__DisplayClass2'::'<Main>b__1'

  } // end of class '<>c__DisplayClass2'

Because methods can not see each other's locals, the compiler has to resort to using fields to exchange information between them. So how is this class used inside our Main method?

First, it stores an instance of this class as a local variable of Main, instantiating it right at the start of the method. Yes, that means that you have an instance of the nested class created per call to the method! This is needed to get the correct semantics in multi-threaded code.

Whenever we reference the Maximum variable in our method, even outside of the lambda expression, it actually gets compiled down to a store field or load field instruction. This means that it's not, at the bytecode level, a local variable any more, which means it will be more expensive to access at runtime.

=== Code That Uses ref ===
So what does code that uses ref look like at an Intermediate Language level? Once again, we'll write a small test program:

class Program
{
    static void Main(string[] args)
    {
        var Test = 2;
        DoubleMe(ref Test);
        Console.WriteLine(Test);
    }

    static void DoubleMe(ref int Val)
    {
        Val *= 2;
    }
}

This program produces 4. Next up, we'll disassemble it:

.class private auto ansi beforefieldinit Program
       extends [mscorlib]System.Object
{
  .method private hidebysig static void  Main(string[] args) cil managed
  {
    .entrypoint
    // Code size       19 (0x13)
    .maxstack  1
    .locals init ([0] int32 Test)
    IL_0000:  nop
    IL_0001:  ldc.i4.2
    IL_0002:  stloc.0
    IL_0003:  ldloca.s   Test
    IL_0005:  call       void Program::DoubleMe(int32&)
    IL_000a:  nop
    IL_000b:  ldloc.0
    IL_000c:  call       void [mscorlib]System.Console::WriteLine(int32)
    IL_0011:  nop
    IL_0012:  ret
  } // end of method Program::Main

  .method private hidebysig static void  DoubleMe(int32& Val) cil managed
  {
    // Code size       8 (0x8)
    .maxstack  8
    IL_0000:  nop
    IL_0001:  ldarg.0
    IL_0002:  dup
    IL_0003:  ldind.i4
    IL_0004:  ldc.i4.2
    IL_0005:  mul
    IL_0006:  stind.i4
    IL_0007:  ret
  } // end of method Program::DoubleMe

  .method public hidebysig specialname rtspecialname
          instance void  .ctor() cil managed
  {
    // Code size       7 (0x7)
    .maxstack  8
    IL_0000:  ldarg.0
    IL_0001:  call       instance void [mscorlib]System.Object::.ctor()
    IL_0006:  ret
  } // end of method Program::.ctor

} // end of class Program

I have colored the interesting lines in red. The first line shows us using the lodloca instruction rather than ldloc, which loads the address of the local variable onto the stack rather than the value of the local variable. Then, inside DoubleMe, it uses ldind and stind to load and store the value indirectly, that is, using the address.

=== So could they have made it work? ===
If all the IL has made you feel woozy, it's about to get worse. Now we've seen all the previous examples, let's try hand-editing some IL to see if we can actually get ref parameters to work in lambdas, and if not working out exactly why we can't. The first step is to write a program that is as close to what we want as we can get C# to compile, since editing IL is a pain.

class Program
{
    static void Main(string[] args)
    {
        var TestList = new List<int>() { 1, 3, 8, 9, 10 };
        int MaxVal = 5;
        RemoveVal(TestList, MaxVal);
        foreach (var Val in TestList)
            Console.WriteLine(Val.ToString());
    }

    static void RemoveVal(List<int> TheList, int Maximum)
    {
        TheList.RemoveAll(x => x <= Maximum);
    }
}

Next we can used ILDASM to get the IL code. Now let's start doing some small changes. First, we'll change the signature of RemoveVal to take an int32&, and the call to it to load the address of the MaxVal local and pass that instead of passing the value itself. Then inside RemoveVal, where we previously just loaded the parameter and assigned it to the Maximum field in the generated nested class, we'll add an extra instruction to load the indirect field.

    IL_0007:  ldarg.1
    IL_00071: ldind.i4
    IL_0008:  stfld      int32 Program/'<>c__DisplayClass2'::Maximum

Using ILASM, we can re-assemble this to an executable file, and discover that the program still works. However, that's not really got us any further than a temporary local would have in C#. What we actually want is to store the indirect reference in a field. Can we have a field of type int32&, though? To find out, in the anonymous class let's add one:

    .field public int32& MaximumTest

If we try and assemble this, it produces an EXE file. It all looks good until we try and run it, at which point the CLR crashes, leaving the following message behind:

Unhandled Exception: System.Runtime.InteropServices.COMException (0x801312E4):
Field of ByRef type. (Exception from HRESULT: 0x801312E4)
   at Program.RemoveVal(List`1 TheList, Int32& Maximum)
   at Program.Main(String[] args)

Why, though? Well, the reason is quite simply that, in general, it is unsafe to allow fields of ByRef type. This is because a ByRef is essentially a pointer into a stack frame, which is where local variables are stored. If you store a pointer into it, and that pointer exists beyond the stack frame (that is, until after we return from the method that declared the local we passed a reference to), you are pointing into unknown and very likely invalid memory. That would compromise the type-safety of the VM. I'd run across this very issue before in a different situation, so it was interesting to see it coming up as the answer to the question we set out to answer.

=== Summary ===
So, to summarize all of this drawn-out analysis, the reason we can't use ref and out parameters in an anonymous method (and thus a Lambda expression or a Linq query) is as follows. Anonymous methods compile down to separate methods. If they are accessing parameters or local variables, to share those with the method implementing the lambda expression we are required to put them in a field. To ensure the type safety of the CLR, we can not store a ByRef value (what we get when we use ref and out) in a field.

I suspect that fixing this well would require adding some extra support to the .Net Common Language Runtime itself, which is relatively unlikely to happen; this happened between .Net 1.x and .Net 2.x and I suspect they won't be in a hurry to do that again (the underlying VM between .Net 2.x and .Net 3.x is actually the same, it's just some extra libraries). You could do it by making stack frames garbage-collectible so it's OK for them to live beyond an invocation (but that probably hurts performance a lot), or you could do it by adding support for lexical scopes and closures, so the compiler doesn't have to fake them using fields. The Parrot VM, which I work on, does the second of these.

Anyway, I hope this analysis has given you more of a sense of the Real Answer, rather than the compilers simple, "you can't".

Very Quick Guide To DLinq: Part 2

pheaven — Tue, 25 Mar 2008 02:45:59 -0700

In this part I'm going to take a look at inserts, updates and deletes using DLinq.

=== The Overall Model ===
First, to make changes to a database, you instantiate the data context object, just as we did before doing selections in the previous part.

var DB = new ResourcesDB();

You then do the required changes; we'll look at this part in just a moment. However, the inserts, updates and deletions are not performed immediately. They do not take place until you explicitly submit the changes:

DB.SubmitChanges();

This means that you can make many changes, then send them to the database all at once.

=== Inserting Data Into A Table ===
To make a single new entry to a database table, use the InsertOnSubmit method on the Table object. This takes a single parameter: an instance of the object that represents a row in the the table. You can use the object initializer syntax to write this very neatly.

DB.BookResources.InsertOnSubmit(new BookResource()
{
    Title = "How To Make Fondue",
    Author = "Mr S. Cheese"
});

Of course, you can instantiate and set up that object elsewhere. You may end up with many object to insert, in which case you can build a List<BookResource> (or anything else that implements the IEnumerable<T> interface) and then use the InsertAllOnSubmit method.

DB.BookResources.InsertAllOnSubmit(NewBooksThisWeek);

Remember that the inserts will not take place until you call the SubmitChanges method on the data context (you don't call it on the table itself).

=== Updating Data ===
To update data, first you need to write a Linq query to get the data that is to be updated. If you want to update just one item, you can use .First() on the query result to just get that item.

var ToEdit = (from Book in DB.BookResources
              where Book.BookID == 42
              select Book).First();
ToEdit.Author = "Mr Swiss Cheese";

If you have many items that you want to update, select all of them and then write a foreach loop.

var ToEdit = from Book in DB.BookResources
             where Book.Author == "Mr S Cheese"
             select Book;
foreach (var Book in ToEdit)
    Book.Author = "Mr Swiss Cheese";

And once again, don't forget to call SubmitChanges when you've done your changes, else they won't be saved back to the database.

=== Deleting Data ===
Deletions follow a similar approach to updates. However, rather than changing the object(s) you select from the database you pass them as a parameter to the table's DeleteOnSubmit for one item:

DB.BookResources.DeleteOnSubmit(
    (from Book in DB.BookResources
     where Book.Author == "Mr S Cheese"
     select Book).First());

Or DeleteAllOnSubmit for many items:

DB.BookResources.DeleteAllOnSubmit(
    from Book in DB.BookResources
    where Book.Author == "Mr S Cheese"
    select Book);

You may assign the selection to a temporary variable and then pass that to delete as you wish. Call SubmitChanges to actually delete these items in the database.

=== Next Time ===
Next time we will look at how to handle various special queries that are less obvious how to express using Linq, including the SQL LIKE and IN operators.

Very Quick Guide To DLinq: Part 1

pheaven — Fri, 14 Mar 2008 09:10:53 -0700

We have already written about using Linq, but didn't cover DLinq. This post takes a quick look at getting started with DLinq, which allows you to write Linq queries against databases. Once you're up to speed, you can take a database and be writing Linq queries against it in under five minutes!

In future parts in this series of posts, we'll look at doing database updates, writing more complex queries and debugging. In this one, we'll just look at doing some simple queries.

=== Creating The Linq to SQL Classes ===
The bad news is that before you can use DLinq, you have to create Linq to SQL classes to represent your database tables. The good news is that Visual Studio 2008 (aside from the Express Edition) can do all of the hard work for you in this.

To get started, right click in the Solution Explorer in your project (or a directory in your project), go to the Add sub-menu and choose "New Item...". In the resulting dialog, choose the "Data" category to the left, then "LINQ to SQL Classes" in the resulting list on the right. Enter a name for the group of classes (collectively known as a Data Context), and press OK.

[includeimage:1]

Next, you need to connect to the database holding the tables that you want to do queries on. To do this, open the Server Explorer (if you don't see it, it's on the View Menu). Right-click on "Data Connections", then choose "Add Connection..." Fill out the fields in the resulting dialog, and click OK. You will then have your database in the list.

[includeimage:2]

Open up the Tables subtree beneath the connection you just created. Select the tables that you want to be able to do queries on (select multiple by holding down Control or Shift). Then drag them to the left pane of the data context. Visual Studio will then create the classes for these tables, and you'll see the tables represented in the data context.

[includeimage:3]

Finally, click in the data context somewhere other than on a table. Go to the properties window and set the namespace and the class name for the class that represents the data context as a whole.

[includeimage:4]

Finally, save this.

=== Writing Queries ===
For the sake of this article, I've just created DLinq classes for a couple of bits of the Programmer's Heaven resources database, where we store details of files, links, articles, books and so forth. I'm using a Console Application to play with this.

First, we need to instantiate the data context.

var DB = new ResourcesDB();

The members of DB represent, amongst other things, the tables in the database. We can use these to write queries on those tables.

var Result = from Book in DB.BookResources
             where Book.Published == "1997"
             select Book;

Remember that we use C# syntax rather than SQL syntax; it's all to easy to forget this and write SQL "=" rather than C# "==". Thankfully, if you make this mistake, the compiler will complain. Of course, you don't have to just select the Book; you can instantiate and populate any class that you wish to instead.

=== Getting At The Query Results ===
In the previous query we selected all books in our database published in 1997. We can now iterate over the Result collection and print the title of the books.

foreach (var Found in Result)
    Console.WriteLine(Found.Title);

Which produces a list of titles of the books that were found.

=== The Next Steps ===
If you have already read our Linq tutorial, you'll have seen how you can write some pretty complex queries over collections of objects. All of that same power is available in DLinq too, so if you want to learn how to write more complex queries then you should refer to that. In future parts of this series, I will look at other DLinq specific bits.

Float, Double and Decimal: do you know the differences?

pheaven — Tue, 11 Mar 2008 03:17:36 -0700

There are a number of data types available in the .Net framework for storing numbers with fractional parts. They are each appropriate for different situations, and using the wrong one can lead to errors in calculations.

=== Thinking About Accuracy ===
In some applications, you require that calculations involving numbers with fractional parts are exact. Examples of this are in financial applications, where losing the odd cent here and there in a calculation is unacceptable. Customers expect their accounts to be completely accurate, not to mention the tax man.

In other applications, you don't care about exact results, but you are interested in having an answer that is correct to a certain number of significant digits. This is the case in experimental physics, for example. When you make a measurement of something, you can only measure it as accurately as your equipment will allow you. Therefore, you have an inherent error in that value already, which is going to cascade through any calculations you do with it. This means that while your computed answer may be 5.1826, you may know that the values this answer was calculated from were only accurate to three significant digits, and so the last two digits in this result (the 0.0026) don't matter. If there is a computational error in them, so be it - it's not going to hurt us.

=== Number Representations ===
There is more than one way to represent a number with fractional parts. Decimal represents the number as an integer, then also stores an integer power of ten (the exponent) that states where the decimal point is. The .Net decimal type allows you to shift it between 0 and 28 places to the left, so the most fractional places you can have is 28. Since both the value and the position of the point are represented as integer types, and because (with enough bits) we can represent any integer value exactly in binary, we can store the number exactly.

Single and Double (also known as float and double, depending on your language) work differently. Glossing over a few details (but if you want them, look up the IEEE 754 standard), they are stored with both a mantissa (which specifies the digits making up the number) and an exponent, which specifies where the binary point goes. That is, we're working in base two for everything. Further, the mantissa is normalized, so your number is always stored as something of the form 1.10010101 (with only a single bit to the left of the point) followed by binary columns that represent a half, a quarter, and eighth and so forth.

The important thing to realize here is that many decimal numbers are not exactly representable in binary. Consider decimal 0.3, for example. If we try to represent it as the sum of powers of two, we end up with a sequence like 1/4 + 1/32 + 1/64 + 1/512 + ... - in fact, we never actually get to an exact value. We'd need infinitely many bits. This is not unique to binary; in decimal, for example, we can not specify 1/3 exactly. However, it does mean that we have the potential to lose data.

=== Range ===
The decimal data type uses 96 bits to store the number itself. There is a special bit for storing the sign, meaning that you get a range of -79,228,162,514,264,337,593,543,950,335 to 79,228,162,514,264,337,593,543,950,335, if you have no fractional parts. This is 29 digits, meaning that you can move decimal point all the way to just after the initial digit. This is, with the maximum exponent you get a range of -7.9228162514264337593543950335 to 7.9228162514264337593543950335.

With floating point types (float and double), you store the mantissa in a normalized form, meaning you can shift the binary point in either direction to get really small or really large numbers. However, remember that unlike the decimal data type, you are not able to represent every value in the range. You get it accurate to a certain number of significant bits.

With a float, you can store numbers from -3.402823e38 to 3.402823e38, where e38 means "10 raised to the power of 38" - a bit of a wider range than with decimal. However, also interesting is the smallest positive or smallest negative number you can represent, which is around 1.4e-45 - really very tiny.

With a double, the range is -1.79769313486232e308 to 1.79769313486232e308, and the smallest is around 5e-324! While the range is greater, the main advantage of the double type isn't so much the extra range of values - due to a small increase in exponent size - but a big increase in precision by having a much larger mantissa. That is, you get a lot more significant digits.

=== Size In Memory ===
A float (Single) takes 32 bits of memory, a double takes 64 bits of memory and a decimal takes 128 bits of memory. Note that the reason a decimal is so much larger is because it can store a huge number of significant digits, giving a great deal of accuracy/precision. Note that a 32-bit float can only represent as many values as a 32-bit integer; you are trading in precision to get an increased range. It's worth noting that about half of the values you can represent with a floating point number are between -1.0 and 1.0.

=== Performance ===
In any modern PC, your CPU will have a dedicated Floating Point Unit, which is a chunk of hardware that performs operations on floating point numbers (floats and doubles). However, in embedded environments you may not have an FPU available. If you do have an FPU, operations on floats and doubles will be fast (and the non-floating point execution units can also be used to do other integer computation in parallel, thanks to in-hardware optimization). If you don't and it is being emulated in software, it will be much slower.

The exact performance differences in operations on floats and doubles is highly platform and application specific. We can certainly say that floats require double the memory, and thus twice as much CPU cache space, but how much that actually impacts performance is specific to how your application uses memory. Due to the fact that cache lines store multiple words anyway, there's not an obvious answer.

The decimal type does integer operations. Normally that would be faster than floating point ones, apart from there isn't dedicated hardware to deal with the 96-bit values, not to mention handling differences in the exponent, so it can wind up being much slower. If you didn't have an FPU, fixed point would likely beat floating point. It's the hardware support that makes the big difference here.

=== Conclusions ===
Always remember that floating point numbers can't represent every value in the range they cover, while decimal numbers can (though the range is somewhat smaller).

The inability of floating point numbers to represent every possible value in the range they cover can have important consequences in application design. For example, adding a bunch of numbers smallest first to largest first can end up with a different result to if you had done the largest first and finished up adding on the smallest one. The reasons behind that are complex and for another post, but it's good to be aware that there are a lot of subtleties to worry about when working with floating point. The other big thing to avoid in most circumstances is comparing floating point values exactly, rather than checking that the difference between them doesn't lie in an acceptably small range.

As a general rule, if you're dealing with currency and/or need exact results, use decimal. If you are writing scientific applications, you likely want to be using a floating point type (either single or double).

Can a language make programming more fun?

pheaven — Mon, 03 Mar 2008 07:38:14 -0700

I've programmed in a range of languages. There are some that I really enjoy coding in, there are some I can put up with and there are some that frustrate and bore me. Having an interest in language design, I got somewhat curious about what it is that makes working with some languages more enjoyable than others.

A couple of questions came to me right away. First, there is probably a lot of personal preference here. What makes a language enjoyable for me to work in may well be what makes it frustrating to someone else, and vice versa. Second, languages are tools. I've said often enough that different languages are suited to different tasks, and you should pick the one that best fits the job at hand. How important a factor is whether one enjoys working in the language when trying to choose whether to use it in a project or not?

=== Personal Preference ===
I find the more limited range of language features in Java frustrating. I enjoy working in Perl and C# because there are a wider range of language features available to me. I enjoy being able to mix and match paradigms to the task at hand, and like the thought process associated with going through the options and picking the best one (or working out why other people chose a particular one when reading other people's code).

However, someone else may find Perl and, probably to a lesser degree, C# frustrating because the languages have many constructs that you have to learn and be aware of the interactions between to understand the code. They may therefore enjoy working in Java more because it takes less time to work out what is going on with a chunk of unfamiliar code, and you can dig into the changes or fixes that need doing more quickly.

While I know what my preference is on the rich vs. simple language continuum, I can understand why someone may have an opposite preference to me. I'm not sure that one is more correct than the other. I just know what I like.

There are many other things that people like or dislike about languages. Some of them include:
* Strong vs weak typing: those who prefer strong typing probably like it that the compilers makes them insert clear coercions because it catches some more mistakes, whereas those who prefer weak typing may be frustrated that the compiler can't just be smart enough to insert the coercion that they most likely meant for them in most cases
* More wordy vs more symbolic: some people prefer code to read like English, in which case they prefer more "wordy" languages like BASIC; others prefer to use more non-alphanumeric characters for syntax so it looks different to names of things
* Single paradigm vs multi-paradigm: some people feel it is better to have on paradigm almost enforced upon them, such as object orientation. Others find that this limits their creativity and want a much wider choice of language constructs.
There are many, many more; I'm sure people will point out others in the comments to this post.

=== Does it matter? ===
I think that, in general, people are more productive when they are enjoying what they are doing. I know that when I'm really enjoying a coding job, I'll tear through it, sometimes churning out hundreds of lines of code per day (working and with tests). Equally, when I'm bored with what I'm working on, I find myself glancing at the latest posts on Slashdot, checking if there's anyone I can bug on MSN or going to brew up yet more coffee. If you've got a language that you enjoy working in, you're potentially going to be more productive.

However, there's a flip side. If you put too much emphasis on what you enjoy working with, you may end up choosing a language that's not well suited to the task at hand. For example, no matter how much I enjoy working with Perl, it's just not the best thing to work with for building a Windows GUI application. If I instead work with one of the .Net languages or another language that has a really good tool chain associated with it for building Windows GUI applications, I'll will likely get the job done much more quickly.

=== Conclusions ===
I think it's safe to say that language preference is a secondary issue to choosing the correct tool for the job. First, identify those languages that are suitable for the task. That may be in part related to the language features, availability of compilers or interpreters for platforms you need to run on, what libraries and tools are on offer and corporate or business constraints. Then, if there's still more than one choice available to you, consider personal preference.

Google Trends of Programming Languages

pheaven — Tue, 26 Feb 2008 15:23:33 -0700

Google Trends allows you to compare the popularity of various search terms. A friend pointed me at this, having put Java into it and noted a "dropping trend". I played around with it for a while, sticking in various language names, and came away with more questions than answers.

=== How to write stupid conclusions ===
The obvious but stupid way to write a blog post based upon Google trends would be to look at the graph for Java, look at the graph for Lua (a recent scripting language suited to embedding) and spurt out things like, "OMGWTFBBQ LOOK! Java is getting less popular and Lua is getting more popular!". Let's look more carefully at this, though.

The giveaway that we have a bogus conclusion here can be found by taking a look at the list of related news stories to the right. For Java, most of them - all but one at the time of writing - are not about the Java programming language, but instead about an unfortunately disaster-prone Indonesian island. Looking at the ones for Lua are even more telling - none of them are about the programming language, but instead about a football player. Also, spot the correlation of the main graph with the rise in news stories. This makes our analysis of the Lua language becoming hugely more popular based upon this graph bogus. It may be gaining popularity, but we aren't going to learn that from Google Trends.

Finally, plotting them on the same graph shows that the scales of the two previous graphs were completely different, and that the increase in searches for Lua, even if it they were to be about the programming language, is barely visible once you put the two on the same scale.

=== Why would people search by language name anyway? ===
I think the two main reasons people would search for a programming language are:
* They are considering using it or learning it and are looking for resources to help with this
* They are already using it and want help solving a problem or want to learn more
The first of these is indicative of growing interest in a language, and an increasing number of searches may point at that. The second may not be such a good indicator of usage, however. Once you have a number of experienced users, and if the language ships with good documentation, it is possible to have high usage with a relatively low number of people going to search results to find resources. There may well be good resource sites that a lot of developers go to rather than Googling it.

Therefore, it's not completely unreasonable to suggest that a declining number of searches may indicate that a language is retaining its current userbase, but attracting less new users. Therefore its userbase may be remaining relatively constant. Of course, this may not be the case too.

You can make arguments like this all day, but the point I'm trying to make is that you can't necessarily map number of searches onto size of user base and expect to come to correct conclusions.

=== Disappearing Programmers? ===
Let's assume that searches tell us something about take-up of a language, even if not overall usage. I did a plot of the big four dynamic languages (Perl, PHP, Python and Ruby). This showed the Perl and PHP were both being searched for less. However, despite the fanfare about Ruby, with the Rails framework, there is no sign of a proportional increase in searches for Ruby. So where are those that were using Perl and PHP going, if they are declining? Not Java, it would appear, since that would also appear to be getting less searches. C# or .Net? Nope, one is steady and the other shows a slight drop.

In fact, I'm finding it near impossible to find any programming language on Google Trends with a rising trend. Clearly, programmers must be a dying breed, or they are doing less searches, anyway. So what does that mean? If we don't want to believe that less programming is happening overall and just that searches are happening less, then it would suggest that those with declining trends are not declining so steeply, and that those with steady trends are actually rising in usage. There still seems to be something of a gap between the falling and rising. Then, I Am Not A Statistician.

=== Conclusions ===
So after an hour of playing with Google Trends, I'm left feeling like the most interesting thing I've learned is that you're more likely to be taking the weekend off if you're a C# programmer rather than a C++ one. So there's one reason for me to be happy I'm using C#, anyway. But in all seriousness, I'm very much left with the feeling that the Google Trends data is probably not especially useful for trying to get an idea of relative programming language usage. Now, is it the weekend yet?

Do try...catch blocks hurt runtime performance?

pheaven — Tue, 19 Feb 2008 06:23:04 -0700

Recently, a friend mentioned that someone had told him that writing a try...catch block in C# (or substitute some other .Net language here) resulted in a "huge penalty" in terms of performance compared to if you had not written it. That is, merely writing such a block actually hurt program performance, even if an exception was never thrown. He didn't believe this was true, and rightly so - it's completely wrong. This post is a tidied up version of my explanation.

=== Inside a .Net Assembly ===
A .Net assembly, if we ignore various headers, consists of three things:
* Bytecode: a sequence of low-level instructions that specify the body of a method
* Metadata: a set of tables, a little bit like a database, describing higher level constructs such as classes, methods, signatures and so forth
* Heaps: places where string constants and other such things are stored
That is, bytecode is what we actually execute and the rest is there to describe the extra details. To give you an example of the interplay between them, consider a method call. The bytecode stream contains the call instruction, followed by a method token, which is actually an index into the methods table. The methods table in turn allows us to find out where the bytecode for the method we're calling starts, so we can execute it.

(As an aside, these lookups wouldn't be all that expensive to do per call, though in reality they likely won't actually happen more than once per call site, unless it's done through reflection or a delegate. That's because when the CLR JIT-compiles the bytecode, it can compile the method call down to call instructions at the CPU level that refer directly to the method being called. You don't need to know this bit; it's just here for the curious.)

=== How Exception Handlers Are Stored ===
The .Net CLR has a concept known as a "protected region". A protected region is a sequence of instructions that has an associated handler. In C# or VB.NET, the instructions in a protected region correspond to code in a try block. There are various types of handlers, including typed ones (that catch only a certain type of exception) and finally ones. Note that a try...catch...finally will result in two protected regions at the CLR level, one for the catch and one for the finally. The protected regions will cover the exact same sequence of instructions, just have different handler addresses.

Each method that has protected regions comes with a table of them. For each protected region, it contains four entries:
* The starting instruction in the bytecode for this protected region
* The number of bytes worth of instructions from the starting point that are protected
* The type of handler
* The location in the bytecode of the handler
They are ordered with innermost handlers coming first.

=== What happens at runtime ===
Here is the important part when it comes to performance when an exception is not thrown. Since the protected regions are stored in a table and are not in the bytecode, and because the CLR does not need to worry about the exception handlers unless an exception is thrown, then there is no runtime penalty for having a try...catch block. For finally handlers it is a little different, because we do need to run those even when we don't have exceptions. However, since we can compute what we will need to run when statically, the JIT compiler can still emit code that jumps to the finally handler at the appropriate time, making the execution overhead of one of those most likely just a jump and a return.

So that's most of the answer to the question that was asked, but for interest let's take a look at what happens when an exception is thrown. Provided the current method has protected regions, you scan through the table to find if there are any that cover the instruction where the exception was thrown and that are capable of handling it (they are looking for an exception of the correct type, for example). If there are multiple nested handlers that could handle the exception, the ordering of innermost first means that we will find the correct handler.

If we find a handler, we execute it. If we don't find one, we move down to the next method in the call stack and check if it has a suitable handler, keeping going until we find one (or we discover that the exception is user-unhandled and terminate the program). This means that the cost of throwing and catching an exception is dependent on how far down the call stack you have to go to find a handler.

From this we can conclude that .Net is optimized for the case where you do not get exceptions. Since exceptions are intended to happen only in exceptional (that is, unexpected) circumstances, this is a sensible design decision. If you inclined to use exceptions for ordinary flow control, this should give you another reason not to.

=== And the cost of a try...catch block is... ===
So what can we conclude? The overall cost of a try...catch block that never handles an exception is a few bytes of memory - or at worst a few words - for the entry in the protected regions table. The only possible runtime penalty is the extra time to load those extra few bytes into memory. Since they are stored way away from the JITted bytecode stream, it's highly unlikely you're going to incur any additional cache-misses at runtime as a result of the handler too. Thus, the cost is essentially nothing.

The cost of not handling an exception that you should have may well be that your program crashes. This results in unhappy customers, a hit to your reputation and development time to go and do a bug-fix, which will almost certainly be much greater than if you had put it in there in the first place. Obviously, protecting code that can not throw an exception under any circumstances is a waste of your development time. But otherwise, it's best to be safe rather than sorry, safe in the knowledge that even if an exception never does occur in that bit of code, it's not really costing anything anyway.

C#.NET or VB.NET?

pheaven — Mon, 11 Feb 2008 09:10:09 -0700

We get a lot of emails from Programmer's Heaven users. Amongst the requests that we do people's homework, which we ignore, we also get some more interesting questions sent to us too. We figured that when we do, we may as well share our answers more widely, and also this gives other folks a chance to give their input too. Recently we got one from someone who had discovered us through the C# School e-book and asked, "please kindly advise which of the two, C# and VB .NET is better for developing commercial applications like Payroll".

=== The Similarities ===
Both languages run in the Common Language Runtime and allow you to have full access to the .Net Class Library. This means that pretty much whatever is in the .Net Class Library is equally available to both VB.NET and C#.NET. Both languages have a similar feature set too, providing strong support for Object Oriented Programming, which is good for developing database-backed applications, as your Payroll surely will be.

If you take the list of additions to C# 3.0 and the list of additions to VB 9.0, both released at the same time along with Visual Studio 2008, they are not too far from identical. Both added support for Linq, both added type inference, both added anonymous types and so on. In fact, there's only one thing I see in the VB 9.0 list that wasn't in the C# 3.0 list: they've supported XLinq (queries over XML) with some special syntax in addition to normal queries. If you're not working with XML, you may find that of limited value, though.

=== Spot The Difference ===
The single biggest difference is syntax. Basically, VB.NET uses syntax derived from the BASIC language family, whereas C# uses syntax derived from the C language family. What you prefer will be just that: a preference. I know that I find VB.NET a bit harder to read, basically because it's more verbose and it's that little bit harder to pick out names of variables, methods and types (because more of the syntax is expressed in words, whereas C# uses non-alphanumerics). However, other people may find that more real words being used in the syntax makes it more readable.

There is the odd feature difference. For example, C# has no equivalent to the With blocks in VB, which let you topicalize. I wish it did have one, so that's a minor point I have to award to VB. More importantly, C# allows you to write unsafe code (that is, code that does direct memory operations and pointer arithmetic), while VB provides no way to do this. However, for a payroll application you just won't need to do this. Heck, I've been writing C# for a couple of years and haven't ever needed to use this feature. There are, of course, use cases, but they aren't ones you're going to run across while implementing a payroll application.

Then there are just different ways to do things. C# chose to let you use the "this" keyword on a parameter to denote an extension method, whereas VB gets you to mark it with an attribute. But at the end of the day, that's not much more than a syntactic difference either.

=== Does it matter? ===
From a technology point of view, I really don't think that it matters which language you choose. I know which I prefer and I can give reasons why I prefer it, but it's little more than that: a preference. You're going to be able to write the same application in VB.NET or C#.NET. The differences are really not much more than syntax deep, so much so that there are translators out there that will take your code in one and turn it into the other, retaining comments. The other thing to realize is that you've only got one set of concepts to learn to know both languages, and two lots of syntax. And syntax is easier to learn than semantics.

Other than your own personal preference, you might want to consider the backgrounds of anyone else who will be working on the project, or may maintain or contribute to it in the future. If they have programmed before, their background may suggest which language they'll be most comfortable with, and if it doesn't matter much to you then then you may want to base your decision on that.

=== Conclusion ===
The differences between the languages are not much more than skin deep. They have a near identical feature list, and the features have near identical semantics - basically, those provided by the underlying .Net Common Languages Runtime. Both are suitable for your project, and I doubt the outcome will be greatly affected by which one you choose. Pick one and go for it. If you need to, you'll be able to learn the other one easily later anyway.

What is REST, anyway?

pheaven — Wed, 06 Feb 2008 11:09:05 -0700

REST is short for Representational State Transfer. I'd place that into the category of names that makes more sense once you understand what REST is, but isn't quite so helpful in terms of explaining it. REST is often heard in the context of web services. When it was suggested as a topic for me to write about it was described to me as a web service but doing "the query string thing" to pass it parameters. That's not really what it's about, though.

=== Verbs and nouns ===
Verbs are doing words. Examples in natural language are "pay", "drink" and "shock". In most programming paradigms, we spend a lot of time defining how to do things, by writing methods, subroutines or functions. These are our programming equivalent of verbs.

Nouns represent entities. Examples in natural language are "money", "beer" and "mother". Whenever we have data in our program and we somehow identify it (for example, by name binding or variable declaration), these are just like nouns in natural language.

So what does this have to do with REST? With REST, you have full control over how you define the nouns, but the verbs (that is, the operations available to perform on nouns) are constrained by the system and not extensible. You perform an operation by locating an entity by knowing its "name", then do one of the limited number of operations available on it. The focus is on the nouns or, you could say, on the data.

This is a contrast to a RPC (Remote Procedure Call) model, which is commonly implemented using SOAP or the simpler XML-RPC. There, you expose a range of methods or procedures that can be called and supply them with data to operate on. The focus here is on verbs - the operation that is being performed.

=== Representations ===
The "RE" part of REST is an abbreviation of the word "representational". In REST, we deal with representations of entities. The way we choose to represent an entity - called the content type - is specific to the type of entity involved. However, for a given system these content types are generally constrained to relatively few widely readable formats (for example, RSS, SVG, HTML and so forth). While applications of XML or SGML are the most common representations, you can use others.

=== URIs ===
The nouns in REST are actually URIs (Uniform Resource Identifiers). URIs encompass URNs (Uniform Resource Names) and the more familiar URLs (Uniform Resource Locators). A URN is a pure name; it gives us no information on how to locate the entity in question. An example would be a book's ISBN number; it lets you uniquely identify the book, but alone it doesn't enable you to find a copy of it. A URL, on the other hand, describes how to locate an entity. We most often see them on the web, when they tell us all we need to know in order to locate a file (the protocol to use, the name or IP address of the server its on and the path to it).

One resource can refer to another using its URI. For example, a category resource could return an XML document containing a list of URIs (which may also be URLs) representing the products in that category. Similarly, an RSS document contains URLs referencing the original documents that are represented in the feed.

=== State ===
Another key concept of REST is that it is stateless. That is, you make a request by specifying a noun (in the form of a URI) and a verb. The operation happens, you get a result back, and then the exchange between the client and the server is over. The server doesn't retain any specific information about the client, and the client doesn't retain any specific information about the server. The only information exchanged is that described by the noun.

=== REST and the Web ===
The world wide web can, to some degree, be seen as a RESTful system (that is, a system implemented on the principles of REST). We have resources (that is, data) identified by nouns - URLs. We are free to define the structure of these resources by building up directory structures. Then we have the HTTP protocol, which defines a limited set of verbs.

The verbs in HTTP are more often known as the method. If you have ever written a form tag, you will have seen them.

<form action="something.pl" method="POST">

There are a limited number of verbs we can use, though more than we commonly see. You'll probably only have written GET or POST as the method, and may be aware that there is also a HEAD method. However, HTTP does specify several more, including PUT and DELETE.

Unfortunately, most systems don't support anything other than GET, HEAD and POST. Instead, we tend to use POST to "tunnel" other operations. That is, to create or delete something, we send a POST request specifying that we wish to do this. Trouble is, the temptation is then there to implement all kinds of operations, not just a constrained set like the REST model calls for.

Generally, however, many web operations are RESTful. Generally, a GET operation will be, unless you pass parameters in the query string that request a given operation be performed other than just requesting the data in a particular way, in which case you're doing something more like RPC.

The web doesn't always live up to the stateless qualities of REST, however. Cookies are a form of state maintained by the client, and generally the cookie will be mapped to some session data on the server. Thus, if you are implementing a true REST model, you shouldn't be using HTTP cookies to maintain sessions. Authentication and authorization is still possible - you just have to pass credentials with each request rather than a session identifier. In fact, that's what HTTP basic authentication does.

=== Advantages and Disadvantages ===
So far it feels like REST places a lot of constraints on us. Is there anything to be gained? The answer is yes, and for some systems they are desirable properties.
* Since you have no state, you don't need to tie a particular server to a particular client. Thus any server can handle a client's request, which allows you to scale more easily.
* Since the set of verbs is constrained, there are far fewer compatibility issues arising from the server or client not supporting a given operation than if you took an RPC approach.
* Since the representations used in RESTful systems are usually things like XML, it's easy to add support for new fields without forcing all systems to know about them. For example, even the oldest browsers with no CSS support can still render a modern HTML document in a readable way.
* By using URIs, and in turn URLs, we can take advantage of existing infrastructure to do resource location.
* If we use HTTP, then we already have a limited but widespread client that can access the data too (web browsers), as well as caching infrastructure to aid performance.
However, REST is not suitable for every circumstance. In fact, it doesn't intend to be. Sometimes your set of operations on different entities won't fall at all neatly into a create/read/update/delete model, or you will need to maintain state between requests. You may have full control over the clients and servers and explicitly want to know when they aren't both aware of the same set of operations and the parameters required by those operations. In these cases, RPC is a likely to be a better bet.

REST is good when you have got widely distributed data and no centralized control over clients, servers and representations. This is very much the case with things like RSS feeds, where many people are publishing them and there are many different readers. If every reader had to support every single field in documents provided by servers, we'd be unable to evolve RSS without causing widespread breakage and forcing everyone to upgrade their readers.

=== Conclusion ===
REST provides us with an approach to client-server interaction. There are relatively few widely used examples of it to observe: the semi-RESTful web aside, two of most common are RSS (mentioned in this article) and WebDAV (not mentioned, but basically an extension of HTTP's methods to be expressive enough to provide services such as version control over). It's certainly not appropriate for all situations, but the constraints it lays down are very much worth keeping in mind when designing distributed architectures with little centralized control.

=== Further Reading ===
If you want to know a lot more about REST and the thinking behind it, probably the best place to turn is the PhD thesis that coined the term.
http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm

Divvy up my list? Aye!

pheaven — Thu, 31 Jan 2008 04:28:52 -0700

I wasn't born in Yorkshire, but I spent most of my childhood growing up there. Yorkshire, a large county in the north east of England, has its own collection of slang, not least including the word "aye", which it appears is appropriate in response to pretty much any situation. Another slang phrase is "divvy up". Basically, this just means to divide a bunch of things up somehow.

These days, I spend a lot of my time writing C#. If I'm dealing with a bunch of things in C#, I'll usually store them in a generic List collection, or some other collection. This got me thinking: how can I divvy up a List in C# into multiple Lists? And that's when I decided to implement the Divvy extension method.

=== What we're aiming for ===
Suppose we have a list of scores:

var Scores = new List<int>()
    { 87, 32, 45, 60, 91, 10, 58, 77, 66, 71 };

We want to categorize them by grade. For example, anything over and including 85 is a grade A, anything from 70 to 84 is a grade B and so on. This is where our Divvy extension method is going to come in useful. We'll look at what the following code means and how to implement the Divvy method in a moment, but using it will look like this:

// Split them up by grade.
var GradeScores = Scores.Divvy(new Dictionary<string, Predicate<int>>()
{
    { "Grade A", x => x >= 85 },
    { "Grade B", x => x >= 70 && x < 85 },
    { "Grade C", x => x >= 50 && x < 70 },
    { "Fail",    x => x < 50 }
});

It will give us back a Dictionary where the keys are the names of the categories we supplied when calling Divvy and the values are lists of items in that category. Therefore, we can loop over like this:

// Display results.
foreach (var Grade in GradeScores.Keys)
{
    Console.Write(Grade + ": ");
    foreach (var Score in GradeScores[Grade])
        Console.Write(Score.ToString() + " ");
    Console.WriteLine();
}

Which will give us the following output:

Grade A: 87 91
Grade B: 77 71
Grade C: 60 58 66
Fail: 32 45 10

Neat, huh? Let's take a look at how on earth we make this work.

=== Extension Methods ===
The first problem is that we want to have a method on the List class. However, that class is in the .Net class library, so we can't go changing it. We also don't want to have to start using a subclass of List all over the place just to get the extra method.

C# 3.0 allows you to write extension methods (detailed tutorial here). These are static methods that can be called as if they were instance methods. This way, we can write a method that can be called on instances of the List class. So our starting point is:

namespace Jnthn.ListExtensions
{
    public static class DivvyUp
    {
        public static void Divvy<T>(this List<T> TheList)
        {
            
        }
    }
}

Note the use of the "this" keyword to mark this as an extension method. Also note that List is a generic type - it takes a type parameter T. However, we don't know what this type is, and we want to make Divvy generic too. Thus we declare it as Divvy<T>.

=== The Return Type ===
We are going to split the List into many other Lists, based upon the category the item gets placed into. The categories are named by strings, e.g. "Grade A" and "Grade B". Therefore, for our return type we can use:

Dictionary<string, List<T>>

That is, a Dictionary where the keys are strings and the values are List<T>s, where T is the type of the original List that we were called on. Therefore, our method is now:

public static Dictionary<string, List<T>> Divvy<T>(
    this List<T> TheList)
{
    // Implementation goes here.
}

=== The Categorizers Parameter ===
Now we need a way to specify how to divvy up the list into the different categories. This is a mapping (from category name to something that determines whether an item is in that category), so we can use a Dictionary. The keys will be category names, which are of type string:

Dictionary<string,?>

However, what should the type of the values be? Well, they will be functions that take an item of type T and return a bool (true or false) depending on whether the item is in that category or not. There is a delegate in the .Net class library named Predicate<T> that specifies this type of function. Therefore, our parameter's type will be:

Dictionary<string, Predicate<T>>

Meaning that our full method declaration is:

public static Dictionary<string, List<T>> Divvy<T>(
    this List<T> TheList,
    Dictionary<string, Predicate<T>> Categorizers)
{
    // Implementation goes here.
}

You'll be glad to know that the implementation is simpler than the signature.

=== Initializing The Result Dictionary ===
Before we start categorizing the items, we need to set up the result Dictionary. We'll instantiate a Dictionary with the same type as the return type we worked out earlier. Then we will loop over the keys of the Categorizers parameter, which are the names of the categories, and create mappings in the Result dictionary from the names to empty Lists.

var Result = new Dictionary<string, List<T>>();
foreach (var Category in Categorizers.Keys)
    Result.Add(Category, new List<T>());

=== Divvying Up ===
Finally, we get to the code that's going to do the real work, and it's actually really quite straightforward.

foreach (var Item in TheList)
    foreach (var Category in Categorizers.Keys)
        if (Categorizers[Category](Item))
        {
            Result[Category].Add(Item);
            break; // Since we've divvy'd it into a list.
        }

The outer loop iterates over each item in the List that we were invoked on. The inner loop iterates over the names of the categories. The only slightly tricky part is the condition. Remember that the keys of the Dictionary are actually Predicate<T>s - functions that will return a bool when passed a value of type T. So we just call that function, passing the current Item as a parameter. If it returns true, then we know the item is in the current category. We add it to the correct result list, and then break. This takes us out of the inner loop, and we continue onto the next item in the List.

=== Putting it all together ===
Here's the Divvy method as a whole.

public static Dictionary<string, List<T>> Divvy<T>(
    this List<T> TheList,
    Dictionary<string, Predicate<T>> Categorizers)
{
    // Initialize result dictionary of lists.
    var Result = new Dictionary<string, List<T>>();
    foreach (var Category in Categorizers.Keys)
        Result.Add(Category, new List<T>());

    // Go through list items and divvy 'em up.
    foreach (var Item in TheList)
        foreach (var Category in Categorizers.Keys)
            if (Categorizers[Category](Item))
            {
                Result[Category].Add(Item);
                break; // Since we've divvy'd it into a list.
            }

    // And that's it.
    return Result;
}

=== Looking Back At The Call To Divvy ===
Let's take another quick look at the call to Divvy to better understand what is going on.

var GradeScores = Scores.Divvy(new Dictionary<string, Predicate<int>>()
{
    { "Grade A", x => x >= 85 },
    { "Grade B", x => x >= 70 && x < 85 },
    { "Grade C", x => x >= 50 && x < 70 },
    { "Fail",    x => x < 50 }
});

Here we are using two features of C# 3.0: collection initializers (tutorial here) and lambda expressions (tutorial here). First we instantiate a new Dictionary, with keys of type string and parameters of type Predicate<T> - exactly what we declared the Categorizers parameter of Divvy to be. Then we use a collection initializers to specify a list of key/value pairs. Each goes in curly brackets, separating the key from the value by a comma. Therefore, "Grade A", "Grade B" and so on are keys.

The values are lambda expressions. If you read where "=>" as "where", you can read the first one as, "x where x is greater than or equal to 85". Note that we are declaring x here - it is just a parameter name. Remember that a lambda expression declares an anonymous method (or function), and what comes to the left of the => is a parameter list. "x =>" declares the x, and "x >= 85" uses it.

=== Aye! ===
And that's your lot. Enjoy divvying up your lists, and I'm off to tuck into a nice Yorkshire pudding.

Higher Order Programming Is Easy!

pheaven — Tue, 22 Jan 2008 07:09:07 -0700

Higher Order Programming is one of those things that to many people sounds weird, magical, mysterious or just too hard for them to be able to do. It's not. If you have ever passed a function or method as a parameter to another function or method, then you have done higher order programming. If you have ever used a function pointer in C or a delegate in C# or some kind of callback mechanism, then you have done higher order programming.

=== Define it! ===
We are all used to writing code that takes parameters. We do it whenever we write subroutines, functions or methods. Normally these parameters are data. In higher order programming, one or more of these parameters is code rather than data. That is, you can pass code, functions and/or methods around, just as you can pass data around. That's all there is to it.

=== Show me an example ===
Let's look at C#, because the syntax will not be too unusual to programmers in many languages. Suppose we have a List of scores for a test.

var Scores = new List<int>() { 2, 7, 9, 5, 6, 10, 8 };

We want to get all scores that were considered passes. Suppose the test is out of ten, and to pass you need to get more than half marks. We can write a method to determine whether a score is a pass and that returns a boolean (true or false value).

public bool Pass(int Score)
{
    return Score > 5;
}

Then we can use the FindAll method to find all scores that were considered passes.

var Passed = Scores.FindAll(Pass);

Notice what we have done here. We did not call the Pass method ourself. Instead, we passed the Pass method to the FindAll method as a parameter. Internally, it looped over the values in the list and called the Pass method on each of them, and put those that it returned true for into a new list and returned it.

You are probably now thinking, why go to all the trouble when I could have just written a loop? And it's true - you could have written a loop that did the test in an if statement and build a new list.

var Passed = new List<int>();
foreach (var Score in Scores)
    if (Score > 5)
        Passed.Add(Score);

However, step back and think about what we just achieved from a software engineering perspective. It is a well known principle that separation of concerns helps to reduce code complexity. Here we have separated the concerns of:
* Testing each element of a list and building a new list of elements that passed the test
* The test itself
Therefore, higher order programming is a tool that enables us to reduce our code complexity by separating concerns, and also increase code re-use along the way.

=== Code without a name ===
You might be thinking, "huh, I just wrote more code than I would have; I had to define a whole extra method!" You can also argue, quite reasonably, that we have actually made the program harder to understand now too, because you have to go and find the Pass method to fully understand the code.

While sometimes we may have complex conditions that we wish to re-use in many places in our program - in which case writing a separate function or method is the right thing to do - in this case it would be nice if we could take advantage of higher order programming without having to resort to writing a separate method.

Enter anonymous methods, or lambda expressions. These allow you to define a method (or function, depending on your language) without a name and inside another method. Let's re-write the previous example using a lambda expression.

var Passed = Scores.FindAll(Score => Score > 5);

The "=>" is the syntax for defining an anonymous method in C# 3.0. Anything to the left of it is a parameter that the method takes. In this case, it takes a single parameter named "Score". To the right of it is the body of the method, with an implicit "return" (that is, it will return the boolean result of testing whether its parameter Score is greater than 5).

You could read the "=>" quite nicely in this case as "where". That is, "find all scores where the score is greater than five".

=== Yes, you can return code too ===
Suppose that I want to work out how many scores got different grades. For example, Grade A was given for scores between 8 and 10, Grade B for given for between 5 and 7, and so forth. We could write:

var GradeA = Scores.FindAll(Score => Score >= 8 && Score <= 10);
var GradeB = Scores.FindAll(Score => Score >= 5 && Score <= 7);

However, it would be nice if we could make something that read more clearly, such as:

var GradeA = Scores.FindAll(Range(8,10));
var GradeB = Scores.FindAll(Range(5,7));

How could this work? Well, FindAll expects a method that takes an int and returns a bool. This type of method - that is, a method with that signature - has a name: Predicate. Therefore, the Range method needs to have a return type of Predicate (actually, Predicate<int>, which means that the parameter it takes is of type int).

public Predicate<int> Range(int LowerLimit, int UpperLimit)
{
}

And what do we write inside the Range method? Well, we return an anonymous function that takes a single parameter and tests if it is squeezed between the two values we supplied as parameters to the Range method.

public Predicate<int> Range(int LowerLimit, int UpperLimit)
{
    return x => x >= LowerLimit && x <= UpperLimit;
}

And that's it.

=== What languages support this? ===
Here's the stinger: not all languages support these kinds of programming techniques. C# started out with the ability to do higher order programming, but without anonymous methods. In C# 2.0 they introduced anonymous methods, but with a fairly cumbersome syntax. In C# 3.0, they introduced the nice "lambda" syntax that I have shown here. Essentially, the language designers have realized more and more the value of higher order programming and worked to make it as easy as they make object oriented programming. The C# of today is a truly multi-paradigm language.

Many of the dynamic languages provide good support for this, including Perl, Python and Ruby. C enables it through function pointers, though there's no anonymous method or lambda style syntax. Java, however, has so far chosen to stick to its object oriented roots rather than embracing the multi-paradigm approach. We'll leave arguing about whether that's a good or a bad thing for another time. And primarily functional languages, such as ML and Haskell, make extremely heavy use of higher order programming techniques.

=== Taking it further ===
The examples I've shown here are trivial, but they are enough to give you a taste of what higher order programming is about. Try writing some code like this yourself, or reading up on higher order programming in a language of your choice. And when you're trying to work out a good way to get better code reuse and reduce code complexity, keep it in mind as one of the tools in your toolbox, alongside the other paradigms (such as object oriented programming, procedural programming and generic programming) that you are probably already more familiar with.

Why automated testing matters

pheaven — Thu, 17 Jan 2008 02:55:08 -0700

Test Driven Development is a development methodology that encourages you to write tests before writing the code that is being tested. Following it completely strictly means that you never make a functional change to the code (that is, adding a new feature or fixing a bug) until you have a failing test that implementing the feature or fixing the bug will cause to pass.

For those who are not used to writing tests, such a strict methodology can feel quite daunting, potentially unproductive and perhaps fun-spoiling. I can empathize with all three points of view, but at the same time it's worth trying almost everything at least once to see how it works out (though I'm still not so keen on trying a bungee jump).

In this article I'm going to look at the idea of automated testing more generally. I'll share a couple of my experiences of it, then I'll look at some thoughts for giving it a go in your own environment.

=== Why Automate Testing? ===
Testing involves working out whether something you have implemented behaves as expected. We all test software at some level. For example, if you are implementing an e-commerce site, you will have a shopping cart where people can add products. You will probably try adding some products to the basket, updating them, deleting them and so on, in the web interface, to see that it actually works before shipping it off to the client for them to play with.

As development continues, you may find that you have to do some changes that in turn affect the shopping cart. You may or may not be conscious that the changes will affect it; in the case where you are not, going and checking that the shopping cart still works will likely not be something you think of doing.

This is how bugs can creep in to systems without getting noticed. It means that you will get interrupted at some point in the future when the bug is discovered to go and fix it. It also means that you may well not know what change introduced it, making finding the problem and resolving it somewhat more difficult.

It is too costly to go and manually check every feature of the site after every change. If, on the other hand, you could issue one command or press one button and run a whole set of automated tests that will exercise many features of the site, you can afford to do testing after every change. Therefore, automated testing enables you to quickly detect bugs introduced through changes.

Whenever you add a new feature, you need to test it too. If it is complex, you will want to test many use cases. Writing an automated test means that you can see, at a glance, what use cases are and aren't being tested. Keeping track of that in your head, or trying to remember if you tested a certain corner case manually, is much harder than just being able to look and see.

=== Study 1: My Bytecode Translation Project ===
One of my areas of interest is Virtual Machines for high level languages, such as the JVM, the .Net CLR and Parrot. For my degree project I looked into the problem of virtual machine interoperability: how can you use a library written to run on one VM on top of another. The solution I implemented was bytecode translation: you take the program in one bytecode format and translate it to another.

It is of great importance that the translated code has the exact same behavior on the target VM as the original code has on the source VM. I quickly realized that as I was going to add more and more features to the translator, so it could translate more and more complex programs, I needed to be able to check that the latest change I had made did not break things that had worked previously. Therefore, right at the start I implemented automated testing.

It was pretty trivial to implement. I'd call the translator to translate the bytecode representing a class on the source VM, then write tests that made method calls on the translated class in the target VM. If the method calls returned the expected values, then the test passed.

This served me very well throughout the project. There were more than a couple of times when I made a change or introduced a new feature and the tests pointed out that I had broken something that had worked previously. Furthermore, looking at which tests had failed often gave a large hint as to where the bug was to be found, cutting the amount of time spent debugging.

With the advantages of tests becoming quickly apparently, I vouched to never, ever check a new feature into source control without writing a test for it. The test didn't have to pass; I can cope with failing tests if they point out real bugs that will be fixed later. However, it meant I did not check in bad code without it being apparent the next time I worked on the project. I followed this model, and it served me well.

As I went on, I found that if I wrote the tests first then it made me think through the use cases and code paths that were going to occur. Having got that running through my mind, I found it was often a easier to write a good implementation of the feature that didn't miss so many edge cases. And of course, before you write the code you are thinking in terms of what the code should ideally do. After you've written it, you're aware of its strengths and limitations, even if you aren't consciously missing out the tests that it has a good chance of failing.

I think this story is pretty suggestive of the "conversion" process that you go through as you embrace automated testing. Some people can read about it, see it's a good idea and just start doing it. For most people - myself included - I think the only way to see its value is to give it a go and find out.

=== Study 2: An Outsourced Web Project ===
I was managing an outsourced project on behalf of one of my clients. After putting the project out for bidding and getting various offers, I looked through the profiles of the freelancers that were behind them. There was quite a mixture in there, but one that caught my eye mentioned that he used automated testing. Figuring this would give a better chance of quality code, I went with this bid.

The code was indeed supplied with a test suite, which I was very pleased to see. However, actually running the program quickly showed up a variety of bugs. In this case, there were tests, but either they were not comprehensive enough, or covered the wrong things.

Would there have been more bugs if the test suite hadn't been written? Very possibly, and very likely. However, I mention this example because it highlights an important issue. You can only consider passing the test suite to be evidence that there aren't problems with the software if it provides a high level of coverage (that is, it exercises most or all of the code paths), and covers a wide range of input values - both the expected and the unexpected.

Be careful not to be lulled into a false sense of security by the fact that all the tests are passing. It doesn't mean there aren't any bugs; it just means there aren't any bugs in the areas of the code base that get tested, for the inputs you are testing it with.

=== So Was It Worth It? ===
One question that comes up with automated testing is whether the time we took in writing the tests, most of which may well have passed straight away because our implementation was good, actually saved us time overall? That is, would it have been cheaper (time wise, and thus cost wise) not to write tests, but just to debug the code when we try to use it and find it doesn't work?

I think that the answer is, even if it does take a bit more time to write the code and a good test suite, it's still well worth it.

First, if you're writing tests alongside writing implementation, you'll have the code base in your head. Any bugs that the tests highlighted can be fixed while the code base is fresh in your mind. If, on the other hand, you discover the bug a week, a month or a year down the line instead, chances are that you'll have moved on to something else, and will have to dig through the code, having to work out what it does before you can identify the problem and work out a fix.

Second, it's good for project and time management purposes. Knowing that you have implemented a chunk of a project and that it actually works because you've got tests to show that it does is a good indicator of your progress. Having a bunch of code that you know compiles, but have no idea if it really works, is a much less reliable measure of your progress. This is especially true when developing libraries and not having any user interface that makes use of them available to test with yet.

Third, it's an investment for future maintenance programming work. At some point, chances are the code will need fixes, have new features added or have to be ported to be compiled under a different compiler or run on a different platform. Chances are that the maintenance programmer may well not be you, or it may be a five year older version of you. Having a test suite that gives at least some assurance that the code still works with the changes or after the porting will be a big help in this situation.

=== Jumping Into It Yourself ===
There are plenty of frameworks out there for automated testing. I can't look at all of them, so I'm just going to list a few for some popular languages and platforms, and then look at some general hints for writing good tests. Note that these are just some of the options, and a little searching will reveal more options than these for the language or platform of your choice.

* .Net Platform: NUnit
* Java: JUnit
* Perl: Test::Simple (or Test::More) and Test::Harness
* Python: PyUnit
* Ruby: Test::Unit
* C: Check
* C++: CppTest

==== Test Organization ====
The first thing to think about is test organization. In most frameworks, you can spread tests across multiple files (which you can in turn organize in some kind of directory structure), and each file can contain many tests. Exactly how you organize your tests will depend on what you are testing and the structure of your project.

One way that can work well for some object-oriented software is to structure your tests in the same way as your source tree, with a test file for each class. A more general principle is to give each feature (or set of related features) its own test file, putting tests for that feature inside that test file.

The reason for this kind of division is that it helps you to see exactly what is broken by looking at which test files contained failures. It also means that you can run the tests for that individual feature alone more easily, which is useful when you want to see if a fix you applied to try and make that test pass again worked. Of course, you need to do a full test run at some point to make sure it didn't break other things, but if it takes a few times to get right then just being able to run a single test file rather than the entire suite of tests can be a big time saving. It also helps you to work out more easily where the test for a particular feature are, if you want to review them.

==== Writing Tests ====
When writing an individual test, try to make it test a certain aspect of an individual feature. If it depends on other features, they should also have tests. You may want to order tests in such a way as to make sure that features depended on by later tests are tested first.

A test should either have a descriptive name that makes the purpose of the test clear or a comment stating that (or both). Just like code, don't assume that some maintenance programmer staring at your test in five years time will know just know what the intention is. Tests are code too.

Additionally, a test should try to leave the overall environment as it found it, whether it passes or fails. This can be decidedly tricky (or maybe even impossible) in some cases, when code that modifies a database or changes files is being tested. In these cases, perhaps consider supplying a test database or test set of files. The key criteria is, if you run the test through once, and then run it again without any changes to the code or the test, does it always give the same result?

==== What To Test ====
It's quite obvious that you should write tests that check that a feature behaves as expected when given sensible input values. However, if there is any way that data can reach it from the "outside world" without being sanitized, or if you are not sure whether that is the case, you should also test it with invalid input and make sure it handles the situation gracefully.

The other important kind of tests to write are boundary condition tests. For example, if you are testing a function or method that processes a list, check how it handles an empty list. If it is meant to take a range of values, check the values at either end of that range. If it is handling a date or time, check it with the first and last day of the year, leap years, midnight and so on. If it is working with a string, check how it handles if an empty string is passed.

=== Closing Thoughts ===
If you weren't doing any testing already, I hope this quick look at automated testing has inspired you to give it a go. If you do, why not write in your PH blog about your experiences of it, or leave a comment here? I'll certainly be interested to read anyone's accounts of this, as well as success and horror stories.

This article is far from the definitive guide to testing. If it were, there wouldn't be books hundreds of pages long on the subject. I hope that it has given you an appetite for the area, and you'll find plenty more material out there on the web and in books, some general and some specific to individual testing frameworks or testing certain types of application. Good luck!

Sun acquires MySQL

pheaven — Wed, 16 Jan 2008 08:27:14 -0700

Sun have today announced their plans to acquire MySQL, the company behind the popular MySQL database server. This gives Sun a large stake in the database market. MySQL already has a range of high-profile clients, with some household names including Facebook, Yahoo!, BBC and Nokia.

MySQL is an Open Source and Free Software database, and members of the community will be somewhat reassured by Sun's background. They are the largest contributors of open source code, well known for Open Office and more recently for releasing Java under the GNU GPL. The MySQL VP of Community wrote a blog post to reassure the community about the acquisition.

With the deal totaling a $1 Billion, it's a clear indicator of the financial potential of companies founded on the principles of free and open source software. And who knows, many of the clients that first started out small using MySQL because it was freely available and grew up to become clients of the MySQL company may well not have done so if they'd had to pay out for it in the first place.

What are the next generation of programmers learning?

pheaven — Mon, 14 Jan 2008 07:22:00 -0700

Recently I read an article asking, where are the software engineers of tomorrow? Written by a President and Vice-President of the company AdaCore, they lament the spread of Java as a first programming language taught on many Computer Science courses and a reduction in the formal and theoretical background that students are being given.

While I can't say I agree with every word they say (for example, they appear somewhat dismissive of dynamic languages), they certainly have a lot of good points. You certainly don't learn to be a good programmer, developer or architect by learning one or two programming languages. To do that you need to understand the whole stack of abstractions form top to bottom, be aware of many programming paradigms and understand (or at least be aware of the existence of) the theory that explains it.

It's important to know what assembly language is and have a rough idea of what the hardware is doing. There's no need to be proficient in the assembly for any one CPU, though seeing the trade-offs between different architectures is useful. Layered on top of that are high level languages like C: they are decidedly high level languages, but they are still pretty close to the metal. I'm not saying that people need to be taught C, but there are some things that people really should understand, especially explicit memory management and pointers. For both assembly and C, people graduating from a computer science course should be able to grab the appropriate reference material and use either of them effectively, because they understand the underlying concepts.

Above this level, we have languages that either sit atop of virtual machines or run with notable runtimes. These languages, free of low-level issues, are good for teaching different paradigms, illustrating algorithms and exploring software engineering issues. No one language should be taught exclusively here. There's a simple reason for this: no one language exemplifies all paradigms and approaches. Programming languages are tools, and they are suited to different jobs.

Java is useful for teaching object oriented programming and reflective programming, and perhaps generic programming too. However, it has nothing to offer with regard to functional and higher order programming, and only shows one approach to typing: static typing with explicit type annotations. If Java's capabilities are all a programmer has even seen on their university course, then they are under-equipped to deal with C# 3.0, which offers both higher order programming and limited type inferencing. Someone who had seen a range of languages and paradigms, however, will be able to C# 3.0 to good effect, rather than writing it like they would write Java and complaining about how anonymous methods and lambda expressions are damaging what was once a "pure object oriented language". (Also, anyone who says that doesn't know what pure object orientation is; neither C# nor Java have ever been pure object oriented languages.)

So far I've just talked about programming, because it's what most computer scientists are recruited to do. However, ideally the ability to program should be a natural fallout of an understanding of computer science and software engineering. That is, programming should not seem that big a deal once you understand the concepts behind hardware architectures, have knowledge of a range of data structures and algorithms and comprehend fundamental issues such as algorithmic complexity (and sorry, but if your computer science course didn't explain at least that, the word "science" deserves no place in the course name). To build real world systems takes an understanding of managing software complexity, safety and security engineering and a range of concepts such as concurrency, databases and so on - all things that should have a place on a computer science course.

If there's any word that could describe a computer science course, it would probably be "holistic". Being equipped to deal with the challenges the computing field faces requires a wide skill set, encompassing mathematics, electronics, engineering, psychology and more. And it involves taking the word "science" seriously and teaching concepts, rather than bringing up a bunch of code monkeys who are ill equipped to deal with what lies ten years down the line, because it's not Java or PHP.