Random header image... Refresh for more!

Posts from — January 2011

dynamic Has a Use! Partial Verification of Complex Classes in Test Automation

I don’t like dynamic in C#.  It’s one of those features that seems to serve only to confuse the language.  And yeah, I get that Ruby and Python are doing it, but if Ruby and Python jumped off a bridge, would you?

When I first picked up C#1, I liked it for three reasons: 

  1. Garbage collection.
  2. It had a built-in string class.
  3. It actively prevented you from doing stupid things that C++ was fine with.

When they added “var“ to the language, reason #3 died a little.  People started using “var” to declare ordinary variables where the type is known.  Sure, the compiler can figure out what the type is, but a human can’t, at least not by just reading the code without a knowledge of the return type of the function you just called. 

And then along came “dynamic” and walloped reason #3 over the head with a large stick.  Now, not even the compiler knows what you’re doing.  You have to wait until runtime to tell if your code is going to work.  It’s a bit like programming in Basic on a Commodore 64, except you’ll get a RuntimeBinderException instead of “?SYNTAX ERROR IN 150”2.  The end result is the same, a stupid typo broke your code when someone else tried to run it. 

It doesn't make any sense here, either.

 The idea behind dynamic is that it lets you write code against certain types of COM objects and dynamic objects from languages like Python and Ruby.  Of course, COM should be dead by now and that maybe MS Office’s Automation should look at this new-fangled .Net thing that everyone else in Redmond has been using FOR TEN YEARS, and that maybe if you want to use features from Python or Ruby that maybe you should actually be using Python or Ruby, but whatever…  The way dynamic works is that you can declare a variable as type “dynamic”, just like you’d declare it a “string” or an “int”, and this tells the compiler that you have no idea what the object is.  Since you don’t know what it is and since the compiler doesn’t know what it is, you can go ahead and call any method you’d like on it or use and property on it and theoretically, it’ll work.  It’s called “Duck Typing”, as in “quacks-like-a”.  In other words, I don’t care what object X really is and I don’t want to force it to be a subclass of Y or implement interface Z.  All I care about is whether or not it can “Quack()”.  If it can “Quack()”, then it’s enough of a Duck for me to use in my function. 

In most cases, dynamic in C# doesn’t actually mean that you can dynamically change the structure of the underlying object.  Whatever that object is, it’s still that type, even though you’re hiding it behind the dynamic keyword.  Think of dynamic as if it’s “var” with amnesia.  “var” knows what type it is at compile time, so you can ask it “Do you know how to quack?” and it can tell you.  With dynamic, it won’t know whether or not it can quack until the code actually runs, at which point its memory comes back and it remembers how to quack.  Either way, whether it knows how to quack at run-time or compile-time, the thing quacking is still a duck. 

Now, I don’t know about you, but I don’t trust amnesiac ducks. 

You see, since you can call any method or property on the dynamic object, you have to simply hope that whatever the object turns out to be when you run it actually implements that method or property, or else you’re totally screwed.  Also, since there’s no Intellisense to guide you, you’d better hope that you spelled the property or method correctly, or else you’re totally screwed.  Oh, yeah, and since there’s no type checking, you’d better hope that the type of that property or the types of the parameters or return value of the method are what you think they are, or else you’re totally screwed. 

It’s this whole “…or else you’re totally screwed” bit that turns me off about dynamic.  There’s a lot of ways this can go wrong, and when it goes wrong, you don’t know it’s gone wrong until the program is running and it dies.  And, of course, it won’t do that until it’s out live in production and a customer comes across the bug, and the whole site goes down and the closest tester gets blamed for missing the bug.  You can see how that would be a problem for me, so you can understand why I don’t like dynamic. 

But then I found a way to actually make use of dynamic to solve a testing problem I’ve been having. 

Before I go into details, let me first say that I’m a bit reluctant to share this idea, for several reasons: 

  1. I don’t like dynamic, so it feels dirty to use it in this way.
  2. I haven’t really fully explored this idea, so it could be a complete load of nonsense.
  3. It’s voodoo magic code and I don’t like using voodoo magic code because no one can understand voodoo magic code.  I know that I’m going to spend more time explaining how it works to other people than they’re going to spend using it, because they’re going to look at it and be scared away.
  4. “…or else you’re totally screwed”.  It would not be good to have test cases that fail at runtime because of typos.
  5. It feels like this is half of a good idea, and that once I figure out what the missing half of the puzzle is, I’ll have something that’s useful and won’t scare people away and won’t feel dirty.  I don’t like writing about half an idea.
  6. I don’t like dynamic.

However, I’m sharing it because I want it to be a good idea and hope that writing about it will get me thinking about how to solve those remaining problems. 

Anyway, here’s the scenario: 

Where I work, we have services that return complex objects.  Sometimes, these objects can have upwards of 20 properties, and some of those properties are classes that have more properties on them.  In other words, there’s more properties floating around in these objects than in most Florida land scams.  Testing these objects is relatively straightforward, though.  Simply create a parallel object with the expected values, and walk through the expected and actual objects property by property, comparing the values.  If something doesn’t match, fail the test. 

The problem I have is that for most of my tests, I don’t really care about most of those properties.  Each test only focuses on a handful of values and the rest don’t matter.  The one-size-fits-all approach of walking the properties required an exact match.  If I only care about four or five properties, why should I have to specify values for the other 15?  I might not even know what they are.  One of the properties could be something like response time, which I have no way of knowing at the time I write the test.   But, I don’t want to have to write custom validation for each test to check only the values it cares about, because that’s inefficient and difficult to maintain. 

Let’s give a more concrete example class to work off of here. 

public class SearchResult
        public string Title { get; set; }
        public string Excerpt { get; set; }
        public string URL { get; set; }
        public double Score { get; set; }
        public int Size { get; set; }
        public int Position { get; set; }
        public DateTime IndexedTime { get; set; }
        public string CacheURL { get; set; }


This class is a simplification of a result returned from a search engine.  Each result has a title, an excerpt of text from the page, a URL for the page and internal information, like the confidence score of the result, the size of the page, the date the page was last indexed, the position of the result, and, if available, a URL for a cached copy of the page. 

Here’s the property by property checker: 

public void CompareResults(SearchResult expected, SearchResult actual)
    Assert.AreEqual(expected.CacheURL, actual.CacheURL, "CacheURL mismatch");
    Assert.AreEqual(expected.Excerpt, actual.Excerpt, "Excerpt mismatch");
    Assert.AreEqual(expected.IndexedTime, actual.IndexedTime, "IndexedTime mismatch");
    Assert.AreEqual(expected.Position, actual.Position, "Position mismatch");
    Assert.AreEqual(expected.Score, actual.Score, "Score mismatch");
    Assert.AreEqual(expected.Size, actual.Size, "Size mismatch");
    Assert.AreEqual(expected.Title, actual.Title, "Title mismatch");
    Assert.AreEqual(expected.URL, actual.URL, "URL mismatch");


Now, let’s write a test:  If I search for “dogs”, then I expect to get the Wikipedia entry for “Dogs” as my top result. 

public static void CheckThatWikipediaIsInResultsForDogs()
    SearchResult expected = new SearchResult();
    expected.URL = "http://en.wikipedia.org/wiki/Dogs";
    expected.Title = "Dog - Wikipedia, the free encyclopedia"; 
    expected.Position = 0;

    SearchEngine engine = new SearchEngine();
    SearchResult actual = engine.Search("dogs"); 

    CompareResults(expected, actual);


Now we run it and…  Aw crap.

Assert Failed! != https://mathpirate.net/cache?=http%3A%2F%2Fen.wikipedia.org%2Fwiki%2FDogs : CacheURL mismatch

The test failed, not because the Wikipedia entry wasn’t the result, which is what I cared about, but because the CacheURL was wrong.  I don’t care about the CacheURL.  There are other tests somewhere else that will deal with CacheURL.  Now, I can specify it if I want to, but I also have to specify the excerpt and the page size and the last indexed time, etc.  But I’m not going to .  I don’t give a flying monkey dance about those values in this test, so it’s stupid to specify them.  The “CompareResults” method has to change.

One fairly simple way to take care of that problem is to check for non-null or non-default values and not check the values if the value is the default.  Here’s the new and improved CompareResults:

public static void CompareResults(SearchResult expected, SearchResult actual)
    if (expected.CacheURL != null) { Assert.AreEqual(expected.CacheURL, actual.CacheURL, "CacheURL mismatch"); }
    if (expected.Excerpt != null) { Assert.AreEqual(expected.Excerpt, actual.Excerpt, "Excerpt mismatch"); }
    if (expected.IndexedTime != default(DateTime)) { Assert.AreEqual(expected.IndexedTime, actual.IndexedTime, "IndexedTime mismatch"); }
    if (expected.Position != default(int)) { Assert.AreEqual(expected.Position, actual.Position, "Position mismatch"); }
    if (expected.Score != default(double)) { Assert.AreEqual(expected.Score, actual.Score, "Score mismatch"); }
    if (expected.Size != default(int)) { Assert.AreEqual(expected.Size, actual.Size, "Size mismatch"); }
    if (expected.Title != null) { Assert.AreEqual(expected.Title, actual.Title, "Title mismatch"); }
    if (expected.URL != null) { Assert.AreEqual(expected.URL, actual.URL, "URL mismatch"); }

Run it now and it passes.  It didn’t care about the missing CacheURL.  Hey, problem solved!

Except…   The new and improved CompareResults is new and improved and still broken.

Here’s what was returned from the Search method:

Notice that Position is set to 5.  In my test, I explicitly set that position should be 0.  CompareResults missed it because  of the check that I added: if(expected.Position != default(int)).  default(int) isn’t some magic value, like “undefined”.  It’s “0”.  Plain old zero.  So when that line executed, it became if(0 != 0)3.  And so, the check was skipped and it missed the fact that the test should have failed.  The same problem will occur if you’re explicitly checking for a null property value.

So…  What to do?

There are multiple options to handle this case, none of them very pleasant:

  • Live with it.  One-size-fits-all never actually fits you.  Deal.
  • Write custom validation for every test that needs to check a default or null value explicitly.  Hope that everyone remembers this needs to be done.
  • Write a parallel set of classes that mimic the actual class, but have boolean properties like “PositionSet” for every single property on the object. 
  • Go back to the original method that checks everything exactly, but write custom pre-treater methods that can go in and clear out values that you don’t care about before you do the comparison. 4


Use dynamic.

Here’s how it works:

Instead of creating an instance of the type of object that you want to create, you create a dynamic object, and only set the properties you care about.  Then, in the comparison method, it can check if you’ve set the value, and then do the comparison if you have.  If not, it gets ignored.  This solves the problem of forcing a check on values you don’t care about as well as the problem of default values.  If you didn’t set the value up front, it’s not there, so the verification method won’t look at it.

Let’s see it in action:

public static void CheckThatWikipediaIsInResultsForDogsDynamic()
    dynamic expected = new ExpandoObject();
    expected.URL = "http://en.wikipedia.org/wiki/Dogs";
    expected.Title = "Dog - Wikipedia, the free encyclopedia";
    expected.Position = 0;

    SearchEngine engine = new SearchEngine();
    SearchResult actual = engine.Search("dogs");

    CompareResultsDynamic(expected, actual);

The test method looks almost identical.  The only notable difference is the first line.  I changed “SearchResult expected = new SearchResult();” to “dynamic expected = new ExpandoObject();”5.  The properties are set on the object the same way, the function is called the same way.

public static void CompareResultsDynamic(dynamic expected, SearchResult actual)
    if (PropertyExists(expected, "CacheURL")) { Assert.AreEqual(expected.CacheURL, actual.CacheURL, "CacheURL mismatch"); }
    if (PropertyExists(expected, "Excerpt")) { Assert.AreEqual(expected.Excerpt, actual.Excerpt, "Excerpt mismatch"); }
    if (PropertyExists(expected, "IndexedTime")) { Assert.AreEqual(expected.IndexedTime, actual.IndexedTime, "IndexedTime mismatch"); }
    if (PropertyExists(expected, "Position")) { Assert.AreEqual(expected.Position, actual.Position, "Position mismatch"); }
    if (PropertyExists(expected, "Score")) { Assert.AreEqual(expected.Score, actual.Score, "Score mismatch"); }
    if (PropertyExists(expected, "Size")) { Assert.AreEqual(expected.Size, actual.Size, "Size mismatch"); }
    if (PropertyExists(expected, "Title")) { Assert.AreEqual(expected.Title, actual.Title, "Title mismatch"); }
    if (PropertyExists(expected, "URL")) { Assert.AreEqual(expected.URL, actual.URL, "URL mismatch"); }

This is the new CompareResults function.  First, the “expected” parameter has changed.  It’s now “dynamic” instead of “SearchResult”.  The Asserts themselves are identical, and they’re still wrapped inside of if statements, but the if conditions have changed.  Instead of looking for nulls or default values, they’re checking that the property exists on the dynamic object.

This is one part where the whole dynamic thing is sorely lacking.  There’s no built-in way to tell if an object can do what you want it to do without trying to do it.  It’s like you’re commanding the duck to “Quack, damn you!”, without first asking “Can you actually quack?”.  Ruby’s got “respond_to?(‘Quack’)”, while C# has “Do it yourself and don’t forget to account for all of the different possibilities of how the quacking can be done, otherwise just try to  ‘Quack’ and catch the exception”.  Here’s a very abbreviated and probably error-prone example of how to ask if an object supports the property you want to call:

public static bool PropertyExists(object dynamicObject, string propertyName)
    if (dynamicObject is ExpandoObject)
        IDictionary<string, object> expando = (IDictionary<string, object>)dynamicObject;
        return expando.ContainsKey(propertyName);
        return dynamicObject.GetType().GetProperty(propertyName) != null;

First, it checks if the object is an ExpandoObject.  If it is, then it can use the explicit IDictionary implementation on IDictionary in order to find out if the property exists or not.  Otherwise, it uses basic Reflection to ask if the object has the property you’re looking for.  This PropertyExists method will probably need a bit of work for general consumption, but it’s a start and works well enough for this example.

But…  Does it work?

Assert Failed! 0 != 5 : Position mismatch

It didn’t complain about the CacheURL, like the first solution did, but it did catch the Position mismatch, which the second solution missed.  I’d call that a success!

There are, of course, problems with this solution…

  • There’s no Intellisense, so you have to know what the object looks like.  Of course, what you can do there is write the initialization with the actual class, then change it to dynamic once you have all of the properties in place.
  • There’s no compile-time checking, so if you make a typo or if a property disappears or changes names, you don’t know about it until everything breaks at runtime.  Or worse, nothing breaks at all, but you end up not checking the values you think you’re checking. 6
  • It’s voodoo magic code.  You’re going to be the only one who understands how it works, so you’re on the hook for making sure it actually does work and you’ll going to be the one that gets blamed or cursed every time something goes wrong.

Like I said, it’s half of a good idea and I haven’t really explored all of its benefits and consequences yet.  The core is here and I know this is a good place to start and build something awesome on top of.  I already have some ideas of where this could go.  Like toward a generic exapandable data-driven object comparison framework.

But that’s for another time…

Here’s the code from this post:  https://mathpirate.net/svn/Projects/AutomationUsingDynamic/

  1. I came through C++ and hadn’t been exposed to Java at that time. []
  2. Okay, it’s more like writing JavaScript, but whatever. []
  3. if(0!=0){ throw new UniverseDestroyingParadoxException(“Critical Mathematical Foundation Error.  Please Reboot.”); } []
  4. I admit that I’ve done this once.  It was not one of my better moments. []
  5. ExpandoObject is a type that does some magic with the C# dynamic runtime binder.  I’m not going to explain how it works, after all, there are search engines for that.  However, using ExpandoObject lets you dynamically add members to your dynamic object.  If you tried to do this with an ordinary object, you’d get a RuntimeBinderException because the property you’re setting isn’t there. []
  6. Why, oh why didn’t they do something like “dynamic<T>” for at least some compile-time checking? []

January 1, 2011   No Comments