More VSTS Unit Testing learnings

I was having problems getting a very simple test to work this morning using VSTT, and I distilled my problem down to its most simple form:

using System;
using Microsoft.VisualStudio.QualityTools.UnitTesting.Framework;
using System.IO;

namespace FileCopyingBug
{
    [TestClass]
    public class FileCopyFixture
    {
        [TestMethod]
        public void WillCopyFileWhenRun()
        {
            Assert.IsTrue(File.Exists("TestFile.txt"));
        }
    }
}

And I created the external TestFile.txt in my project directory and set its properties to have it copied to the output directory always.

Depending on how I chose to run this test, it would either work or fail. If I ran it using the Test View pane in VSTT and I chose “Run” or “Debug” the test would fail. If I ran it through TestDriven.Net, it would work if I ran the test, but fail if I debugged the test. And if I translated the test to NUnit, it would work through TestDriven.Net all the time. Very interesting.

It would seem that, according to the Principal of Least Surprise, this test should just work.

But it turns out that VSTT copies your executable and its dependencies to another directory whenever it runs your tests, so just because you made sure some files showed up in your binDebug directory doesn’t actually mean that they showed up where your program was being executed from. This seems a bit fishy to me, but at least I know about it now.

The Fix — Deployment Items

Many thanks to Tom Arnold, the PM for the unit testing tools in VSTS, and the people who work with him for giving me the answer to this very quickly. I really do appreciate it!

There is an attribute that you can apply to your TestMethods called the DeploymentItemAttribute. If you apply this attribute and give it the name of the file you want copied, it will make sure that it copies the named file to the real output directory. Once I did this, my test worked just fine.

You can also apply this attribute to your TestClasses, but one web site I saw said that there was a bug in this. If you ran your tests through the IDE, it would not respect this attribute. If you ran them from the command line using MSTest.exe, it would work fine. I did not verify this, so it may apply to an earlier verion, etc.

Finally, you can also make this fix by editing the Run Configuration for your test. If you open the Test menu and choose Edit Test Run Configurations, a dialog box pops up. On this dialog, you can detail which files you would like treated as deployment items. I fixed my problem like this, and once I did, everything worked great.

Conclusion

Once again, I really do feel that these tests should just work. The fact that VSTT is doing magic behind the scenes for whatever personal reasons shouldn’t matter to me, as that was an implementation decision that they made. Fortunately there is this very simple fix for it, however, and I wanted to write about it so I could remember it a month from now when it bit me again

— bab

 

Slides from Enterprise Library presentation at St. Louis .Net User Group, June 26th

On Monday, June 26th, I had the pleasure of addressing the St. Louis .Net User Group. I had been invited to speak about Enterprise Library, as several people had been asking questions in the group about how to get started using it and were looking for an overview. I had already given a talk on Enterprise Library to the C# and VB.Net SIGs of this group, so it was only natural that they might call on me to give the talk at the main meeting at well.

I think the talk went really well. There were probably 50 people or so in attendance, and they listened to my talk, watched me write some really simple code that I then taught lots of new tricks through configuration, and they even asked a bunch of really good questions. I do have to say that the questions that this group asked were better and harder than those in my previous talks, because several of the questioners had been using Enterprise Library for a bit. The questions they asked were pretty particular about individual facets of the library, which made my life a lot harder  Seriously, though, they were really good questions, and I’m glad I was able to answer most of them.

As I promised in the talk, here are the slides. In them, you’ll find all the links I was talking about, including the download site for Enterprise Library, the Hands On Labs, the great webcasts and podcasts, and all the blogs, as well as descriptions of each block, and a really general overview of what is coming next in the Enterprise Library family.

Knock yourself out, and if you have any questions, you have my email address

Thanks again for having me and listening. I really appreciated the opportunity, and I’m looking forward to my next series of talks to these groups.

— bab

 

WMI, LUA, and a surprise for me!

Note to self: WMI and LUA don’t play together nicely.

I’ve been researching a bug that many of you Enterprise Library users may have encountered. According to our instructions, you have to run installutil against all EL assemblies, to allow them to load up their performance counters, event log sources, and WMI schema. If you do this as an admin, things work beautifully. If you run your EL-based app with less than admin privileges, interesting things happen with WMI, but that has been covered on other blogs.

What I want to cover today is what happens during development when you are creating code that attempts to publish WMI events or data.

Example Test

Here is an example test to show what I’m talking about. The behavior of this test is different, depending on whether you’re running as an admin or not. But the difference it not something that you might intuitively expect to happen.

   [TestClass]
    public class WMIExplorationFixture
    {
        List inputData = new List();

        [TestCleanup]
        public void RevokeAllInstances()
        {
            foreach (MyData instance in inputData)
            {
                Instrumentation.Revoke(instance);
            }
        }

        [TestMethod]
        public void CanSendSinglePieceOfData()
        {
            MyData myData = Create("foo!!!", 7);
            myData.Published = true;

            int instanceCount = GetInstanceCount();
            List results = GetInstances();

            Assert.AreEqual(1, instanceCount);
            Assert.AreEqual(myData, results[0]);
        }

        private MyData Create(string myFoo, int myCount)
        {
            MyData instance = new MyData(myFoo, myCount);
            inputData.Add(instance);

            return instance;
        }

        private List GetInstances()
        {
            List results = new List();
            using (ManagementClass wmiClass = new ManagementClass(@"\.rootWMIExploration:MyData"))
            {
                foreach (ManagementObject instance in wmiClass.GetInstances())
                {
                    int count = (int)instance["MyCount"];
                    string value = (string)instance["MyFoo"];

                    results.Add(new MyData(value, count));

                    instance.Dispose();
                }
            }
            return results;
        }

        private int GetInstanceCount()
        {
            int instanceCount = 0;
            using (ManagementClass wmiClass = new ManagementClass(@"\.rootWMIExploration:MyData"))
            {
                instanceCount = wmiClass.GetInstances().Count;
            }
            return instanceCount;
        }
    }

And here is the class that represents the WMI schema:

   public class MyData : Instance
    {
        private string myFoo;
        private int myCount;

        public MyData(string myFoo, int myCount)
        {
            this.myFoo = myFoo;
            this.myCount = myCount;
        }

        public string MyFoo { get { return myFoo;  } }
        public int MyCount { get { return myCount; } }

        public override bool Equals(object obj)
        {
            if (obj.GetType() != typeof(MyData)) return false;

            MyData other = (MyData)obj;
            return this.MyCount == other.MyCount && this.MyFoo == other.MyFoo;
        }

        public override int GetHashCode()
        {
            return 1;
        }
    }

So, if you’re LUA, this test never passes. But, depending on what action that were taken, it can fail for one of two basic reasons.

The first reason that it can fail is that the schema may never have been installed into WMI. This is the exception you see in that case:

------ Test started: Assembly: WMISpikes.dll ------

TestCase '' failed: Test method WMISpikes.WMIExplorationFixture.CanSendSinglePieceOfData threw exception:  System.Exception: This schema for this assembly has not been registered with WMI..
    at System.Management.Instrumentation.Instrumentation.Initialize(Assembly assembly)
    at System.Management.Instrumentation.Instrumentation.GetInstrumentedAssembly(Assembly assembly)
    at System.Management.Instrumentation.Instance.set_Published(Boolean value)
    C:DevelopmentSourceDepotEnterpriseLibrarySpikesWMISpikesWMISpikesWMIExplorationFixture.cs(28,0): at WMISpikes.WMIExplorationFixture.CanSendSinglePieceOfData()


0 succeeded, 1 failed, 0 skipped, took 0.00 seconds.

You can fix this by installing the schema, using installutil. Once you’ve done that as an admin and gone back to running your tests as LUA, your tests now fail for this reason:

------ Test started: Assembly: WMISpikes.dll ------

TestCase '' failed: Test method WMISpikes.WMIExplorationFixture.CanSendSinglePieceOfData threw exception:  System.Runtime.InteropServices.COMException: Exception from HRESULT: 0x80041003.
    at System.Management.ThreadDispatch.Start()
    at System.Management.Instrumentation.InstrumentedAssembly..ctor(Assembly assembly, SchemaNaming naming)
    at System.Management.Instrumentation.Instrumentation.Initialize(Assembly assembly)
    at System.Management.Instrumentation.Instrumentation.GetInstrumentedAssembly(Assembly assembly)
    at System.Management.Instrumentation.Instance.set_Published(Boolean value)
    C:DevelopmentSourceDepotEnterpriseLibrarySpikesWMISpikesWMISpikesWMIExplorationFixture.cs(28,0): at WMISpikes.WMIExplorationFixture.CanSendSinglePieceOfData()


0 succeeded, 1 failed, 0 skipped, took 0.00 seconds.

This error (0x80041003) says that you have no permission to be sending this data to WMI, which is true if you’re running LUA. By default, only a very limited set of users is allowed to send data into WMI, and changing this appears to be rather twisted and convoluted (it can be done, but probably only by experts).

Now for the surprising twist

If I recompile my program and rerun the tests, while still running as LUA, I go back to the first exception message. What apparently happens is that compiling your program invalidates the schema registration and you have to rerun installutil. Surprise!

Surprising twist #2

If you are running your tests as a Local Administrator, registration of your schema class happens automatically for you when the program is run the first time. So the above test works all the time, without ever having to run installutil. Surprise!

Why am I posting this?

I’m posting this because I’ve seen a lot of Enterprise Library users having this exact problem. I spent a while today researching it, and I wanted to write my conclusions down somewhere so that I can refer to them later when I have this problem again 🙂 The page I used to find most of this out was here.

My main conclusion is that it is very difficult to run WMI-publishing code as LUA. It seems to almost require being an admin to run correctly and easily. You can add some sort of security descriptor to your data or event class to allow it to be given to WMI, but that’s pretty confusing from the explanations I’ve seen. You can use wmimgmt.msc to give certain users permission to write to particular WMI namespaces, but this also seems to be a pretty big hurdle for Enterprise Library customers to have to climb. It just seems like WMI needs to be run as admin to work easily and correctly. Not real happy about that answer.

I hope that this information is going to help someone, sometime to avoid wasting a lot of time tracking down this issue. With any luck, that person will be me 🙂

— bab

 

Now playing: Alice In Chains – Dirt – Rain When I Die

My Myers-Briggs Profile

Some of my friends here at Microsoft have been involved with some personality analysis to find out how red, blue, green, or yellow they were. These colors reflect some sort of behavior (think aggressive, depressing, party, and mom :)). Since I’m just a consultant here, I didn’t get a chance to do this, but I did go back to a Myers-Briggs testing site and took one of the on-line quizzes. I turned out to be an INTJ, which chances are most of you are, too.

I’m 22% introverted, 25% intuitive, 50% thinking, and 22% judging, which according to their clustering system makes me a Mastermind Rational.

No idea what this means, but in reading the page above, it really does sound like me. Here is the quiz that I took — take it yourself and see if it matches you!

— bab

 

Whidbey System.Configuration, the Principle of Least Surprise, and a dash of TDD

What in the world do these three topics have in common??? I would have thought nothing until yesterday. But then I learned better…

The Principle of Least Surprise

Do me a favor, really quickly. Google “Principle of Least Surprise”. Back already? Good. It sure seems like there are a lot of folks out there who are pretty hot on using software that does what they expect it to do. And doing something unexpected, like making Ctrl-Q quit your program instead of ending macro recording, makes some folks pretty angry.

I’m a pretty firm believer in this principle as well. I believe that well designed software should do the most expected thing at all times, and never, ever surprise me.

Whidbey System.Configuration

And then yesterday, I started playing around with System.Configuration in Visual Studio 2005, and found the Principle of Least Surprise had become more of a guideline than a rule (paraphrase of a movie quote — anyone recognize its original genesis??? Think Bill Murray, Dan Akroyd, Harold Ramis).

I wrote these tests, each of which I think has a reasonable expectation of passing:

    [TestClass]
    public class RawConstructionFixture
    {
        [TestMethod]
        public void CanBeConstructedFromSingleArgumentConstructor()
        {
            SettingsStore settingsStore = new SettingsStore("foo");
            Assert.AreEqual("foo", settingsStore.Name);
        }

        [TestMethod, ExpectedException(typeof(ConfigurationErrorsException))]
        public void CannotRetrieveValueIfItIsInvalid()
        {
            SettingsStore settingsStore = new SettingsStore();
            Assert.IsNull(settingsStore.Name);
        }

        [TestMethod]
        public void CanSetValueInCodeBeforeGettingAndGetValueBackOut()
        {
            SettingsStore settingsStore = new SettingsStore();
            settingsStore.Name = "foo";

            Assert.AreEqual("foo", settingsStore.Name);
        }
    }

and here is the class under test:

   public class SettingsStore : ConfigurationElement
    {
        public SettingsStore()
        {
        }

        public SettingsStore(string name)
        {
            this["name"] = name;
        }

        [ConfigurationProperty("name")]
        [StringValidator(MinLength = 1)]
        public string Name
        {
            get
            {
                return (string)this["name"];
            }
            set
            {
                this["name"] = value;
            }
        }
    }

This seems like a pretty simple class. The only things to note about it are that it derives from ConfigurationElement, which is what ties it into the System.Configuration system, and that the Name property has two attributes on it. Of these two attributes, it is the StringValidatorAttribute that we are particularly interested in.

Expected Principle of Least Surprise Behavior

Let’s look at each of these tests individually and discuss my expectations, and later we’ll talk about what actually happened. And at the very end, we’ll talk about the effect of this on being able to write code through TDD that uses any configuration information with validations at all.

So, our first test, CanBeConstructedFromSingleArgumentConstructor, seems pretty simple. I expect to be able to construct a class and initialize its members in the constructor. After all, that’s how objects work. The field being initialized in the constructor is Name, which is defined in the SettingsStore class as a string, with a validator that enforces the constraint that any value that Name ever holds will be at least one character long. No problem, I think to myself. I’m initializing Name to “foo”, which is certainly more than one character long. This test should really, really pass. If it doesn’t, I think I would be Surprised.

In the second test, CannotRetrieveValueIfItIsInvalid, I create the object using the default constructor and try to access its value before it is initialized. There is a bit of an argument here about what the least surprising behavior would be. My partner thinks it should return NULL, and I think that getting its value at this point would violate the constraint, and an exception should be thrown. He eventually acquiesced to me, and I wrote the test as you see. This seems reasonable to me. Yea!

Finally, in the third test, CanSetValueInCodeBeforeGettingAndGetValueBackOut,I instantiate the object using the default constructor and properly initialize the value to something reasonable, “foo”, which satisfies the length constraint. I then try to assert that the value I set actually was set. I fully and completely expect this test to pass, as would anyone else, I believe.

What Really Happened

The first and third tests failed.

In each case, the test failed on the line where I attempted to assign a value to the Name property. Here is the first part of the stack trace:

TestCase ” failed: Test method ConfigurationAttributeSpike.RawConstructionFixture.CanBeConstructedFromSingleArgumentConstructor threw exception:  System.Configuration.ConfigurationErrorsException: The value for the property ‘name’ is not valid. The error is: The string must be at least 1 characters long..
 at System.Configuration.ConfigurationProperty.Validate(Object value)
 at System.Configuration.ConfigurationProperty.SetDefaultValue(Object value)
 at System.Configuration.ConfigurationProperty.InitDefaultValueFromTypeInfo(ConfigurationPropertyAttribute attribProperty, DefaultValueAttribute attribStdDefault)

What it looks like is happening is that at construction, ConfigurationElement, our base class, initializes all properties attributed with ConfigurationPropertyAttribute to some default value. We have the ability to provide a default value as another field to ConfigurationPropertyAttribute, which will get used. But if we don’t, ConfigurationElement provides its own. It then, if there are any validations for that field, validates this value against any sort of constraint that has been applied to the property under question. If you don’t provide a default value, one is chosen for you, and the validation is done against that value.

The interesting problem that comes up is what happens when the default value that is chosen for you is invalid according to your constraints. This is the situation my tests are in. I have no earthly idea what a good default value would be, so I don’t set one. But not setting one leads me down the road of having these tests not work. Quite a quandry 🙂

And this is the crux of my issue. If you have any validation at all on your configuration classes, your ability to instantiate them outside the framework seems to be greatly compromised. It seems you need to provide a valid default value for them, even in those cases (most case?) where there is no default that makes sense.

What This Means for TDD

When writing code in a TDD style, your tests/examples are much easier to read and understand if everything you need to know is right there in your tests. If you can find a way to put all your data, all your behavior, and all your asserts into a single place, each test begins to stand alone. I believe that this makes it serve its many purposes as a unit test better.

Given that I can’t seem to instantiate a ConfigurationElement that has any validations but has no default values in any way other than having its data reside in an actual XML configuration file, I don’t think I can fulfill my initial goal of having everything in one place. I’m not at all sure how prevelant ConfigurationElements that have validations but no default values are, but it seems like a fairly routine thing to have to me. And in these cases, I’m going to have to externalize my configuration, possibly many times to account for all the different test cases I’m going to want to create.

Am I Just Whining?

Probably. That’s what my wife says I do best 🙂

Looking back over what I wrote, I have to wonder if you think I’m doing more whining about something that is making my life marginally harder 🙂 Well, maybe I am! But this one thing seems like it would be so easy to fix, and would make my tests that involve configuration so much easier to write and understand. I’m a little shy about relying on configuration files for my unit testing, as we incrementally went down that road in Enterprise Library, and it ended up making things very difficult. Tests became very hard to write because you first had to find the right configuration file, and then create or modify it. It became really hard to inject mock objects, because they had to be created by the configuration system, which meant defining them in configuration files, which made them a pain to use and instantiate. Things would have been much easier if the configuration system and the application logic would have been kept separate!

OK, I take back what I said before — it is not just whining — it is the Voice of Experience speaking. Having to work with configuration files in your unit tests is a royal PITA and should be avoided at all costs. I really think this surprising behavior of these objects should be fixed.

I’m planning on filing a bug on this through MSDN, and if you agree that it’s a problem, please vote for that bug. I’ll post the link as soon as I have it.

UPDATE — Here is the link to the bug: FDBK29746. Vote away!!

— bab

3-day public Test Driven Development course in St. Louis, MO, July 13-15

Sorry for the crass commercialism, but I thought I’d post this to my blog to see if anyone was interested.

Agile Solutions Group will be offering another in its series of public Test Driven Development classes, July 13–15 in St. Louis, MO. The class covers both refactoring and test first programming, which together make up TDD. The session is about 20% lecture, 30% example, and 50% lab, so you’ll get a chance to really get your hands dirty with this stuff. I promise close, personal attention for each student and lots of coaching, both during class and afterwards, once you return to work.

Please contact me at bbutton@agilestl.com if you are interested or have further questions.

I return you to your regularly scheduled rants 🙂

— bab

 

Now playing: Dream Theater – Octavarium – The Answer Lies Within

Double Duty — My “Better Software” article as PDF

Thanks to the kind folks at Better Software Magazine and Sticky Minds, I have my article, as published, as a PDF.

In this article, I discuss how to write programmer tests in a little different way to support them being used as API documentation. Tests needs to stand alone a little bit more, you need to pay more attention to using better names, and you need to have some way to let you map from application method to the tests that define it.

Enjoy!

— bab

 

Update:

I know I mentioned my TestMap software in the article, but I got distracted while I was creating it, and I never had a chance to finish it. It is still on my list of things to do, but it is not in any sort of presentable format right now. I’ll post here when I do have a chance to get back to it.