My first day using VSTS

My group at MS is considering changing over to using the new unit testing stuff built into the newest version of Visual Studio, so I decided to give it a try. I’ll be keeping a diary of how things go today, and hopefully I’ll learn something interesting.

The first thing I’ve learned is that I can’t wait for the ReSharper EAP  I need the navigation and refactoring tools I get with that. Boy I miss them in Whidbey…

So, on to the show!

How does this thing work?

I have no idea how it works. I know how NUnit works. I know how JUnit works. I don’t know how this works. Google, however, pointed me to here: In this link, the author explains how to create, run, and manage tests. Read it before you go on.

My first test

I’m rewriting the instrumentation stuff that goes along inside Enterprise Library. The first version of it didn’t do everything that we wanted, so I volunteered to rewrite it to make it more OO, more easily separable from the calling code, and hopefully able to be driven from configuration. I’m not sure how I’m going to do any of those things yet, so I plan to play around and learn how to do it today.

So, my first learning is that there are things called test projects, and they have some sort of relationship to your regular projects. There is a wizard to create them, which means I don’t have to. We’ll see if I like that or not — I’m generally not a wizard fan. And it turns out that you don’t need to use the wizard to create these things. You can just do a regular Add Project and choose Testing Project. The nice part about working with a TestingProject is that you get some additional choices on the Add menu — things like adding new tests of various kinds. In fact, I just did that, and I appear to have broken the IDE. It locked up, and it is currently using 99% of my CPU. Thus is the way of beta software…

So, restarting… Doh! It came back just as I was about to kill it…

So, saying Add New Test brings up a dialog where you can choose the kind of test you want to add. There are generic tests, load tests, manual tests, web tests, and finally, unit tests. There is a choice to just create a test and one that will drive you through a wizard that will create test stubs for you for all existing methods. I’m going to use the manual test generation method, because I hate tools the generate tests for me. One annoyance about this dialog box is that I have to put a .cs suffix on my Test Name before OK is enabled.

After choosing OK, I’m presented with this test shell:

using System;
using Microsoft.VisualStudio.QualityTools.UnitTesting.Framework;

namespace InstrumentationTests
    /// Summary description for InstrumentationExplorationFixture
    public class InstrumentationExplorationFixture
        public InstrumentationExplorationFixture()
            // TODO: Add constructor logic here

        /// Initialize() is called once during test execution before
        /// test methods in this test class are executed.
        public void Initialize()
            //  TODO: Add test initialization code

        /// Cleanup() is called once during test execution after
        /// test methods in this class have executed unless the
        /// corresponding Initialize() call threw an exception.
        public void Cleanup()
            //  TODO: Add test cleanup code

        public void TestMethod1()
            // TODO: Add test logic here

Not bad, although I’ll probably change the template to get rid of those annoying comments. In this test you see some attributes that are very similar to NUnit’s attributes. TestClass appears to map to TestFixture, TestInitialize => SetUp, TestCleanup => TearDown, TestMethod => Test. It also supports ExpectedException, just like NUnit, and it has the equivalents for TestFixtureSetUp and TestFixtureTearDown, and one new one, something that lets you set up just after your assembly loads and right before it unloads. These are called AssemblyInitialize/AssemblyCleanup and ClassInitialize/Cleanup.

At this point, I started writing my first test. And I was thinking so hard about the test, that I forgot I was using the VSTS testing framework (btw, what is the short name for this thing? NUnit versus ???). I just started writing a test, and everything just worked for me. The first change I saw was that the Assert.AreEqual method is not a generic. The old one still exists, but with the new one, you can specify the type of the things to compare, and a certain class of errors is now caught at compile time. Here is my first test:

        public void CreateAndClearCounter()
            EnterpriseLibraryPerformanceCounter counter = new EnterpriseLibraryPerformanceCounter();

            Assert.AreEqual(0, counter.Value);

I’m following a practice here called Coding by Intention. I have not written a thing yet, but I’m writing my code as if I have all of these methods and classes already in place. I’ll create them as I go to make this test pass. But coding like this helps you think about the abstractions you need to perform your task. So I obviously need to create an EnterpriseLibraryPerformanceCounter (I doubt this name will last very long. We have nothing else called EnterpriseLibrary*, so why should this be called this).

On to the second test

OK, so I’ve written and implemented that test, and it was pretty easy. It really felt as simple as using NUnit. The only problem right now is that TestDriven.Net isn’t working. I’ve talked with Jamie, and he’s working on a solution for that. So now its time for that second test. Here it is:

        public void CounterCanBeIncremented()
            EnterpriseLibraryPerformanceCounter counter = new EnterpriseLibraryPerformanceCounter();

            Assert.AreEqual(1, counter.Value);

Again, a really simple test. Now I try to run it…It’s only running one test. Damn. I’m trying to figure out why it is only running one test, but it shouldn’t be this hard.

Well, I got the other test to run. I had to physically move it before the first test, and then it got picked up. Then I moved it back to where it was, and it worked there, too. Very strange, but I’ll definitely keep an eye on this behavior.

Test number 3

OK, now that I have some basic counting functionality in place, I want to get it so that I can have multiple instances of the same counter. I write a new test, write the code to get it to compile and fail, and it does. In fact, all three tests fail now. And I have no idea how to know where they’re failing.

The way that this test stuff works here is that there is a pane called Test Results. Double-clicking on a test gives you a new view that shows the test name, how long it took, etc, as well as any error message and the stack trace. But the stack trace isn’t clickable. I’m trying to figure out how I can get from my stack trace to the line of code that failed. There are no line numbers in the stack trace either, so I can’t get there manually. This is more than a little frustrating. It is not working as I expect or want at this point. Not happy…


I’m wrapping up my work for the day now, and I have some conclusions. Please take these conclusions with a huge grain of salt, since I just started working with this tool. I’m uncomfortable with it, it is not natural yet, so it is to be expected that I wasn’t as productive as I would be using my old tools.


I find test authoring to be the same. For the basic kinds of tests I’m doing, it is exactly like using NUnit. There are a bunch more attributes that you can use to let you do things that may or may not be wise, but they are available. What I don’t like, at all, is how I have to run the tests. This is completely a product of not having TestDriven.Net installed. I can’t pick and choose the tests I want to run easily, and I can’t run just a single test easily. I know I can click on the check boxes in the Test Results windows to choose which tests to run, but that just seems foreign to me right now. And when the tests run, they seem to take a long time. The reported time is only a fraction of a fraction of a second, but the subjective time the tool spends starting stuff up seems to be pretty long,and it doesn’t get shorter for subsequent runs of the same tests. TD.Net has these same issues at times, and it might just be spinning up the AppDomain for the test, etc. It just feels like a long time.

The biggest disappointment to me is how hard it is to figure out how to find the line where a test broke. I’m sure I’m just missing something simple, as no one would create a test tool where you couldn’t jump around in the stack — I’m just missing how to. But if I’m missing it, does that mean that it’s too hard to find?

Stay tuned for more tales of my travels through this testing framework. There is a lot to learn with it, as it is much more than a TDD framework. My problem is going to be finding and choosing just those parts I need.

— bab


9 thoughts to “My first day using VSTS”

  1. I tried the VS unit stuff in the Oct CTP. It was just as hard, then. I spent a lot of time to reorder and reconfigure several windows, just to discover that VS didn’t care to remember my settings.

    No need to say it didn’t take long before NUnit and TestDriven was installed.

    Your tale suggests the usability problems still has to be solved.

  2. I have begun checking out the testing in vs too. With NUnit, you can test ui. NUnitAsp.NET for web apps and NUnitForms for Windows forms. Do you know if there is anything similar in vs ? I have not seen anything yet but I am just starting.

  3. Brian,

    Thanks for looking at our unit testing functionality and sharing your comments.

    Your disappoinment about not seeing a line number for a broken test is a bug we’re working on fixing right now. Hopefully this makes your disappointment short-lived.

    Regarding the ease of running tests, be sure to make use of the Test View window. This window defaults to snapping to the same region as the Solution Explorer so it doesn’t take any additional real estate from your coding window. What it provides you, though, is a way to group your tests by type, owner, and other properties so that you can run one or more tests quickly while you’re authoring tests.

    Have you had a chance to run tests in debug mode? This is an option available in Beta 2 as well, and helpful when you want to see exactly what’s happening as your tests are executing.

    Regarding spin-up time for the test execution, this is likely the "code coverage" support we have that is enabled by default. We instrument the code that you’re testing to help you see the effectiveness of your tests. You can edit your active Run Configuration and turn off code coverage support to avoid this step when executing tests.

    I hope this helps. Best regards,

    Tom Arnold

    PM, VSTT

  4. Tom, what are your testing scenarios? Referring to the Test View Window is not very user friendly. Why should I have to go to another window to run the test I was just writing? And why do I have to spend time to group my tests?

    Here’s what I do with the TD.NET addin:

    1. Add a test class

    2. Add a test method

    3. Right-click inside of the test method, select Run Test(s)

    4. Add another test method, right-click etc.

    5. Right-click on the class level, select Run Test(s) and see all the test runs.

    6. Right-click on the namespace or the project, right-click etc…

    A short-cut would be nice, but the right-click works very well. And there is no need to group anything, the namespaces do just that.

  5. I love the integrated code coverage you get from this stuff too. I can see my unit tests missing code, which causes to either improve a test or create a new one to cause a new execution path in the target code. Cool.

  6. I guess being in QA for all these years has warped my sense of ease-of-use, hehe. When A .webtest is open it is not possible to do a find/replace it took me 30 minutes to slow down and read the stupid warning about "do you want to close this file", well when I finally got so pissed and clicked Yes close the file, an entirly different view of the file was opened and the replace worked as expected.

    Odd, very odd.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.