I have not worked with a lot of mocking or fakes frameworks. It always seems to be a topic just low enough on my priority list that I think I should probably do that sometime, but then never actually get to it.
Earlier on in my development career (but after the realization of how important testing is), I wrote a lot of custom test implementations of my interfaces. I’m a pretty large code purist at times, and I loved the feeling of being able to write not just good tests, but control every single aspect of my hand coded mock interface implementation. But truthfully, the added bloat around each test behavior makes your entire job just maintaining the testing code and possibly even afraid to refactor your test code later down the road because you are afraid of breaking it (hmmm, maybe you should have tests for your test… ?).
None of this is new, and you know exactly what I mean. The longer it takes you to maintain a test or write a test increases the likelihood of not even writing the test. Getting rid of the all the boilerplate in test writing is key to keeping on top of maintenance in my workflow. Hence, the rise of the mocking frameworks.
One of the most popular frameworks “Moq” (https://github.com/Moq/moq4) became my first mocking framework that I used years ago. It was free, popular, and did a good job. I had no real quarrel with Moq at all, but some of the more powerful “paid” products, such as TypeMock (http://www.typemock.com), offered additional hooks for shim’s and other faking frameworks that would allow you to even fake out a static usage of practically anything you can imagine – it is still pretty much magic in mind. While using shim’s is a last resort (or generally should be), it is nice to know if all else fails in a tough situation it is there (we’ll assume it is someone else’s code and not your own pristinely dependency injected service model).
In just about all of my experience I have not been in a position to use any paid product for unit testing, which leads us to Microsoft Fakes Framework.
Microsoft Moles launched in Visual Studio 2010, later renamed to the Fakes Framework we enjoy (and curse) today. Initially, The Fakes Framework was ONLY available in Visual Studio Ultimate, but then jumped down to the larger user base on Premium as of Visual Studio 2012 Update 2. This level of accessibility, meant an organization could adopt a full featured Fakes framework (including powerful shim options) for free and fully integrated into Visual Studio context menu to boot (which is exactly what we did at the company I worked at). Usually, when Microsoft adds features into their IDE and Framework they are fantastic and there is no looking back. Unfortunately, this appears not to be the experience with the Fakes framework. I jumped in head first without a second look, and found myself too far invested in it with certain projects to change mocking frameworks. I FEEL TRAPPED! SOMEONE GET ME OUT OF HERE!
The Microsoft Fakes Framework, has the allure of complete framework integration but just falls apart with so many mysterious bugs and lack of documentation. Even more appealing to it is that it seems to work SO GOOD with a small project and has a lot to promise. With my experiences using it on a long term project, I simply won’t go near it again.
Some of the issue’s that I’ve faced that has stolen so much of my precious development and testing time are:
– Random failures to regenerate the fakes DLLs.
– Visual Studio 2013 Update 2 introduced an updated version to the assembly causing some DLL location issues with no help to inform you of such.
– .NET Framework 4.6 install lots of mysterious failure to generate fakes DLLs from System and System.Web.
– Performance in building and running the tests is noticeable, especially with shims.
– Test that use the framework generally suffer from heavy nesting from lambda’s, and complexity greater than the system under test itself.
– Has a high learning curve in comparison to other Frameworks. Mocking a contract method by full method signature is just confusing and hard to read. Seriously, How do this look:
– Random build server failures to generate fakes leave you looking like an idiot after checking in your code that passed ALL tests locally (ummmm… works on my machine).
– Build output seems to spit out lots of different warnings about Fakes Framework usage.
– IDE visualization will fail to recognize the Fakes DLL references sometimes, and show you a screen full of RED unrecognized objects, and zero intelli-sense help, but everything continues to build.
Last week, we were kicking of integration and unit tests for a new project, and I wasn’t about to make the same mistake again with the Fakes Framework. I was settled to go back and use Moq again, but a colleague suggested a framework he had been eyeing up recently called “NSubstitute” (http://nsubstitute.github.io/). Appears to be very stable, and has been around since 2011. Looking at the documentation, it just seems TOO SIMPLE! How can this do everything I need? Well quite frankly, it is simple, and doesn’t do everything, but it gets you most of the way there, which is pretty far, and deals away all previous hassles I had to deal with.
I was delighted by the fact that the mocking behaviors that you add to each method are all built with extension methods, and the mocked interfaces are simply your contracts only. There are no extra special mocking methods it adds in. This makes the code so much more readable and also brief since you are not fooling around with the syntax of the mocking framework. In fact, that’s what NSubstitute boasts about is that their syntax (or syntactic sugar) is second to none, and I like it!
Check out how simple it is to stub a service contract method:
It is literally that simple. Check out the documentation:
It is well organized, easy to navigate and allows you to be up and running in 20 minutes with a wide understanding of the entire framework. This makes integration with teams and others much easier.
There was a single thing, so far, that I found myself wanting from NSubstitute that I didn’t see immediately how to get. That was the ability to mock a return from a method based on the incoming parameters that were sent to the mock (which could be anything, completely dynamic). While I didn’t see it right away, the framework does all for this in a pretty simple way (perhaps just wording of the API through me off as I am use to Fakes Framework).
If I wanted to return from the email service a success indicator on sending based on the length of the email being sent I could do something like this:
With this type of mock, ensure you are using “ReturnsForAnyArgs” instead of just “Returns”, and then locate the argument by index (alongside the generic ArgAt helper) and you are good to go. You could also perform assertions inside the lambda, or assign to variables outside the lambda as you would expect to finish your assertions a bit later.
Great job to the team behind NSubstitute. I am looking forward to utilizing this framework in some sticky situations ahead and seeing how it holds up.
What testing frameworks are you utilizing? How are they holding up?