How to test dependencies on external services?
I’ve been writing a lot of code lately that depends on external services, and I’ve really been struggling with how to test it well.
Tinder
The tests for Tinder, the unofficial Campfire “API”, are pathetic. Partially because it was originally just a proof of concept and partially because I just didn’t have a clue about how to write tests that would actually tests and not just execute the code.
The problem with Tinder is that it is really hard to isolate the tests. By default, every test will have 2 points of failure, the code to set up the test and the code to verify the result.
For example, here would be the test for deleting a room:
c = Tinder::Campfire.new 'mycampfire'
c.login('email', 'pass')
room = c.create_room 'My Test Room'
assert room.destroy
assert !c.find_room_by_name('My Test Room')How do I know that it is working? All I’m testing is that #destroy returns true and that Tinder thinks that it is gone. Is that good enough? To make matters worse, every other test will depend on at least login with whatever other setup code is required.
Ideally, each API method would be tested by making assertions on the HTTP request/response.
Graticule
With Graticule, I decided that instead of requiring API keys for every service and hitting up against them with every test (especially since some services are pay-per-request), I would just save sample responses and mock the HTTP connection. This works really well as far as unit testing goes, but poses problems if services change their APIs.
Ideally, what I need is to be able to run the exact same test suite locally and remotely, but I haven’t figured out a good way to do the setup.
What is a wanna-be tester to do?
I could keep talking about examples (LDAP, CalDAV, etc.), but you get the point.
Testing isn’t supposed to worry about the internals of the code and only test the API, but with something like Tinder, how do you properly test it without peeking into the guts? What are some things you are doing to handle external dependencies?
4 comments
My preferred method is to cache the service’s responses locally, and then flush the cache periodically in order to detect breakage on the service’s end. Ideally, I’d run a CI server that would flush the cache and re-run the tests with every checkin. That way, my local tests could always used the cached responses (keeping them speedy, and able to run off-line), but I’d still get quick notification if the remote service changed.
April 25, 2007 at 03:52 PM
Using Mock Objects will helps me tremendously. Some people view it as “peeking into the guts” however, I strongly believe that you are asserting behavior. When you mock out the external dependencies you are allowed to test the object in isolation.
You become more intentional on what you want to test as you can easily simulate behaviors the external dependencies should exhibit. Just as important, you can assert your code’s interaction with the dependencies (asserting the methods / parameters used).
I’m not sure about the underlying code for Tinder, but seeing the snipbit given I’m going to assume that objects are instantiated within Tinder::Campfire. This becomes a thorn in mock testing since it’s not as easy to inject the dependencies. Two solutions that I generally take are a) Use some type of Dependency Injection like Needle or b) use stubbing capabilities to whichever mock library you’re using, for example both Mocha and Flexmock provide these. Just be wary of overuse of stubbing.
April 25, 2007 at 04:59 PM
Scott: Excellent idea.
Zach, mocking, stubbing, and dependency injection work great when you’re testing code that uses other code that is the external dependency, but when your code
ISthe dependency, it doesn’t help much. Somehow I need to prove that my code does what it’s supposed to when interacting with the external service.April 25, 2007 at 05:24 PM
At first I thought it would help you to read the “Don’t mock third-party libraries” section of this article. But then I realised Tinder is actually meant to be the “adapter layer” that Steve Freeman talks about.
So then I thought you might be better off doing something like what’s described in this article
I hope that’s of some help.
April 25, 2007 at 06:14 PM
Speak your mind: