opensoul.org

For those that haven’t heard: Cucumber is pretty much the greatest thing since sliced bread. It dramatically improved the quality and stability of our applications, and the outside-in approach that it encourages forces you to stay focused on what’s important.

When we started working on a Facebook application a few months ago, we couldn’t fathom not using Cucumber. So we had to figure out a way to test it. It took us a few months to evolve it to a point where we could extract it, but this week we pushed a change to Facebooker to make life a little easier. So grab the latest version of Facebooker and keep on reading…

First, in features/support/env.rb, replace the default Rails world with the one in Facebooker:

# require 'cucumber/rails/world'
require 'facebooker/rails/cucumber'

Given I am logged in as a Facebook user

Most of our Facebook application requires that a user be logged in. So most of our scenarios started with “Given I am logged in as a Facebook user”.

Scenario: Uploading a video
  Given I am logged in as a Facebook user
  When I upload a video
  Then I can see a video on my blog

And here is our implementation for that step:

Given "I am logged in as a Facebook user" do
  @current_user = User.create! :facebook_id => 1
  @current_user.facebook_user.friends = [
    Facebooker::User.new(:id => 2, :name => 'Bob'),
    Facebooker::User.new(:id => 3, :name => 'Sam')
  ]
  @integration_session.default_request_params.merge!(
    :fb_sig_user => @current_user.facebook_id,
    :fb_sig_friends => @current_user.facebook_user.friends.map(&:id).join(',')
  )
end

Our application has a User model with a facebook_id attribute and a #facebook_user method which returned an instance of Facebooker::User. Due to how the Facebook session is being mocked, it is important that we set our fake user’s id to 1 for now (I’ll try to figure out a way around this). We also manually add some friends for our application to use. Lastly, we merge in our user’s id and friend ids into the default request params so that any requests we make include those parameters.

Drop your…canvas

There were some places in our app where requests don’t go through the canvas. For example, we have a few multipart forms, which have to submit directly to your application. To mimic this, wrap webrat calls in #without_canvas:

When "I upload a video" do
  visit root_path

  without_canvas do
    fill_in 'Title', :with => 'A video'
    fill_in 'Description', :with => 'Caption for video'
    attach_file 'Video', "#{RAILS_ROOT}/features/support/sample.mpg", "video/mpeg"
    click_button 'Upload Video'
  end
  follow_redirect!
end

Note that if your action redirects to a URL with :canvas => true, webrat will see that as an “external” redirect and won’t follow it. Just call #follow_redirect! and it’ll go on it’s merry way.

Accessing the Facebook API

Instead of making requests to the Facebook API in your tests, Facebooker will try to read a canned response fixture from features/support/facebook/. It will give you a friendly error whenever this happens, so just follow the directions and you should be on your way.

Feedback & Patches

I’m sure there are things that don’t work right, so let me know if you run into any troubles. If you have any ideas for making the Cucumber support better, please share them here or on the Facebooker mailing list.

The Great Lakes Ruby Bash was a good time. There were several quality presentations, including Jim Weirich’s talk “Playing it Safe – How to write library friendly code in Ruby”, Larry Karnowski’s talk “Usability on Rails”, and Brandon Dimcheff’s “Metawhat? A look into the mysterious metaclass”.

I presented a talk titled “Behavior Driven Development with Cucumber”. Despite the fact that half of the audience didn’t know what the hell I was talking about, I think it went well. I uploaded the slides from my talk to slideshare if you’re interested. I’m not sure that they’ll really be very helpful, but I may try to record audio to go with them at some point.

As far as I know, no audio or video was captured at the conference.

A while ago I posted a configuration for getting autotest to work with Test::Unit outside of Rails. Ryan Davis, author of autotest, commented on that post saying that it should “Just Work™” without any custom configuration. I was perplexed because I’ve never been able to get it to work on my gems and Rails plugins.

I finally took time to look into the issue, and realized it’s because I always use the Rails naming conventions for my test files. I name them foo_test.rb, instead of test_foo.rb, which is what Autotest looks for.

That’s easily solvable. Here’s an Autotest configuration, tested with ZenTest 3.10.0, that should make it work for either naming convention. You can throw this in your ~/.autotest file, or in a .autotest file inside your project.

Autotest.add_hook :initialize do |at|
  at.clear_mappings

  at.add_mapping %r%/^lib/(.*)\.rb$% do |_, m|
    possible = File.basename(m[1])
    files_matching %r%^test/.*(#{possible}_test|test_#{possible})\.rb$%
  end

  at.add_mapping(%r%^test/.*\.rb$%) {|filename, _| filename }
end

Happy autotesting.

All the cool kids tell me that fixtures just aren’t the “in” thing anymore. Speaking of which, when did fanny-packs stop being cool? I wish someone would have said something…

Anyway, while I still haven’t fallen out of love with (foxy) fixtures, I have taken some of the home-grown alternative methods for a spin, but so far, I’ve been slightly frustrated with my experience.

The predominant replacement seems to be the “factory” pattern, talked about by Dan Manges and then followed up by the FixtureReplacement plugin. The basic idea is that all your tests/specs should setup the data that they require, and there is a factory that makes it easy to create the necessary valid test data. For example, a homegrown solution might look like this:

class Generate
  def self.person(options={})
    p = Person.create!({
      :name     => 'Brandon',
      :login    => 'brandon',
      :password => 'testing'
    }.merge(options))
    p.account ||= Generate.account
  end
end

describe Person, '#authenticate' do
  it 'should return the person record if successful' do
    person = Generate.person
    Person.authenticate(person.login, 'testing').should == person
  end
end

This method scales pretty well. There are some issues, such as the use of create!. Sometimes you intentionally want an invalid record, or you want a new record with just some valid attributes initialized. It’s also slower than fixtures, but I don’t care too much about that right now.

But there’s a bigger issue. Like a good little BDDer, I have also been stubbing and mocking all my “unit tests” so there is no interaction with the database or code outside of what is being tested. To make the factory method work, I now need to define 3 different methods depending on what I’m testing: one using #create!, one using #new, and another one using stubbing.

What do we do about it?

I think the FixtureReplacement plugin is on the right track. It handles the new vs. create problem very nicely.

module FixtureReplacement
  attributes_for :person do |u|
    u.name     = String.random
    u.login    = String.random
    u.password = String.random
    u.account  = default_account
  end
end

@person = new_person(:login => 'brandon')
@person = create_person

Before I heard about the FixtureReplacement plugin, I actually concocted my own little solution that handles all three scenarios. I’m not crazy about the syntax, but it works for my needs.

Generate.add Person, :name => 'Brandon', :login => 'brandon' do |generator, u|
  u.account ||= generator.account
end

@person = Generate.person(:login => 'brandon')
@person = Generate.new_person
@person = Generate.stub_person

Anywho, for the fixture-less approach to work, I think stubs need to be supported.

Is all this really worth it?

That’s the question I find myself asking. What’s wrong with fixtures anyway?

One thing I really like about fixtures, when done right, is that they help tell the story of your application. You get to know the fixture data as you work on the app.

The factory method also seems to go against convention-over-configuration. Instead of having a default “configuration” when your tests run, you have to configure each test. Call me lazy, but that just seems like too much work.

I find that testing takes a lot more effort now than it did with fixtures. And as a result, I’m more hesitant to test everything. So while the factory method and stubbing is theoretically supposed to help you test better, I feel like they’ve had the opposite effect.

What do you think? Have you had success using the factory pattern for tests?

Autotest is a gem (well, technically ZenTest is a gem that includes autotest). But seriously, autotest is a gem! I can’t write code without it anymore. It has become my coding security blanket.

I started going through withdrawal last night while I was working on a Ruby library because autotest doesn’t work out of the box with plain ol’ Ruby projects. It assumes you’re using one of the Ruby web frameworks.

Here’s how to remedy that.

Aizatto shows us how to use autotest with your Rails plugins and RSpec. That’s a great start, but unfortunately, not all of my projects use RSpec (yet), and not all of them are Rails plugins.

To make autotest work with Test::Unit, we need to tell it how to map files to their tests. Here is a simple mapping that maps each ruby file in lib/ to the a corresponding test test/*_test.rb. For example, lib/foo/bar.rb would map to to a test in test/foo/bar_test.rb.

# autotest/testunit.rb
require 'autotest'

class Autotest::Testunit < Autotest
  def initialize # :nodoc:
    super
    @exceptions = /^\.\/(?:config|doc|log|tmp|website)/

    @test_mappings = {
      %r%^test/.*\.rb$% => proc { |filename, _|
        filename
      },
      %r%^lib/(.*)\.rb$% => proc { |_, m|
        ["test/#{m[1]}_test.rb"]
      },
      %r%^test/test_helper.rb$% => proc {
        files_matching %r%^test/.*_test\.rb$%
      },
    }
  end

  # Given the string filename as the path, determine
  # the corresponding tests for it, in an array.
  def tests_for_file(filename)
    super.select { |f| @files.has_key? f }
  end
end

We have the mapping, now all we need to do is tell autotest about it.

# autotest/discover.rb
require File.dirname(__FILE__) + '/testunit'

Autotest.add_discovery { "testunit" }

Throw both of those files into a autotest/ directory in your plain ol’ Ruby project and autotest should just work for you.

Extra credit: Make it work with RSpec on plain ol’ Ruby projects.

The theory is that tests are supposed to be agnostic of the implementation. This leads to less brittle tests and actually tests the outcome (or behavior).

With RSpec, I feel like the common approach of completely mocking your models to test your controllers ends up forcing you to look too much into the implementation of your controller.

Here is an example of a spec that is generated by the rspec_scaffold generator:

describe ThingsController, "handling POST /things" do

  before do
    @params = {}
    @thing = mock_model(Thing, :to_param => "1", :save => true)
    Thing.stub!(:new).and_return(@thing)
  end

  def do_post
    post :create, :thing => @params
  end

  it "should create a new thing" do
    Thing.should_receive(:new).with(@params).and_return(@thing)
    do_post
  end

  it "should redirect to the new thing" do
    do_post
    response.should redirect_to(@thing)
  end
end

This by itself is not too bad, but the problem is that it peers too much into the controller to dictate how the model is used. Why does it matter if my controller calls Thing.new? What if my controller decides to take the Thing.create! and rescue route? What if my model has a special initializer method, like Thing.build_with_foo? My spec for behavior should not fail if I change the implementation.

This problem gets even worse when you have nested resources and are creating multiple models per controller. Some of my setup methods end up being 15 or more lines long and VERY fragile.

RSpec’s intention is to completely isolate your controller logic from your models, which sounds good in theory, but almost runs against the grain for an integrated stack like Rails. Especially if you practice the skinny controller/fat model discipline, the amount of logic in the controller becomes very small, and the setup becomes huge.

So what’s a BDD-wannabe to do? Taking a step back, the behavior that I really want to test is not that my controller calls Thing.new, but that given parameters X, it creates a new thing and redirects to it.

Here’s the approach I’ve been taking:

# spec_helper.rb
def valid_thing_attrs(attrs = {})
  {:name => 'Foo'}.merge(attrs)
end

# things_controller_spec.rb
describe ThingsController, "handling POST /things" do

  def do_post
    post :create, :thing => valid_thing_attrs
  end

  it "should create a new thing" do
    lambda { do_post }.should change { Thing.count }.by(1)
  end

  it "should redirect to the new thing" do
    do_post
    response.should redirect_to(@thing)
  end
end

My spec is now testing that posting certain attributes creates a new thing (in the database) and redirects to it. You’ll notice that I have a method called valid_thing_attrs. I don’t remember where I picked up this pattern, but it is something that has allowed me to minimize all of my test’s dependencies on the model.

My spec is now somewhat dependent on my model–and thus will break if my model gets hosed–but that is a small price to pay, in my opinion, for an implementation-agnostic spec with significantly less overhead.

What do you think?

I’ve been writing a lot of code lately that depends on external services, and I’ve really been struggling with how to test it well.

Tinder

The tests for Tinder, the unofficial Campfire “API”, are pathetic. Partially because it was originally just a proof of concept and partially because I just didn’t have a clue about how to write tests that would actually tests and not just execute the code.

The problem with Tinder is that it is really hard to isolate the tests. By default, every test will have 2 points of failure, the code to set up the test and the code to verify the result.

For example, here would be the test for deleting a room:

c = Tinder::Campfire.new 'mycampfire'
c.login('email', 'pass')
room = c.create_room 'My Test Room'
assert room.destroy
assert !c.find_room_by_name('My Test Room')

How do I know that it is working? All I’m testing is that #destroy returns true and that Tinder thinks that it is gone. Is that good enough? To make matters worse, every other test will depend on at least login with whatever other setup code is required.

Ideally, each API method would be tested by making assertions on the HTTP request/response.

Graticule

With Graticule, I decided that instead of requiring API keys for every service and hitting up against them with every test (especially since some services are pay-per-request), I would just save sample responses and mock the HTTP connection. This works really well as far as unit testing goes, but poses problems if services change their APIs.

Ideally, what I need is to be able to run the exact same test suite locally and remotely, but I haven’t figured out a good way to do the setup.

What is a wanna-be tester to do?

I could keep talking about examples (LDAP, CalDAV, etc.), but you get the point.

Testing isn’t supposed to worry about the internals of the code and only test the API, but with something like Tinder, how do you properly test it without peeking into the guts? What are some things you are doing to handle external dependencies?

One of the things I always check when spec’ing is that the model that I just created is valid.

@mymodel.should be_valid

This uses RSpec’s predicate support (any call to be_something? calls something? on the target object), and returns a correct but very unhelpful error:

expected valid? to return true, got false

Here is a little rspec matcher that I threw together this morning to check if a model is valid and spit out the validation errors:

module Spec
  module Rails
    module Matchers
      class BeValid  #:nodoc:

        def matches?(model)
          @model = model
          @model.valid?
        end

        def failure_message
          "#{@model.class} expected to be valid but had errors:\n  #{@model.errors.full_messages.join("\n  ")}"
        end

        def negative_failure_message
          "#{@model.class} expected to have errors, but it did not"
        end

        def description
          "be valid"
        end

      end

      def be_valid
        BeValid.new
      end
    end
  end
end

Now, I get:

MyModel expected to be valid but had errors:
  Password confirmation can't be blank
  Email has already been taken

This is nothing fancy, but it’s the small stuff that make life meaningful, so I thought I would share it. Throw the matcher code above into your spec_helper and check if your models are valid.

update: added negative_failure_message so “model.should_not be_valid” will work

Build still broken and getting worse

Today Cerberus officially became my tool of choice for continuous integration when it sent me an email with that subject. I’ve been using Cerberus recently and I really like it. It’s simple and it works, which is more than I can say for any other continuous integration tool that I’ve ever used.

I played with breakpointer for a few minutes to check it out when I was first learning Rails, but I’ve never actually used it for serious debugging. But yesterday I had a unit test that was failing and I could not figure why, so I decided I would try it out. I discovered that breakpointer works really well for debugging tests! I had only used it in the context of debugging actions.

Using the breakpointer is really easy, just insert breakpoint at any point in your code. When you run your tests, it will open up an irb session whenever it encounters a breakpoint in your code.

.....Executing break point at ./test/unit/event_test.rb:36 in `test_duration'
irb(test_duration_with_hash=(EventTest)):001:0>