opensoul.org

I’ve pushed out the delayed_job 2.0 gem from the Collective Idea fork on GitHub. See the changelog for a summary of changes, or see the full list changes.

Multiple Backends

One of the most significant changes was adding support for multiple backends. You can now use Active Record, MongoMapper, or DataMapper as backends for your job queue. See the README for more info.

Benchmarks

The Active Record backend in delayed_job 2.0 is much faster (6x in the benchmarks I ran), primarily due to reversing the priority column and adding an index. Here are benchmarks for running 10,000 simple jobs on my laptop:

                      user     system      total        real
delayed_job 1.8.5 195.670000  14.020000  209.690000 (230.887172)
delayed_job 2.0    36.200000   0.940000  37.140000  ( 39.959233)

While we’re looking at benchmarks, here is how the current backends compare:

                     user     system      total        real
active_record      36.200000   0.940000  37.140000 ( 39.959233)
mongo_mapper       69.270000   3.220000  72.490000 ( 90.783220)
data_mapper       255.620000   2.880000 258.500000 (275.550383)

I have not done anything to optimize the mongo_mapper or data_mapper backend, so performance patches would be appreciated.

Upgrading

To take full advantage of the Active Record performance improvements, you’ll want to add an index:

add_index :delayed_jobs, [:priority, :run_at], :name => 'delayed_jobs_priority'

The only other issue that most people will run into is that all of the configuration options have been moved to Delayed::Worker. Here’s how to change the options now:

Delayed::Worker.destroy_failed_jobs = false   # Delayed::Job.destroy_failed_jobs = false
Delayed::Worker.max_attempts = 3              # Delayed::Job.const_set("MAX_ATTEMPTS", 3)
Delayed::Worker.max_run_time = 5.minutes      # Delayed::Job.const_set("MAX_RUN_TIME", 5.minutes)
Delayed::Worker.sleep_delay = 60              # Delayed::Worker.const_set("SLEEP", 60)

Feel free to post any comments or questions on the mailing list.

The Great Lakes Ruby Bash is now accepting talk proposals. The conference will be held on Michigan State University’s campus in East Lansing, Michigan on Saturday, April 17th.

We’re looking for passionate speakers to give 25 and 40 minute presentations about their experiences with Ruby and related technologies. Our goal is to engage attendees and inspire them to create great software, empower users, and continue learning with others.

Proposals are due Feb 28th, 2010 at 12:59pm EST. We hope to have all proposals reviewed and speakers chosen by March 8th, 2010. Visit the website for more info.

Sponsors

We are also looking for companies and freelancers to sponsor the event. This year’s conference will be the third Ruby conference in this region. The previous two conferences attracted 60-70 attendees each. As a result of more interest, greater community involvement, and a more aggressive marketing campaign, we are anticipating 100-150 attendees this year.

Visit the sponsorship page to become a sponsor or find additional information about available sponsorship packages.

Sphinx, everybody’s favorite search engine, has support for location-based search, giving you geo-aware, full-text searching. So now you can find all of the garage sales on Saturday within 20 miles that have LPs and a reel mower.

All you need to do is add latitude and longitude (in radians) to the index, allowing you to limit the results to records within a distance of a point. The hardest part is getting the coordinates, but acts_as_geocodable makes that really easy.

To start, install acts_as_geocodable. Once you have that configured properly, install ThinkingSphinx, define an index on your geocodable model and add the coordinates to the index:

class Listing < ActiveRecord::Base
  acts_as_geocodable

  define_index do
    indexes title
    indexes description

    has geocoding.geocode(:id), :as => :geocode_id
    has 'RADIANS(geocodes.latitude)', :as => :latitude, :type => :float
    has 'RADIANS(geocodes.longitude)', :as => :longitude, :type => :float
  end
end

The three lines that start with has add the geocode id, and the latitude and longitude in radians to the index. Our index doesn’t need the geocode id, but we have to add it so that ThinkingSphinx properly joins the geocodes table.

After you rebuild the index and start the daemon, you can search for records by location. Here’s an example of taking a zip code from a user and finding all records with in 20 miles. (Note: you will need to grab the latest version, 0.2.9, of Graticule for this next bit of code to work)

def search
  location = Geocode.geocoder.locate(params[:zip]).coordinates.map(&:to_radians)
  @listings = Listing.search(params[:q], :geo => location,
    :with => {'@geodist' => 0.0..(20 * 1610.0)})
end

After looking up the coordinates of the zip code that the user entered, we do a search with the :geo parameter, limiting the results using the special @geodist condition. We have to pass in a range of floats that represent the distance of the points in meters (and since the US is in the stone age, I’m converting from miles).

That’s all there is to it. Now go write some cool location-based search and comment here about it.

The interwebs are all a’buzz about Rack. For those that haven’t been following along, Rack is a specification and a library for connecting web servers to Ruby libraries (a.k.a. “Middleware”). It’s basically the Web 2.0 version of CGI, except that it doesn’t suck and it’s just for Ruby.

Rails 2.3 has Rack baked in. It uses Rack for things like sessions and parameter parsing. But what if you want to add your own middleware to a Rails app?

It’s really easy! In the init.rb of a plugin, or just in a Rails initializer:

ActionController::Dispatcher.middleware.use MyMiddleware

This will append your middleware to the end of the “stack”, so it will be executed after all of Rails’ middleware, but before anything else in the Rails framework.

But what if you need to massage request parameters before Rails parses them? All you need to do is insert your middleware in the stack before Rails parses the params. You can find the current list of middleware by inspecting ActionController::Dispatcher.middleware. In there you’ll find ActionController::ParamsParser, which is what we want to insert our middleware before. Unfortunately, there’s not an insert_before method (yet), so we’ll need to use insert_after, giving it a middleware earlier in the stack:

ActionController::Dispatcher.middleware.insert_after 'ActionController::Failsafe', MyMiddleware

If you don’t like the way that one of the existing pieces of middleware handles things, you can swap it out for your own version:

ActionController::Dispatcher.middleware.swap 'Rails::Rack::Metal', HeavyMetal

So there you have it. Eat, drink, and go write middleware.

MapQuest has sent out several emails about their current geocoding API being discontinued. Here is the latest:

Dear MapQuest Platform Customer,

At the start of the New Year, MapQuest will be retiring the MapQuest OpenAPI product, having launched the more feature rich MapQuest Platform: Free Edition product. Since the MapQuest OpenAPI does not use the same backend as our newer APIs, nor does it provide the breadth in functionality, we want to provide you with a better free experience. Don’t wait – make the switch today!

If your application is currently being powered by the MapQuest OpenAPI product, you will need to migrate to one of 6 APIs available in the MapQuest Platform: Free Edition product prior to January 31, 2009.

Our MapQuest Platform: Free Edition product offers more flexibility and ease of development along with providing developer choice with six APIs:

SERVER SIDE APIs Java C++ .NET

CLIENT SIDE APIs JavaScript AS3 (ActionScript 3: Flash, Flex, AIR) FUJAX (Write JavaScript, output Flash)

Our MapQuest Platform: Free Edition product includes many additional features including:

• COLLECTIONS: Support for multiple and remote collections (KML and GeoRSS); allowing easier handling of shape collections.
• ADVANCED SHAPE OVERLAYS: Build applications that allow users to create and interact with a variety of overlays on maps, including custom lines, polygons, rectangles, and ellipses.
• CUSTOM TILE LAYER SUPPORT.
• Add REAL-TIME TRAFFIC to your map.
• GLOBE VIEW: http://globe.mapquest.com.
• AERIAL IMAGERY and HYBRID VIEWS.
• SMART ROLLOVERS: Rollover windows that adapt their size and positioning based on the content placed in the window.
• ADVANCED MAP MARKERS: With the MapQuest “declutter mode,” automatically move collided markers to positions on the map with a leader link pointing to their original location.

The MapQuest OpenAPI product servers will go offline on Saturday, January 31st, 2009. Please plan on migrating your application before this date or applications based on the MapQuest OpenAPI product will stop working.

You can find documentation and downloads for the MapQuest Platform: Free Edition product on our Developer Network: http://developer.mapquest.com.

Additional information can be found on: http://platform.mapquest.com & http://devblog.mapquest.com.

Thank you,

MapQuest, Inc.

Graticule, a ruby wrapper around many popular geocoding APIs, uses the old MapQuest API. I don’t use it, and I don’t know if anyone else does, so I’m leaving it to you, lazyweb: If you use the MapQuest API, or want to in the future, fork graticule on GitHub and update the MapQuest wrapper.

A common criticism of RSpec is that it is very verbose. I don’t necessarily agree (or care), but I thought it would be fun to see how concise I could make my specs.

Here are some simple specs from a client project:

describe Company do
  before do
    @company = Company.new
  end

  it "should have many classifications" do
    @company.should have_many(:classifications)
  end

  it "should have many industries through companies" do
    @company.should have_many(:industries, :through => :classifications)
  end

  it "should have many locations" do
    @company.should have_many(:locations)
  end

  it "should have many leads" do
    @company.should have_many(:leads)
  end

  it "should have many jobs" do
    @company.should have_many(:jobs)
  end

  it "should have many notes" do
    @company.should have_many(:notes)
  end

  it "should have many phones ordered by phone type position" do
    @company.should have_many(:phones, :as => :phonable,
      :include => :phone_type, :order => 'phone_types.position')
  end

  it "should have many events" do
    @company.should have_many(:events)
  end

  it "should have many interests" do
    @company.should have_many(:interests)
  end

  it "should belong_to an exchange" do
    @company.should belong_to(:exchange)
  end

  it 'should belong_to a primary_contact' do
    @company.should belong_to(:primary_contact, :class_name => 'Job')
  end

  it 'should have many articles' do
    @company.should have_many(:articles,
      :order => 'articles.date DESC, articles.created_at DESC')
  end
end

These specs check the declared associations on our company model using some custom matchers. They are not very complicated, but are somewhat repetitive. Each example has a description that is basically a duplication of the implementation.

Step 1: remove the description

For a while now, RSpec has had the ability for matchers to be self describing. If you don’t pass a block to #it, it uses the description provided by the matcher.

it do
  @company.should have_many(:jobs)
end

When that spec is run, it gives the output “should have a has_many association called :jobs”. Depending on what you’re speccing, the built in description isn’t always clear, but in this case it’s great.

See #simple_matcher if you want to create custom matchers with useful error messages.

Step 2: remove the subject

So the duplication within each example is gone, but if you look at the full spec above, each example calls @company.should. Accessing an instance variable isn’t what I would consider “duplication”, but thanks to a nifty new feature added to RSpec today, it’s now unnecessary noise. We can simply call #should within our example, and it will use a new instance of the described type as the “subject”.

describe Company do
  it do
    should have_many(:jobs)
  end
end

You can customize the subject if you don’t simply want a new instance.

describe Company, 'validations' do
  subject { Company.new(valid_company_attributes) }

  it do
    should be_valid
  end
end

Note: As David Chelimsky points out in the comments, this is not released yet and is subject to change.

Step 3: One-liner

Lastly, we can use the one line block:

describe Company do
  it { should have_many(:classifications) }
  it { should have_many(:events) }
  it { should have_many(:interests) }
  it { should have_many(:jobs) }
  it { should have_many(:leads) }
  it { should have_many(:locations) }
  it { should have_many(:notes) }
  it { should have_many(:articles, :order => 'articles.date DESC, articles.created_at DESC') }
  it { should have_many(:industries, :through => :classifications) }
  it { should have_many(:phones, :as => :phonable, :include => :phone_type, :order => 'phone_types.position') }
  it { should belong_to(:exchange) }
  it { should belong_to(:primary_contact, :class_name => 'Job') }
end

That’s pretty sexy. I wasn’t able to do this with all of the specs in my app, but it worked with quite a few of them.

While chatting with Dr. John Nunemaker at RubyConf, I realized that I have several problems. Ignoring the many character flaws that are beyond the scope of this post, my problems are:

1. I tag things I find useful on Delicious. But I rarely look back to delicious because it’s just easier to search Google, resorting to Delicious if I can’t find something that I remember tagging.
2. It’s easier to search Google because not everything that I find useful is in Delicious. I don’t want to have to think about where useful things are, I just want to search for them.

I want a search engine that prioritizes things that I’ve found useful in the past. Ideally, google would let me tag things and take that into account when calculating page rank. But, in the mean time…

Let’s have a search party!

I threw together a simple search interface that pulls in results from Google, Delicious, and just for fun, GitHub. It shows primarily Google results, but then in the sidebar, it shows results from Delicious, with my bookmarks highlighted at the top, and also results from GitHub. Check it out.

And if you use Firefox, you can add it as your search provider:

It is an extremely simple app that has 2 pages and uses JavaScript to read in JSON from all of the services. There is one Ruby class that does screen-scraping since Delicious doesn’t provide an API to their full search.

I first implemented it in Merb, but upon realizing how simplistic it is, I switched it over to Sinatra. And I used jQuery to do the JSON and other JavaScripty goodness. Thanks to Mark Van Holstyn for helping implement it.

The code is available on GitHub, so check it out, fork it, and make it more awesome.

I will be making a few other posts about specific things I learned when building this, including deploying sinatra apps and using jQuery to do JSON with Merb and Sinatra.

Scenario 1: You’re half way through a really productive day on a wicked new feature for an app and everything is going smoothly. Then, from out of nowhere…SMACK! A nasty bug in a library you depend on splatters right in your face. “Seriously, nobody has come across this before?” you mumble.

Scenario 2: You’re using some fancy library and you think to yourself, “Geez, wouldn’t it be sweet if it did <insert fancy feature here> for you?”

The temptation, especially with Ruby, is to solve both of these problems in your app’s code base by modifying your copy of the library. This may seem like the path of least resistance and the quickest way to move forward in the short term, but in the long term all you are doing is delaying the cost of properly solving that problem. You are incurring debt, which over time compounds and will end up costing you more to fix.

First, when you’re making the change just for yourself, you’re less likely to solve the problem properly or do the appropriate testing. You’ll do some hack job for the problem you have, without considering how it might affect other scenarios.

Second, if you found a bug or have a desired feature, chances are that someone else has or will run into the same situation. Fixing it locally doesn’t help everyone else.

Lastly, fixing the bug or adding a new feature directly in your app makes it nearly impossible to upgrade the library. I spent over 4 hours this week trying to upgrade an open source app to the latest version of Ruby on Rails, and still have over 100 failing tests. The biggest problem isn’t Rails itself, but all the plugins that also need upgraded, many of which had been patched in some form.

At Collective Idea, we use git submodules for as many external libraries as possible. While submodules have their own issues, and probably aren’t a recommended approach, the positive side-effect is that it strongly discourages us from patching libraries in place. If we do need to make a change, each library is already its own git repository, so we can push our changes and send a pull request to the original maintainer.

If you need to modify an open source library, don’t just do it in place. Push your changes upstream. Irresponsible patching hurts us all.

I’ve been working on a project that needs to store mileage reimbursement rates to the nearest tenth of a cent. We were using the money gem, which stores money amounts in cents, so it looked like it was going to be a pain.

But without too much suffering, I modified the money gem to take a precision (in powers of 10), which defaults to 2. It can now store amounts in any precision.

>> amount = 20.to_money + 0.505.to_money
=> #<Money @precision=3, @currency="USD", @cents=20505>
>> amount.to_s
=> "20.505"
>> amount.format
=> "$20.51"

You can also store amounts in negative precisions, like millions:

>> amount = Money.new(50, 'USD', -6)
>> amount.to_s
=> "50"
>> amount.format
=> "$50000000.00"

Check out our fork of the money gem on github. There are lots of other goodies in there.

The Great Lakes Ruby Bash was a good time. There were several quality presentations, including Jim Weirich’s talk “Playing it Safe – How to write library friendly code in Ruby”, Larry Karnowski’s talk “Usability on Rails”, and Brandon Dimcheff’s “Metawhat? A look into the mysterious metaclass”.

I presented a talk titled “Behavior Driven Development with Cucumber”. Despite the fact that half of the audience didn’t know what the hell I was talking about, I think it went well. I uploaded the slides from my talk to slideshare if you’re interested. I’m not sure that they’ll really be very helpful, but I may try to record audio to go with them at some point.

As far as I know, no audio or video was captured at the conference.

A while ago I posted a configuration for getting autotest to work with Test::Unit outside of Rails. Ryan Davis, author of autotest, commented on that post saying that it should “Just Work™” without any custom configuration. I was perplexed because I’ve never been able to get it to work on my gems and Rails plugins.

I finally took time to look into the issue, and realized it’s because I always use the Rails naming conventions for my test files. I name them foo_test.rb, instead of test_foo.rb, which is what Autotest looks for.

That’s easily solvable. Here’s an Autotest configuration, tested with ZenTest 3.10.0, that should make it work for either naming convention. You can throw this in your ~/.autotest file, or in a .autotest file inside your project.

Autotest.add_hook :initialize do |at|
  at.clear_mappings

  at.add_mapping %r%/^lib/(.*)\.rb$% do |_, m|
    possible = File.basename(m[1])
    files_matching %r%^test/.*(#{possible}_test|test_#{possible})\.rb$%
  end

  at.add_mapping(%r%^test/.*\.rb$%) {|filename, _| filename }
end

Happy autotesting.

All the cool kids tell me that fixtures just aren’t the “in” thing anymore. Speaking of which, when did fanny-packs stop being cool? I wish someone would have said something…

Anyway, while I still haven’t fallen out of love with (foxy) fixtures, I have taken some of the home-grown alternative methods for a spin, but so far, I’ve been slightly frustrated with my experience.

The predominant replacement seems to be the “factory” pattern, talked about by Dan Manges and then followed up by the FixtureReplacement plugin. The basic idea is that all your tests/specs should setup the data that they require, and there is a factory that makes it easy to create the necessary valid test data. For example, a homegrown solution might look like this:

class Generate
  def self.person(options={})
    p = Person.create!({
      :name     => 'Brandon',
      :login    => 'brandon',
      :password => 'testing'
    }.merge(options))
    p.account ||= Generate.account
  end
end

describe Person, '#authenticate' do
  it 'should return the person record if successful' do
    person = Generate.person
    Person.authenticate(person.login, 'testing').should == person
  end
end

This method scales pretty well. There are some issues, such as the use of create!. Sometimes you intentionally want an invalid record, or you want a new record with just some valid attributes initialized. It’s also slower than fixtures, but I don’t care too much about that right now.

But there’s a bigger issue. Like a good little BDDer, I have also been stubbing and mocking all my “unit tests” so there is no interaction with the database or code outside of what is being tested. To make the factory method work, I now need to define 3 different methods depending on what I’m testing: one using #create!, one using #new, and another one using stubbing.

What do we do about it?

I think the FixtureReplacement plugin is on the right track. It handles the new vs. create problem very nicely.

module FixtureReplacement
  attributes_for :person do |u|
    u.name     = String.random
    u.login    = String.random
    u.password = String.random
    u.account  = default_account
  end
end

@person = new_person(:login => 'brandon')
@person = create_person

Before I heard about the FixtureReplacement plugin, I actually concocted my own little solution that handles all three scenarios. I’m not crazy about the syntax, but it works for my needs.

Generate.add Person, :name => 'Brandon', :login => 'brandon' do |generator, u|
  u.account ||= generator.account
end

@person = Generate.person(:login => 'brandon')
@person = Generate.new_person
@person = Generate.stub_person

Anywho, for the fixture-less approach to work, I think stubs need to be supported.

Is all this really worth it?

That’s the question I find myself asking. What’s wrong with fixtures anyway?

One thing I really like about fixtures, when done right, is that they help tell the story of your application. You get to know the fixture data as you work on the app.

The factory method also seems to go against convention-over-configuration. Instead of having a default “configuration” when your tests run, you have to configure each test. Call me lazy, but that just seems like too much work.

I find that testing takes a lot more effort now than it did with fixtures. And as a result, I’m more hesitant to test everything. So while the factory method and stubbing is theoretically supposed to help you test better, I feel like they’ve had the opposite effect.

What do you think? Have you had success using the factory pattern for tests?

A week or so ago my gem command magically stopped working. Honestly, it was magic! There is no other logical explanation. The fetcher’s socket marshaled all over, spewing this error:

$ gem --version
1.1.0
$ sudo gem update --system
Updating RubyGems
ERROR:  While executing gem ... (Gem::RemoteFetcher::FetchError)
    Hostname not known: gems.rubyforge.org (SocketError)
    getting size of http://gems.rubyforge.org/Marshal.4.8

Googling different parts of that error didn’t yield anything useful. Lo, I post my solution in case some other poor soul has the same problem. Upgrade to 1.2.0.

Since gem upgrade --system is broken, download the rubygems-update manually.

$ sudo gem install ~/path/to/rubygems-update-1.2.0.gem
$ sudo update_rubygems

Ah, all better. Oh, by the way, rubygems 1.2 rocks!

Am I just dumb, or is it really a lot harder than it should be to break an array up into a set number of chunks?

For example, I have a list of 8 items that I want to break into 3 arrays, each displayed in their own unordered list, like this:

Brian and I spent a ridiculous amount of time (20 minutes, at least) trying to come up with a clean solution to this seemingly simple problem. The closest thing there is to a solution is Enumerable#each_slice in Ruby core or Array#in_groups_of in Active Support.

<% my_array.each_slice((my_array.size.to_f / 3).ceil) do |list| %>
  <ul>
    <% list.each do |item| %>
      <li><%= item %></li>
    <% end %>
  </ul>
<% end %>

or

<% my_array.in_groups_of((my_array.size.to_f / 3).ceil, false) do |list| %>
  <ul>
    <% list.each do |item| %>
      <li><%= item %></li>
    <% end %>
  </ul>
<% end %>

There’s not really a difference between either solution. Both requires that we calculate how many items we want in each list. (We convert the size to a float, divide by the number of columns, then round up. This gives us the same number of items in each column, with the last column having fewer.)

Our solution

We didn’t like having that much logic in the view, so we added a method to enumerable; we thought the division (/) method seemed appropriate since we’re dividing the array into equal parts.

module Enumerable
  # Divide into groups
  def /(num)
    returning [] do |result|
      each_slice((size.to_f / num).ceil) {|a| result << a }
    end
  end
end

Note: this method is now in our awesomeness plugin.

So now we can just divide our array into chunks in the view.

<% (my_array / 3).each do |list| %>
  <ul>
    <% list.each do |item| %>
      <li><%= item %></li>
    <% end %>
  </ul>
<% end %>

Are we dumb? Is there already a way to do this that wasn’t obvious to us and we just wasted our time (and I wasted even more time blogging about it)?

Update: Thanks to Aaron Pfeifer for pointing out the discussion on Jay Field’s blog about something similar. I’ve refactored this code in awesomeness to be more “robust’ (read: convoluted).