What followed instead was a search for the cause of a build failure– or, rather, the cause of a ruby gem that was causing the build failure.

The Problem

The only hint I had to go on was this message in my Jenkins console:

Installing acts_permissive (0.3.2) /tmp/hudson7424891008093103684.sh: line 5: 12423
Killed
bundle
Build step 'Execute shell' marked build as failure

It couldn’t install the gem? Why? To the console:

jenkins@li128-183:~/workspace/musicstand
$ bundle update acts_permissive
Fetching source index for http://rubygems.org/
Using rake (0.9.2.2)
Using RedCloth (4.2.9)
...

NoMemoryError: failed to allocate memory
An error occured while installing acts_permissive (0.3.2), and Bundler cannot continue.
Make sure that `gem install acts_permissive -v '0.3.2'` succeeds before bundling.

What the what? Sadly, trying it manually was no help:

jenkins@li128-183:~/workspace/musicstand
$ gem install acts_permissive -v '0.3.2'
Killed

That’s it. Nothing else.

The Investigation

Luckily¹ I am the writer of the Gem in question. ActsPermissive is an instance-specific permissions system based on “circles of trust” (a bit like Google+ circles).

I went to my dev folder for the gem and explored it. Nothing seemed out of the ordinary. So I created a new gemset and ran “bundle install” forcing an installation of acts_permissive. It installed fine; however, I did notice that Bundler seemed to “pause” while installing it. On a hunch, I went to look at what the size of the .gem file was.

The Solution

Embarrassing as it is, the .gem file weighed in at a whopping 23M. That’s just ridiculous for a few text files! Further investigation showed me that my development directory was a full Gig! What the what?

Log files.

Without thinking, I was including the spec/dummy rails application, which contains a log directory, and log/test.log was nearly a gigabyte in size! Consequently, that was all getting packaged up, and then this 1G gem was trying to be compressed on the server and borking because of failed memory!

Lazy, John!

I had the normal “s.test_files = Dir["spec/**/*"] line in my gemspec, but need to exclude some files. The “test_files” attribute is an array, thus, I added the following lines to my .gemspec:

  s.test_files = Dir["spec/**/*"]
  s.test_files.delete("spec/dummy/log")
  s.test_files.delete("spec/dummy/log/development.log")
  s.test_files.delete("spec/dummy/log/test.log")

This dropped the .gem file back to a few kilobytes, which is all it should ever be.

Coda

It took a bit of detective work, but the “bundle update ‘killed’” problem was a crisis averted. This would’ve been grueling to troubleshoot if I didn’t own the acts_permissive gem. However, in the interest of helping others, it can be pretty easy to at least rule this problem out if your ‘gem install’ or ‘bundle update’ fails mysteriously. Grab the .gem file from RubyGems and check the size. If it’s crazy huge like acts_permissive was, then you might just be filling up RAM when it’s extracted.

1. or, unluckily, as the case may be

You love the acts_as_follower gem, right? Sure, everyone does. Wouldn’t it be better if it was based in Redis? Sure, of course it would.

This weekend I found the great Amico gem, a low-level Ruby gem for Redis backed followers, and I decided to create Acts_as_Amico, a Rails injectable gem that uses that back-end ability, so now you can have a followers gem that is completely in Redis right in your Rails app.

The code has full-ish, but ever expanding, documentation on the Github page, but here’s a quick rundown

ActiveRecord Objects

class User < ActiveRecord::Base
 acts_as_amico
end

usera = User.create
userb = user.create

usera.follow! userb
=> nil

usera.following? userb
 => true

Advanced Usage

class Admin < ActiveRecord::Base
  acts_as_amico :amico_key => :name
  validates_uniqueness_of :name  # -> do this or be sorry
  validates_presence_of :name # -> this too, you've been warned
end

usera = User.create

puts usera.id
 => 18

admin = Admin.create :name => "frank"

usera.follow! admin
 => nil

admin.follow! usera
 => [1, 1]

admin.followers
 => ["18"]

usera.followers
 => ["frank"]

ActiveResource Objects

class RestObject < ActiveResource::Base
  self.site = "http://mettadore.com/junk"
  acts_as_amico :amico_key => :title
end

usera = User.create

rest_object = RestObject.find(123)

rest_object.title
 => "Bread and Circus"

usera.follow! rest_object

usera.following? rest_object
 => true

usera.following
 => ["Bread and Circus"]

Issues

I have every expectation that there will be problems and bugs. If you find one, please don’t hesitate to file an issue if you’re on Github, or send me a message on Google+ if you’re not.

Hacking Github

This weekend, a Github user named Egor Homakov hacked Github in such a way that allowed him to commit directly to the Rails core project. Since Homakov is not a Rails team member, this is a really big deal.

Since this happened, there’s been a lot of talk about the Rails core being fundamentally insecure. In fact, Homakov has been harping on this for at least 1000 years. A few days ago he filed an issue about a mass assignment vulnerability in the Rails core, and later he illustrated this vulnerability by filing an issue report from the future. He was illustrating that you can inject attributes into a Rails model with an HTTP request.

So, when no-one took him seriously, he took the next step. He created HTTP PUT requests adding his SSH key to a Rails core user id, then pushed a commit directly to Rails core.

How to Hack Rails

It turns out that Rails apps, by default, are easy to hack. Peter Nixey wrote a very detailed post about how Homakov hacked Github and the one line of code that could have prevented it, so if you want the full details, read there. The summary is much shorter.

Let’s create a simple User model which has two attributes, name and role. By default, every attribute of every model can be modified by using update_attributes. We all know this, and we’ve known it for a while. What it means is that any User, even if they don’t have permission, can update their own role. Imagine if I log in to our example app and submit a PUT request to the UsersController with the package {'params': {'id': 23, 'role': 'superadmin'} }. By default, the app will accept this and update the user with the new role.

This is exactly what Homakov did. He sent an HTTP request to Github and used update_attributes to change the Github database. All of this could, he argued, be prevented by adding attr_accessor to the User model.

Rails: Convention over Configuration Security

Now, the philosophical point I want to make about this is that the Rails core team seems to be ignoring their own mantra. All of us know that Rails adheres strongly to the Convention over Configuration design pattern. It’s a pattern that Rails users are taught from day one. In fact, it’s embedded in the framework’s name.¹

One of the outcomes of Convention over Configuration is that sensible defaults should be, well, sensible. The Rails core team feel strongly that attr_accessor should not be a default² and that security is the responsibility of the app developer. I agree that security is our responsibility, but disagree that the dominant Rails design-pattern-come-mantra supports this.

It’s almost as if no-one wants to say “Yeah, we should really take care of this.” Everyone is saying “You know, you should really take care of this.”

Stop the world! We’ve found a SQL injection

There’s a term that makes all software developers give pause: “SQL Injection.” It’s a phrase that keeps us lying awake at night, and gives us nightmares when we finally fall asleep. The idea that someone, using nothing more than a web browser, can change our database willy nilly. It’s a terrifying thought.

I can’t help but take an initial read of all the hubbub and think that we’re not giving it the importance that it’s due. Everyone sees a headline saying Rails has a ‘mass assignment vulnerability’ and says to themselves ‘I should probably look into that at some point.’ It’s too vague, to uncertain. To unemotional.³

I have to assume that everyone would be treating this differently if we called it what it is. Imagine your thoughts (and actions) if you read a different headline, something like “Rails apps prone to SQL injection by default”

Don’t you think you’d get something done? You should, because that’s what we’re talking about. This is a path to SQL injection, full stop.

Who owns this issue

I love Ruby on Rails. Like Python, Scala, Node.js, and a host of other technologies that we have at our disposal, Ruby’s web framework makes being a developer both powerful and fun. There’s so much we can do– and so much we can do very quickly. But there’s a cost to pay. Convention over Configuration is a good thing, but we can’t expect people to learn fast, develop fast– create fast– if we don’t respect the logical outcome of that philosophy.

That outcome is this: We, Rails developers, understand that we are responsible for our security; however, we also believe in you, the Rails core team. We trust you. We believe it when you tell us to follow “Convention over Configuration,” and so we naturally believe that the defaults you give us will be a safe, or at least not horribly, dangerously wrong. We, all of us– developers and Rails core team both– can’t have it both ways. We are telling ourselves to follow Convention over Configuration, but we’re also telling ourselves that SQL Injection is a viable convention, thus, that we have security as a configuration.

Now, personally, I don’t believe that attr_accessor belongs in the model– or at least that security from view-based actions belongs in the model. Rails is an MVC framework, and therefore I don’t like forcing myself to define my model based on actions in the view. I don’t want my view owning control of my model that way. I think the controller should manage these permissions and there are others who feel the same. In fact Yehuda Katz argues this as well.

Still, the overall question remains: How can we reconcile the mantra of Convention over Configuration if we support the standard of dangerous insecurity by convention?

1. Look, just stay on these rails and you’ll move fast. We’re not responsible for what happens if you go over there
2. Incidentally, I don’t either, but you can set this as the default by using ActiveRecord::Base.send(:attr_accessible, nil), but this causes all sorts of problems
3. In fact, to be honest, that’s how I was reading it.

The Power of Good Questions

Some history.

Years ago, I was starting to question my decision to become a hydrologist. Not because I hated hydrology, I did– and still do– deeply love it, but because I couldn’t find a job. Since I’d been a programmer for 20+ years (indeed, I programmed all the way through 10 years in the sciences), I decided to see what options were out there in that field.

So I interviewed for this software development company. I was nervous because I hadn’t actually worked as a programmer in a long time. My title had been “researcher” and “scientist” for a decade. Even though I’d been programming that entire time, I was nervous about being labeled as a “wannabe.”

My interviewer saw through that nervousness right away and immediately started asking me questions that were not the basic “do you actually know programming” but were much more geared toward “what do you think of this subtle programming construct.”  It was, without a doubt, the best interview experience I ever had, and made me respect the interviewer and want to work with him even if not at this job– which I didn’t even get.

I never forgot that interview.

What Goes Around

Fast forward a few years, and last year I happened to see that same interviewer at the coffee shop. He said that he’d just left his last job and was looking for work. That, to me, was a wide open door. Finally, a chance to work with this guy!

“Really,” I said, “It’s not much, but I’ve got some Ruby work I could use some help with for my company if you’re interested”

“I don’t know Ruby, I’ve been learning a little bit, but I can’t do work in it.”

“Tell you what, work with me. I’ll help you learn Ruby while you do small bits of work on my site. Fair trade?”

As I suspected from this now barely remembered interview so long ago, he was a sharp tack and things progressed fast enough that he never even did any work on my site. Rather, I took another contract that was a wee bit too big for me, and brought him on as a sub. And so, finally, I was working with this guy, and helping him learn Ruby, and getting him paid to do it.

A Favor, Returned

A couple months later, I got contacted by another company who wanted me to create a web app. I was just too busy, so I brought this fellow to the first meeting. My response was, basically, “I’m too busy to do this, but I can speak from experience that this fellow here is a good candidate. I’ll let you take it from here.” And I walked away. Now, here we are, a couple months after that, and MyLunchIn.com has launched for our local restaurant Nora’s Table. It’s another Rails app that he created, start to finish, entirely free of my involvement.

I don’t know I started to cry when I placed my first lunch order today. I didn’t create that site, and honestly, I didn’t teach this guy Ruby– I just happened to be around while he was teaching himself. But it still feels good.

All those years ago, I was a scientist scared that I wouldn’t be accepted by “serious” developers and I was. It feels good to think that I brought some happiness back to that person, a favor returned. It just feels like this new website is part of a rising tide that floats all of our boats when we work together, when we look out for each other. When we remember a good conversation.

I’m on my way now to pick up my first ever lunch ordered on this new site. It’ll be the best tasting lunch I’ve had in a while.

Recently, I had need to create a plugin that held a polymorphic :has_many, :through relationship. I spent a day or so trying to get this to work, only to realize that it is essentially impossible to do inside a pluggable Rails module, but possible if you templatize the classes.

PolymorphicPlugin Example Project

There’s a StackOverflow thread that describes this problem and the solution a bit; however, Based on the difficulty, and the fact that I couldn’t find any documentation specific to this problem, I figured I’d write it up as an example, and create a PolymorphicPlugin gem to illustrate it better.

The PolymorphicPlugin inside-module branch is an example of the way you’d distribute a pluggable Rails module with all classes contained in the module. The generator only creates a migration in the Rails project.

The PolymorphicPlugin outside-module branch is an example of the way you’d distribute a pluggable Rails module in such a way that the generator creates a migration in and also copies model files to the Rails project.

Problem Description

The idea is that there’s an “acts_as” like plugin that I want to add to my Rails models:

module PolymorphicPlugin
  module PolymorphicHolder

    def self.included(base)
      base.extend ClassMethods
    end

    module ClassMethods
      def holds_things
        has_many :thing_groupings, :as => :thingable
        has_many :things, :through => :thing_groupings
        include PolymorphicPlugin::PolymorphicHolder::InstanceMethods
      end
    end

    module InstanceMethods

      def has_things?
        true
      end

    end
  end
end

You can see that anything that calls holds_things should have a :has_many relationship with Thing through the ThingGrouping join table. This is a standard paradigm for a polymorphic :has_many, :through relationship.

Basically, I want to do this:

class User < ActiveRecord::Base
  holds_things
end

class Admin < ActiveRecord::Base
  holds_things
end

user = User.new
user.has_things? -> true

admin = Admin.new
admin.has_things? -> true

inside-module plugin layout

The failing case

Seems pretty simple, right?

Turns out it’s not. Take a look at the directory layout that I tried to use. This seems like a standard for a gemmable plugin. The model files are contained in the gem library, and accessed as part of the module. When you use gem 'polymorphic_plugin', the only thing that gets copied to your Rails project is the migration file. Nice and clean.

Unfortunately, that doesn’t work. If you clone this branch and run rake test you’ll get exceptions. Specifically, a NameError: uninitialized constant User::ThingGrouping because our User model, by extending the PolymorphicHolder module, doesn’t have access to this unknown model called ThingGrouping.

And no, using things like :class_name and :source were no help. I tried everything, or seemingly everything. Nothing would make this work.

outside-module plugin layout

The passing case

That is until I pulled the models outside of the plugin module.

Take a look at the directory layout now and you’ll notice that the Thing and ThingGrouping models are no longer in the plugin’s lib directory. Moving them to the templates directory is the unfortunate answer.

I say “unfortunate” because this means that the models need to be distributed by copying them to the Rails project. In other words, there’s a line in my PolymorphicPluginGenerator that reads: template "thing.rb", File.join('app/models', "thing.rb"). In other words: Copy the Thing model into this project so it can be accidentally futzed with by users and maybe, if we’re lucky, cause some interesting name collisions.

Thing and ThingGroup are no longer contained inside my PolymorphicPlugin module. They are top-level classes equivalent to anything generated with rails generate model. Not a solution that I’m fond of, but one that passes my tests. We can at least have a polymorphic :has_many, :through relationship now.

Outcome

Polymorphic :has_many, :through relationships are available in pluggable Rails modules, but it’s a bit of a kludge. You’ll need to make sure you name your polymorphic classes smartly to avoid name collisions with the calling Rails project. You’d hate to create a plugin with a model like Book only to find that Book already exists in someone’s Rails project!

I’d love to be proven wrong on this, by the way. Go ahead and grab the Github repository and play with the inside-module branch. If you can make it work, I’ll buy you a beer!

Currently, the new Rails 3.1 mountable plugin architecture will not work with RSpec. There’s a gem called Enginex that will work with RSpec, but it won’t work with Rails 3.1. So it looks like you either get RSpec or Rails 3.1. What’s the deal?

Playing around, I figured out that combining the two will work:

$ gem install enginex
…
$ enginex my_plugin --test-framework=rspec
…
$ rails -v
Rails 3.1.1
$ rails plugin new stock_plugin --full
…
$ rm -rf my_plugin/spec/dummy
$ cp -r stock_plugin/test/dummy my_plugin/spec

After that, it’s just a matter of modifying my_plugin/spec/dummy/config/application.rb to require your gem, and clean up the spec_helper.rb file. I also changed the Gem files so that my_plugin.gemspec held the dependencies, and Gemfile just called that. This is the way the stock Rails 3.1 generator works. Enginex uses Gemfile as the canonical source, and I like the new way better.

Using this I’ve currently got a Rails 3.1 gemmable plugin with RSpec, Cucumber, Capybara, Spork, Guard, and Simplecov working just fine.

The basic problem that I’ve found is that ActiveResource is a great Rails model to inherit from, except when it isn’t. And it seems that “when it isn’t” basically means “anytime you are using associations.” To be fair, associations don’t always break, it just turns out that my architecture illustrates exactly when they do break.

To lay out the situation, I have a backend server running Rails with three models, User, Service, and Groups. I have a Rails frontend server without a database hitting this (as well as other) backend servers.

Backend

class User  :members
  has_many :managed_groups, :class_name => "Group"
  accepts_nested_attributes_for :managed_groups, :services, :groups
end

class Group  :members
  belongs_to :manager, :class_name => "User", :foreign_key => :user_id
  accepts_nested_attributes_for :users  
end

class Service < ActiveRecord::Base
  belongs_to :user
end

class Member < ActiveRecord::Base
  belongs_to :group
  belongs_to :user
end

You can see it’s really basic setup. Users can have many Services, but Service can only have one User. By contrast, users can Manage many groups, and can belong to many Groups, while Groups can have many users who belong to it.

Users -> Services is a basic :has_many relationship and Users -> Groups is a :has_many, :through relationship. There is nothing complicated about this setup at all…

Frontend

…until you get to an ActiveResource frontend

class User < ActiveResource::Base
  self.site = "http://localhost:8801"
end

class Group < ActiveResource::Base
  self.site = "http://localhost:8801"
end

class Service < ActiveResource::Base
  self.site = "http://localhost:8801"
end

The frontend server sees all of these models as ActiveResources, which works fine if you’re accessing User.name, but which doesn’t work so well if you’re trying to access User.services because User.services doesn’t exist. ActiveResource simply doesn’t know anything about associations, so you have to work around a big problem.

:include => :associated_model

One easy way to make User.services available to ActiveResource is by including it in the UsersController. Here’s a sample UsersController#Show method on the backend:

def show
@user = User.find(params[:id])
  respond_to do |format|
    format.xml  { render  ml => @user.to_xml(:include => [:managed_groups, :services, :groups]) }
    format.json  { render :json => @user.to_json(:include => [:managed_groups, :services, :groups]) }
  end
end

We just include a list of whatever associated objects we want to include in the returned user package. Easy enough. Starting up the Rails console on the frontend now allows you to access User.services. However, what if you tried the following:

service = Service.create(:name => "Test Service")
user = User.first
user.services << service

Turns out that we can’t do that. We get an exception that the User model expects a Service class, but was supplied with an IndifferentHash.

With a :has_many/:belongs_to relationship, this is easy to work around:

service = Service.create(:name => "Test Service", :user_id => User.first.id)

Of course, user_id is a “primitive” type, a direct attribute of Server. So we can force it by going to the other side of the relationship and set that attribute and not have a problem.

:has_many, :through => :needles_in_your_eye

Working around the :belongs_to issue was easy, working around :has_many, :through is as fun as… well, you get the idea.

The problem is that there’s no backdoor to setting the association. You simply can’t access an array of objects as an association through ActiveResource. Look at what we see when the frontend accesses the backend server with a PUT request at /users/1.xml:

{"user"=>
  {"created_at"=>2011-06-23 02:58:15 UTC,
   "id"=>1,
   "name"=>"John Metta",
   "updated_at"=>2011-06-23 02:58:15 UTC,
   "managed_groups"=>[],
   "services"=>[
     {"created_at"=>2011-06-23 02:58:15 UTC,
      "id"=>1,
      "updated_at"=>2011-06-23 02:58:15 UTC,
      "user_id"=>1}
   ],
   "groups"=>[
     {"created_at"=>2011-06-23 02:56:37 UTC,
      "id"=>1,
      "updated_at"=>2011-06-23 02:56:37 UTC,
      "user_id"=>nil,
      "users"=>[
        {"created_at"=>2011-06-23 03:36:28 UTC,
         "id"=>6,
         "name"=>"Jefferey",
         "updated_at"=>2011-06-23 03:36:28 UTC},
        {"created_at"=>2011-06-23 02:59:36 UTC,
         "id"=>2,
         "name"=>"George",
         "updated_at"=>2011-06-23 03:05:13 UTC}
       ]
     },
     {"created_at"=>2011-06-23 02:56:37 UTC,
      "id"=>1,
      "name"=>"Site Admin",
      "updated_at"=>2011-06-23 02:56:37 UTC,
      "user_id"=>nil,
      "users"=>[
        {"created_at"=>2011-06-23 03:36:28 UTC,
         "id"=>6,
         "name"=>"Jefferey",
         "updated_at"=>2011-06-23 03:36:28 UTC},
        {"created_at"=>2011-06-23 02:59:36 UTC,
         "id"=>2,
         "name"=>"George",
         "updated_at"=>2011-06-23 03:05:13 UTC},
        {"created_at"=>2011-06-23 02:58:15 UTC,
         "id"=>1,
         "name"=>"John Metta",
         "updated_at"=>2011-06-23 02:58:15 UTC}
       ]
     }
   ]
  },
  "id"=>"1"}

Cats talking to dogs, here. They’d both be happy eating a squirrel, sure, but they just speak different languages.

Basically, the backend tries to find the “groups” and “services” properties on the User model. ActiveRecord expects groups to be, well Group objects- but it’s a Hash, not a Group. In order for this to work, those should be named “groups_attributes” and “services_attributes.”

This is basically telling me that ActiveRecord and ActiveResource basically don’t know how to talk to one another. What the hell was the Rails core team thinking?

ActiveResource should be patched to format it’s JSON string to use MODEL_attributes, right? Well, that would break a lot of stuff. ActiveResource is supposed to talk fine to my Scala backend server too, which it wouldn’t.

ActiveRecord should automatically treat MODEL: {ASSOCIATED_MODEL: …}} as something that can be converted to an ActiveRecord, right? I’m not sure of this one- it seems too possible to break things down the line that we didn’t expect.

So, you can do it from either direction without possibly horribly breaking something down the line.

Ahh, apparently the Rails core team was thinking “You know, we shouldn’t do this because we might horribly break something down the line.”

Wrestling ActiveResource to the ground

In order to fix this, I’ve added a simple hack to my UsersController on the backend. I say it’s simple because it’s pretty dumb. I say it’s a hack because I think it’s really ugly and inelegant– something I don’t expect from Rails.

Here’s the full UsersController#Update method on my backend server:

  def update
    @user = User.find(params[:id])

    respond_to do |format|
      begin
        groups = []
        params[:user][:groups].each do |g|
          groups << Group.find(g[:id])
        end
        params[:user][:groups] = groups
      rescue
      end

      begin
        services = []
        params[:user][:services].each do |s|
          services < @user.errors, :status => :unprocessable_entity }
        format.json  { render :json => @user.errors, :status => :unprocessable_entity }
      end
    end
  end

As you can see, we grab that rogue IndifferentHash groups list and, basically, walk through it and find the local groups that correspond to the Group that’s in the list. This works- it allows ActiveResource to barely communicate with ActiveRecord. It’s no longer cats and dogs talking, it’s more like dogs and wolfs.

Yes, I know, wolves bite.

One Model, One Resource

This hack works, but it’s just that- A hack. If something seems this inelegant to me, then I have to question whether it is just that inelegant.

Today a friend mentioned that he’s never used accepts_nested_attributes_for. Never. He has never used nested resources like this because his strong philosophy is “One model, one resource.” I’m going to explore this for a few reasons, not the least of which is that I don’t think I want to pass the groups around inside the user if the user is a member of 3000 groups that we don’t care about. That’s just stupid.

Trying this method would mean that the User model on the frontend would require a groups method that would search the join table based on the user, and an additional add_group method which would go through the Group model to create a new group and somehow add that group to the users list of groups.

There’s a hole in my bucket, dear Liza, dear Liza, a hole in my bucket, dear Liza, a hole…

Really not sure how that would work– at least without some inclusion of my hack above. It seems like there’s just no way to get around this problem elegantly. I hope I’m wrong about that.

Someone please tell me I’m wrong.

The problem is that Ruby and Ruby-based tools like Chef are becoming so natural that they feel like they’re just part of the BASH shell. It took me a while to figure out the following error:

% knife cookbook create somename

/Users/john/.rvm/gems/ruby-1.8.7-p302/gems/chef-0.9.12/lib/chef/knife/cookbook_create.rb:90:in `initialize': No such file or directory -
 ~/Dev/Chef/chef-repo/cookbooks/sti-rewrites/recipes/default.rb (Errno::ENOENT)
	from /Users/john/.rvm/gems/ruby-1.8.7-p302/gems/chef-0.9.12/lib/chef/knife/cookbook_create.rb:90:in `open'
	from /Users/john/.rvm/gems/ruby-1.8.7-p302/gems/chef-0.9.12/lib/chef/knife/cookbook_create.rb:90:in `create_cookbook'
	from /Users/john/.rvm/gems/ruby-1.8.7-p302/gems/chef-0.9.12/lib/chef/knife/cookbook_create.rb:74:in `run'
	from /Users/john/.rvm/gems/ruby-1.8.7-p302/gems/chef-0.9.12/lib/chef/knife.rb:127:in `run'
	from /Users/john/.rvm/gems/ruby-1.8.7-p302/gems/chef-0.9.12/lib/chef/application/knife.rb:121:in `run'
	from /Users/john/.rvm/gems/ruby-1.8.7-p302/gems/chef-0.9.12/bin/knife:25
	from /Users/john/.rvm/gems/ruby-1.8.7-p302/bin/knife:19:in `load'
	from /Users/john/.rvm/gems/ruby-1.8.7-p302/bin/knife:19

Because, no matter how much I confirmed that the directory existed, it still failed. Why?

Because Ruby, as anyone really thinking about it would naturally realize, doesn’t expand Bash shortcuts.

To wit: The following entry in my ~/.chef/knife.rb file:

...
cookbook_path [ '~/Dev/Chef/chef-repo/cookbooks', '~/Dev/Chef/chef-repo/cookbooks' ]
...

Which I generated by running a command at the bash shell:

knife configure

As anyone would expect, changing that line to:

...
cookbook_path [ '/Users/john/Dev/Chef/chef-repo/cookbooks', '/Users/john/Dev/Chef/chef-repo/cookbooks' ]
...

fixed the problem. D’oh!

As I use more Ruby-based server configuration tools, I think I need to be a bit more conscious of when I’m with the purview of the Bash shell, and when I’m not.

I’m a consultant, and that means I work on multiple projects, and have to bill those projects. Recently, I began working on a contract that required weekly billing, using an Excel spreadsheet that can’t be modified, and a signature from a project manager in another city. Talk about tedious! My workflow for just filling out a weekly timesheet is this:

1. Use Freshbooks daily to track time
2. On Friday, open Freshbooks and pull the hours on that project
3. Open Excel (in VMWare) to update the read-only Master Timesheet file by changing the date, and adding my hours into the day slots
   1. I can’t use Open Office or anything on the Mac since the formatting changes when I save it as an Excel file- I’ve tried a bunch of ways to do this, but as I usually have VMWare open to work in Visual Studio for this project anyway, I just use MS Excel on Windows
4. Save the timesheet under another name
5. Drag the timesheet into another folder in my Dropbox (so that I have a record)
6. Open my email client and create an email to my project manager that says little more than “Timesheet, Cheers, -J”
7. Drag the timesheet into the email (I Google Mail’s ability to drag a file to attach!) and hit send
8. When he sends it back, it’s an attachment as an image- save the image to my project folder locally
9. Print out the image
10. Sign it (in blue pen, mind you)
11. Scan the signed copy
12. Move the scanned copy file to my local computer
13. Open a web browser to connect to their email system, which I’m supposed to use, but which is proxied, so I can’t use my normal email client.
14. Create another email to the timesheet recipient
15. Attach the file using the dialog boxes (Drag is not possible) and send it
16. Grab the timesheet original off my scanner and put in an envelope to mail to New York.

Now, if you think about doing all of that, it’s not like it takes half the day. Still, it takes enough time that a good bit of my morning is lost to it. There’s not only the time spent “actively doing everything” but the lost time where I could have been concentrating on something like our object model or database abstraction layer, but which I’m distracted by simultaneously dealing with the timesheet issue. Anyone working in a “mental sphere” such as programming knows that a few seconds of distraction with something will pull you out of the mindset for 20 minutes or more.

Plus, there’s the added issue of this just being boring. It’s something a robot should be doing- or at least a secretary. More importantly, it’s something that could happen automatically! We programmers tend to get annoyed when things “should happen automatically” and they don’t. It is, in fact, why many of us became programmers.

Enter Hazel

For a while, I’ve been simplifying things a wee bit using Hazel. For those who don’t know, Hazel is a Mac program that watches a folder and performs actions based on the folder contents. I first heard about it on the Mac Power Users podcast and bought the software pretty much immediately following the podcast.

Hazel rules screen

Hazel is one amazing piece of time-saving software. It has a super simple graphical dialog box to let you set things like which folder, what attributes (“if files are older than” and “if file is a:” (e.g. movie, soundfile, pdf, etc) and “if the name contains:”) and what actions (Open, move, delete, archive, change name, import into iTunes, etc.) to perform on files in that folder.

One of the ways that I use it is on my desktop. If there are any pictures or movies on my desktop that are older than 1 day, Hazel takes them off my desktop and puts them on my remote hard drive, in appropriate folders, and simultaneously imports them into iTunes (I don’t have iTunes copy files to the local drive, so they’re watched from the remote drive).

When dealing with my timesheet, I have Hazel set up so that anything that goes into the “Timesheets” folder and is an Excel file gets automatically moved to a subfolder called “archive” and gets renamed with the pattern “John_Metta_<date added>.xls.” That way, I have an easily recognizable location and file to drag into an email to my project manager.

Using Hazel to automatically move, delete and organize your files can save you a lot of time. It’s tiny little chunks of time, sure, but it’s time. In reality, it’s saving you more the headache of having to do the same thing over and over again. Hazel is powerful, check it out.

Add A Little Ruby

Because I love Ruby, I decided to automate the creation of the Excel spreadsheet a bit. The Windows version of Ruby has the win32ole module, which can open and handle spreadsheets really easily. So I created a small script that opened a read-only version of the master timesheet spreadsheet and changed the date and the hours/day based on the arguments.

Here’s the whole script:

require 'win32ole'

xl = WIN32OLE.new('Excel.Application')
xls = xl.Workbooks.open("c:\timesheets\Timesheet")
sheet = xls.Worksheets(1)

sheet.Cells(4,11).Value = ARGV[0]

#Time cell index
t = 6
ARGV.each_with_index do |a, i|
  sheet.Cells(10,t+i).Value = a
end

xls.SaveAs("Z:\john\Dropbox\Consulting Stuff\Nova\timesheets\timesheet")
xls.Close
xl.Quit

With this, the command:

 C:timesheets$ ruby timesheet.rb "10/16" 0 0 8 7 6 7 8

will create the correct timesheet for the week beginning 10/16, with the correct dates all lined up for the hours, and will save that timesheet to my Dropbox/timesheets directory- afterwhich Hazel will do the rest.¹

More with Automator

Automator: More "secretary" than I'd thought!

Earlier this week, I was still frustrated at the workflow of timesheets, however, because I still had a bunch of steps. Hazel was good at moving files and renaming them, but I still had to email them, print them, etc. So I decided to get my programming game on… then I decided that I didn’t even need to. Like all Macs, my Mac has Automator- a program that will create “workflows” and “applications” which basically amount to my own personal robotic secretary. Hazel can run both applications and Automator workflows on files, so I decided to give it a try.

Mailing to my PM

The first thing I do when I have my timesheet is move it to the timesheets folder where it gets renamed and archived so that I can send it to my PM. It would only be a single step to mail the renamed file. So I opened my Automator script and used the “create mail message,” “add attachment to open messages,” and “send mail” actions to create a script. Now, whenever a folder goes into Timesheets, it gets moved, renamed, and automatically mailed to my PM in one step!²

Printing and Signing

When my PM signs and mails it back it’s an image. So, I created another automator script to take any images with the name pattern “John_Metta_*” and the extension JPG in my “Downloads” directory and automatically move them to the “timesheets/images” directory and send them to the printer. That means all I have to do is open the email and click “download” and I’ll soon have a printed timesheet awaiting my signature. Sounds pretty secretary-like to me.

Mailing the final

My Canon all-in-one will scan automatically save files to a specific directory, so I could just set up Hazel to monitor that directory. The problem with this is that ANYTHING I scan will get processed. Rather than that, I’ve decided to have Hazel monitor that directory for a specific file name. Thus, when I scan any document, I see a dialog box on my screen prompting me to add a name. I can add a name in the rare occasions I want to scan something else. If I just hit <enter>, I’ve set Hazel up with another Automator script: Take the file, rename it with the pattern “John_Metta_Timesheet_<date addedd>”, mail it to the company as an attachment, and then throw the local copy in the trash (because Google Apps mail will store it for me).

A Robotic Secretary

The function of a secretary is to take care of everything that doesn’t absolutely need your action. A secretary should take care of everything in a “Do everything and let me just sign the paper.” That’s almost what I have now.

With this workflow, I’ve gone from 16 tedious steps to

1. Track time daily in Freshbooks
2. Friday morning, while working, run the ruby command with my hours³
3. Work until lunch/break, before stopping, check my email (have an email from my PM waiting)
4. Click “download”⁴
5. Sign, scan, then hit enter

That’s it!⁵

Coda

I highly suggest looking into combining Hazel, Automator, and any other talents you may bring to creating robotic secretaries for your automated tasks. This is just an example of a way you can take a morning’s worth of frustration and productivity vacuum and turn it into a few steps. Best of all, it took me about as long as two of these frustrating mornings to set everything up– meaning that it pays itself off in about half a month, on a year-long contract.⁶

1. There’s actually an API library for Freshbooks in Ruby that is written and maintained by Ben Curren at Outright.com, so you could pull the hours straight from the project in a Ruby script- I decided against this step
2. In full disclosure, the Automator script actually uses “Save to a variable” “create new mail message” “get from variable” “add attachments” and “send mail”– in that order. For some reason, the “add attachments” phase was throwing an error otherwise.
3. This automatically does everything through emailing my timesheet to my PM
4. This automatically files and prints the timesheet, it’s just awaiting my signature
5. Well, I still have to mail it to New York, but that final step is a lot less annoying when it’s not preceded by so much tedium!
6. I’m sure this’ll only get easier as more companies have APIs available. As I mentioned, I could probably have an automated task in Windows start the Ruby  script and use output from a Freshbooks API query to automate step 2 above, meaning that I’d do nothing until I got an email from my PM. However, I wanted the additional steps of adding my hours manually- more as a double check than anything.

Like tomato sandwiches, Celtic music, beer, and programming- Github is something that, try as I might, I just can’t make myself sick of.¹

Recently, I took the Git survey, and it contained an interesting question along the lines of “What do you use Git for?” The answers were things like “configuration files” and “large binary files.”

I use Git and Github for a lot- configuration files are one use (I love the “raw” path), but it was only a small percentage of what that list suggested, so I started wondering how else I could use it.

Today, I thought of an interesting way: Github as a CDN.²

If you know what a Content Delivery Network (CDN) is, you don’t want me to explain. If you don’t, then it’s good enough to say that it’s a way to server up pictures or other content from another server. It’s often used to, say, serve the main portion of an html page from a webserver, but serve the images, CSS files, etc. from something like Amazon to speed things up.

I recently finished work on The Great John Metta Branding Project and, like with so many projects, stored the images in Dropbox. Then, deciding I might want to version them, I stored them in Github. Then, just as I was starting to begin the rather annoying process of copying all these logos and textmarks to my various sites throughout the web and updating all the image tags, I thought to myself “Hey, if these are already in Github, and I can access the raw files, why don’t I just use that as the image source?” But I don’t want to have to remember the URL each time. What if it was somewhat dynamically generated?

A RESTful CDN system for a logo

This is exactly why I made johnmetta.com a Rails project, to play with stuff like this and then have a place to have it be live if I wanted to use it.

I’ve got a lot of different styles that my logo can take. Stacked, Icon only, textmark only, icon one-color grey, textmark one-color blue, etc. Ideally, I wouldn’t have to remember the path to each, I could type something like http://johnmetta.com/brand/icon/blue/transparent/logo.png³ and have the app return the image that is strictly the icon (no words) with blue coloring and transparency inside the logo (instead of the normal white).

Since none of these actual images is created on the fly, I created a sensible directory structure and embedded all the files in this structure, naming them all logo.png and putting the default logo in the top level. So the above image would be {base}/full/logo.png, while a version with the icon on top would be {base}/full/stacked/logo.png. A one-color version of only the icon might be {base}/icon/blue/logo.png

So I created a Brander class which would turn arguments into an appropriate path and added some new routes such as “brand/:type/:color/logo.png” and created a call to brander in the controller’s “brand” action that ends with a redirect to the returned URL.

It’s all really simple, but it allows me to have a url like http://johnmetta.com/brand/icon/logo.png and supply my logo, and http://johnmetta.com/brand/stacked/blue/logo.png for a one-color blue stacked logo. Then, if I update the colors (move the files, whatever) they update automatically. At a maximum, I just check in the changes and make minimal changes to the routes- but don’t have to copy files to servers.

A Good Illustration

Right now, the Brander class is pretty dumb- little more than a way to translate a route into the correct external URL. It was built up not as an illustration of how to accomplish “Github as a CDN,” but merely as a toy to test one way that it might be done on a small scale. It might be better done as a class where instance attributes make more sense, and I’ll clean it up a bit, but not much more than that.

Also, it’s cool to know that stuff like this can be set up so easily, and for my uses- where bandwith might reach a whopping few requests per day, it’s useful while being low impact. But for a real CDN, you’d still want to use something like AWS- or at least talk to the Github guys before swamping their webservers.

1. I’ve tried for the past two summers to make myself sick of eating tomato sandwiches. I just can’t.
2. The title is with Rails because that’s what I used, but it would take 2.4 minutes to make this a “With Django” or “With Lift” example
3. Actually having “logo.png” is unnecessary, however, sometimes the software using the URL will depend on that, so I just made it convention so that I don’t have to think about when sometimes is.