Bundle update “killed:” Watch those log files in your gems!

Today was supposed to be a normal day. A simple pull-request merge, an automated Jenkins build, and then a database migration.

What followed instead was a search for the cause of a build failure– or, rather, the cause of a ruby gem that was causing the build failure.

The Problem

The only hint I had to go on was this message in my Jenkins console:

Installing acts_permissive (0.3.2) /tmp/hudson7424891008093103684.sh: line 5: 12423 
Killed 
bundle
Build step 'Execute shell' marked build as failure

It couldn’t install the gem? Why? To the console:

jenkins@li128-183:~/workspace/musicstand
$ bundle update acts_permissive
Fetching source index for http://rubygems.org/
Using rake (0.9.2.2) 
Using RedCloth (4.2.9) 
...

NoMemoryError: failed to allocate memory
An error occured while installing acts_permissive (0.3.2), and Bundler cannot continue.
Make sure that `gem install acts_permissive -v '0.3.2'` succeeds before bundling.

What the what? Sadly, trying it manually was no help:

jenkins@li128-183:~/workspace/musicstand
$ gem install acts_permissive -v '0.3.2'
Killed

That’s it. Nothing else.

The Investigation

Luckily¹ I am the writer of the Gem in question. ActsPermissive is an instance-specific permissions system based on “circles of trust” (a bit like Google+ circles).

I went to my dev folder for the gem and explored it. Nothing seemed out of the ordinary. So I created a new gemset and ran “bundle install” forcing an installation of acts_permissive. It installed fine; however, I did notice that Bundler seemed to “pause” while installing it. On a hunch, I went to look at what the size of the .gem file was.

The Solution

Embarrassing as it is, the .gem file weighed in at a whopping 23M. That’s just ridiculous for a few text files! Further investigation showed me that my development directory was a full Gig! What the what?

Log files.

Without thinking, I was including the spec/dummy rails application, which contains a log directory, and log/test.log was nearly a gigabyte in size! Consequently, that was all getting packaged up, and then this 1G gem was trying to be compressed on the server and borking because of failed memory!

Lazy, John!

I had the normal “s.test_files = Dir["spec/**/*"] line in my gemspec, but need to exclude some files. The “test_files” attribute is an array, thus, I added the following lines to my .gemspec:

  s.test_files = Dir["spec/**/*"]
  s.test_files.delete("spec/dummy/log")
  s.test_files.delete("spec/dummy/log/development.log")
  s.test_files.delete("spec/dummy/log/test.log")

This dropped the .gem file back to a few kilobytes, which is all it should ever be.

Coda

It took a bit of detective work, but the “bundle update ‘killed’” problem was a crisis averted. This would’ve been grueling to troubleshoot if I didn’t own the acts_permissive gem. However, in the interest of helping others, it can be pretty easy to at least rule this problem out if your ‘gem install’ or ‘bundle update’ fails mysteriously. Grab the .gem file from RubyGems and check the size. If it’s crazy huge like acts_permissive was, then you might just be filling up RAM when it’s extracted.

1. or, unluckily, as the case may be [↩]

Comments are closed.