Today was supposed to be a normal day. A simple pull-request merge, an automated Jenkins build, and then a database migration.
What followed instead was a search for the cause of a build failure– or, rather, the cause of a ruby gem that was causing the build failure.
The only hint I had to go on was this message in my Jenkins console:
Installing acts_permissive (0.3.2) /tmp/hudson7424891008093103684.sh: line 5: 12423 Killed bundle Build step 'Execute shell' marked build as failure
It couldn’t install the gem? Why? To the console:
jenkins@li128-183:~/workspace/musicstand $ bundle update acts_permissive Fetching source index for http://rubygems.org/ Using rake (0.9.2.2) Using RedCloth (4.2.9) ... NoMemoryError: failed to allocate memory An error occured while installing acts_permissive (0.3.2), and Bundler cannot continue. Make sure that `gem install acts_permissive -v '0.3.2'` succeeds before bundling.
What the what? Sadly, trying it manually was no help:
jenkins@li128-183:~/workspace/musicstand $ gem install acts_permissive -v '0.3.2' Killed
That’s it. Nothing else.
I went to my dev folder for the gem and explored it. Nothing seemed out of the ordinary. So I created a new gemset and ran “bundle install” forcing an installation of acts_permissive. It installed fine; however, I did notice that Bundler seemed to “pause” while installing it. On a hunch, I went to look at what the size of the .gem file was.
Embarrassing as it is, the .gem file weighed in at a whopping 23M. That’s just ridiculous for a few text files! Further investigation showed me that my development directory was a full Gig! What the what?
Without thinking, I was including the spec/dummy rails application, which contains a log directory, and log/test.log was nearly a gigabyte in size! Consequently, that was all getting packaged up, and then this 1G gem was trying to be compressed on the server and borking because of failed memory!
I had the normal “s.test_files = Dir["spec/**/*"] line in my gemspec, but need to exclude some files. The “test_files” attribute is an array, thus, I added the following lines to my .gemspec:
s.test_files = Dir["spec/**/*"] s.test_files.delete("spec/dummy/log") s.test_files.delete("spec/dummy/log/development.log") s.test_files.delete("spec/dummy/log/test.log")
This dropped the .gem file back to a few kilobytes, which is all it should ever be.
It took a bit of detective work, but the “bundle update ‘killed’” problem was a crisis averted. This would’ve been grueling to troubleshoot if I didn’t own the acts_permissive gem. However, in the interest of helping others, it can be pretty easy to at least rule this problem out if your ‘gem install’ or ‘bundle update’ fails mysteriously. Grab the .gem file from RubyGems and check the size. If it’s crazy huge like acts_permissive was, then you might just be filling up RAM when it’s extracted.