What followed instead was a search for the cause of a build failure– or, rather, the cause of a ruby gem that was causing the build failure.

The Problem

The only hint I had to go on was this message in my Jenkins console:

Installing acts_permissive (0.3.2) /tmp/hudson7424891008093103684.sh: line 5: 12423
Killed
bundle
Build step 'Execute shell' marked build as failure

It couldn’t install the gem? Why? To the console:

jenkins@li128-183:~/workspace/musicstand
$ bundle update acts_permissive
Fetching source index for http://rubygems.org/
Using rake (0.9.2.2)
Using RedCloth (4.2.9)
...

NoMemoryError: failed to allocate memory
An error occured while installing acts_permissive (0.3.2), and Bundler cannot continue.
Make sure that `gem install acts_permissive -v '0.3.2'` succeeds before bundling.

What the what? Sadly, trying it manually was no help:

jenkins@li128-183:~/workspace/musicstand
$ gem install acts_permissive -v '0.3.2'
Killed

That’s it. Nothing else.

The Investigation

Luckily¹ I am the writer of the Gem in question. ActsPermissive is an instance-specific permissions system based on “circles of trust” (a bit like Google+ circles).

I went to my dev folder for the gem and explored it. Nothing seemed out of the ordinary. So I created a new gemset and ran “bundle install” forcing an installation of acts_permissive. It installed fine; however, I did notice that Bundler seemed to “pause” while installing it. On a hunch, I went to look at what the size of the .gem file was.

The Solution

Embarrassing as it is, the .gem file weighed in at a whopping 23M. That’s just ridiculous for a few text files! Further investigation showed me that my development directory was a full Gig! What the what?

Log files.

Without thinking, I was including the spec/dummy rails application, which contains a log directory, and log/test.log was nearly a gigabyte in size! Consequently, that was all getting packaged up, and then this 1G gem was trying to be compressed on the server and borking because of failed memory!

Lazy, John!

I had the normal “s.test_files = Dir["spec/**/*"] line in my gemspec, but need to exclude some files. The “test_files” attribute is an array, thus, I added the following lines to my .gemspec:

  s.test_files = Dir["spec/**/*"]
  s.test_files.delete("spec/dummy/log")
  s.test_files.delete("spec/dummy/log/development.log")
  s.test_files.delete("spec/dummy/log/test.log")

This dropped the .gem file back to a few kilobytes, which is all it should ever be.

Coda

It took a bit of detective work, but the “bundle update ‘killed’” problem was a crisis averted. This would’ve been grueling to troubleshoot if I didn’t own the acts_permissive gem. However, in the interest of helping others, it can be pretty easy to at least rule this problem out if your ‘gem install’ or ‘bundle update’ fails mysteriously. Grab the .gem file from RubyGems and check the size. If it’s crazy huge like acts_permissive was, then you might just be filling up RAM when it’s extracted.

1. or, unluckily, as the case may be

Sqoot is just a light on what’s seemingly rampant in a field where I just assume there are awesome women doing awesome things. How much of this stupidity am I unaware of? How much do I add to it. Does it really make us feel better to treat other people like shit?

Sexism At Hydrasi

Kerridra and SI1138, Hydrasi's lovable mascots

It strikes me as somewhat odd that I am a man, in an really male dominated field, in a male dominated world, who’d write about this. I mean, what do I know? Just as with white people who don’t see things that every person of color takes for granted, I’m sure there are a great many things than I just blindly ignore in my happy placement of gender-based privilege. In fact, I know there are.

Recently, we were having some graphic design work done for Hydrasi.¹ Hydrasi has a lot of inside jokes based on our name² and our mascots, one of which is a Sasquatch and one is a cute, blue robot.³

The short story is that there are tiers of service, and for the top “enterprise” tier we were trying to make yet another cultural reference joke that could have gone bad.

For every tier, we have a different image. The free tier shows our Sasquatch, Kerridra, holding a pitchfork, the next tier– the lowest paid tier– shows Kerridra with his robotic companion SI1138. The idea, of course, is that you get more robots, more power, with higher subscription rates.

Attempt at referencing Madmen

At the top tier, the enterprise tier, I thought it would be fun to have a reference to the TV show Madmen, where rich advertising executives have martini parties and red-headed secretaries wear mini skirts. My thought here was not “focus on red-headed chics in mini-skirts” but rather on “At the enterprise level, we take care of everything for you so you can sit back and drink a martini.” Here’s the first rough cut of that design.

The final version. You get a computing rack, NOT a secretary

Again, the idea was to have something immediately identifiable as from the show Madmen, but we looked at it and felt that it came off as sexist, pointless, and sort of missing the point of what it means to be at the upper level of service. In the end we axed my Madmen idea and went with another reference, this one not to a cultural reference, but to service. We put a rack of servers behind Kerridra and decided to say “You are getting people, robots, and servers” instead of “You can get a hot red-head.”

Mindless Sexism

There’s a large part of me that’s embarrassed at the whole idea. I mean, I was just trying to go for a funny pop-culture reference in the same way as we have Star Wars references and WALL-E references. In the end, we recognized it as a misplaced, and overly sexist image, and decided to pull it before it ever went anywhere, but how many things like this don’t get pulled?

I guess the take away is that we did sit down, and make a conscious decision to axe that train of thought because if it’s overtones. We thought about it, we decided. That’s sort of what worries me. How much do we– all of us– throw out there because we don’t think about it. How much is just simply cultural privilege that we swim in like water?

Seeking The Positive

This week kind of got me down, both because of the shitty nature of the way things are, and because of my worry (as my wife and I watch her belly grow and I, at least, hope for two daughters) that I add to it as much as anyone.

I guess one thing to look toward positively is a responses to everything this week. Responses like Selena’s. She chose not to battle, but to, almost quietly, Storify the interaction and then ask for feedback (http://bit.ly/GW1XT7). This was an incredibly mature and awesome way to handle a troll (Cheers to you, Selena, for so much restraint and thoughtfulness). And there was a lot of outrage about Sqoot’s mindless objectification of women, including posts like this by Tom Morris. There’s a lot of outrage and a lot of people, not only women, saying “Why is it this way? It doesn’t have to be, and you need to stop perpetuating it!” That’s positive.

For what it’s worth, Sqoot apologized with a “we can do better” post. This made me feel worse instead of better because it just seemed way too late and seemed to radiate insincerity. The only thing I could think of while reading that was an image of Steve Carrell from The Office, spouting an apology speech which had all of the right words that he knew other people would want him to say.

Coda

Overall, it’s been a disappointing week in Tech. I’m trying to see the positive. Good, strong, well-reasoned responses to stupidity and sexism. That should make me feel good. But the fact that there’s such a culture of “brogramming” makes me feel sad. I hope we can change that, but I’d really expected it to have been changed by now, so am not sure how much faith I have in me and my fellow man.

1. These images are secret, don’t tell anyone you saw them!
2. Oh no, it broke! It’s Brokedrasi!
3. The Star Wars connection here is obvious, but there are others we play with as well

You love the acts_as_follower gem, right? Sure, everyone does. Wouldn’t it be better if it was based in Redis? Sure, of course it would.

This weekend I found the great Amico gem, a low-level Ruby gem for Redis backed followers, and I decided to create Acts_as_Amico, a Rails injectable gem that uses that back-end ability, so now you can have a followers gem that is completely in Redis right in your Rails app.

The code has full-ish, but ever expanding, documentation on the Github page, but here’s a quick rundown

ActiveRecord Objects

class User < ActiveRecord::Base
 acts_as_amico
end

usera = User.create
userb = user.create

usera.follow! userb
=> nil

usera.following? userb
 => true

Advanced Usage

class Admin < ActiveRecord::Base
  acts_as_amico :amico_key => :name
  validates_uniqueness_of :name  # -> do this or be sorry
  validates_presence_of :name # -> this too, you've been warned
end

usera = User.create

puts usera.id
 => 18

admin = Admin.create :name => "frank"

usera.follow! admin
 => nil

admin.follow! usera
 => [1, 1]

admin.followers
 => ["18"]

usera.followers
 => ["frank"]

ActiveResource Objects

class RestObject < ActiveResource::Base
  self.site = "http://mettadore.com/junk"
  acts_as_amico :amico_key => :title
end

usera = User.create

rest_object = RestObject.find(123)

rest_object.title
 => "Bread and Circus"

usera.follow! rest_object

usera.following? rest_object
 => true

usera.following
 => ["Bread and Circus"]

Issues

I have every expectation that there will be problems and bugs. If you find one, please don’t hesitate to file an issue if you’re on Github, or send me a message on Google+ if you’re not.

Hacking Github

This weekend, a Github user named Egor Homakov hacked Github in such a way that allowed him to commit directly to the Rails core project. Since Homakov is not a Rails team member, this is a really big deal.

Since this happened, there’s been a lot of talk about the Rails core being fundamentally insecure. In fact, Homakov has been harping on this for at least 1000 years. A few days ago he filed an issue about a mass assignment vulnerability in the Rails core, and later he illustrated this vulnerability by filing an issue report from the future. He was illustrating that you can inject attributes into a Rails model with an HTTP request.

So, when no-one took him seriously, he took the next step. He created HTTP PUT requests adding his SSH key to a Rails core user id, then pushed a commit directly to Rails core.

How to Hack Rails

It turns out that Rails apps, by default, are easy to hack. Peter Nixey wrote a very detailed post about how Homakov hacked Github and the one line of code that could have prevented it, so if you want the full details, read there. The summary is much shorter.

Let’s create a simple User model which has two attributes, name and role. By default, every attribute of every model can be modified by using update_attributes. We all know this, and we’ve known it for a while. What it means is that any User, even if they don’t have permission, can update their own role. Imagine if I log in to our example app and submit a PUT request to the UsersController with the package {'params': {'id': 23, 'role': 'superadmin'} }. By default, the app will accept this and update the user with the new role.

This is exactly what Homakov did. He sent an HTTP request to Github and used update_attributes to change the Github database. All of this could, he argued, be prevented by adding attr_accessor to the User model.

Rails: Convention over Configuration Security

Now, the philosophical point I want to make about this is that the Rails core team seems to be ignoring their own mantra. All of us know that Rails adheres strongly to the Convention over Configuration design pattern. It’s a pattern that Rails users are taught from day one. In fact, it’s embedded in the framework’s name.¹

One of the outcomes of Convention over Configuration is that sensible defaults should be, well, sensible. The Rails core team feel strongly that attr_accessor should not be a default² and that security is the responsibility of the app developer. I agree that security is our responsibility, but disagree that the dominant Rails design-pattern-come-mantra supports this.

It’s almost as if no-one wants to say “Yeah, we should really take care of this.” Everyone is saying “You know, you should really take care of this.”

Stop the world! We’ve found a SQL injection

There’s a term that makes all software developers give pause: “SQL Injection.” It’s a phrase that keeps us lying awake at night, and gives us nightmares when we finally fall asleep. The idea that someone, using nothing more than a web browser, can change our database willy nilly. It’s a terrifying thought.

I can’t help but take an initial read of all the hubbub and think that we’re not giving it the importance that it’s due. Everyone sees a headline saying Rails has a ‘mass assignment vulnerability’ and says to themselves ‘I should probably look into that at some point.’ It’s too vague, to uncertain. To unemotional.³

I have to assume that everyone would be treating this differently if we called it what it is. Imagine your thoughts (and actions) if you read a different headline, something like “Rails apps prone to SQL injection by default”

Don’t you think you’d get something done? You should, because that’s what we’re talking about. This is a path to SQL injection, full stop.

Who owns this issue

I love Ruby on Rails. Like Python, Scala, Node.js, and a host of other technologies that we have at our disposal, Ruby’s web framework makes being a developer both powerful and fun. There’s so much we can do– and so much we can do very quickly. But there’s a cost to pay. Convention over Configuration is a good thing, but we can’t expect people to learn fast, develop fast– create fast– if we don’t respect the logical outcome of that philosophy.

That outcome is this: We, Rails developers, understand that we are responsible for our security; however, we also believe in you, the Rails core team. We trust you. We believe it when you tell us to follow “Convention over Configuration,” and so we naturally believe that the defaults you give us will be a safe, or at least not horribly, dangerously wrong. We, all of us– developers and Rails core team both– can’t have it both ways. We are telling ourselves to follow Convention over Configuration, but we’re also telling ourselves that SQL Injection is a viable convention, thus, that we have security as a configuration.

Now, personally, I don’t believe that attr_accessor belongs in the model– or at least that security from view-based actions belongs in the model. Rails is an MVC framework, and therefore I don’t like forcing myself to define my model based on actions in the view. I don’t want my view owning control of my model that way. I think the controller should manage these permissions and there are others who feel the same. In fact Yehuda Katz argues this as well.

Still, the overall question remains: How can we reconcile the mantra of Convention over Configuration if we support the standard of dangerous insecurity by convention?

1. Look, just stay on these rails and you’ll move fast. We’re not responsible for what happens if you go over there
2. Incidentally, I don’t either, but you can set this as the default by using ActiveRecord::Base.send(:attr_accessible, nil), but this causes all sorts of problems
3. In fact, to be honest, that’s how I was reading it.

The Power of Good Questions

Some history.

Years ago, I was starting to question my decision to become a hydrologist. Not because I hated hydrology, I did– and still do– deeply love it, but because I couldn’t find a job. Since I’d been a programmer for 20+ years (indeed, I programmed all the way through 10 years in the sciences), I decided to see what options were out there in that field.

So I interviewed for this software development company. I was nervous because I hadn’t actually worked as a programmer in a long time. My title had been “researcher” and “scientist” for a decade. Even though I’d been programming that entire time, I was nervous about being labeled as a “wannabe.”

My interviewer saw through that nervousness right away and immediately started asking me questions that were not the basic “do you actually know programming” but were much more geared toward “what do you think of this subtle programming construct.”  It was, without a doubt, the best interview experience I ever had, and made me respect the interviewer and want to work with him even if not at this job– which I didn’t even get.

I never forgot that interview.

What Goes Around

Fast forward a few years, and last year I happened to see that same interviewer at the coffee shop. He said that he’d just left his last job and was looking for work. That, to me, was a wide open door. Finally, a chance to work with this guy!

“Really,” I said, “It’s not much, but I’ve got some Ruby work I could use some help with for my company if you’re interested”

“I don’t know Ruby, I’ve been learning a little bit, but I can’t do work in it.”

“Tell you what, work with me. I’ll help you learn Ruby while you do small bits of work on my site. Fair trade?”

As I suspected from this now barely remembered interview so long ago, he was a sharp tack and things progressed fast enough that he never even did any work on my site. Rather, I took another contract that was a wee bit too big for me, and brought him on as a sub. And so, finally, I was working with this guy, and helping him learn Ruby, and getting him paid to do it.

A Favor, Returned

A couple months later, I got contacted by another company who wanted me to create a web app. I was just too busy, so I brought this fellow to the first meeting. My response was, basically, “I’m too busy to do this, but I can speak from experience that this fellow here is a good candidate. I’ll let you take it from here.” And I walked away. Now, here we are, a couple months after that, and MyLunchIn.com has launched for our local restaurant Nora’s Table. It’s another Rails app that he created, start to finish, entirely free of my involvement.

I don’t know I started to cry when I placed my first lunch order today. I didn’t create that site, and honestly, I didn’t teach this guy Ruby– I just happened to be around while he was teaching himself. But it still feels good.

All those years ago, I was a scientist scared that I wouldn’t be accepted by “serious” developers and I was. It feels good to think that I brought some happiness back to that person, a favor returned. It just feels like this new website is part of a rising tide that floats all of our boats when we work together, when we look out for each other. When we remember a good conversation.

I’m on my way now to pick up my first ever lunch ordered on this new site. It’ll be the best tasting lunch I’ve had in a while.

The Research Works Act

Unlike many bills, The Research Works Act has very short text:

No Federal agency may adopt, implement, maintain, continue, or otherwise engage in any policy, program, or other activity that–

1. (1) causes, permits, or authorizes network dissemination of any private-sector research work without the prior consent of the publisher of such work; or
2. (2) requires that any actual or prospective author, or the employer of such an actual or prospective author, assent to network dissemination of a private-sector research work.

Now, this seems like a reasonable request at first glance. It sounds like we’re saying “If private people paid for it, private people should own it,” right?

Wrong. Let’s look at the definitions in this bill:

In this Act:

1. (1) AUTHOR- The term `author’ means a person who writes a private-sector research work. Such term does not include an officer or employee of the United States Government acting in the regular course of his or her duties.
2. (2) NETWORK DISSEMINATION- The term `network dissemination’ means distributing, making available, or otherwise offering or disseminating a private-sector research work through the Internet or by a closed, limited, or other digital or electronic network or arrangement.
3. (3) PRIVATE-SECTOR RESEARCH WORK- The term `private-sector research work’ means an article intended to be published in a scholarly or scientific publication, or any version of such an article, that is not a work of the United States Government (as defined in section 101 of title 17, United States Code), describing or interpreting research funded in whole or in part by a Federal agency and to which a commercial or nonprofit publisher has made or has entered into an arrangement to make a value-added contribution, including peer review or editing. Such term does not include progress reports or raw data outputs routinely required to be created for and submitted directly to a funding agency in the course of research.

What’s the gist of this? Let’s look at the National Institutes of Health for an example:

The NIH is a publicly funded institution. Taxpayers– you– pay for its existence. You also pay for the research they do. Because the NIH knows that you pay for their research, when researchers publish papers² the require that the researchers also post the research for free, to you, on their website.

You pay for the reasearch, you should have access to it.

The Research Works Act would make that illegal. Under the act, any “network dissemination” of that research is illegal. The NIH can’t offer it to you for free, the scientist can’t give it away for free. The publishing company owns the information.

You pay for reasearch, you might, if the company wishes it, be able to pay dearly for it… again.

This is wrong.

Life in a Free Market

Don’t get me wrong, I’m a free market citizen. In fact, I’m something of a corporate zealot. After spending years working for the government– a proud publicly paid employee– I got completely fed up with the government’s inability to accomplish what I thought was possible. I left to start a company instead.

I’m a believer in the corporation. It’s the nearly perfect problem-solving entity. In fact, I don’t think of a corporations purpose as “revenue generation,” I see their purpose as “problem solution.” But they solve those problems within a market, and people pay for the solution, so everyone wins. I believe corporations should be allowed to make money within that market. What I do not believe, is that our government should enshrine a market for the corporation. That’s not a market, that’s a baby crib. Nice, protect, safe, with mommy’s breast readily available.

If the market does not exist, or is not strong enough, then the corporation needs to put its big-boy panties on and get lean, or get lost. Period.

Government should not make laws that take tax-payer funded information and require taxpayers to pay corporations for it again. This is not the protection of the real market, and it is not the protection of market values. This is the governmental protection of corporate profits by creating an unreal market.³

Publishers argue that they add value to the work because it is their peer review process that confirms scientific legitimacy.

That’s complete bullshit. I’ve been part of this process. It involves me volunteering my time to review a scientific paper. If this is a “service” that the publishing companies offer, then they are offering it because thousands of people are working for them for free. The publishing company doesn’t do this, university scientists, while they are getting public salaries, donate their time to do this. And they’re mandated to do it, and do it for free. Not by the government, but by the community. I dare you to try to be a scientist without participation in the peer-review process.

The “added value” that the publishers offer is essentially the time of public scientists who are volunteering because they have to. The publishing companies make a mint off of this public time. The researchers do the work, do all the writing, and do all the peer review. Meanwhile, the publishing companies rake in amazing profits.⁴

I’m not against all of that. It’s a great system. Companies taking advantage of a market? Fine. Sounds like a good deal.

Government Should Not Coddle Corporations!

But now we have this thing called The Internet. Now we have this idea of “openness.” Now we have a public that demands open information, that demands access to the research that they paid for– and we have the tools to give them that access, easily.

In short, we have a market that changed– and a publishing industry that, rather than changing and growing up, sits in its crib crying for mommy’s breast of federal subsidies.

This bill is not about protecting a real market. This bill is about enshrining a federal subsidy of the scientific publishing industry. Let’s call it what it is.

If the federal government wants to subsidize the publishing industry to protect its profits, it should say that. If the federal government wants to allow a publishing company to use old-market strategies, by making the new market reality illegal, it should say that. If the federal government wants to restrict access to publicly financed scientific research by putting that research into corporate ownership, it should say that.

And then we’d all acknowledge that this bill should fail.

If we keep passing federal laws to protect and subsidize companies and markets solely to protect their historic profits in a changing market, we risk ruining the entire idea of a free market altogether. If I have a company and can talk the federal government into passing arbitrary laws that make my company profitable despite market forces, it may seem like a good deal to me. It is emphatically not a good deal to our country.

Publicly funded scientific research should be public. Period. If a company cannot remain profitable given that constraint then the result is simple: The company perishes. It is not Congress’ mission to protect the lives of private companies as if they were fragile fluffy bunnies, protecting them from the big-scary coyote called ‘a free market economy.’

1. HR3699
2. as they are essentially obligated to do by the scientific community
3. Maybe I’m wrong. Maybe we should do this. After all, China is doing loads of this and it’s working out great for them, right?
4. in the billions, people. Lots of these journal subscriptions are in the thousands or tens of thousands of dollars per year

It’s not that a tactical programmer will never write tests, it’s just not their focus. They are on the front lines, working, well, tactically. Tests are a strategic issue. To many tactical programmers, tests are not something that are necessary. To the tactical programmer, unit tests are this vague, unimportant, time-consuming thing that someone else tells you that you really should do if you want to have good dental health into your later years. They are not something you need to make this feature work, they are something someone else wants for some unimportant reason.

I’m not going to argue why testing is important. There are enough flies on that carcass. What I want to give, to my team of great tactical programmers, especially, is a concept about writing tests that will make it more palatable.¹

Why Starting To Write Tests Sucks

The fundamental thing is this: The best way to actually accomplish testing is to start writing tests.

The problem is that you look at this huge codebase you’ve written without a single test, and you start feeling sick to your stomach and grumpy about your chair and angry at the fact that you’re coffee is too cold and you say “Screw that, I don’t have time for that stupid shit.”

So, what do you do? You write another feature, that adds more untested code to your codebase, that makes you even more sick to your stomach the next time you think about tests.

And before you know it, the only thing you actually know about tests is this: Those fucking things make me sick to my stomach!

The One-Dish Theory of Work

When I clean my house, I don’t start and say “I’m going to clean the house.” There’s too much to do, too much to even think about. I start in the kitchen– but I hate cleaning, so I can’t even think about the kitchen. When I clean my house, I start by going to the sink and washing, not everything in the sink, but one, single dish.

I tell myself, “I’m going to wash one dish and see where I go.”

The thing is, of course, that soaping the sponge, washing that dish, and putting it away requires a small amount of set-up. Thus, when I’m done washing that one dish, it’s much easier to wash just one more dish. I mean, I’m already there, and I’ve got the sponge in my hand.

Before I know it, all the dishes are washed, and the counter next to my sink is clean. Well, with one clean counter, it’s easier to clean off this other counter. I’ll just clean this one other counter.

Just Write One Test

That’s how a non-test writing, tactical programmer can think about writing tests. Just write one. Because here’s the thing: It’s not that bad. The reason it’s not that bad is that the best way to accomplish testing is to start writing tests.

Not all of the tests. Just one test.

The beauty of testing is that they work no matter how many you have. You don’t have to wait until you have time to go back and write tests for everything all at once. If you just start writing tests for the things you work on today, now, then it’s not such a big task. Don’t write tests for every class in your codebase, but write tests for this one, small class you’re working on today.

Write one test, for just this one method in this one, small class, that you are working on today. Tomorrow, write one test for what you’re working on then. Maybe write two if you’re feeling cheeky.

What you’ll find is that, as you write tests here and there, for just the thing you’re working on, it gets easier and easier. Very soon, you’ll say “Well, I was going to just write a test for just that one method, but since I’m here in the test class, it’ll only take me another 20 minutes to write tests for _all_ the methods, so I might as well.” By the time you have a few weeks to actually fill in the rest of the tests, it might only take a few days, because you’ve been doing it and are more familiar.

Just write one test. Just one. You can decide after you write that one whether you want to write another, but at least write one.

Today.

1. This is not news for seasoned developers, I’m not saying that it’s new