MettaProgramming

I took a trip to Portland recently to traipse through OSCON. I was mostly in the exhibition hall with all the great schwag and company booths– many which had posted job announcements. While there, I was once again frustrated by a trend I keep seeing. The trend can be described as an “arms race of job announcements,” and has gotten to the point where it’s difficult to find a development job listed by a company that is not seeking a “ninja,” or a “rockstar,” or some similarly absurdly described candidate.

Smart and Gets Things Done

As best I can tell, this trend really took off– even if it didn’t start– with Joel Spolsky’s blog article titled “The Guerrilla Guide to Interviewing.” In that article, he stated that there was one primary requirement for working at Fog Creek Software: “Smart, and gets things done.”

His basic argument, with which I tend to agree, is that it doesn’t really matter what your past qualifications are as much as it matters that you can do to things: 1) Learn shit, and 2) Actually do shit.

His point? If you can’t learn, you’ll be stuck trying to write software in Visual Basic and someone will eventually shove you off of a roof out of frustration. Furthermore, if you don’t actually DO anything, but rather just talk about it, or think about it, or tell others why you could do it better– then you won’t even write software in Visual Basic. You won’t write any software at all, and someone will eventually shove you off of a roof out of frustration.

But if you can learn, it doesn’t matter what you come into the job knowing, because that’s merely a starting point. It’s how your knowledge and skills translate over time that’s important. You can learn <insert whatever you need to do your job here> quickly. I say quickly because you are a DOER, and doers always learn new stuff, so they can DO more stuff.

The most important thing here: The job will probably change, and when it changes, it’s the people who can LEARN, ADAPT and DO that will help the company succeed. The person who came to the job with one incredible skill but can’t learn probably has that one incredible skill in a stupid technology like Visual Basic, and someone will eventually shove–

well, you get the idea.

The problem with catchy words

So, herein lies a bit of the problem. Joel also talks about hiring the best people, treating them like “rockstars,” etc. Again, I think he has a point, but that there are a great deal of people who are stupidly picking up the hot new terms like “rockstar” and “ninja” and using the words, basically, without using their brains.

I see them a lot. Those posting that sound all hip and cool: “Are you a Python Guru?” or “We’re looking for a Ruby Rockstar for-” or “We want a PHP ninja to-” Everytime I see one of them I want to slap them in the head with the nearest O’Reilly book and then vomit.

This type of job posting proves one thing to the very people that you want hire: That they don’t want to work for you.

Why?

Because the very people you want to hire are the ones who describe themselves as “hard working” or “with a lot to learn” or even “not as good as many, but loyal, friendly and likes to learn new things.”

Do you really want to hire a ninja?

What’s a ninja?

It’s such a stupid, overused buzzword! Do you even know what it means? It’s either an assassin or a stupid 14 year old jumping out of a dumpster brandishing a medieval sword.

You don’t want either of those things!

Ninja: It's a word that's THIS overused

Yes, there’s some evidence that a well trained ninja was, if absolutely nothing else, a competent assassin (though the Samurai made fun of them). A ninja is basically a person who does one single thing really well, they kill people, and the rest of life- including the whole “getting along with people” part- they could give a rat’s ass about.

Human interaction to a ninja is “kill them!”

Conflict resolution to a ninja: “Kill them!”

For those who miss my subtly: A ninja programmer’s response to pretty much anything is going to be this: “Do it my way and no-one gets stabbed in the face with my medieval sword.”¹

Think about it. Do you really want to hire someone who does one single thing really well? To the exclusion of things like “showering” and, say “talking to other human beings?”

Ninjas are Zombies!

Here’s a neat trick: think of another stupid, overused buzzword: Zombie. What if I told you that ninjas are just zombies with black bags over their heads?

They do one thing really well: kill people (eating their brains- mostly the brains of your team if you hire them), and they could give a rat’s ass about things like “human interaction” and “conflict resolution.” What’s their solution to everything? “Kill them! (and, since we’re here, we could maybe snack on their brains too…)”

Ninjas are zombies! They’re mindless idiots going around trying to do one thing.

That’s it.

When you say “I want a ninja” you’re saying “you can be a complete asshole, refuse to learn anything new, refuse to respond to other human needs or the needs of the business, and also be a social trainwreck. Oh, and you don’t really need to know, or care, about 90% of programming or computer work, but if you can sit in your hole and <do that one thing well> and not talk to anyone, you’re the guy for us!”

You want a ninja-zombie as your lead developer, don’t you? Admit it.

Rockstars: They go to 11

Alright, I think you get my point, but let me touch upon rockstars for a moment. Here’s another place where I think Joel’s point was completely missed by a lot of people. Joel says “hire the best people you can find and treat them like rockstars” and in pure politician-like “I don’t really want to put any actual cognitive thought into this process, so I’ll just pull a buzzword”-style, job announcements start popping up for undefinable qualities such as “Ruby Rockstar.”

You want these guys in your company?

The mistake here is that all the corporate bozos think “Hey, ‘Rockstar’ is the current buzzword, so I’ll use that too!” without stopping to think about one thing: “Rockstars don’t usually make the best employee material.”

Think of the words “punctual” and “hardworking.” Okay, now, keep those words in your head, and think of the word “rockstar.”

Yeah, your neck hurts doesn’t it?

See what happens when you take things out of context? Joel’s statement was “treat them like rockstars.” In otherwords, treat them like they mean something, like they matter, like the company depends on their happiness… so they will be happy… and do really good work… and make you more money.

The line was emphatically NOT “Make them into rockstars.”

Seriously! Think about your average caricature of a rockstar.² They show up late, drunk, stoned, with a 16 year old’s bra stuck to their belt, and give you a loud, shitty performance of something they don’t feel like playing before hopping back into the bus for more sex, booze, and food.

Okay people, repeat this after me:

“I don’t want to hire a Rockstar!”

A rockstar is probably worse than a ninja, because at least a ninja- or a zombie, for that matter- can do one freakin’ thing well. The only thing the rockstar can do is pray to god that the sound team can mix together the shit they call a studio performance, and that the stage crew can hit blow the fireworks early to cover the guitarist tripping and falling because they’re too freakin stoned to remember that there’s a drumset behind them.

They think of themselves as the best thing that’s ever happened to you, and if you tell them otherwise, they’ll freak out.

Rockstars look good. Full stop.

You know who the rockstar is? It’s the young programmer I met at the Open Source Bridge conference who immediately berated me for using a different database strategy than him– without ever stopping to listen to the problem I was solving, to hear about my application’s design, or to learn why I would choose one over the other.

I was wrong, they were right, and I should really just stop being stupid and do it their way.

Yeah, that’s who I want on my team.

Fucking rockstars.

Are you an incompetent programmer with an overblown sense of self-worth?

Here’s the part that bugs me the most: The people posting these job announcements are actively selecting for people with a tendency to overstate their abilities while understating their faults.

It’s called the Dunning-Kruger effect and it’s well documented in both the scientific literature andpopular journalism.

The basic Gist is this: Incompetent people tend to overstate their abilities and think they are amazing, while highly competent people tend to downplay their skills and think that they are less than amazing.

And you know it’s true.

Go pick the first male rockstar-ninja-zombie programmer you can find and I’ll pick the first quiet, understated female programmer I can find. Then we’ll see which one can actually shut the hell up about how great they are and get something done.

The best people for the job are not the ones who are going to feel comfortable applying for the “Amazing Rockstar!!1!” position for the very reason that they are the ones that you want to hire: because they are too busy being amazed at all the stuff they don’t know to be rockstars who tell you everything that they do know.

They don’t think they know everything. In fact, with everything there is out there, they realize that they know basically nothing. Most importantly: They know that there’s a lot to learn, and they are trying to learn it.

The rockstars? The ninjas? They already know it, and they’ll tell you.

In fact, They’re more than happy to tell you how much they know.

Every single time you interact with them.

Folk Developer: Will learn and be nice for food

I don’t apply to any of these positions because, while there are a lot of things that I am and can do, there are certain things that I am emphatically not. A partial list of the things that I’m not is:

• Rockstar
• Ninja
• Zombie
• Best person in the world at <fill in whatever you want here>

Me, I’m not any of that. And that makes me think, from the majority of jobs I see lately, that I won’t be good enough to make the cut. And let’s face it, if they’re advertising for a rockstar, then I am probably not, because I do a lot of things competently, not one thing better than everyone else including you.

See here’s the thing: I’ve been programming for 25+ years, I’m competent in at least 10 different languages, and really good in at least 4.³ I’ve built everything from robotic control systems to mathematical models, from spatial applications to social web applications. I’ve taught programming, and am about to do so again, and I do other things like start a Ruby users group.

By many-if-not-all accounts, I’d be one hell of a developer to have on a team; yet many teams are seeking rockstars.

I am not a rockstar.

I’m not the guy on stage with lights and explosions and a screaming electric guitar.

I’m the guy who goes to a party and sits on the couch singing a song on a guitar. A really fun song, with really good guitar work, but not anything that needs lights and explosions. I may also pick up a banjo or back someone up on a number of other instruments, but mostly I just stay at the party and hang out with people.

No lights. No explosions.

I’m a folk developer.

I’m a seeker, I’m a learner. I’m a hard worker who will spend his free time coming up to speed on a technology for curiosity as well as success. I’m loyal, I’m friendly, I’ve got a sense of humor and would much rather laugh at myself than at anyone else. I spend my off time programming, like lots of people who love programming, but I also spend my off time playing the Irish flute and banjo, cycling, brewing cider and mead, working in community theater, and lots of other social pursuits.

Despite this (or, rather, because of it) I see myself as little more than “a decent programmer who’s probably not as good as most, but might be better than some.” In fact, I don’t have enough fingers to count the number of jobs I’ve actually turned down because I thought that I was probably not good enough– only to find that someone else was hired whom I actually know that I can outperform.

I am not going to apply for a job as a Python ninja or a Ruby Rockstar, because I’m not a ninja or a rockstar. I’m a person who knows a heck of a lot less about Ruby than many other Ruby programmers.

I’m a person with a lot to learn.

Exactly zero of my qualities describe a rockstar.

Are you an incompetent company with an overblown sense of self-worth?

And I know that’s also true of many of my developer friends and colleagues. Companies select for people who are not them.

Here’s the clincher. Most of those companies who are hiring ninjas and rockstars are probably doing so because they see themselves as ninja and rockstar companies.

They are the companies that say things like “do you want to work in our cool-ass company where everything is better than any other company you’ve ever worked for, where we have foosball all day and are all awesome and badass about everything we do?”

Sound familiar? I’ll bet it does.

It sounds like a rockstar of a company.

1. No, don’t ask your ninja programmer why they have a European weapon, you’ll get stabbed in the face. [↩]
2. which is all we’re really talking about in either case- neither Joel or you are talking about Dar Williams, here. [↩]
3. 5 if we count Ruby, which I’m learning more about every day [↩]

We had a great meeting last night for the RubyGorge Block Party. Pasta, beers, great talk, and programming. What more could a guy want? (Wait, don’t answer that… “No, honey! I–” <smack!>)

We stumbled a bit on our discussion of blocks, mostly because I threw Jason to the wolves (“Uh, so let’s start. Jason, go!”). Partly, it was because blocks are such an unusual concept to non-Ruby programmers, that it’s just hard to wrap your head around them.

So, I thought I’d write a few posts on Lambdas and Blocks– mostly so I can work through some things to try to understand them myself. Because I’ve been a Pythonista for so long, I thought I’d try to discuss Ruby blocks by describing them in terms of Python code. Strange idea? Of course, but let’s see how it goes.

Let’s start with a focus on lambdas, and let’s start in full-on Python code.

Building a Stream Reach

There are times that you want to access all the elements in an array, and perhaps operate on the attributes of those elements, instead of the elements themselves. To wit:¹

from __future__ import division # if using Python 2.6, let's divide to floats
import random

class StreamSection:
 def __init__(self,l,w,d,s):
  self.length = l
  self.width = w
  self.depth = d
  self.slope = s

 def __repr__(self):
  return "SS:(%i, %i, %i, %0.3f)" % (self.length,
          self.width, self.depth, self.slope)

reach = []

for i in range(10):
 l = 50
 w = random.randint(5,10)
 d = random.randint(1,5)
 s = random.random()
 reach.append(StreamSection(l,w,d,s))

print(reach)

>>> [SS:(50, 10, 3, 0.252), SS:(50, 5, 1, 0.843), SS:(50, 8, 3, 0.723),
SS:(50, 7, 2, 0.548), SS:(50, 8, 2, 0.573), SS:(50, 6, 2, 0.218),
SS:(50, 9, 3, 0.149), SS:(50, 6, 1, 0.555), SS:(50, 6, 5, 0.425),
SS:(50, 8, 2, 0.088)]

So here we create a stream reach comprised of 10 rectangular stream channel sections. Each section has a length of 50 meters, a random width, depth and slope.² Let’s say we want to come up with some average values for width and length. One way to do this is:

def average(lst):
 return sum(lst)/len(lst)

widths = []
depths = []

for sec in reach:
 widths.append(sec.width)
 depths.append(sec.depth)

ave_w = average(widths)
ave_d = average(depths)

print(ave_w, ave_d)

>>> (7.2999999999999998, 2.3999999999999999)

What we do here is create a list of widths and depths, then average those lists. We need to iterate over the reach to accomplish this because we need to access the width and depth of each section (Of course, Python has list comprehension, but that’s something we’ll get into in a bit).

Intro To Lambda

What we want to explore is a way to do this without the need for the steps of “create a function, iterate through the list, build a new list, then send that new list to the function.” This is the perfect use for a lambda.

A lambda is a “nameless function.” It’s a function that can be passed around like an object, need not be defined before hand, and is garbage collected automatically when we leave the scope. Lambdas are like really stupid, super simple, one-line functions. For example here are two really simple lambdas:

def go:
 first = lambda x: int(x)
 second = lambda x,y: x*y

 print(first(2.3))
 print(second(2, 3))

go()
>>> 2
>>> 6

first(3.5)
>>> ERROR! Undefined variable 'first'

Lambdas take arguments and do one line’s amount of stuff to them, automatically returning the result. For instance, the following is equivalent to the average() function above:

ave = lambda x: sum(x)/len(x)

print(ave(widths), ave(depths))

>>> (7.2999999999999998, 2.3999999999999999)

While this functionality isn’t really too useful here, it becomes moreso when we start to combine it with other functionality.

For instance, Python’s map() function applies a function to each element in an iterable collection. To use map(), we need to call it as map(function, collection). This means that we need to pass a function into the call itself. We can write a function, store it, and then send that, of course, but sometimes we just want to do something quickly and throw it away. With lambdas, we can create temporary nameless functions on the fly.

List Comprehension

Previously, when we wanted a list of stream widths, we needed to create an empty list, iterate over the entire reach, and pull the width from each reach, adding it to the empty list of widths. Now, we can use map() to apply a lambda function to the entire list, and that lambda function can simply return the “width” attribute of each element:

widths = map(lamba x: x.width, reach)

print(ave(widths))

>>> 7.3

It turns out that this is so powerful a concept, that Python implemented list comprehension, which is essentially a way of doing this same thing:

[i.width for i in reach]

>>> [10, 5, 8, 7, 8, 6, 9, 6, 6, 8]

print(ave([i.width for i in reach]))

>>> 7.3

Making it Zippy

Using this nameless function, we can forgo the step of creating methods just to do something simple once. We can also do a lot in a single line of code. For instance, we can use Python’s zip() method to really reduce our code. zip() returns a list of tuples where the i-th tuple is the i-th element of each tuple. From the Python documentation:

>>> x = [1, 2, 3]
>>> y = [4, 5, 6]
>>> zipped = zip(x, y)
>>> zipped
[(1, 4), (2, 5), (3, 6)]
>>> x2, y2 = zip(*zipped)
>>> x == list(x2) and y == list(y2)
True

So, let’s add zip() to our new understanding of map() and lambda, then print the averages of our stream reach in one line. Yes, all the code from our first average attempt in a single line:

print map(lambda x: sum(x)/len(x),zip(*map(lambda x: (x.width, x.depth), reach)))

>>> [7.2999999999999998, 2.3999999999999999]

Scary? Not so! Let’s look at it like maths and start innermost parentheses and working out. I’ll describe each element, and then replace that element with ellipses (…) in the following element. This way, you can build it up yourself and try each part in IDLE or the commandline:

1. Use lambda to give us a tuple of element.width and element.depth for whatever duck-typed element we send it: (lambda x: (x.width, x.depth))
2. Use map() to apply that nameless lambda function to the entire list of stream reaches (map(…, reach))
3. Use zip(*args) to take the resulting list of two-element tuples and return two lists of the elements (i.e. a list of widths and a list of depths) (zip(*…))
4. Then we create another nameless lambda function (lambda x: sum(x)/len(x))
5. Finally, use map() again to apply the second lambda to whatever is returned from zip() (map(…, zip(*…)))

Coda

So, here we see the power of lambdas: nameless, temporary functions that get immediately garbage collected upon leaving scope. It helps Python become a much more functional language. When programming in Python, I tend to use lambdas (along with map() and zip()) pretty much all the time. Whatever I’m doing, I find the need to iterate over the elements in a list and perform some operation on them, and there’s always a need for these non-reusable bits of code that I use to perform those operations.

Recently, I’ve learned that while lambdas are something I used daily in Python, they’re something I will very rarely use in Ruby. This is because while lambdas are something that were added to Python, they are something fundamental to Ruby. I’ll discuss that in the next post, where I try to write an example to help me wrap my head around blocks.

1. Because actual examples are better than fake ones, I’ll use a subset of the code I used when I was programming a water quality modeling application for Oregon DEQ. [↩]
2. The fact that they’re stream channel sections is unimportant. Replace “stream channel sections” with “donuts” if you want. [↩]

Pythonistas all know about Unladen Swallow, the blazingly fast heavily optimized implementation of Python. It’s sponsored by Google¹ and used extensively internally as well as in the App Engine. Like App Engine itself, it’s limited to Python 2.5, which is annoying at best for those of us who really love Py3k.

A couple days ago, I stumbled onto a post by Jesse Noller that had some really great news. Unladen Swallow is going to be merged into the Py3k branch!

…a PEP is coming, proposing Unladen Swallow be merged back to Python-core – specifically, the Python 3k branch. Yeah, that’s right – a PEP to merge Unladen Swallow back into the mothership, but to Py3k only. Talk about a shot in the arm!²

Back when I worked at Oregon DEQ, I programmed a stream temperature modeling application in Python³ and was forced to create C modules of many functions that were just too slow when run 14,000 times in a row. Unladen Swallow and Py3k would make that kind of necessity nearly vanish. This is definitely something to look forward to.

1. well, maybe not. We can at least say that it’s “encouraged” [↩]
2. via jessenoller.com – Unladen Swallow: Python 3’s Best Feature. [↩]
3. actually, I was re-programming spaghetti-code Visual Basic extension to Excel… unfortunately, we had to keep the Excel interface. I’m working on that [↩]

Over the past few months, I’ve been working with Rediviva Magazine on a special (somewhat secret) community project.  We’re getting closer to a possible launch and I’d like to get some more people involved so I thought I’d put the word out there to see whether there are any local programmers interested in working on it with me. Feel free to pass this around.

Overview

Ideally, I’d like to build a small team of programmers who are interested in building an Open Source Software web application that can hopefully become something bigger than what we at Rediviva want to start with. It’d be ideal if a couple high school hot shots are around who want to work up their resumes for later development jobs, eventually becoming a Project Manager on this one.

Bottom Line

1) This is not a money making opportunity, full stop.

Like most Open Source Software projects, this is is geared to the person who is a programmer because they really love to program; because they love making really cool stuff for the mortals. At best, we’ll be able to buy you lunch during code sprints. If we run into a hot shot high school hacker or two, we can definitely work out some sort of “Web Development Internship” credit, which might be useful to you.

2) You will be working as part of a team.

Sorry, but if you are the “my way or the whine-way” coder, or one who does great things sitting in his basement but can’t talk to a normal human, I can’t use you. I’ll take personality over skill any day of the week, because I can teach someone the skills they don’t have.

Speaking of good team players, a note to the girls: Female coders rock, you know you do. You communicate and work as a team by default- rather than force. Rediviva is run by 3 Women, with one more new on staff. I’m the token male. If there are some women out there who want to code, you are most welcome.

3) This isn’t “work.”

We’re not talking “a job,” we’re talking about “a project.” It’s Open Source Software, so I’m expecting you’ll be working on it in your spare time and during code sprints- not 8 hours/day (unless you want to). The Open Source coders out there are familiar with this, still, it’s been a question so I’ll state it outright.

The Details… Or Hints Thereof

Okay, if you’re still reading, you’re definitely an Open Source coder. So what are we doing? Well, I can’t give too many details,¹ but here are the things you need to be able to do or able to learn:

1. Programming: If it’s not obvious, you need to be a programmer. We’ll be using Python, PHP and Java, so knowledge of those languages would be great, but if you’re a programmer in any two other languages, you can probably learn what we need fast enough.
2. Web Development: We’re programming for the web, and creating an application that’s easily deployed by anyone. Here’s what we’re using:
   1. WordPress Custom Programming (PHP): We’ll be modifying a WordPress theme for Rediviva and eventually creating a Plugin.
   2. CodeIgniter (PHP): We’re developing a web app using this framework to start with.²
   3. Google App Engine (Python or Java): We’ll be building the web app in this eventually, so anyone can easily deploy and use it for free.
   4. Google Data API (PHP): Were working mostly with the Calendar API currently.
      
   5. Google Wave API (Python or Java): You’ll get a developer preview invitation to Google Wave if you don’t have access already. Go watch the developer preview to see what Wave can do. Oh, and request access to the Wave Developer Sandbox now- it takes a while.
3. PivotalTracker: It’s an Agile project management tool that we’ll be using to organize our development. It’s free, so if you don’t have an account, we’ll just set up one when we start and you can learn to use it on the fly.You don’t need to know all of this, or any of it for that matter, you just have to be interested in learning how to- and able to learn how. Any programmer can pick up the details of a framework, this list is just here so you know what you’ll be picking up.
4. Github: Source control is hosted at Github, so get a free account if you don’t already have one. Git’s pretty easy to use, and if you haven’t, it’s a good opportunity to learn.

By the way, I’d much rather program that have flame wars, so arguments about why these tools suck and other/completely different tools are the greatest thing ever will fall on deaf ears.³

What’s Next?

Interested in playing along, or at least learning more about the project? Contact me (John) via any method you see fit, the best being email or Twitter links at the top of the page. We’ll set up a meeting around Hood River or somewhere else in The Gorge and talk a bit. That way I can give you more info on what we’re trying to accomplish and see whether it’d be a good fit to work on it together.

Let’s get coding, yo.

1. And the good coders will be able to figure out a lot of details by seeing this anyway
2. Don’t bother flaming that Cake is better. I don’t really care.
3. Yes, I know there are other tools. Yes, I know there may be better tools. But honestly, tools are tools– and all are, for the most part, just as good as others. Choice of tools is basically a matter of opinion, despite the flame wars. These are what I started using when building this project, so this is what we’re going for. Not trying to be mean, just hate spending time on “what to use and why” when that could be spent building something useful.