A word of warning: I suspect that I may be completely off-base here. My understanding of this subject is a mix of antiquated theory and limited anecdote. I’m throwing this out as a snapshot of my current understanding and perceptions.

I’ve been thinking about the stock market lately. Not in the sense of me making investments, but in terms of how it works, how it allocates money to commercial ventures. This is part of a general curiosity about what makes the world tick, but there’s also a specific self-interest in wanting to understand what motivates the companies I work for. My background coming into this is from historical reading and personal experience.

But first, why do I care about the stock market? Well, I don’t, really – not the day-to-day of it. I’m not one of those guys who keeps a stock ticker running on his desktop. But it’s important – these days, it’s responsible for a lot of what gets done in the world. Most of our food, clothing, shelter, medicine, transportation, entertainment, etc. comes from publicly traded companies. Also, the origins and theory of it are kinda cool. Essentially, it’s a mechanism that allows a whole mess of strangers to work together on a venture that’s far to big for any one of them to tackle. When you look at it this way, it’s a really elaborate form of human communication. And humans, I find fascinating. I’m really only interested in economics to the extent that it’s about people. I’m most interested in the microeconomic end of the scale, because it’s really the psychology of decision making. Why do people do what they do? How much is internal motivation, and how much is external constraints? Group behavior is also interesting, as in companies. But as economics gets more macro, it gets away from people and eventually ends up in a bad place full of aerodynamics-grade math.

The other reason for caring about the stock market is that a lot of the companies I might work for are motivated by the stock market. If they’re public companies, they end up making a lot of decisions based on how it will affect their stock. If they’re private, they’re probably thinking about becoming public companies, or getting acquired by public companies, which influences them in similar ways.

My limited understanding of stocks comes from reading about 17th century trading ventures. You want to build a ship, outfit it and crew it, and spend the better part of a year sailing it halfway round the world to bring back spices or silver or lumber or whatever. These are huge up-front expenses and there’s a very real chance that the ship will be lost to storms or pirates. The upside is that if all goes well, you’ll make a ton of money. Almost nobody has the kind of money you need for that initial investment. Even if they do, they can’t afford to lose it all. But if you get a big group of people together, then each can bet a share they can afford to lose. They buy shares, and they get a cut of the profits. Now if you get a really big group of people together, you can fund a bunch of ships and play the averages. Instead of feast or famine, you get a nice, predictable return on investment. Instead of shares in a one-time venture, you have an ongoing business, and the shareholders get yearly dividends. More importantly, without some mechanism like this, the happy consumers in your corner of the world will never know the joys of coffee, chocolate, or potatoes.

So stocks and dividends make sense. You invest in a company. You give them money up front, and they give you a piece of the action. It even makes sense to trade stocks. If you decide you’d rather have a chunk of money now, you sell your share to someone, and they get the dividend every year. It makes sense that the value of that share could go up, if the company prospers and grows, and the yearly dividend goes up. The important thing here is the company’s focus on profit. Profit is a measure of how useful the company is, how well it provides a good or service that people are willing to pay for.

But now we get to the point where I think it starts to peel loose from reality. If you think a company will be successful, and its shares will pay higher dividends in the future, it makes sense to buy them now and sell them for more later. It’s logical, but it’s a critically different way of valuing the stock. You’re buying shares based not on the dividend itself, on the profit from the venture, but on how much you expect to be able to resell the share for later. It’s the shift from investment to speculation. The value of the stock is no longer tied to the expected dividends, but to what people think they’ll be able to resell it for, what they think other people will think it’s worth, which is what they think they’ll be able to sell it for. Profitability, useful products, business value, all of that may still influence the stock price, but they’re secondary factors.

We got a prime example of this in the dot-com boom, where there was really no business justification for the value of stocks. There were companies with no product or customers. They didn’t own anything; they hardly had any employees. They didn’t do anything useful. But they’d go public with a lot of buzz and their stock would skyrocket just because people thought everyone else wanted to buy it. Once investors thought it had stopped going up, they scrambled to get out before it crashed, which ensured that it did. Then the cycle would start over with the next cool idea – there was no end of those.

The key is that the investors don’t actually care about the success of the business, as long as they can get out in time. They want the business to look promising, so others will want to invest and drive the price up, but it doesn’t have to deliver. The rewards for investors became largely detached from value creation, from profit. The fundamental promise of capitalism, that businesses are rewarded for doing useful stuff, is broken.

Something strange and bad also happens to the company’s motivations. Remember that they’re always competing with other companies for investment money. The things they should be focusing on – doing useful stuff – take a back seat to activities oriented around market perception. A lot of that seems to do with growth, or the perception of. If a company is doing useful stuff, making a profit, and plowing that back into the business (buying more ships), it will grow. It will have more equipment and people than it did last year. All well and good. But this gets turned around, and growth becomes the measure of success. So companies start doing stupid things in order to have good growth numbers.

Mostly, they go out and buy a bunch of smaller companies who have little to do with their core business. They end up being less than the sum of their parts. The guys at the little company are less motivated, because now they’re just working to make a bunch of strangers rich. That’s even if you haven’t demoralized them by laying off a bunch of their staff. Even if there are synergies to be leveraged (shudder), there are a lot of hurdles to overcome – cultural, political, technical, and organizational – and no guarantee that everyone will ever work effectively together.

So I don’t know what the answer is, or if I even understand the problem. A lot of this is idle curiosity: There’s a complex and important system that looks kinda broken, and I want to understand how it got that way. It’s not like I run a company, or even manage investments.

But the practical side is that my understanding of this will inform decisions about where I work. I’ve come to appreciate how much the business environment and priorities have a real impact on my day to day job. I’ve worked at the little company that got bought. I have friends who’ve worked for the big company doing the buying. I have friends who have worked for the company that’s hiring lots of useless people to make the numbers look good, and others at the one that’s laying off lots of useful people to make the numbers look good.

If I can go into an interview and ask the right questions about their business, I’ll end up a lot happier. I’m not job hunting now, but understanding this will also help me recognize if my company is going off the rails. Maybe the answer is to find a company that’s not playing this game, where the owners aren’t just looking to cash out, where they’re really engaged in the business. I’ve actually worked at a place like that years ago, a little boutique software shop. It was really nice in a lot of ways. I left it then because I had a lot to learn, but I might be happy going back to that kinda gig now.

Since getting my new MacBook, I’ve been trying to pull off the pure Mac play – sticking with the stock applications, choosing integration over individual features, trying to work with what I’m given. It’s mostly worked out pretty well, but it’s taken a bit of tweaking. At this point, I’m working on a combined Mac/Google solution.

The ingredients are: MacBook at home, Ubuntu Linux at work, iPod, cellphone, old Palm Vx. Of the last three, I figured I should be able to at least ditch the Palm. It’s a workhorse, but it’s old and doesn’t play well with the Mac. I could go for the iPhone 3G in a few days. That would kill off all three, plus my digital camera. Tempting at $199, but the $100 $60 a month service plan is nuts for the amount I’d use it. It would take some sort of radical lifestyle or professional change to justify that. I’m not above throwing money at problems, but that’s just profligate.

Getting all my info off the Palm was pretty straightforward. Jpilot let me save all the calendar and to-do info as .ics files. After a small but annoying bit of tweaking to DOS-format and rename them, they uploaded smoothly into iCal. Similarly for the contacts and Address Book. That was one of the big wins – having all of my addresses, phone numbers, email and IM in the same place. Once that was all done, it all synched seamlessly to my iPod. Nice. The only snag (Apple developers, listen up!) is that the To Do list displays on the iPod in alphabetical order, not by priority. Come on guys; what are you thinking? Notes were also a bit awkward. You have to mount the iPod as a drive and save them as files in a “Notes” folder. No application integration there. Still, it might make sense to start storing To Do lists as Notes. If I work with them regularly, I may get used to them. So that’s awkward, but not a killer.

The killer turned out to be that I can only sync the iPod to one machine. The whole philosophy of the Palm is that you can go from home to work and keep your core info in sync. Even on Linux, there was a decent client that I could run either at home or at work. The iPod will really only talk to my home machine. That’s kinda control-freakish. It means that when I’m away from my MacBook, I would need to write down appointments on paper or something, and enter them in when I got back. That’s annoying enough that it got me playing with Google Calendars.

Google Calendars are pretty slick. You can have multiple calendars, set up different access constraints for them, and view them in a single display. My girlfriend and I each have personal calendars that we share with each other, and then I have an “events” calendar that’s open to the public. There’s probably also a middle ground for friends: Stuff that isn’t really private, but I don’t care to have strangers knowing, like who I’m having dinner with tomorrow.

iCal lets you set up read-only calendar “subscriptions”. So I re-exported my calendars, uploaded them to Google, deleted them from iCal, and slurped them back down as subscriptions. From there, they synched to my iPod just like before. I was also able to set up a subscription to our calendaring software at work (Zimbra). That’s gravy. Normally, I only have to worry about meetings and such when I’m in the office and have access to the calendar directly. But it is nice to have them on the iPod with audible alarms and all.

While I was at it, I discovered that Google Calendars has a handy text messaging (SMS) interface. You register your phone with them, and then you just send messages to “GVENT” (48368). Sending “day” gets a response with today’s events. “happy hour at 5pm tomorrow” creates a new event. You can set up events (individually or by default) to send you a text message alert. So if I’m going out without my iPod, I can have my cell phone remind me when it’s time to move on to the next round of fun. I don’t actually see myself using that a lot, but it’s nice to have as backup.

I watched Merlin Mann’s “Inbox Zero” Google Tech Talk video the other weekend, and I’ve gotten all fired up on that. I’d been letting stuff just pile up in my Gmail inbox because I don’t actually get much mail there. Gmail doesn’t use the normal folder setup, but it has Labels, which let you do much the same thing, and a bit more besides. So I got everything tagged and archived in short order. Set up a few basic filters to handle list mail that I only need to check every day or two. They’re not super sophisticated, but they do the trick. An empty inbox is a nice feeling.

I also started playing with Google Docs. I haven’t done spreadsheets or presentations yet, but the documents are nice. They strike a good balance between features and simplicity. I’m a little paranoid about storing all this stuff on a free “beta” service, so it was nice to find that you can easily export them as HTML (and relatively clean HTML at that).

So I think that may be the winning combination. I could also look at the iPod Touch option, which would give me a way to enter data on the go. Right now, the only problem situation on the horizon is DragonCon, where I have to track a whole lot of events, and I’m not planning on lugging my MacBook along. It could be handy if they would publish an ics file with the complete schedule before we got down there. Maybe a separate one for each programming track or room. Hmm…

I’m deploying a vanilla webapp with its own log4j.xml in WEB-INF/classes. Why does this not just work out of the box? Because, hey, maybe you’d want to use some other logging library. Not that there are any out there that don’t suck, but y’know, maybe someday. So we need to leave this all open-ended and configurable.

You see, once upon a time, we had Log4J. It was good. In fact, it was perfectly sufficient. It solved the problem. It did just about everything you could want.

Unfortunately, Sun was also developing their java.util.logging (JUL) around that time. They went through the whole JSR community process, so they started earlier and finished later. It sucked, but Sun owns Java in a very literal sense, so we were pretty much stuck with it. Not in the sense of actually having to use it, thank god, but any poor bastard writing logging tools had to accommodate it.

So then we got Apache Commons Logging. It’s a diplomatic facade that stands in front of either of them so your code can be agnostic about whether you’re using Log4J or JUL under the hood. In practice, anyone who doesn’t have a paycheck from Sun and a gun to their head uses Log4J. Java Developer’s Journal did a comparison of the two. They tried to give a “fair and balanced” report, as in, “there are situations in which JUL might be the better choice.” If you read between the lines, those situations were essentially if you’re writing a toy application that would probably be fine with System.out.println().

There’s something of the paradox of choice here. Trying to leave all your options open is ultimately less effective. Your code has to be way more complex to support everything, and you almost always have some sort of mismatch that forces you into a limited-functionality common denominator. It’s not worth it. Pick something that works and go with it.

So today’s pain came about because rather than doing that, Tomcat 5.5 seems to have slapped yet another layer of indirection and indecision on things. It seems that one of the ways in which java.util.logging sucks is that you can’t have multiple configurations within an application. Why on earth would you want to do that? Well, if your application is a web application server, it needs to run a bunch of java libraries as if they were standalone applications. Kinda killer flaw there, huh? But who can pass up the challenge of hacking around something that thoroughly broken? And hey, it’ll make it standards-compliant.

So you end up with these gems from the Tomcat 5.5 logging page:

the default Tomcat configuration will use java.util.logging.

And a bit later on:

The default implementation of java.util.logging provided in the JDK is too limited to be useful.

How can you say that and still build in support for it? Grow a spine, people! How much simpler would this world be if you just followed that up with, “…so Tomcat uses Log4J exclusively. Don’t like it? Write your own app server.”

But no, what did they do? They hacked together their own implementation of the java.util.logging API. Apparently, it’s not “too limited to be useful,” but it “is not intended to be a fully-featured logging library.” Why bother? You’ve had a fully-featured logging library for the last 6 years. Why would you make the default logging configuration one which by your own admission is really not adequate? Why not make Log4J the default? If you really have to, then you can provide instructions on how to plug in your half-assed substitute.

Apollo 13

Everybody brings up Apollo 13. That seems to be the role model for a high-performance, results-oriented team working under high pressure. Could we maybe pick something a little less dramatic? Remember, this is the same crew of people who put the first man on the moon less than a year earlier. They were working at the height of American Big Science. They were educated and trained as if the future of America, Democracy, and the world rested in their hands, which it arguably did. At the very least, three lives, millions of dollars of equipment, and the scientific prestige of the nation hung in the balance. I don’t care how cool your business plan is, or how much VC money you have, you’re not getting that kind of talent and motivation.

Cost of Creativity

There’s a rant in the “Creativity is not Enough” article about how organizations aren’t supposed to be creative. They are anti-creativity, anti-chaos. The whole point of an organization is to constrain and coordinate people’s actions to focus them on a task. Creativity is mostly a distraction.

It’s not that this guy is an idiot and you should just ignore him; they wouldn’t have published him. He was writing in a different time, witha different focus. For large industrial firms, the goal is well understood, and the main effort is in execution. He wouldn’t think of incremental process innovations as Creativity – that’s just doing things better/faster/cheaper. In a service economy, most of the effort is in figuring out what to do. Success is based more on adapting to the customers – understanding and meeting their needs. And the folks in the trenches have more insight into that than the big boss back in HQ.

But the main point that he’s trying to make is still valid – that creativity and change have a cost. The time you spend figuring out how to do things better is time not getting things done. The time you spend on one innovation is time not spent on others. The payoff has to justify that. And trying to do a dozen new things at once probably means that none of them will be well done.

Summary

Now a quick summary of the key points of the book. Since it’s a number of essays written by different people on the same topic, there’s a fair amount of overlap. There’s also a wealth of supporting detail for these principles.

Pressure always hinders creativity. Don’t confuse a surge of relief at dodging a bullet for the true rush of creativity. If you have real and meaningful causes of pressure, make sure they’re communicated. Meaningless or arbitrary pressure is even worse.

Focus helps. Pressure plus task switching equals stress without productivity. If you have a bunch of different things to do, sequence them; don’t multiplex. Meetings with more than one or two others are fragmented, undirected.

To motivate people, pick an enemy and cast yourselves as the underdogs. The enemy can be imaginary, or just a concept.

Innovation should be constant and pervasive. Everyone should always be thinking of how to improve, how to do their work better. Build a portfolio of ongoing experiments at different stages of maturity. A portfolio spreads the risk. You need both short-term/incremental and long-term/disruptive. You need both blue-sky research and radical solutions to known problems. You need to be willing both to take risks and to abandon unsuccessful projects. Allow for failure; make it cheap. Once an idea is proven, go all-out to make it happen.

Innovation needs to focus on competitive advantage. It has to either benefit your customers, expand your market (meet the new needs of your not-yet-customers), or give you an edge over your competition. Be alert for solutions in search of a problem, innovation for the sake of novelty. Measure performance, cost, benefits, risks. You compete on price, performance, or features. You may need to segment your customers so you can focus on the specialized needs of one group. Plan for your competitors’ response to your innovation.

Build networks of innovation through trusted third parties – investors, executive search firms.

Collaborate with your customers. Quick feedback is better than a polished release. Get them involved before the big investment. Get to know them better than they know themselves. Learn what they do, not what they say they do. Give them what they need, not what they ask for. Show them how it will be used – mock it up, tell a story.

Try to make technology invisible to normal users. Make failure transparent – give the user the information they need to solve the problem. Build tools for power users, so they can scratch their own itches and help others. People learn from experience and peers, not so much from formal training.

Each innovation needs four kinds of support: An on-the-ground champion, an executive logistical supporter, creative idea generators, and practical implementors.

Get outsiders’ perspectives. If you want people to think outside the box, hire from outside the box. Get fresh eyes on your business. Varied backgrounds produce better problem-solving skills. Focus on people who are bright, verbal, assertive, and creative. You’ll need enough outsiders to form a critical mass.

Look at your company through the eyes of your competitors. What are your limits, weaknesses and vulnerabilities? Your competitors are doing your market research for you. If they do better, it’s because they’re meeting a customer demand that you aren’t. If they don’t, you know not to try that.

Learn from the potential customers that don’t choose you. What are you failing to provide them?

Project groups focus on goals, not resources. They minimize turf battles. Align the organization’s goals with employees’ intrinsic motivations.

It worked fantastically well in the industrial age. As we moved into more of a service economy, the principle still applied, though not as strongly. In your typical office, you have specialization of skills: Managers, admin assistants, sales, marketing, and so on. In a one person company, that one person can cover all those roles. Things will go smoother when they can divide the work up among several people, but not by a factor of hundreds.

As software development has become a bigger and bigger business, the obvious thing has been to try to apply the same assembly line model: analyze, subdivide, specialize. And so we come to Enterprise Software development, which pretty much follows the old waterfall model: Requirements, architecture, design, implementation, test, support. The business analysts don’t write code, the architect doesn’t write test plans, and the programmers don’t field support calls. The business analyst writes a requirements document, the architect writes an architecture document, and so on. Each stage does their piece and throws it over the wall to the next team. The principle is sound: The more narrowly you divide up the skill set, the more specialized each group will be, and the better at their particular task. Push this far enough and eventually you can replace them with trained monkeys. Very clean, very elegant. Very efficient? Umm… no.

The problem is that there’s a significant difference between an Enterprise Software solution and a pin. A pin is a very simple thing. Everyone knows how it works and can tell at a glance if a given pin will do its job. Unless it’s bent, blunt or missing its head, it’s good to go. If an end user finds that one pin is poorly made, they just throw it away and get another.

An Enterprise Software system is not simple or obvious. Aside from the fact that even the customer often doesn’t entirely understand how they want it to work until they start using it, the amount of information that has to be communicated from stage to stage is huge. Requirements documents can be hundreds of page. None of the workers has any innate sense of whether the information handed to them from the previous stage is complete or accurate. Every misunderstanding is multiplied down the chain. It’s like playing “telephone” with War and Peace.

The key here is that when you divide up the labor in a process, you also incur a communications cost between each stage. It’s a trade-off: Each stage can be done more efficiently, but information about the work has to be passed between them. In the pin factory, the benefit of specialization is huge, and the communications transfer is tiny – as I said, the quality of a pin is obvious. In a software shop, the benefit of specialization may be significant, but the communications overhead is enormous. To come at it from a different angle, I’ve heard it said that a programmer is lucky if they get to spend 20% of their time actually writing code. The other 80% is communications overhead: reading documents, sitting in meetings, writing documents. Add in the time you spend on rework due to miscommunication, and that 20% starts to sound generous. Here, the assembly line is actually less efficient.

So what do you do? The popularity of “Agile” tools and techniques is clearly a response to this, along with component or service oriented architectures. Break the system up into independent, well-defined pieces, and put them in front of the users as quickly as possible. The catch here is that you can’t use trained monkeys. You need people who can talk to the end users and write code, who can carry the product all the way through the process. Maybe they need help from experts like domain specialists or QA engineers, maybe they can divvy up some of the work to tech writers or programmers, but the key is to capture an end-to-end understanding of the product in as few heads as possible.

What they do is interesting. They look at the address for XYZ Corp and say, “Hmm… that’s out of town. It needs to go to Central.” So they put your envelope inside another envelope and write “Central Post Office” on it.

When it gets to the central post office, they open the outer envelope and read the address on your letter. They say, “Oh, this is going to Chicago,” or wherever. So they put your envelope inside another envelope and write “Central Post Office, Chicago” on it.

Then it gets to the Central Post Office in Chicago. They open up the envelope addressed to them and see the address for XYZ Corp. So they put it in another envelope that just has the 9-digit zip code for the XYZ Corp building on it.

It shows up at XYZ Corp, and the guys in their mail room open up that envelope and see that it’s addressed to Joe Smith. Somebody runs it upstairs to Joe’s secretary, and she opens the envelope and hands Joe your letter. When Joe sends a reply back, it works the same way.

This is how the Internet works.

It’s actually more complicated – there are more middlemen – but that’s fundamentally how it all works. It’s all these little letters (called “packets”) flying around a very, very fast postal system. This is a pretty clear match for email, but it’s also how everything from web pages to streaming video to Voice Over IP works.

When you “go to” a web site, you’re really mailing out a request for a web page. It’s like writing off to a mail-order catalog company. There’s a standard form that defines how you ask for web pages. You fill it out and send it in. You’re sending this little form that says, “I want to see http://www.spuriouspundit.com/index.html”. That request goes out through this metaphorical postal system to the spuriouspundit.com server, and some little toiling minion there xeroxes off another copy of the index.html document and mails it back to you.

Like I said, it’s more complicated than that. How do you keep email and web pages and FTP sites all running on the same machine without tripping over each other? Imagine your office building has a bunch of different departments in it, but they all share the same mail room. Instead of a billing department and a sales department, you’ve got an email department and a web department and an FTP department. The way it works is that they each have different post office box numbers. Whenever a letter comes into your building, the mailroom guys just have to drop it in the right box. One of the rules of this postal system is that the addresses on letters have to have a box number. Furthermore, these box numbers are standardized, so that box 80 is the normal box number for the web department, box 25 is for email, etc. So when you send off your web page request, you know that it’s a web page request, and wherever it’s going, it should be going to box 80.

This is also how you keep your responses straight. Even if you’re the only guy at your company, you could be downloading a couple of mp3s in the background while you’re popping up new browser windows right and left. If all that stuff is landing in the same inbox, you’ll never sort it out. So each time you ask for a web page, you set up a new post office box just for its responses. Your downloads go to boxes 5001 and 5002, and your web pages end up in 5003, 5004, and so on.

Here’s the next wrinkle. Say you send off a request for some big, fat mp3 file that won’t all fit in one envelope. So it gets broken up into a whole bunch of separate letters. Now on top of that, like the real post office, stuff can get lost: Mail trucks get stolen; Some yutz in New Jersey cuts through a long-distance fiber-optic cable with a backhoe. Even if nothing gets lost, there’s no guarantee that everything is going to show up in the order you sent it.

So what do you do? First, you send a letter that effectively says, “Hey, I want to send a whole bunch of letters back and forth with you. Here’s the address you should send all the replies to.” This is where your web browser says, “Connecting to …” in the status bar. Once you’ve got an OK back, you say, “Send me that mp3 file.” You get a whole flood of letters back, numbered “1 of 23″, “2 of 23″ and so on. You count through them and realize that you’re missing number 17, so you send another message saying to re-send it. When you finally have all the letters, you can put them in the right order, open them up, and glom the mp3 file together.

This little procedure we’re going through here is called a Protocol. It’s not part of the postal system itself, but it’s a set of rules that people have agreed on for how to use the postal system. In this case, it’s a way of communicating reliably through a system that isn’t reliable. It’s a layer of communications on top of a layer of communications. It’s very meta.

The internet is built up of layers of these protocols. Like the envelopes inside envelopes, at each stage, you’re only concerned with the outermost layer. You slip on or peel off your envelope, and everything else is just the stuff inside it, be it one layer or many. IP, the Internet Protocol, is the postal system – simple, but not entirely reliable. TCP, the Transmission Control Protocol, provides the reliable, ordered delivery on top of IP. Email (SMTP), the web (HTTP), and others are actually another layer on top of TCP. Essentially, they all define standard forms for different mail-order requests.

Again, it’s even more complicated than that. But for now, lots and lots of little letters zipping back and forth across the world. That’s the way to think of it.

Remember that Picture Hanging essay, about how you aren’t aware of how much you take for granted once you’ve learned something? Yeah. I had no idea how much stuff you have to learn before you can even start programming. You have to understand the environment – what’s going on in a computer – at a much deeper level than most users need. (As the tourist/business traveler/expatriate metaphor rears its head again.)

First, there’s the whole command-line thing. Fortunately, she had that down, ‘cuz I have no idea where I’d even start. Your whole frame of reference has to shift to deal with that. You have to explain about opening and reading from files – that a file is really a sequence of characters (don’t get started on bytes vs. characters). And by the way, the end of a line is actually marked by a character, but you can’t see it. Oh, and pipes: They’re like files, but they don’t really live anywhere. Ummm… yeah.

Then there’s the whole utility program thing: The idea that programs can be little tiny things, that they don’t have to have graphic interfaces – they “talk” to each other.

To most people, an application is something you see and interact with. Even when you do deal with things like the filesystem, it’s mediated by a graphic interface – a metaphor. It’s a good enough metaphor that people aren’t really aware that it is a metaphor, let alone what it’s a metaphor for. To someone whose idea of programs is Word and Excel, the idea of ‘piping’ the output from one program into another just doesn’t make any sense.

OK, so that’s all the environment. Those are the building blocks you have to work with, the walls you live within. Like I said, I got off fairly easy on most of that; I had someone who knew her way around the command line, and had even used grep a few times. The next step was explaining what a program is.

A program is a sequence of instructions, kinda like a recipe. But that doesn’t really capture the issue, because recipes are written for humans. Humans are smart. Computers are very, very stupid. Fast, but stupid.

A computer is like some sort of high-speed, idiot-savant imp. It works very quickly, and it can remember a huge amount of stuff, but it’s fundamentally dumb as a bucket of mud. It can’t think for itself at all. If you want it to do something, you have to explain in minute detail precisely how to do it. You also have to think of all the things that could go wrong and how to deal with them.

Let’s imagine you’ve got this sort of imp, and you want it to go get your groceries. It has to go to the store, find all the stuff on the list, pay for it, and come home. Simple, right? You’re underestimating how stupid your computer imp is. If it knows how to walk, open doors, follow directions, and cross streets without getting run over, it’s only because someone else programmed that much for you.

Let’s assume you’ve got an imp programming language that gives you that much. So you say, “Go to the grocery store, get everything on the list, pay for it, and come home.” First, it goes off and never comes back. You go to the grocery store to see what’s up, and your imp is standing in the freezer section. You told it to get pistachio ice cream. They’re out. The imp is waiting for them to restock.

So now you have to instruct it to try another store. You have to give the imp a list of stores, in addition to the list of groceries. If it gets to the last store without finding everything, it should just come home.

This seems too work, but it always takes your imp a really long time to get groceries. You spend a while following him around (debugging) and discover that he’s going to every store every time, even if he already has everything. Ooops, another fix to make.

You write a bunch more imp code, and send it off. It still never comes home. This time, it’s stuck at the cash register. Spaghetti sauce went up 5 cents from last time, and now your imp doesn’t have enough money, and is stuck in the act of paying. You can just give the imp some extra money, but you’ll always have to deal with the possibility that he won’t have enough. Even though it seems less than ideal, you tell him to just come home if he doesn’t have enough money for everything.

Everything works fine for a while, but eventually, you realize that you haven’t had vanilla ice cream for ages. It’s on the list. You check the stores, and sure enough, they’ve got tons of it. You double-check the list, and it’s on there as “Vannilla”. Okay, maybe you could come up with a system for correcting your typos – one that won’t result in your imp buying a gun when the store is out of gum – but that’s a lot of work. It’s easier for you to just type things right. But if you ever sell (or even give) your imp code to other people, you know there will be an unending stream of complaints about how your stupid imp won’t buy vanilla ice cream.

Programming is all about developing this mindset – learning to think through all of this stuff in advance, figuring out all the ways your imp could screw up anything you tell it to do.

Let’s go through the basics here. You are working at a job. Your employers employ you. They pay you a salary. They pay you this salary because you do something that they consider useful. “Useful” generally means “makes money for them”. There may be a fairly indirect path to it, but what you do ultimately brings money in the door. Your usefulness to them is measured by how much money that is. Your perceived usefulness also depends on how visible the connection is between your work and money coming in.

This is why sales people make so much more than programmers. Their work is directly tied to money coming in. The sales guy can say, “I made X in sales this month.” You can argue that because you’re a really good programmer, it’s easy for the sales guys to sell your work. You can argue that the clever work you did six months ago has resulted in a better product, and increased sales now. But that’s very hand-wavey. It’s hard to put dollar figures on that.

There are two things you can do to increase your perceived usefulness. You can publicise your accomplishments, strengthening in the minds of your bosses the connection between your work and money coming in. You can also focus your work on things that make (or save) the company money. Both of these require understanding at least a bit about the business side of things.

Let’s say you’re a sysadmin (because if you aren’t, you probably know one), and you implemented an automated patch management system. Well, that sure made your life easier, and gave you more time for your other work (‘cuz there’s no end of that). And it’s a much more scalable solution, to boot. But how do you sell that to your boss?

Your boss thinks in terms of money. It’s not that he’s inhumane or clueless, it’s just that money is how you measure things. People and equipment are manifestations of money. You are a person, and your time costs them money. Therefore things that save you time save him money. Now the hard part, the maddeningly imprecise part, is figuring out how much.

In this example, you’ll have to have some idea how much time you used to spend patching machines. Guess – you can preface your report with, “According to our estimates…” Say you used to spend 3 hours a week patching machines, it took you a week (40 hours) to set up the system, and now you only have to spend 5 minutes a day on it. So your system “cut patch management staffing requirements by 85%” and will “earn back its initial investment” after about 4 months. Yes, that’s management-speak, but “automated a bunch of stuff” is meaningless to anyone outside of the tech staff. You have to translate it – you have to explain the business impact of what you’re doing.

Note that this cuts both ways. If you only used to spend half an hour a week patching boxes, and it took you a month to set up the system, you’ll have a hell of a time explaining that. Ultimately, you need to learn to do the math up front.

So now what do you do with your new-found spare time? Make yourself useful. First, start learning the bigger picture. Learn more about why you do what you do, and then why your group does what it does. Then start thinking about how that might change, or how they should change. Research new tools and techniques you could use – not just “cool” stuff, but things that will help you do your job better. Learn to apply that business logic.

You can also learn more about the other parts of your company, how and why they do what they do. First off, this is just interesting, like exploring a new world or some weird parallel universe. But there may also be things you can do to help them out. I’m not talking about doing their work for them; I’m talking about situations where you say, “Hey, we could write a tool to do that,” or “Don’t you have a database to track all that?” – situations where a little work by you could save them a lot of time and trouble. You’d be amazed at how much tedious, repetitive crap other people put up with in their work – things that programmers would find some way to automate.

At one job, we had a graphic artist who got stuck doing a monthly update of a web page from a PDF file. It was table after table of financial data. She opened up the PDF file in one window, the HTML in another, and copied it field by field, by hand. Click, drag, CTRL-C, click, CTRL-V. For every single number in every table. The first time she did it, it took her two full days. She griped about it to me, and I said, “that’s insane.” I spent the next day hacking together a couple of utilities and a perl script, and came up with something that parsed the PDF file and generated the web pages. It took about 5 seconds to run, and after that, she just had about 15 minutes of reviewing the pages and making special format tweaks. I made a new best friend that day.

You can earn a lot of good karma that way. It can also help break up the monotony of your regular work. You pick little projects and do them incrementally, on your own terms. You’ve got your “customer” right there, so you can try out all the XP and agile methodology stuff you’ve been reading about. And it can give you an arena for learning new skills or technology. A quick and simple web-based form entry application may be perfect for teaching yourself Ruby On Rails. A report generation tool may be a good way to brush up on your Perl.

When you’re done, make sure everyone knows. Tell your boss, and get him to talk to your new-found friends over in graphic design or accounting or wherever. It’s one thing to toot your own horn, but it another when people outside your department are singing your praises.

Negotiating 102: Deal from a position of strength. How can an employee get the upper hand? By making themselves essential. It’s not good for the company to have essential employees. Employees get sick. Employees go on vacation (or try to). Employees get pregnant. Employees have children, which is a whole world of conflicting priorities. The company needs to be able to run smoothly while they’re away.

If employees have to bargain, they’ll strengthen their hand by hoarding information. You become an essential employee by not sharing knowledge. You become “the only person who knows …” In tech companies, it’s pretty easy to be the only person who understands a critical piece of code, or knows how to configure specific servers. You generally have to work to not find yourself in that position. But almost anyone can do it. You do it by not writing things down. You don’t document requirements, decisions, client requests, lessons learned, policies and procedures, or the reasons behind them. You keep it all in your head. You prevent it from becoming institutional knowledge.

The people who write everything down, the ones who work up procedures that everyone can follow, write code that everyone can read, and explain all the whys and wherefores of what they do? These are the ones who really create value for the company in terms of institutional knowledge. And if they’ve done a really good job of it, you can replace them at a moment’s notice. Again, the ones who are the most valuable are the ones you can pay the least. But which do you want to encourage?

Employees, particularly skilled, creative workers (as we programmers flatter ourselves to be) should not have to negotiate against our managers for pay and benefits.

This isn’t some sort of hippie socialist crap – there are two real, clear business reasons for this. The first is that it introduces a perverse incentive. I’m speaking in economic terms – this has nothing to do with anyone’s sex life. Not directly. The second, which is sort of related, is that it’s shitty for morale. If that sounds fuzzy-headed to you, you shouldn’t be managing anyone who’s smarter than a chimp and doing anything more complex than stamping out license plates.

The perverse incentive is simply that salary negotiations set up a system that rewards people for not caring about their jobs. Negotiating 101: The most important factor in any negotiation is the willingness to walk away. You go to buy a house, and that’s the first thing anyone tells you. Don’t fall in love with it. If you aren’t willing to walk away from it, they can take you to the cleaners.

So the people who don’t really care about their jobs, the clock-watchers, the ones who can take it or leave it, who haven’t really bonded with their co-workers, who don’t really believe in the product or take pride in their work? They’re the ones who can walk away. The only thing keeping them there is the cash, so it’s going to cost you more. The ones who aren’t like that, who are emotionally invested in the company? You’re turning that against them. You can’t imagine that’s going to make them happy when they find out.

Now, that part applies to everyone. Technical staff in particular have another disadvantage: The personality traits that make them good at their jobs work against them at the negotiating table. It’s not just through lack of people skills or anything – in order to be good technicians, they have to be bad at selling themselves.

Allow me a brief digression here. Technical staff actually operate in a different universe from the business side of the house. They’re “reality-based” in the disparaging way that our current administration uses the term. Put simply, shit will either work or fail based on the properties of the observable universe. No amount of personal motivation or focus-group work will let a spare, obsolete PC handle a million page hits a day. There’s math there. You can prove things.

On the business side of the house, to an alarming extent, believing things makes them true. If the sales guys are confident and self-assured, if they really believe that they’re great salesmen and they have a great product, people will trust them and buy stuff from them. If the managers believe that the current project is destined for success, and can keep everyone enthusiastic and focused, it’s much more likely to be successful. By contrast, once people start believing it’s doomed, they don’t work as effectively, maybe they leave, and the project fails. (This is that morale thing I mentioned earlier.) It’s all unsettlingly self-referential.

These traits which managers and sales people need to be effective are actually bad for technical folks. Arrogance means you overlook things. You miss subtleties that you’d catch if you were more cautious. Two heads are always better than one; three or four better still. (Okay, yes, there’s a limit to this.) But any technical solution will be better if you can put your ego aside and plunk your idea down in front of some other folks for a good, harsh critical review. Being able to sell them on it by force of personality doesn’t make it better. Being aware of your own failings and limitations makes it better. Being able to let go of your proprietary defensiveness makes it better. Learning to value the people who will poke holes in your designs – the ones who are “not team players” – makes it better. A considerable amount of humility is called for here.

This all makes you pretty easy meat in a salary review.

So, point two: Shitty for morale. Yes, we live in a capitalist economy. Companies compete against each other, often fiercely. But within a company, you need to cooperate, to work together, and to trust each other. Without that, a lot of energy gets wasted on internal politics, turf wars, and jockeying for position.

This is particularly acute for tech workers. Technology is complex and huge. People tend to be experts in a very narrow field. Odds are that you’ll have to cross a lot of boundaries to get anything done. Competition – feuding between fiefdoms – can slow everything to a crawl.

Also, it’s one thing if you’re managing guys who stamp out license plates. You know how to stamp license plates, you can watch them doing it, and you can measure their work by the number of license plates they stamp out. Tech guys, not so much. Much of the time, management has no real idea what these guys are really doing. Even if you could watch them all the time, it might not do you much good. It can be hard or even impossible to distinguish between smart, hard-working guys and slackers. Is the problem really hard, or are they just lazy? Is the system running well because your admin set it up right and maintains it carefully, or is he just lucky so far? Maybe he automated the crap out of everything, and now he just has to sit around, waiting for alarm bells to go off. No way to know. (A hint: If you ever have to choose between the guy who puts in 12-hours days doing everything by hand, and the guy who has a bunch of programs doing his work for him, you want the programmer. Brains scale better than brawn.)

So you have to trust these guys. You’re best bet is to put a bunch of them together and hope they’ll keep an eye on each other. That’s still a lot of trust, and why should they look out for you if you’re not looking out for them? If they think you’re chiseling them on pay, they’re likely to chisel back. It’s pretty easy to goof off and look like you’re working, particularly if you program for fun. At best, they’re not going to put a whole lot of brain power into figuring the best way to do things. The surest sign that this has happened is that the really good ones, the ones with a work ethic and motivation, will leave. They have better things to do than put up with this bullshit.

Don’t go down that road. Keep an eye on the market and pay your folks what they’re worth.