Hey there. You're reading Simpler Machines, a weekly newsletter about the human side of software development. I'm Nat Bennett, a former Cloud Foundry engineer, current writer, and occasional software consultant.

This is going to be the last issue until the new year, when I'll start "season two." (I'm doing a bit of consulting and want to focus on that– and I want to give myself some time to think about topics and content without the constant pulse of newsletter creation driving me forward.)

In the meantime, I'm sending out a more playful newsletter once or twice a week called If I Could Give It No Stars, I Would, where I review Google Maps reviews for things like grocery stores and cemeteries.

I talked to Jesse Heitler and Theo Cushion about what it's like to work at their company, Ignition Works. We talked about why they pay everyone at their company the same, how to make decisions quickly while also maintaining an inclusive environment, and why you need a guy who has an arc welder on speed dial.

The thing that impressed me about this conversation was that they're making decisions and structuring their company like they actually believe that performance is a function of systems, not individuals. Most people who talk about software engineering being a "team sport," or about a lot more than writing code, still obsess over individual performance evaluation, who was "responsible for" what features, and making sure everyone is paid exactly right according to their "contribution." Which, if you're paying attention, is obviously impossible to do right, but it's still really rare to see people acting like that's true.

If you're in a UK-friendly time zone, at all interested in enterprise infrastructure problems, I highly recommend getting in touch. (There's a link at the bottom of the page.)

This interview has been edited and condensed.

Nat Bennett: So what is Ignition Works?

Jesse Heitler: Our company has two threads. We help to operate a large PCF installation at a client. And we are developing a technology to remove the shackles of governance compliance from change-producing teams at enterprises. That's the new tagline.

Theo Cushion: We've spent a long time in Pivotal and in other jobs, working with big enterprises, and they're completely hobbled. There's a lot of talk about agile and things like that. And with all of the best will in the world, we don't think it's possible for them to adopt those practices unless there's areas in the governance of the organization that are changed. And so we feel quite passionately that that's a big problem that needs solving.

Jesse: The practices are easy once there's space for them.

Nat: So you're operating a very large Cloud Foundry instance. And then you're also kind of your own first customer, or you're working with your own first customer very closely, to do this continuous compliance piece.

Jesse: Yeah, that's the gist of it.

Nat: What does that look like on a day to day basis? What are the team's daily activities like?

Jesse: It’s evolved a lot through time, depending on our staffing levels, and work levels and stuff like that. At the moment, we're configured, mostly as a result of the pandemic, with all of us spending most of our time on the client operations work, although we're actively hiring at the moment, to get the people hours to be able to have folks focusing on the product development as well.

So in the past, we've been configured, split half-half either developing the product or doing consulting in this area of governance.

Theo: Our business model basically means that so many people on the ops side fund so many on the development side, and we're at that point where we're hiring for those.

We have the typical meetings, the stand ups, the iteration planning meetings, but each one of those has actually been refined and changed over time to accommodate exactly what it is for operating a platform, which is a bit different than running a product.

So for example, when we're dealing with lots of support, we also have to deal with rolling out changes to a whole bunch of different foundations. There’s lots of scheduling, and there's lots of interaction with other teams. We have multiple backlogs optimized for each one of those problems.

Jesse: I think a common theme in the way we work is continuous improvement. As you mentioned, there's-- what's that notion, Kaizen or something like that?

Nat: Kaizen, yes.

Jesse: I find it amusing that I only rediscovered that word last week, but it's very much embedded in our mindset. We're always picking something up trying to run an experiment on it. And so the practices evolve on a regular basis.

Nat: What’s a change that you've made in your process lately? Walk me through the full lifecycle. What problem were you trying to solve? How did you identify the problem? And then what change did you make?

Jesse: Should we talk about consensus?

Theo: Yeah, let’s talk about consensus.

Jesse: So we had an interesting realization, as we started to scale up. When it's three people in a room, making decisions isn't that much of a problem. As you start getting more and more people, the burden of consensus becomes significant, the n-by-n matrix of conversations that have to happen. We just started grinding to a halt in terms of like, forward progress, and the fun of just getting stuff done.

Theo: But also balancing the whole idea of people having input into things and belonging to something and not simply being a dictatorship. How do you marry the ability to make decisions quickly, but also have an inclusive environment?

Jesse: Right, and we had no interest in dictatorial hierarchy. That's not the kind of environment that inspires me.

Theo: And I don't think it gets the most out of other people. We want to work with really, really good people. And part of that is getting out of our way and providing them the tools that they need and the space that they need.

Nat: So both, you don't personally want to be working in a very top down environment. And you also don't think that the people who you want to work with want to work in that environment.

Jesse: And I don't think that environment is very effective.

None of us are smart enough to out-think a whole larger group of people. A team is far more effective when you have people who are sensitive to different signals, and can integrate that and come up with optimal responses that one person might not have come up with.

Theo: And they have the power to execute that. And they have the backing of everybody else to do it, and the encouragement.

Jesse: So the theory is that a high performing team is going to yield a much better result, even if the individuals are in some measures, not the superstars.

A basket of investments of various risk levels is going to yield a larger return, then a single asset with one risk level, because you have some things that will work well in one environment and other things will work well in other environments. And they'll always somewhat be doing good and somebody doing bad that mix has a much better result than placing one bet on one thing.

Nat: So instead of relying on one central person to make all the decisions, you're betting that you will get better decisions, in aggregate, if everybody has a hand in them,

Jesse: Right. So that's the justification for Ignition Works. Personally, I would like to work in that kind of team. One that's oriented around respect and contribution. So we changed everything based on that.

First, we pay everybody the same. Because one of the huge barriers to an effective team is if there's, I mean, pay touches so many things, if there's secrecy around what people are earning, if there's a need for one person to place a value on people, if you have this unrealistic, implied connection between the perception of contribution and the money they make.

Nat: You’re taking very seriously that 90% of performance is the system.

Jesse: I've never heard that idea. Yeah, I like it.

Nat: Oh, yeah. This is Deming. I don't know which one of his books. That’s his pithy comment on performance management, basically, is that measuring individual performance, you're measuring just 10% of what's going on and you're trying to optimize 10%. Why would you focus on that 10% when you could be focusing on the 90%?

Jesse: Yeah. So we pay ourselves the same as everybody else. Every time we mention this, I have to caveat, that doesn't mean that everybody will earn the same from the company. We still maintain a distinction between equity and salary, or salary and bonus and employee compensation, but the employee comp rotation is all fixed. The intent is that everybody should be paid well enough, that money isn't what we're talking about.

So that was step one. Step two was, we changed our thinking to be oriented around who responds for Ignition Works, and who "counts" for Ignition Works on what topics.

We have this tree [of responsibilities]. We replaced all of our backlog and IPM and that kind of ritual for tracking what needs to get done for an organization. We started orienting ourselves around this tree. [At the top] it starts with the company, and the company counts on the board of directors for managing the company's business.

The board of directors has a few topics that it reserves as board level decisions, like hiring. Hiring is so important that you need a consensus on it. So we need to have at least two or three of us to agree on that.

But everything else, Ignition Works relies on Jesse, me, for managing the day to day business. We say “counts on,” derived from “is accountable for,” but the point is that the company counts on me, but I'm not necessarily the person who's doing it. And that's what the tree is.

So the tree then carries on and says, Ignition works counts on Theo for bookkeeping, and he in turn counts on Hannah, for invoicing timesheets. So any decisions that are in that sphere, in invoicing timesheets and billing, she's the voice of the company. All of us can offer ideas to her whatever but when she makes a decision, that is the company's decision.

Likewise, Theo handles all other aspects of bookkeeping. In every branch of the tree there has to be a catch all.

So the game goes on, like I’m responsible to Ignition Works about recruiting, apart from the final “do we make an offer or not,” which is at that board level. But I've been having a blast developing our recruiting process, without fear that like, “Oh, I have to get a whole bunch of other people on board.”

Nat: How does that work out to engineering decisions?

Jesse: Ignition Works counts on Theo for the client account in our relationship with VMware. The important thing is, even though my name is further to the left, it’s not my call for what we're doing. It’s not simple to do real delegation like that.

And then Theo has said, “Okay, I'm gonna delegate to Tom,” or, actually, the way it works in our company is Tom says, “I would like to have delegated to me responsibility for Ignition Works for improving the PCF service offering.” So Tom said, “Please give me this sphere."

Nat: How long have you been running this way? Has there been anything that surprised you or been unexpectedly challenging? Or that you've had to like, modify mid-flight?

Jesse: We revisit the tree every week. So we’re always modifying, and that's part of the power of it is that it’s responsive. But still, it's clear, who has what decision.

Theo: I've really enjoyed people putting their hand up for responsibility. I really hate arbitrarily selecting somebody and then saying, “Oh, hey, can you take care of this,” and you've got no idea where their energy lies on it, or trying to figure out who's the perfect person for it. It makes delegation really difficult.

But with this tree, everybody can see what people’s workloads are like, where we are as a company and what we need to succeed, and I can go, “Okay, I don't love doing this thing. But I've got some capacity.” So I'll stick my hand up. And I can see that other people are probably a bit swamped. And it's allowed that to kind of happen in quite a nice way.

Nat: You’re using a pull system rather than a push system.

Theo: Yeah, totally. It allows people to be empathetic. Everybody's a manager in the sense that, in some roles, Jessie and I are kind of servants to other people. And in other roles, it kind of goes the other way. I don’t think we’ve been running it long enough to run into a serious issue with it.

Jesse: To me the most interesting positive has been sort of the emergence of… the word I'm not saying is leadership. Because this is slightly different. Confidence. Amongst our team. Where people who would have waited for someone to tell them what to do, are now saying, “I think I'm going to do this and I'm proud of it.” In totally unexpected places.

Theo: They’ve even started saying, “This is what I've done.” And it's 100% aligned in the direction that everybody knows about. I sort of have to pinch myself every time it happens.

Jesse: We've had a few decisions that I've been like, “Oh. So that's what we’re doing.”

Nat: Oh! [laughs] So how do you handle that?

Jesse: More often than not, they were good decisions, and consistently, they yielded learning, which is in and of itself worthwhile. And likewise, I've asked for a whole lot of-- of forgiveness, not permission-- you know, experimenting, in my recruiting realm, or in the client service desk. People will give me the benefit of the doubt.

Our rule is that if you're further up in the tree, the grounds for intervening is whether it's going to likely cause harm to Ignition Works. As opposed to, you don't think it's a good idea.

Theo: “I wouldn’t have done it like that” is nowhere near good enough.

Nat: I want to zoom way in for a second. Do y'all pair? Does Ignition Works pair?

Theo: So in the client, it's very difficult to pair. And also, we were pairing when we weren't physically located, but being remotely located, not so much of it is happening. We’ve been experimenting with other things. We're trying to get JetBrains “Code with Me” into the client, because we think that is very valuable.

But stuff that we're already doing at the moment, we’ve been experimenting with hurdles on Slack, whereby you've just always got a channel open, so it's just kind of cheap to talk to other people. And that's not quite comparable to pairing but there is an element of community about it and discussing ideas as they come up and making it kind of cheap to interact.

Jesse: We also have the mob room.

Theo: Yeah. So we have a Zoom Room that's always open. When people want to discuss stuff or they want a bit more detail, they join the mob room. And we can all look at the same problem.

So, for instance, there might be an incident around, say SSL certificates, people who have an interest in that will jump in and share their ideas and whatnot. So it’s not as much as we used to do in labs. It's something we'd like to do more of.

I don't think it's something that we'd like to do as religiously as labs did it. We’ve found some benefits of-- There's a lot of different area we have to cover in and we have to wear a lot of different hats. Would you add to that Jesse?

Jesse: I would just add, when we were introducing the notion of the tree to the whole team, and whether we wanted to have that direction or not, this is one of the things we talked about. There's a place and a time for a different number of people in a given box. And pairing is two people in a box.

Are we getting the benefits? And okay with the risks or consequences of that? Or not. It’s about individual responsibility, but what it's also about is the right responsibility in the right place. So that board is a three person responsibility, and with coding, we could very well say like, “Oh, well, that's really a two person responsibility. And we're going to do it by pairing.” But, yeah, that'll be up to the person who's responsible for the software we're building there.

Theo: Our teams are quite a different structure than what would have been used in somewhere like Pivotal Labs, which had an even number of engineers and Product Manager. We’ve got a mix of people. In the wider client team, there's lots of different backgrounds. People with big operations backgrounds who know how the systems work, and understand the kind of larger concerns and the users and how to get things. There's software engineers, there's kind of more-- what word would you use to describe Rich's role? Jesse?

Jesse: He's a problem preventer.

Theo: Yeah, he's kind of like the guy who knocks down all of the trees and gets the hurdles out of the way. Which is really valuable somewhere like an enterprise.

Nat: Like what’s an example?

Theo: Oh, it could be anything, it could be figuring out the damn phone number that you need to ring to get into some archaic system.

Nat: So he's the guy who brings out the like, the arc welder to cut into the data center, like at Facebook the other day.

Theo: Oh, my God, I haven't read enough into this incident.

But yeah, he’s got the arc welder on speed dial. I think I've done Richard a disservice, actually, he’s really key.

Jesse: That’s also back to, everyone paid the same, as there's no way to distinguish whether it's some fancy engineer that makes the team succeed or Richard happening to know the number of an arc welder. It's the team that precedes, not the individual. I think it's false to allocate value to individuals like that.

Theo: That diversity in the team has also kind of made pairing a little bit different than sitting down as engineers doing you know, going to write my tests, gonna do this, gonna do that. There's just a lot more variety in what we cover.

Nat: This is going to be a difficult question to answer, then, but this is a question I always ask when I'm interviewing.

What’s the difference, in your context, between a good engineer and a great engineer?

Theo: We’re sort of similar to Pivotal in the way that we raise empathy very highly, and the kind of how people communicate with one another and how they handle themselves and deal with difficult problems and all of that type of stuff. Because the thing is, there's a huge range of technologies that we run into, on the coding side, there's some really mathy stuff. There's a lot of computer science in there. There’s what we’re doing at the client, which is loads of infrastructure and networks and stuff like that. So we deal with a massive, massive range of technologies. And some people are super knowledgeable about one chunk, and other people are more knowledgeable about another chunk. And it's impossible to measure, like, where the utility is.

That's where the communication comes into it, figuring out okay, well, we have these problems, how are we as a team going to solve this thing? And also, people who contribute a lot and are confident putting their hand up and giving an opinion. And yeah, I think that's what makes a great engineer. What would you say, Jesse?

Jesse: I'm largely aligned. I would question the premise. I just watched this great Elon Musk video about, oh, “requirements are stupid,” or something like that. And he was saying that we don't often say it was a bad question. So I'm gonna try that.

I don't think the notion of privileging engineers makes sense in our context, or put differently, I think our context works much better if engineering is a skill. What makes a good contributor to the team is a helpful question, right? And a good contributor to the team is very much what Theo just described somebody who's looking around saying, “How can I contribute? What can I do that’s valuable?”

I worked on a construction site for a while, when I was a kid. And someone's got to shovel the dirt. It doesn't matter how many big fancy machines you have, there's always something that requires a shovel. And having someone who's like, “well, I'm gonna shovel,” is as important to building the building as the person that's driving the 20 tonne digger. So we need people who have that recognition and that style.

I also don't think there is a distinction between a great and a good because where someone's going to excel is super environment dependent. Someone might be the person you need in an emergency, and not so useful in normal times, and another person might be the person you need in normal times. And together you have a great team.

At any moment in time different people's strengths and weaknesses are playing out in different ways. So the real challenge for us is what's a bad member and we need to get very good at our immune system in identifying and removing disruptive elements. And not too bothered about getting “the best of the best."

Nat: How do you identify that somebody is not working out? What does people leaving look like?

Jesse: So, there's a couple of dimensions to that. It's very much in development. We have had one engineer that we let go. After working with him for a few months, it was pretty clear, it wasn't good for him, it wasn't good for us. He kind of made this declaration, “I'm not going to work in the client, I don't like that environment, I will only work in this environment.” And that signaled his inability to align himself with a larger team mission, see his success as a team success.

Nat: There was a values mismatch.

Jesse: Right. So that's a piece of it. Another piece that we do with the pay is, as a company, all companies have to do this, you have to keep a bunch of money in reserve for sick days. In our model, those reserves are out of that amount that the company pays to each individual. We have to keep that money, but when someone leaves they get it. And so everyone sort of has this light, push out the door,

Theo: The golden goodbye.

Jesse: We haven't implemented it yet, but I would love to make that much bigger. So people have three months, six months of pay when they choose to leave. Because it's way better for the company to make it easy for people to depart, than to have what I see at the client all the time. People who want their pension and so they're hanging around, dialing it in.

Nat: Yeah. I've seen that turn toxic in a couple of situations where you've just got a lot of people hanging around who don't want to be there anymore, but they feel really stuck. And then that compounds, it sort of spreads for the organization

Jesse:If it's not the right place, it's not the right place. And that's not a measure of a person. It's a measure of a situation.

Links, etc.

"Loving Your Job is a Capitalist Trap." Reflects a lot of my own thinking on the topic these days.

Jobs

Near Form is an all-remote consultancy with a specialty in Node.js. They actively contribute to open source, including Node Core and GraphQL. Encourages regular but flexible working hours, and "maternity, paternity, and adoption leave." "NearFormers shape work around life, not the other way around." They're hiring mostly Senior Frontend and Fullstack developers, but also devops engineers, principal consultants, engineering managers, and more.

Curious Cardinals is looking for a Head of Engineering – open to a first-time Director, and they don't explicitly mention manager experience. If you enjoy the "people side" of technical leadership (coaching, recruiting, planning) and are looking for a bigger challenge, and have relatively broad technical skills including DevOps experience, might be a good opportunity.