How I see Joel's list 18 years later
Back in 2000 Joel Spolsky published a test to help gauge the quality of a software development team. It is a list of 12 items where if you had 10 or less meant you (your company) was in trouble.
The first time I've read it made me think about the company I worked for at the time, how I wanted to shape my engineering life and of course how far some of what was discussed there was from the reality in Brazil.
This week I came across an ad for VP of Engineering at StackOverflow and one of the items in the ad was how the org rated itself against this test. In case this ad goes away, I took a screenshot of the test result for SO:
Almost 18 years later the company Mr Spolsky founded is looking for a VP of Engineering, which in itself is a demanding position elsewhere but I figure it carries more weight in a place like StackOverflow. And they still use the test to inform how they are setup. Cool stuff !
It brought me warm feelings to make a retrospective on how the importance of items in this list changed for me since I saw it for the first time. Lately the subject of people, engineering management and how to cope with tech dramas have been of interest to me so I set to write how I see this list today.
I believe that the importance of this test has not diminished but new trends and changes in engineering can reshape some of the items and make a good grade easier to achieve without forgetting the human factor in the success of any tech org.
Do you use source control ?
This is the easiest one nowadays. I know there are companies still using zip files or tarballs on Dropbox like MyApp.final.zip and MyApp.1.0.this.is the.good.one.tar.gz somewhere in the universe but thanks to SVN, GIT and services like Github source control and at least a simple branch-to-master workflow is easily adopted.
The social aspect of Github also helps to enroll in projects and communities. At the time of the original test, Sourceforge was the hub for FOSS projects and mailing lists were the medium where you would hopefully have your voice heard or smashed to death by trolls. IRC networks played an important role on getting communities together, as much as Slack and other chat apps are providing. In private projects you could use svn, clearcase or cvs to keep your own track for your artifacts.
So I guess this question still relevant but in a reverse way — as a engineer you won't get or stay at a company so green as to not have any kind of source control. New engineers are born knowing and using some workflow. Can you imagine someone that was born without storing code on zip files ? Just like babies that never saw a world without internet !
Can you make a build in a single step ? Do you make daily builds ?
I've grouped both questions because since that time, and specially after Amazon the role and tools to make a single step build and allow for more than one deploy per day are available to all.
One of the top of the mind trends there is of course DevOps, which are freely interpreted as a role or culture depending on the vague meaning a tech org wants to give and but also a class of engineers that are in the middle of the road there — they don't fully conduct feature and product development but also don't take over operations as a fulltime job.
From Jenkins implementation to Dockerfile and other automations gadgets there is a world of tools and people available that make having Continuous Deployment possible while fixing build pipelines in a way that they are invisible to the developer. The bad thing is that no developer is giving up control of these pipelines until they have to face some kind of outage, in which we all revert to 1980s ops style demanding someone to take care of "production".
Reverting the trend again, I'd ask engineers "Are you able to take care of what you create ?" or "Do you create conditions to Engineers own production ? Do you still have an ops team ?" for companies. Both embed the concern originally expressed.
Do you have a bug database ? Do you fix bugs before writing new code ?
I've collapsed both questions as the software development process has changed a lot over 18 years. Most teams are using a variant of an agile process, or so they think, and with it comes the backlog.
Backlog is a kind of entity that no one can tell exactly what is. Companies become great backlog producers way before being as prolific in delovering software.
Theoretically a backlog has ownership and would contain what matters to the product, including what impacts its users and stakeholders. Items are taken out of this pile and prioritised, which means they gonna get some developer time. The criteria varies — what brings more money, who is asking that, what is the company plan and so on.
Bugs originally would be kept in a separated place with some process around that — probably ITIL or any incident-to-bug/customer calls-to-bug/number-of-customer impacted metric that would help you pull it out in some order. Now they live along with the backlog or outside of it when they are new and someone in the company comes yelling about it.
So the idea of fixing bugs or your tech debt before creating new things is still amazing and desirable but I doubt all companies can vouch they do that in a regular and non-pressure way. One of the failures of the Agile model is that it is not agile, it depends on people to pick what to do and people are subject to pressure.
The funny arguments grew too — from "once you guys fix it our product team will get in to make some more money" to "let's live with this bug and pay for more servers to save engineering side" makes it harder to spot these ninjas of the art of not working.
I'd say that if a company answer yes to both questions you should marry it, treat it good, buy a box of chocolate and really drop the attitude. But if you see someone saying their company does that and it sounds like a brainwashed fanatic, well probably your gut feeling is right.
Do you have an up-to-date schedule ?
Which brings us to schedules. If your code is at all important to the business, there are lots of reasons why it’s important to the business to know when the code is going to be done. Programmers are notoriously crabby about making schedules. “It will be done when it’s done!” they scream at the business people.
Does that creeps you ? 18 years ago, Joel was telling us how shitty the world would be and we were like "whoa PHP 4 and that gtk thing to put cpu usage in my gnome desktop"…
Good developers are diligent, even amidst managerial confusion. Keeping your team's WiP (work in progress) in control by not committing to do too much in face of unplanned work that may come, estimating effort in hours not in points, breaking walls between product teams and engineers so both can have a saying on what is being done can help.
Remember Mastery, Autonomy and Purpose ? Over time people learn how to estimate if they are given a plan that won't change each week. So instead of having a schedule, make sure you have a simple way to estimate how your team work and stick to that adjusting as you learn but keeping it transparent. The energy spent infighting — inside and outside the team — is underrated.
The other crucial thing about having a schedule is that it forces you to decide what features you are going to do, and then it forces you to pick the least important features and cut them rather than slipping into featuritis (a.k.a. scope creep).
Not mentioning things like CI/CD and the value of code that is done, my hunch is that this part of the discussion is more about what to do than how to do it.
Do you have a spec ?
Writing specs is like flossing: everybody agrees that it’s a good thing, but nobody does it.
Looking at your backlog, how was it created ? How many items are "to close a sale" and how much time you can dedicate to refactor your legacy or fix bugs without having to negotiate against such sale items ?
Also
My pet theory is that this problem can be fixed by teaching programmers to be less reluctant writers by sending them off to take an intensive course in writing. Another solution is to hire smart program managers who produce the written spec. In either case, you should enforce the simple rule “no code without spec”.
You'll find a host of formats to write epic specs that result in shitty and patched code. How do you review drifts against a spec ?
Do programmers have quiet working conditions ?
Open floor offices and remote working discussions would fall right in here, along with the great headphones of our era. Companies are still struggling to hire and manage remotely, but there are good tools to help coordinate. The original item looked like a struggle and I still think it is.
I would rephrase that as do programmers have conditions to do work ? I'll leave this post about open offices here which sums what I think about distractions and focus on management.
Remote distributed work demands good management and it shows from how you communicate (slack, irc and other chat) to how work gets to you. In this sense having a checkpoint early on or in a time that makes sense if you cover different timezones and not having your work measured by how many jira tickets you punch in an hour is what I'm looking for.
Having a mindset oriented to products instead of project management is interesting too — as long as having a schedule is something you are used to. Having the team to know and voice product concerns is as important as having a product person that understand technology.
Do you use the best tools the money can buy ?
I think this is an easy one and should be asked to the company: good computers, licenses, cloud providers and conferences are the best tools. An overwhelming amount of coding is done on FOSS languages and tooling, an evolution considering the time the test was conceived. At that time the gnu c compiler, php3, a bit of perl and mysql were my tools of choice, interconnected to Oracle and other very-expensive-for-the-wrong-reasons software and hardware.
Do you have testers ?
It might sound old school but I still think that it is important to have some kind of tester profile in our teams. Someone gonna spend time testing: the developer, product owner, customer support or the customer.
This is different than having the engineer creating testing, test automation and so on. Specially on legacy and overcomplicated systems, testing business rules or how a new feature impacts the current product is important and often neglected.
I would not put this team in the critical path to production or responsible for the test result tho. That can lead to harassment, pressure by pushing responsibility in the hands of people that can't steer development or product delays.
Do new candidates write code during the interview ?
Github for flirting, Hackerrank for dating, Jira for married life.
This always rang the wrong note with me. I think that very few people can implement and conduct a good whiteboard or coding interview. Same as for analyzing a github profile.
Most of the time what I saw is that the interviewer wants to compete or smash the candidate somehow, just so if the person accepts the offer there will be that sense of victory when both of them are picking up jira tickets to fix a legacy crud.
So probably this is a turn off today, not sure I'd include that as a valuable metric anymore. I'd rather fix the candidate to spend an afternoon with the team after proper interviewing rounds and let them have some time together to gather better feedback. I did that and it worked better.
Do you do hallway usability testing ?
This last item to me is complimentary to having testers and also about engaging stakeholders as UX and UI teams. This changed a lot, although it was probably not understood right at the time. Joel mentions UI Design and people over processes (by engaging five or six people) as a practical shortcut to have a quick feedback loop on what you had, regardless of your UI Design skills.
The game changer in this field were mobile applications. You can tell right away how much care a company puts in its products if you use its apps. Latency became an important metric too, with companies as Amazon and Walmart working over milliseconds to increase conversion. That all could not be foretold by the tech available at the time but the instinct was there:
A hallway usability test is where you grab the next person that passes by in the hallway and force them to try to use the code you just wrote. If you do this to five people, you will learn 95% of what there is to learn about usability problems in your code.
Amazingly simple. Mostly forgotten in face of other software development metrics.
Which list should I use to evaluate a company
I'm no Joel to come up with such a meaningful and telling list, but past experiences made me unconsciously distil these teachings in a set of beliefs that I follow when looking to a new prospect or company, mostly around culture, openness and ownership. Tech is easy and a side effect — we deal with more people issues than technical issues.
There will be endless tension and discussion around project management, product management, operations, compliance and prioritisation. Who takes the pager, when are we refactoring out of our legacy, why can't engineers be creative and why we pay more for servers than salaries. This is all due to human decisions and leadership priorities than with your tech stack.
In this sense, Joel's list still a great starting point to breach these problems. It is more than a checklist that you can easily fool to say your tech stack is great but your culture sucks and you have to get a pass to use the restroom in the middle of your 14 hour workday.
From discussing the need for testers you can get straight how your org defines the concept of "done" and how something that was planned is delivered and guaranteed to be what was expected, for example.
A discussion around having a spec can help you spot bikeshedders and tech tourists from a mile besides discussing how to commit to a goal with your team. Same for operations hurdles and how to contribute in a meaningful way without getting mad
Finally, you won't find a perfect place. Part of the adventure is to engage and change what you don't like based on what you believe and want for your life if possible. There will be plenty of opportunities to lead these changes which I strongly advise engineers to do as early as possible.
More where this came from
This story is published in Noteworthy, where thousands come every day to learn about the people & ideas shaping the products we love.
Follow our publication to see more stories featured by the Journal team.
