Hiring software engineers? Here’s how to perfect the technical interview process

This article was originally published on Built In by David Vandegrift.

The single most important asset to any technology company isn’t its inventory or even its intellectual property. It’s the people who make up the company: the engineers, designers, researchers, and managers who harness their inner knowledge and ingenuity to create something out of nothing.

Accordingly, investing in attracting and retaining top-tier talent is a vitally important initiative for these tech companies. And the task must be taken seriously from the beginning: The interview process should ensure you’re bringing in the right people, as well as laying the groundwork for their future success at the company.

While hiring and interviewing is always tricky to get right, no function has seen more variance — and less efficacy — than engineering: The processes used to determine whether a software engineer is the right candidate for a company vary widely across both sectors and companies; they even shift within a company based on the hiring manager or the time of day.

[Read: These 3 tried and tested hacks will land you a corporate client]

Nailing the technical interview process is one of the most important things that a technology company can do. It’s an area that we’ve spent a lot of time refining here at 4Degrees, the Chicago-based software startup I co-founded a few years ago. I would like to share some of what we’ve learned, starting with what not to do.

What does ‘bad’ look like? 

Bad technical interview processes often go wrong at the structural level: The company doesn’t have a good idea of what it’s looking for, so interviews tend to be set up on an ad hoc basis until some hiring manager or committee gets the warm and fuzzies about the candidate. This approach has several immediate negative consequences:

• It often leads to too many interviews, where each subsequent interaction generates marginally less value but racks up costs both internally and with the goodwill of the candidate pool.
• In an ad-hoc process, there’s no guarantee that you’re actually assessing the right characteristics. The interviewers are reacting to what they’re seeing and feeling rather than developing a proactive perspective on what good looks like.
• Your false negative rate goes way up. In other words, you end up declining on too many good candidates. That’s because these processes often end up being consensus driven, and the more interviews you have, the more likely you are to find one person who just doesn’t click for some reason.

Once the right interviews are established to assess specific qualities, the actual assessment methodologies themselves are the next major stumbling block. The tech industry is infamous for some wild interviews: questions about manhole covers, impromptu jogs leaving candidates sweating through their khakis, and bizarre coding environments that look like they were inspired by ’70s-era mainframe and terminal programming. Some of the common bad interviews include:

Brainteasers: These bizarre questions are meant to test creativity and orthogonal thinking. They’ve led to an entire interview prep industry training candidates to deal with them. In reality, there is next to no evidence that brainteasers lead to better interview outcomes.

Whiteboarding: Some companies figure that, since the interview is already an artificial environment, they can capitalize on the situation by having a casual assessment of programming capabilities using a whiteboard and pseudocode. The problem is that the vast majority of engineers never use a whiteboard or pseudocode in their actual job. Not to mention the strangeness of the medium leads to anxiety and lower performance in many otherwise perfectly qualified candidates.

Knowledge quizzes: It seems reasonable that if you know more about a language or framework then you’ll be better at using it. The problem is that the vast majority of engineering work requires only a tiny fraction of the total knowledge of a given technology. Plus, any missing knowledge is typically easy to rectify with a quick Google search. Knowledge quizzes are the equivalent of “teaching to the test” in school — you filter for good memorizers, not good engineers.

Unrealistic situations: Online and timed coding tests are now the norm for low-cost mass filtering of applicants. Algorithmic coding challenges are considered a gold standard for an engineer’s capabilities. The problem with these and other coding assessments is that they don’t reflect what real coding work is like. Most engineers can perform exceptionally well with no formal knowledge of algorithms, and engineers pretty much never have to code with a clock ticking down in an unfamiliar IDE.

Asking too much: Even when companies get the rest of the process right, they will often simply ask too much of their candidates as a part of the process. It’s not unheard of for a process to include a full day (or more!) of at-home work to get an “appropriate” assessment of their capabilities. While this type of assessment will undoubtedly generate more data and better prepare the candidate for the job, it also de facto excludes an enormous population of candidates who simply don’t have the resources to do that much work for free.

What does ‘good’ look like?

An entire book could be written on how not to conduct a technical interview process. But that’s the easy part. The real question is: What does a good process look like?

First, it’s important to remember that every company and situation is different, with specific requirements leading to a different “perfect” approach. No single process will be ideal for every hiring manager in every situation. Instead, it’s crucial to carry a set of good principles into any given situation.

Those principles include:

Intention: The key to a good interview process is being intentional: thinking through what you’re looking for in a candidate and what you need to assess to get confidence on those attributes. Design a process that accommodates those assessments. Then stop there. Don’t keep adding on layers to the process just to try to find more comfort with your decision.

Range: While it’s important to keep the process concise, it’s equally vital to bring a bit of variety to the various interviews to get a range of readings on the candidate. At 4Degrees, every technical process includes one live assessment and one take-home assessment. The live assessment is kept under an hour and is framed as a collaborative exercise, like paired programming. It assesses the candidate’s thinking process and ability to work with others. The take-home assessment is longer, designed to take about three hours and mimic a real task the engineer might do on the job. It assesses the candidate’s ability to tackle a problem from start to finish and some of the more specific skills required for the job. While a long in-person assessment could theoretically measure the same things, many candidates would struggle with the stress of direct oversight for such a long period of time.

Assess the non-technical: Although an engineer’s job is ostensibly to write code, the reality is that every employee in your company also must bring to bear a full suite of non-technical skills in order to be successful. The skills vary from role to role, but include collaboration, communication, resilience, ingenuity, authenticity and commitment.

The ideal template 

Every situation is different and should be approached with thoughtfulness around the specific requirements of the role and the candidate. Still, by combining a basic list of do’s and don’ts, we’ve managed to develop a good template for technical interviews at 4Degrees. This template typically gets us 80% of the way to a real-world process while allowing flexibility to adjust as needed.

The process is as follows:

Initial expectation-setting call: Nearly every process begins with a 30-minute casual chat over the phone. The hiring manager shares the background on 4Degrees, what we’re looking for in the hire, and why. They then ask for the candidate’s background and what they’re looking for in the role. While the call is not strictly evaluative, it actually does end up weeding out about 20% of candidates who otherwise looked good on paper (typically because what they’re looking for isn’t a good match). For the other 80%, the call helps make expectations clear and can tailor the rest of the process to better assess the candidate.

Live technical assessment: The first formal step in the interview process is a 60-minute technical assessment, typically done via screen share. About 45 minutes of the interview are spent in an actual coding environment, with the rest set aside for introductions and questions from the candidate. The coding is meant to be collaborative, somewhat like a paired programming exercise. The coding tasks are meant to be as reflective of the real on-the-job work as possible and often include writing from scratch as well as editing existing code.

Vision/values alignment: The second interview is another hour, this time focused on the non-technical. Specifically the vision, value and mission of the company. This interview is particularly valuable at companies that place a strong focus on the role of these artifacts within the company (I would urge all companies to care about these things and how employees embody them, but that’s an article for another day). If nothing else, this conversation gives the candidate an idea of what the culture of the company is like and can raise an early flag if they depart from these core values in a major way.

Take-home assessment: The penultimate step in the process is for the candidate to tackle a take-home assignment designed to mimic the on-the-job tasks they will be expected to do once hired. The assignment is meant to take the average candidate about three hours, and an additional hour is given as buffer. The assignment is rigidly time-boxed to ensure consistency across different applicants.

Full-team interviews: The final step in the interview process is a full set of in-person (or video) interviews with the team. For 4Degrees, these interviews have historically included every member of the team (so three to five total interviews in one day). We’re quickly growing past the point where including the full team isn’t reasonable; we’ve found that after about five hours the candidate’s performance in an interview can’t reasonably be expected to reflect their on-the-job performance. These interviews are tailored to focus on any as-of-yet unanswered questions from earlier in the process, and therefore have the least consistency across processes.

And that’s it! Five separate touchpoints, with a total of six to nine formal interviews. The interviews range widely across the technical and non-technical, with an emphasis on different skills in each category. The process is demanding and rigorous, but not so much so that it excludes many candidates on any grounds other than capability and fit for the role. You will of course need to tailor your own process to your specific needs, but we have found that this template serves us well for a wide variety of technical roles.