Introduction

I’m a programmer by trade and I’ve been helping my manager recruiting for quite some times. By helping recruiting, I mean separating good applicants from bad ones by having a technical interview with them.

Resumes are often full of approximations and flat-out lies. The idea here is to figure out what their actual programming skills are.

And I needed a tool to do that efficiently and remotely.

Context: The Technical Job Interview Process

I see this interview more as an exchange than an actual test. So I’m fairly flexible with applicants.

My usual process:

1. I introduce myself, and I present quickly to them the overall process. I also state various things:
   • In their future work, they shouldn’t have to work under pressure (it would mean their manager failed at its job). So I don’t want to test them while they are hugely anxious. Therefore, to reduce their anxiety level, I make it immediately clear that they will fail at some point in this test, and that’s expected. A test where applicants would get all the answers right would be useless to me: it would just mean I haven’t been able to figure out the actual limits of their skills.
   • In their future work, they will always have Internet access. So they are free to look whatever they want on Internet, as long as they don’t look specifically for the answers to my exact questions.
2. I ask the applicant to write a very simple function in their preferred programming language. For any good applicant, this should be fairly straightforward (15 minutes max). With junior applicants, you would be surprised how many fail at this first step (some still don’t have a working function after 30 minutes). For examples:
   • A function returning the number of vowels in a string.
   • A function indicating whether one string is an anagram of another.
   • A function returning the second largest element in an integer list.
3. For some programming languages (Python, C, Java), I have some code snippets. The first code snippets are to check their knowledge of some basic elements of their favorite programming language (string/object comparison, operator overload, etc).
4. Then I have some other code snippets that have bugs. The idea is then to see how good the applicants are at spotting bugs and debugging. The bugs are fairly obvious. Some are even strongly emphasized in the code. Still, many applicants still fail to spot them …
5. Then I ask some general programming questions (“what is an abstract class ?”, “what is algorithmic complexity ?”, etc)
6. For the good applicants, if I still have some times, I finish testing with some basic algorithmic problems (almost all applicants fail here).
7. Sometimes if we still have some time at the end (rare), I like discussing their resume with them. It’s redundant with the interviews my manager already do, but it allows me to correlate what I’ve observed during the tests with what they tell me.

There would be no point in tricking or failing them entirely. So during the whole process, if they are stuck, I try helping them without giving them the actual answers. If they really just don’t know it at all, then I tell them. I even like showcasing the answer whenever possible.

I’m French, but I’m also aware that English is an essential programming skill. So if the applicant says they are at ease with English, I do a part of the interview in English.

It usually takes from 1h to 1h30. At the end, I usually have a very good idea of the applicant’s actual skills. On the applicant side, they often leave having learned new things and knowing which skills they should try to improve. You would be surprised how many times I’ve been thanked afterward for those interviews :-).

Then I write a feedback email to my manager.

The need for a testing tool

Those interviews are always done remotely. So for steps 2, 3 and 4, I need a way for them to send me the code they wrote, and a way for me to send them code snippets.

For the step 2, applicants could cheat by looking up the answer on Google. So I need a collaborative editor to see them type the actual answer on-the-fly. This has always been enough to discourage them from cheating.

Initially, I used a simple collaborative editor. But applicants were often annoyed by the lack of compiler/interpreter. In their future work, they will always have a compiler at hand, so this limitation was pointless. So I had to copy/paste their code on my computer, compile it, then paste back the compiler errors in the collaborative editor. This was annoying.

For the step 2, having an integrated compiler/interpreter is also handy to test their function. Good applicants even have this good reflex of writing and running the tests themselves :-).

Overall, the integrated compiler/interpreter is also handy to showcase things. For example when they fail to spot the race-condition in one of my code snippet, I can just run the function 100 times, and the bug becomes obvious.

Of course, I cannot ask applicants to install a whole IDE on their computer. This would be a huge waste of time, and it would be really error-prone. So plugins for well-known IDEs are out of the question. To avoid any installation process, the tool has to be a web interface only.

Public web collaborative IDEs

I tried some public web collaborative IDEs. The one I used the most was CodeCollab as it is free. There are also paying ones, but I haven’t tried them at all.

CodeCollab and the others did work nicely .. when they worked. Unfortunately, they were too often down or only half-working when I needed them. Last time I wanted to use CodeCollab, the C compiler wasn’t working anymore for some reason.

My own collaborative web IDE

I self-host already a lot of personal services (email, personal website, etc). So, out of frustration, I’ve decided to self-host my own collaborative IDE. Unfortunately, at the moment, there are only very few open source options.

The only good option I’ve found is JupyterLab and its experimental real time collaboration mode. While it’s still experimental, it does work very nicely and reliably. Unfortunately, it’s not yet supported by JupyterHub.

For the interview, I need support for a bunch of common programming languages (Python, C, C++, Java, PHP, Javascript). So I’ve made my own JyputerLab Dockerfile that includes many JupyterLab kernels.

Dockerisation

I’m assuming here that you are already familiar with Docker. So without further ado:

• Dockerfile
• docker-compose.yml
• change_password.sh

HTTPS

Of course, this container has to be reachable from Internet. So, to do things right, you will also need a reverse-proxy in front of this JupyterLab container to take care of SSL/HTTPS (and possibly IPv4|IPv6).

I, for one, use Apache. For reference, here is my virtualhost: 025-jupyter.kwain.net.conf

Usage

Before each interview, I use the script change_password.sh to change the JupyterLab’s password. I always set a random 32-characters password (no special characters).

At the beginning of the interview, I send an email to the applicant with the address of the JupyterLab and the password. I also create immediately a first notebook.

Once they are connected, I ask them to open the first notebook. Then I explain quickly that they can type code, and they can compile+execute it with Shift-Enter.

At the end of the interview, I shut down the whole JupyterLab.

Limitations

Unfortunately, since JupyterHub doesn’t support the collaboration mode yet, this approach only work if you have control on the Docker containers. In other words, this can work for one or two interviewers who know Docker, but it doesn’t scale to more.