To start, I think your interviewing practices are significantly better than the stock whiteboard-based version (and your criticisms thereof are spot-on).
That said, some concerns:
Leading with the source graph challenge seems a bit much. My experience has been that it’s necessary to have at least some filter for the unwashed masses before requiring that much work of someone. At my last company, we had a 30-minute non-technical talk with a founder (later replaced by our well-trained recruiter/office-manager). This helped filter out the people who weren’t what we were looking for at a really high level (e.g., an ops engineer applying for a dev role, a really junior person applying when all we have room for right now is a senior one, a dev who only knows pascal, etc) before making them invest the time required for an in-depth take-home test.
Also, what’s with the 24-hour limit? It seems too long to prevent cheating, but short enough to be annoying. For similar challenges, I’ve given people as long as they want (up to a few weeks), but asked for a blow-by-blow writeup of how it went and how long it took them. We may have gotten some liars, but we never had someone arrive on-site who claimed it took them 2 hours when, based on their skill, it probably took them much longer. People seemed reasonably honest about it, and the convenience seemed to be appreciated.
Asking to see source code for a significant project is tricky. I know good developers who wouldn’t have any code to show for that because everything significant they do they got paid for, and can’t in god conscience show to someone else. For example, I worked pretty long hours at my first job and built some interesting stuff, but I didn’t have much time for side-projects because of it. A friend of mine works normal hours now, but does contracting on the side. Plenty of good code, but nothing he could show to someone else. That said, asking to see some source is very high-signal, so it may be worth the tradeoff of only selecting people with significant free side projects.
That said, it’s still a way better process than teasers and whiteboard algorithms. If I were looking for a job, I’d definitely apply. :)
But what if the ops engineer can pass the coding challenge? He maybe trying to change roles.
Or the dev that only knows pascal - if they can pass the challenge using pascal then I'd think they could easily learn whatever coding language your team uses. Learning a language is easier than learning coding.
That's what I like about coding, and about these blind challenges - it's purely skill based. If they want to apply and can pass the test, they they're worth talking to. If the wrong people are applying (and passing) then your job description and challenge are wrong and need to be calibrated.
It is possible that someone with little dev experience could complete our challenge. We were mostly trying to avoid wasting time: theirs to complete the challenge and ours to review it, if we thought it was extremely unlikely they’d complete it.
However, when I referenced the hypothetical ops guy, I was referring more to a misalignment of goals: someone looking for a job where they’d be doing more devops stuff vs our need for a dedicated developer.
Regarding the hypothetical pascal developer, it really depends on your need. I’ve been in the position to hire bright people who can learn our tool set on the job, and that’s great. I’ve also been in the position where we can’t afford a month or two while a new dev learns our language an framework before they become productive. When in the latter position, it makes sense to weed people you’re not interested in out early.
I would love it if a well-written job description would prevent unqualified people from applying, but my experience has been that it just doesn’t. That’s why some people use fizz buzz (http://www.codinghorror.com/blog/2007/02/why-cant-programmer...). That’s why we used a short phone screen.
> However, when I referenced the hypothetical ops guy, I was referring more to a misalignment of goals: someone looking for a job where they’d be doing more devops stuff vs our need for a dedicated developer.
If someone were looking for a devops-oriented job, why would they be applying for your non-devops-oriented position?
That depends on the attitude you go into the conversation with and whether your take-home test can be automatically scored well enough for a coarse filter to reject the people who obviously cannot do it. It also depends on how many applicants you have. Spending half an hour per candidate gets onerous when you have hundreds, so cutting to the chase in a semi-automated way may be a better means of whittling the pile down than having a recruiter, founder, or hiring manager filter it based on conscious or unconscious biases.
I was highlighting that line because I think it indicates a subtle bias towards people based on background and assumes people are not making reasoned decisions about what they are applying to. I understand that there are tons of unqualified resume blasters out there, but I figure nearly all of them will either ignore the take-home or submit such schlock that it will be easy to automatically reject them.
That said, some concerns:
Leading with the source graph challenge seems a bit much. My experience has been that it’s necessary to have at least some filter for the unwashed masses before requiring that much work of someone. At my last company, we had a 30-minute non-technical talk with a founder (later replaced by our well-trained recruiter/office-manager). This helped filter out the people who weren’t what we were looking for at a really high level (e.g., an ops engineer applying for a dev role, a really junior person applying when all we have room for right now is a senior one, a dev who only knows pascal, etc) before making them invest the time required for an in-depth take-home test.
Also, what’s with the 24-hour limit? It seems too long to prevent cheating, but short enough to be annoying. For similar challenges, I’ve given people as long as they want (up to a few weeks), but asked for a blow-by-blow writeup of how it went and how long it took them. We may have gotten some liars, but we never had someone arrive on-site who claimed it took them 2 hours when, based on their skill, it probably took them much longer. People seemed reasonably honest about it, and the convenience seemed to be appreciated.
Asking to see source code for a significant project is tricky. I know good developers who wouldn’t have any code to show for that because everything significant they do they got paid for, and can’t in god conscience show to someone else. For example, I worked pretty long hours at my first job and built some interesting stuff, but I didn’t have much time for side-projects because of it. A friend of mine works normal hours now, but does contracting on the side. Plenty of good code, but nothing he could show to someone else. That said, asking to see some source is very high-signal, so it may be worth the tradeoff of only selecting people with significant free side projects.
That said, it’s still a way better process than teasers and whiteboard algorithms. If I were looking for a job, I’d definitely apply. :)