If you’ve stopped by Code Anthem in the past few days, you’ve noticed our brand new design. That’s the side effect of Code Anthem going into the long-awaited, much anticipated private beta. We’re keeping the PR fanfare to a minimum during the private beta, but at the very least, I am jumping up and down.
Photo by Pink Sherbert
Long time readers probably have a decent idea of what Code Anthem is about, but here’s a no holds barred run-down of how we’re going to save our little nook of the world.
A highly skilled programmer is much more productive and valuable than an average programmer, and the worst programmers can actually end up a net loss. Anyone who’s ever hired programmers can tell you that years of experience, advanced degrees, fancy resumes and certifications don’t translate into productive people and quality work.
Programmers who are skilled want some way of differentiating themselves from the rest. Businesses want to figure out which programmers are skilled and which are not so they can hire and promote the right people. Both sides really want the same thing: a uniform and transparent way to measure programmer skill.
Here’s how Code Anthem works:
1) Programmers take a test in their programming language of choice or multiple tests. Initially we support Java and C#, but will roll out support for new languages too. The test includes straightforward languages that assess real world problem solving through code, and don’t cover language-specific trivia or gotchas. It actually ranks programmers against each other on a scale of 1 to 10, so there is no arbitrary bar.
2) Programmers can share their scores via a public profile page to their prospective employers, current employers and their colleagues. You can use that knowledge to prove their skills, win that job, earn a promotion or just get the respect you deserve. You can link to your profile page from your resume, LinkedIn profile, blog, etc. It will no longer be enough to claim that you are an expert programmer in X language, you’ll need to prove it by ranking high on a Code Anthem test.
3) Employers can search our database of skilled programmers to find qualified candidate in the area. Since they will be paying per contact they will actually value each contact, instead of the mass spam that most developers get. Employers are always eager to find new skilled programmers, so that part is an easy sell. The only comparable way to find a pre-screened technical candidate is through a recruiter, which, among other faults, can easily cost $20K for a senior programmer.
As a programmer, I’ve always wanted a way to cut through the noise and demonstrate my skill level.
As a hiring manager, I’ve always wanted a way to cut through the noise and find great programmers.
Now, Code Anthem is here and I’m so excited to use it and share it with you.
If you’re a programmer, please sign up for the beta on the main page. Invitations are rolling out in waves and current wait time is a few days. I’ll see you on the other side.
I’ve really been enjoying this blog, but isn’t the scale going to be kind of skewed weird? You’ll only get people who are really confident in their knowledge and skills and while some people are OVER confident, most are probably justified… so you’ll be ranking people as 3-5s because they’ll be competing only with other skilled people, whereas unskilled people won’t even do it and so they won’t fill up the bottom ranks. I mean, the number will make it look like they are low to middling, but in reality they’re maybe in the top 12% – but only the top 12% are taking the tests.
The problem with any kind of online test is that there is no guarantee of who is taking it or what materials they have in front of them at the time. Obviously, they have the web available, but they could also have a couple of friends helping.
At that point, rankings mean little since there is no guarantee of the results and anyone lower on the ranking table could easily make the excuse that they took it honestly and without looking up other items.
@Talyssa In general, people think they are more talented than they are, so plenty of bad developers will try it thinking they’ll score well. As CA grows it will be a black mark not to have a score.
@Andy That is true for every way that we evaluate candidates (and life in general) but resumes are way easier to fake.
@Andy: I would bet the type of person who would get someone else to do the test for them is type of person who will be crazy easy to spot as non-hirable during an inteview. If they have to go so far as to cheat on this test, they’ll freeze or it will be obvious they don’t know what they’re talking about when you ask them technical questions during an interview.
I would try that.. at least hoping for good feedback that might show some drawbacks of knowledge / experience I’ve got so far
Have a great time building this up.
YAY! Okay…. so I’ve got to get back into the world of C# and Java! (Thank you for not starting with C++…. I would have wept) I’ve been focused recently on PHP, so have left the other languages on the “I remember when” category…. It’ll be fun to get back into writing little apps in C# or “enjoying” the process of installing Eclipse….
Though I almost laughed when the first two comments were nay sayers.
@Talyssa and Andy: If you don’t think it will work, there’s plenty of resume sites to pour money into. I, on the other hand, look forward to both finding out how rusty I’ve gotten over time, and using CA to hire the best as my company grows.
@Amber: I’ll be blunt, can a developer retake a test, or overwrite a score by doing another test for the same language? I ask this because some days you start of thinking you’re on the top of your game. Then, when an opportunity comes up to excell, you hit a mental brick wall and hilarious carnage ensues. In other words, some days I can’t pass a test asking how I got dressed in the morning, and I’m not always aware of when I’m having one of those mornings. So if someone bombs a test, is there a chance to retake it and redeem ourselves?
A fake resume carries consequences (i.e. being fired) that usually acts as a deterrent. There are no deterrents for an online test, especially when the results are all produced by crunching numbers across all tests taken. A crappy programmer can’t explain a fake resume, but a score from 1-10 in a test can easily be explained by the fact that either other people taking the test are really crap, or your algorithm is off.
A resume is no easier to fake than a test, but a fake resume can easily be detected (i.e. can’t talk about Spring even though you did 2 years of it?). OTOH how is an employer going to argue about whether you are a 5 and not a 3 on the test?
If both fake test takers and fake resumes will be weeded out in an interview, then the usefulness has been reduced to that of a resume that might be fake.
How many of those people on the code-for-hire sites will be lining up to take the tests in groups so they can include a link to their results as part of their proposal to build a CRM web site for $100?
@Andy – Yes, there’s a risk of someone cheating on the CA tests. There’s a risk of cheating/lying at every point in the hiring process. Interviews can be just as easily faked as resumes, and most “resume lies” that I’ve seen, are more along the lines from “gross exaggerations” to “misleading wording”. The main point of CA, and especially the upcoming testing process, is to offer an alternative option to the current process of hiring software developers. If you believe the current method of job posting > resume > interview works, then, I’ll admit, I’m confused at why you’re reading this blog. CA is about how, when the goal is to hire high quality software developers, the current system is somewhere between horribly flawed and just plain broken.
So yeah, there will be people that get scores they don’t deserve. Some higher and some lower. But, when I am looking to hire someone, I will hold CA’s test score higher then a resume that was possibly copy-pasted from a “career helper” site. It won’t be the only method I use to identify someone to hire, but it will be a great way to weed out those that can’t back up their resume with actual skill, and those that aren’t willing to take the initiative to go the extra step to get noticed.
There’s a lot that SHOULD go into hiring a developer. CA’s blog posts, and their upcoming testing system, is all about giving an alternative method of finding someone that can really succeed as part of your team and company. When I look for a developer to join my team, I will be looking for someone who is able to not be restricted by the status-quo. I want someone that is willing to say “I’m not going to just do what everyone else is doing, just because everyone else is doing it.” And going out of their way and taking the CA tests, shows me the developer has that spark in them.
Xain, I’m not even really thinking about the hiring scenario, but why value the possibly faked test above the possibly faked resume when you can get the candidate with the possibly fake resume for free.
I’m actually thinking more of the consultant-types who will use their CA ranking page as a cheap form of marketing for their services who don’t have an interview process to weed out the frauds and it is in their interest financially (at least in their eyes) to have the best score out there even if it means cheating. My point is that they then skew the results unfavorably for everyone else.
My other issue which I haven’t voiced is the metrics by which quality is measured. Programming is far more than syntax and remembering the technicalities of a language. Those that are able to score highly in these areas are not necessarily able to develop an enterprise application.
This has all been done before, I can’t remember the name of the last testing site to try all this (something like Brain Dump), even down to contractors bidding for work putting their scores on their email signatures, but I don’t think they are still around.
@Andy – Ah! I get it now. Yeah, I can definitely see the worry of consultants trying to abuse the system. Personally, as a whole, I would be more likely to outsource a project to a team/company overseas before trying to wade through the see of “consultants” to figure out which one(s) would be a positive experience, but that’s more due to knowing which students from my college chose the “consultant” life and why. (And to any consultants reading this, I know not all consultants are simply computer geeks that couldn’t get a job and/or are allergic to cublicles, but I personally only deal with consultants that friends/peers refer me to, because of the same high risk of misrepresentation of development skill that Andy pointed out)
In reference to metrics and how much the testing will show, I’m holding back any criticism until I actually take the tests. Yeah, there’s a ton more then just knowledge of code syntax that goes into a “senior developer”, but that’s practically a subtitle for most of the blog posts on CA. If anything, the time the CA tests have taken to become available at all is a good thing in my book. I doubt the end result will be just another syntax test.
Sorry for misunderstanding you earlier. Thanks for the insight about “consultants”.
There are entire ecosystems setup for people to dishonestly benefit online (black hat SEO, fake LinkedIn recommendations, etc). If Code Anthem could be “lucky” to have its own, that would mean that so many (honest) people already getting real value from it. For now, I’ll worry about the problems I have, and not the ones I don’t.
Beeing a site about programmers, I would have expected some client-side validation on the only input field which you have
Btw, you’re website does not look good on FF 3.6 / linux
I’m failing to see how this is any different from the Sunoracle Java Certified exams, Microsoft certifications, etc. They might mark a CV out as worth a look, but most developers in the real world do not place much trust in them as a tool for identifying a good developer. Such certifications are basically memory related- can you recall an arbitrary programming construct to solve a problem. In the real world one can remind oneself of classes and functions in an API in a second or two. The real problems with employees such as dedication, drive and motivation, intelligence level, being a team player and not pi**ing the rest of the team off, the primaddonas who think bug fixing is beneath them, general attitude, even simple HR issues such as timekeeping – these are the important things to identify in a candidate, but can be hard to identify. I’d rather have someone on my team who is bright, enthusiastic and skilled but failed a programming test than someone who passed the test but is a PITA. When I interview a candidate, while I need to guage their technical knowledge I’m more interested in forming an impression of them as a person. And where programming tests happen at interview, it is best for them to be tailored to the type of problem solving tasks that are common on that particular job.
@Graham The difference is that certifications are arbitrary pass/fail labels. The value is useless as soon as you see someone who can’t program well but is certified. Furthermore, certifications make money by charging, so their “customers” are actually the test-takers making them a poor judge. Code Anthem ranks developers according to their peers. If you look at a programmer with a 5.0 in Java and they’re not up to your standards, then you look at a 7.0 next. There is no 3rd party labelling programmers as good or bad or qualified or not qualified – it’s simply how they rank next to other developers. Furthermore, our “customers” are employers, we don’t take any money from programmers, so it’s in our best interest to make sure we only send over qualified programmers.
@Mihai That’s a fair point, although I’d guess that you’ve probably not built a startup. Do I know about it? Yes. Is it even near the top of my priority list? Nope. If a programmer can’t manage to enter their email address correctly … do they really deserve to be in my beta? Not sure about the FF thing. I’m on FF 3.6 too but in Windows. I never quite understand why that matters but it usually does…
@Amber – So basically, the first test is “Can you enter your email correctly?” Nice!
@Graham – Yes, overall value of a developers is not defined only by his skill with a language. That’s why the interview process will always be there.
In reference to cert tests not being a valid “skills assessment” I agree, but I don’t agree that they are useless. Worst case scenario, a certified programmer is capable of digging in and getting a job done. Cert test prep involves a considerable effort, and the shorter the time between starting the prep and passing the test, the better for the test taker.
I’m curious about your comments that you would rather have someone with lower programming skills and higher compatibility, then higher skills, and potentially non-compatibility issues with the work environment/team. How do you decide who gets interviewed? Most resumes are designed to throw the person’s skill level out to a reader in a positive manner (No, cover letters don’t count for anything! Anyone under the age of 30 is 99% likely to simply find a cover letter example online and change a few words in it before adding it to the resume) However, we use the resume to “weed out” those who won’t work “technical skill-wise” for the job. If you’re much more focused on personality and team work skill level, then you would need to become much more relaxed with the resume screening process, or you risk not even considering someone that could be a great asset to you and your team.
I’ll use these tests to hire programmers (once they’re in a language I need) because they’re new, there aren’t any “how to cheat the certification test” books that discuss the CA method. It won’t be the whole of my recruiting, but it will be a major part in the initial screening.
Just some food for thought.
So I got a 1 out of 10 and no way to see what I did wrong. I think you should show me the results of running the program at least. Now I’m not sure if my loop variable was off, or my if expression was the wrong way around.
There are two things not discussed in this post (or anywhere on the site that I can see, for that matter) that I would like to know.
First, what are we being “graded” on? After having done the test, it is clear that we are being graded on our ability to write code that compiles and that produces the correct answer. (I’m assuming that’s the case, although it never told me directly whether I had gotten an answer right or wrong. I’ll never know what problems I actually got right and what problems I got wrong.) Aside from code that is free of compile and runtime errors, and getting the correct answer, what else is being factored in? Time it took to write the code? Time it took to run the code (i.e. is performance an issue)?
Second, I took the test and got a score back, but I have no clue what it means, and no way to deduce it. I used to bring home scores on tests to show my parents. I’d say “I got 95%!” My dad, being a math teacher, said “I don’t know what that means. Is that good or bad?” See, if everyone else in the class got 100%, then 95% isn’t that great. But if the class average was 55%, and the second highest score was 75%, then 95% is spectacular. So now I know my score, but I have no idea about what that means or how I compare to other people. Some basic feedback about what the scores mean would be meaningful.
@Chuck @Mike I’ve been making lots of additions, like an addition of a “How this works” screen before launching you into the test. As for Mike, if you confirmed your code works in an IDE then a 0 is probably off. Usually that is due to using some library or code construct that is not supported. More info on that in the How it works page. If you email me your account info I can look at it and see. I will probably add something that provides transparency into your results but that’s in the works. Thanks guys.