I think that's going a bit far. I think we're saying the same thing but using different signals; the ability to say "I don't know" is actually an important gauge of aptitude in my opinion, because anyone that is unable to say it will possibly have too much hubris to take the time to learn something they don't know. It is not the only signal I rely on, and if they truly know the topic, there are other signals I rely on either way. But it is /a/ signal which I don't believe is invalid. I don't actually care if someone agrees with my definition of expertise, I just care about their ability to learn, which starts with admitting they don't know something.
Challenges are another great way to do that, and don't necessarily need to be time-limited; MicroCorruption is a great example. We will occasionally do similar "take home" style challenges. The only time we'll time limit them is specifically when we're looking for how /quickly/ someone can get up to speed on a topic they're unfamiliar with, which is also occasionally a valid question; i.e. do they need to become an expert to build something in it or can they get dangerous enough quickly. Not everyone can. In those cases, we provide plenty of resources, make ourselves fully available, and make the time limits a matter of days, not hours.
Edit: to clarify, it is far more important to me that someone have the ability to learn new things quickly than already contain an existing bit of knowledge in their head.
I think people think I'm talking about "how well people can learn new things" when I talk about this. That's potentially a part of it, but it's not what I interviewed for. I interviewed for, literally, simply, "ability to do the job we had Matasano consultants perform".
Microcorruption and the Crypto Challenges were outreach for Matasano, but they were not part of the hiring process. Our hiring challenges were slightly more boring, and explicitly tuned to generate the exact signal we wanted.
I don't know whether there's any valid signal to recover from candidate psychology. I don't have to reach that argument, because in my universe, there's a giant, very accurate signal available that makes looking at other signals a poor use of my time. The fact that the big signal also keeps me from using dubious signals and making bad decisions is just a knock-on benefit. :)
That's great that you had a very specific job req and set of things you were looking for. Not all of us are that lucky ;)
Startups sometimes look for generalists (i.e. people who can learn new things) specifically because nobody knows what the stack will look like, how the product will change, etc. a year from now.
Challenges are another great way to do that, and don't necessarily need to be time-limited; MicroCorruption is a great example. We will occasionally do similar "take home" style challenges. The only time we'll time limit them is specifically when we're looking for how /quickly/ someone can get up to speed on a topic they're unfamiliar with, which is also occasionally a valid question; i.e. do they need to become an expert to build something in it or can they get dangerous enough quickly. Not everyone can. In those cases, we provide plenty of resources, make ourselves fully available, and make the time limits a matter of days, not hours.
Edit: to clarify, it is far more important to me that someone have the ability to learn new things quickly than already contain an existing bit of knowledge in their head.