The typical multi-round hiring process is structured so that:
Each stage contains more candidates than the successive stage
Each stage costs the employer less to evaluate per applicant than the successive stage
Early on, you’re extracting compressed signals across a higher number of applicants, based on which you screen people into the next stage. These later stages, which are more expensive, are designed to elicit higher bandwidth signals which cover more dimensions (eg. in an in person conversation, the interviewer can follow up on how someone reasoned, and also gauge things like affect, attitude etc). This is based on the assumption that the earlier stages actually generate cheap but useful signals.
The internet made it possible for companies to broadcast to and receive applications from larger pools than before. But this cut both ways, since the internet and email also reduced the effort it took to apply such that the pool became much less self-selected. As Byrne Hobart writes, this increased the return on writing customized cover letters. A customized letter carried two separate signals: that you cared enough about this particular job to put in the effort, and how well you could write.
LLMs now make it trivial to customize cover letters. Agentic workflows can research many companies and teams in a lot of depth and leave evidence of customization that blurs the distinction between genuine interest and mere willingness to burn tokens by casting the widest net possible. Similarly, producing grammatically correct, coherent writing is no longer evidence of an ability to communicate effectively or even a basic command of language.
AI makes the artifacts candidates submit less diagnostic. However, it also reduces the cost of a more thorough reading of the evidence:
LLMs can match on the shape of skill set → job, whereas the previous paradigm often meant recruiters or hiring managers either used crude keyword searches or manually screened each application. Machine learning engineer on a resume means very different things depending on company and context. (insert an example) LLMs can reason about this contextually and help prevent screening out candidates who weren’t a strict match on function/industry keywords but have essentially the same skill set. Separately, LLMs can also encode human intuition about the selectivity of different signals (how good do you have to be to go to school x or work at employer y in z role). Admittedly all of this requires some effort re prompting, testing, validating.
Cover letters and application questions are also not a lost cause. In my experience, running cover letters and answers to application questions through Pangram and filtering out everything it scores as 100% AI-generated can reduce pool size considerably. All that guarantees is that the applicant didn’t delegate the entire artefact production to AI. So if someone just paraphrases ideas that AI generated, you wouldn’t catch it. This is solvable by having LLMs measure the residual over a baseline frontier model run. For each question, you give a frontier model the same prompt, along with whatever context an applicant could reasonably throw in about the company and role, and generate tens of baseline runs. LLMs can then help evaluate whether the answer contains content that isn’t prevalent across the baseline runs.
Work tests/takehomes
Here is a certain category of work tests/take homes (from the software engineering world) that have become basically unusable:
This is anything that’s a closed form problem graded only on the final answer, where AI can reliably reach that answer and you can’t control whether candidates use it. Whether it’s a math problem or specs to build a system that you then assess by running the submission through test cases.
If you try hard, you can come up with problems that can’t be one-shotted by frontier models. Or come up with problems where the upper bound on performance is high, such that a great engineer can still get some uplift over the current frontier models. But this is a race you will likely lose as models make progress on verifiable tasks.
Some have adapted to this by moving to shorter, synchronous assessments. These let you watch someone work and probe their reasoning as it happens, and this is probably the right solution if the time it costs the employer isn’t prohibitive. Another solution is to give people open-ended assessments where the assessment criteria put a lot of weight on what they decided to do relative to how, ie. which priorities they picked and which tradeoffs they accepted. Execution still matters, it just isn’t the whole grade.
One way to operationalize an assignment at this higher level of abstraction is to build a simulated company/project environment, eg. knowledge base of internal docs, data etc. which sits behind a chat interface. “Our largest customer says they’ll leave unless we build Feature X. Decide whether we build it and explain your decision in 250 words.” The candidate has to ask the system for information and work towards a diagnosis and solution. The transcript will contain traces of their thought process which LLMs can help you parse quickly even if it’s hard to fully automate.
It’s admittedly harder to automate the grading of such assessments. One useful trick here is to have any assignment have a written section at the end with a 250 word synthesis that has to be human written. AIs are great at exploring spaces and creating plausible sounding explanations but it is notoriously difficult to get them to compress their reasoning. And this type of compressed reasoning, however, is quite easily evaluated by a human and its the type of thing where a human with judgement is able to spot another one.

