top of page
Talent Pulse Logo - Computer Vision and MLOps Recruitment Agency
  • Writer's pictureSoheil Koohi

Technical Assessment in ChatGPT Era


a human wearing a robot mask to take a technical assessment
a human wearing a robot mask to take a technical assessment

Introduction


From content creation to email editing, from debugging codes to generating arts, generative AI tools like ChatGPT and MidJourney are reshaping how we approach tasks. Their accessibility and efficiency have turned once daunting tasks into simplified ones. But the benefits come with their own challenges. Especially in technical interviews and assessments, where candidates might use these tools to breeze through problem-solving or coding challenges, effectively "cheating" the system.

Imagine a situation where a candidate is asked to solve a coding challenge during a remote assessment. Instead of relying on their knowledge and skills, they could feed the problem statement to an AI model like ChatGPT. The model could then generate a solution which the candidate could present as their own. This isn't just limited to coding. It could also apply to answering technical questions or conceptual problems.

This shift necessitates a reevaluation of how hiring teams conduct technical assessments. The traditional methods might no longer be sufficient to identify truly skilled candidates. In this article, we will explore new strategies and mindsets for interviewers and hiring managers, ensuring they can keep pace in the rapidly evolving recruitment landscape influenced by AI. Let's delve into how we can conduct a technical assessment in the current era.


Accept or Confront?


There are two main ways to approach the challenge posed by generative AI tools in technical assessments.

The first approach is a more confrontational one: develop methods to detect the use of these AI tools. For example, recruiters could monitor candidates' screens during online interviews or design tests that exploit weaknesses of these tools.

The second approach accepts that these AI tools, like ChatGPT, have become an integral part of an engineer's toolkit, much like a modern calculator. Instead of trying to prevent their use, recruiters could provide candidates with access to these tools during assessments. This approach would shift the focus to evaluating how candidates interact with these tools: Can they craft effective prompts? Do they use AI tools effectively as an assistant?

This approach not only acknowledges the reality of these tools' presence in our day-to-day work, but it also allows recruiters to assess a candidate's adaptability to new technology. After all, being able to effectively leverage technology is a critical skill in today's rapidly evolving tech landscape.


Rethinking Technical Assessments: Embracing Real-world Contexts



a traditional technical assessment
a traditional technical assessment Source: HackerRank

Traditionally, coding assessments revolve around resolving a distinct algorithmic or data structure problem. The task? Write a piece of code that outputs the correct result. A classic example would be "Code a function that determines whether a number is a palindrome or not." Now here's the catch – generative AI tools like ChatGPT can handle these tasks almost instantly. They can present solutions that appear remarkably creative and provide step-by-step explanations. In several iterations, they can even optimize the solutions.


So, how can we pivot to make technical assessments more AI-resistant and, importantly, more reflective of actual job roles? The answer lies in embedding more context into the assessment.


Instead of focusing on isolated problems, why not give candidates a real-world codebase filled with diverse files and modules? Explain the codebase to them and ask them to add some modules or modify a specific part of the code. It's a challenging task to feed an entire codebase to an AI tool like ChatGPT and ask it to resolve a problem. Not only is it time-consuming, but it's also difficult for AI to devise a good solution because it's far from a typical problem found on the internet.


This method gives you a richer evaluation of your candidates. You're no longer just testing if they can devise an algorithmic solution. You're assessing their ability to comprehend a codebase, write standard-compliant code, develop test cases, and so much more.


So, dare to be innovative and invest some time in assessment design. Create a test that's not just another input for ChatGPT but a mirror of real-world coding challenges.


This is precisely the approach we champion at TalentPulse. For each client, we tailor a code challenge that's closely aligned with the company's applications and needs. We offer a comprehensive codebase that evaluates multiple facets of a candidate's abilities. Through this methodology, we ensure you discover truly exceptional talent.


Beyond Coding: Assessing Holistic Engineering Skills


 An engineer with her toolbox
An engineer with her toolbox

Now, here's a perspective that might not be widely shared: If a candidate can cheat on an assessment using ChatGPT, perhaps the assessment itself is lacking.


Relying solely on coding problems to evaluate a candidate's competency may not be the most effective approach. Besides the fact that these problems are susceptible to AI-assisted cheating, a good engineer isn't defined by their coding prowess alone.


The ability to analyze problems, communicate effectively, review and enhance others' work, document their ideas, and demonstrate various other skills are equally important. Hence, your assessments should be designed to evaluate these skills along with coding abilities.


How about asking candidates to write documentation for their final code? Or providing them with a functional but unclean piece of code and asking them to refactor it? You could even present them with a codebase and ask them to review it as if it were a GitHub pull request.


These tasks not only make it more difficult for candidates to rely on generative AI tools like ChatGPT in real-time but also test their abilities in essential areas of engineering that are part of their daily tasks.


This way, you're not just assessing a candidate's technical proficiency, but their holistic abilities as a software engineer. Such an approach can lead to better hiring outcomes, ensuring you bring on board not just a coder, but a well-rounded engineer.


Conclusion


In conclusion, the advent of generative AI tools like ChatGPT is revolutionizing many aspects of our lives, including how we conduct technical assessments. While these tools can be utilized for cheating on traditional coding tests, they also present an opportunity to redesign these assessments in a more holistic and effective manner.


Instead of resisting this AI advancement, we can harness it to create more comprehensive and resilient technical assessments. By shifting our focus from solely coding abilities to a candidate's overall engineering skills, we make it harder for AI tools to cheat while also getting a better understanding of the candidate's capabilities.


In a rapidly evolving technological landscape, updating our methods and mindsets for technical assessments is imperative. This is the philosophy we embrace at TalentPulse, where our focus is on creating assessments that not only challenge candidates in a meaningful way but also take into account the technological advancements of the era. We strive to find the best talents by designing tests that reflect real-world tasks and gauge essential engineering skills beyond just coding.


8 views0 comments
bottom of page