Interviewing analysts can be a challenging activity, especially if you are not an analyst yourself. You have probably seen at least a few resumes from young analysts that seem almost too good to be true. Fortunately, my friend, there are several tell-tale signs that the analyst candidate you are interviewing is overstating their skills.

Inability to Explain Analysis Without Using Technical Jargon

Pick a recent project on the candidate’s resume and ask them to explain the project in plain language. This exercise focuses on three skills. First, can the candidate get away from technical jargon to tell others what was done and why? Technical jargon has its place, helping technical staff to communicate more efficiently by encapsulating complex ideas into discipline-specific terminology. Analysts who are unable to break out of technical jargon may be inexperienced with the concepts they are talking about.

Second, the reliance on jargon demonstrates a limited view of the concepts they are discussing and potential challenges drawing linkages between the real world and their analytical world. Third, candidates who cannot explain what they did in plain language when asked may be demonstrating that they didn’t listen to your request or that they can’t communicate well. Either way, the reliance on technical jargon or the struggle to describe the work in plain language is a red flag that our candidate may not be very experienced.

Unclear About Who Wrote the Programming Code

Experienced analysts are typically responsible for writing the programming code used in their analyses. Novice analysts, on the other hand, are often running pre-written code that requires only limited updating. Ask your analyst candidate about their specific role in writing, updating, and running the code used for the project. Drill down if necessary to make them tell you whether someone else wrote the code.

It is far easier for an analyst to run code written by someone else when they are shown what needs to be updated to run the analysis. With a little instruction from an experienced programmer, the person running the code may not even need a strong programming background at all. They may simply need to edit the code to change a few parameters such as filenames or dates before running the code. When you focus on who wrote the code, you’ll be able to determine whether your candidate was responsible for developing the analysis or was simply running the code.

Too Much Experience with Cutting Edge Methods

If your candidate is new to the workplace, but they list a lot of projects using cutting-edge analytic methods, they may be overstating their experience. Cutting-edge methods are, by definition, relatively new and the pool of candidates with a lot of experience will be small. Candidates may have gone to school to work specifically to work with such advanced methods, but this should be clear elsewhere on their resume.

For example, in 2023 the explosion of artificial intelligence created a lot of buzz. The pool of AI researchers and those developing AI analytic solutions is, however, relatively small. If your candidate claims to have extensive experience building AI solutions, this is likely to be an overstatement. Probe their explanations with direct questions to understand how they are defining AI and what precisely their models accomplished.

Academic Experience vs Real-World Experience

Young analysts will often list projects they completed as part of their coursework on their resumes. You can’t really blame them since these academic projects are often their only source of experience. The challenge, however, is that academic projects are typically created to teach a set of programming commands to execute a specific methodology.

The data sets are often “toy” data, meaning they are small, clean data sets created to have characteristics useful for practicing a specific technique. The data given to students in data analysis courses typically do not offer the opportunity to practice the data extraction, cleaning, and manipulation techniques that are required for working with real-world data.

If your candidate includes a lot of academic coursework as experience on their resume, you’ll want to know the type of projects listed. The experience gained varies across projects that were part of a data analysis course, executed as part of a substantive topic course, or thesis projects. Thesis projects and substantive course projects often provide greater practical experience for analyzing real-world data. Confirm where the data set came from and what it contained to get a better sense of the experience.

Young analysts will also sometimes refer to Kaggle projects as experience. Kaggle is an online community that hosts data sets and pre-written code geared toward the data science community as a source for testing out new techniques and participating in coding competitions. There is a lot of information analysts can learn from using Kaggle data and participating in competitions; however, the data sets are often still toy data, and the competitions have very targeted goals.

If your candidate uses Kaggle experience on their resume, you’ll want them to elaborate on the projects more. Ask them to explain what the purpose of the project was, what they learned from participating, and how the experience aligns with the type of work you’ll expect them to perform at your organization.

Are They Highly Focused on Particular Methods?

It is unfortunate, but some new analysts seem to believe that if they load their interview responses with buzzwords about the latest sexy analytic methods they will somehow appear more skilled. Additionally, there is a strong incentive to focus on the most advanced techniques they have used to demonstrate their skill.

The problem is that experienced analysts understand that most of their work lies in cleaning and transforming data and performing small and relatively simple exploratory analyses. Your candidate should be able to recommend simpler analyses when possible to answer the question.

“If the only tool you have is a hammer, you tend to see every problem as a nail.”

– Abraham Maslow

The analytic analogy of Maslow’s famous quote is that when an analyst has limited experience, they tend to see every problem as something they can solve with the most advanced technique they know. If they seem fixated on one method or advanced methods only, they may not have the experience you are looking for.

Bonus Technique #1: Do they Understand the Programming Language They Need to Use?

I have two bonus techniques that are excellent for vetting analyst candidates. The first one is whether they are knowledgeable in the programming language they need to use. While a good programmer can pick up new languages when needed, there is usually a ramp up period for developing solid skills. If your candidate is inexperienced with the tools used in your organization, you can expect some lag time to achieving full productivity.

The best way to tell how much experience a candidate has in a particular programming language is to have them speak with an analyst experienced with the programs required. The purpose of the programming interview is to verify that they understand the syntax and packages they will use.

In some cases, this type of interview may include whiteboarding the candidate by asking them to develop a small piece of code to perform a specific task. The exercise allows the interviewer to see how the analyst thinks, and their comprehension of the analytic tools.

As an alternative, if an analyst is not available to assist with the interview, you can very often obtain programming cheat sheets online for various analytic languages. These cheat sheets will provide you with specific commands and analytic packages used to execute various techniques. As an interviewer, having one of these cheat sheets available for your review can help you determine at a cursory level whether the candidate understands the basics of the language or not.

Bonus Technique #2: Do They Display Intellectual Curiosity?

This is one of my personal favorite techniques for identifying good analysts. Some analysts demonstrate technical expertise, but make it clear they want to be given all of the parameters for an analysis before they execute what is asked. Effectively, they are indicating that they want you to do all the work to develop the solution, so that they can simply write the code you have asked for. If you have an analytic mindset and simply want someone to do the specific analyses you developed, this could still be fruitful.

On the other hand, if you want an analyst who will think for themselves, then you need someone who displays intellectual curiosity. The analyst with intellectual curiosity gets excited about new projects. They ask a lot of questions to help identify business needs and analytic goals. You can tell that intellectually curious candidates are engaged and thinking about the project from various angles to develop a solution. You won’t have to spell everything out for them. They will take the time to confirm that their understanding is consistent with yours and that the solution they are thinking about will meet your needs.

Identifying candidates who demonstrate strong intellectual curiosity has proven to be one of the best predictors I have found for successful hiring.

Conclusion

It is unfortunate that we need to discuss how to spot candidates who are overstating their skills during an interview. Nevertheless, it is a reality. While I try to give latitude to new workers fresh out of school, I have seen plenty of “experienced” analysts overstate their skills as well. If you participate in interviews for analytic candidates, be aware of these red flags to look out for. Do not hesitate to ask probing questions to clarify the details and flush out those who might be exaggerating. You are the one who needs to work with your hire. Make sure they’re a good one!