If you followed my previous suggestion about the discussion you want to have with your analysts at the start of a new project, there is a good chance you are all on the same page. If instead, your analysts have been working on projects you’re not completely aware of, then you are likely to have a bunch of questions about what they did. Here are seven (7) things to ask your analyst when they bring you analytic results.

Pro Tip: Even if you had a pre-analysis discussion with your analysts, it’s great to have them go through these questions after the fact to ensure they were successful.

What is the research question?

Your analysts should be able to tell you what the research question was that they answered. They should be specific in terms of clearly defining the outcome and all other concepts included in the analysis.

“Has our sales funnel improved sales?” is not a particularly specific question. A more specific question would be, “Has the new sales funnel increased new customer conversion rates among qualified prospects relative to the conversion rate of the old funnel?”

Now the question has been framed with clearer specifics. The previous outcome was “improved sales, which could mean volume, dollars, or conversion rates. Stating that the question is about conversion rates clarifies the outcome.

Other concepts have also been clarified. The previous input concept was “our sales funnel”, which could be confusing if you have more than one funnel. By focusing on “the new sales funnel”, the question is framed more narrowly to exclude any previously existing funnels.

Finally, the original question asked if the sales funnel “improved sales”, but didn’t state what sales would be compared to? It’s not clear what relationship your analysts were looking at. The more specific question is explicit to say that you want to compare conversion rates from the “new funnel” to the same conversion rates from the “old funnel”.

You could even get more specific in this question by stating the specific time frames, such as comparing Q3 conversion rates under the new funnel to Q2 conversion rates under the old funnel.

What Is the Population of Interest?

You’ll want to make sure the analysts can clearly state the population they were studying. In the original question above, the population is implied. You could easily assume that they were assessing all sales coming through the sales funnel. It’s not clear if they were focused on new or repeat customers, a specific demographic, or sales of a certain product or service. The more specific example question clarifies some of this to indicate that the analysis focuses on conversion rates for “qualified prospects”. The question could be further clarified to indicate whether the analysis focused on all qualified prospects, or a subset of prospects based on other criteria.

When asking about the population of interest, your analysts should also be able to tell you whether they used data on the entire population or whether they drew a sample of data. If they drew a sample of data, they should tell you how big the sample was and how they confirmed that the sample characteristics were similar to those of the population. In our example, because the question focuses on conversion rates for qualified prospects the analysts should be using data from the population of all prospects. Still, it is best to confirm this.

What Data Was Used for the Analysis?

While the data was used for the analysis may seem obvious, it’s good practice to confirm this with your analytic team. You might be surprised at how often they used data you didn’t expect them to use. You might also be surprised that they omitted data you thought they should have used.

In addition to confirming the data sources with your analysts, ask them to explain how they calculated each of the measures used in the analysis. As with the choice of data sources, you might find differences between how you think certain metrics should be calculated, and how your analysts think they should be calculated. Discussing the measurement strategy explicitly when reviewing the results will highlight any discrepancies and provide an opportunity to consider making potential adjustments.

What Methodology Was Used for the Analysis?

Your analytic team should be able to tell you what analytic methodology they used for the analysis. Be sure to ask them to provide you with a plain language explanation of what the method does. This provides them with additional experience communicating with non-technical audiences. It also offers you an opportunity to ensure that everyone has the same shared understanding of the methodology.

As part of the methodology discussion, ask your analysts to explain how any coefficients or statistics should be interpreted. This will make your effort easier when translating the results into a plain language answer to the research question.

Be sure to have the analysts tell you how they defined a statistically significant result. Did they use a conventional five percent probability, or did they use a larger or smaller standard?

As a final question about the methodology and analysis, you should develop the habit of asking your team one very important question: did they receive any warnings or errors during the analysis? Statistical software packages may stop an analysis if there are fatal flaws in the data, or critical assumptions are violated. This is not, however, always strictly true. Even if a log file is free of warnings and errors, there could still be issues such as incorrectly calculated measures, invalid assumptions, or improper functional forms of a model. It is the analyst’s responsibility to ensure the data and model were prepared properly, and to be able to describe how they verified the results.

How Did the Analysts Identify Any Outlier Observations?

Many statistical analyses can be influenced by outlier observations – those with unusually large or small values on a variable. Consider trying to calculate the average cost of a hospital stay. There will be a few observations that are extremely expensive. We typically see these expensive stays with patients receiving high-cost treatments, surgeries, or spending extended periods of time in the hospital.

To get a more accurate estimate of the typical hospital stay, we could drop those outlier observations from the data set completely. Alternatively, we could keep the observations in the data and cap their costs at a high value, but one that won’t have a major influence on the average value (e.g., $100,000).

The most important question to assess is how much the results change because of the outliers. Whether outliers are dropped or have their values capped, make sure that the analysis is performed both with and without the outlier manipulation. The change in the results will tell you how much of an influence the outliers had on the analysis. You’ll also be able to look at the results without the influence of the outliers and decide whether the results make more sense in the context of the project.

How Did the Analysts Validate Their Results to Ensure Correctness?

The process of validating analytic results is an extremely important part of the analysis. Given the costs of performing the analysis, and the greater costs of making decisions using bad information, you want to confirm the results are as accurate as possible.

Have your analysts document any alternative methods they used to corroborate their results. List out any diagnostic tests that were performed to ensure the results were robust, as well as the results of those checks. Include any sensitivity checks performed to verify the stability of the analysis. Also, confirm with the analysts how they reviewed and confirmed that their programming code transformed the data and performed the analysis correctly.

Tell The Data Story

As a final check, ask your analysts to tell you a story that explains the question driving the analysis and provides the answer. Ask your analysts to tell the story in plain language. Doing so ensures that everyone stays on the same page and agrees on the results. This exercise also provides them with an opportunity to translate their technical craft into a more digestible format for non-technical audiences.

To help coach your analysts through the story, tell them that they do not need to discuss all the technical details of the analysis. Instead, have them focus on framing the motivation for the research question. Next, they should provide a brief statement of the primary methodology. Follow this with the highlights of the results as they relate to answering the question. Again, your analysts do not need to cover every statistical result produced by the analysis; have them stick to the key points.

Conclusion

If you planned out your analytic project thoroughly from the beginning, many of these questions for your analysts should be simple confirmation that the analytic plan was followed. In contrast, if your analysts completed the project without detailed planning input or routine progress updates, then you can use these questions and approaches to help uncover what was done, why it was done, and what the results mean.