How To Measure the Things You Can’t See… A Step by Step Guide
Organizations of all types and size are increasingly tracking metrics to inform decisions, assess progress, and forecast future needs. For example, corporations often track expenses, sales, and headcount while non-profits track overhead, donations, and people served. One common trend associated metric tracking is that organizations typically track the things that are easily measured such as headcount or donations (i.e., the things they can see). However, these same organizations often have strategic plans associated with metrics that aren’t so easily measured such as customer loyalty, client satisfaction, employee engagement, donor trust, leader agility, employee performance, and so on. These metrics are considered hypothetical constructs, which are variables that are not directly observable. I refer to these metrics as the things you can’t see. While it can be a bit more difficult to measure feelings, attitudes, and behavioral intentions than count people or sales, it can – and should – be done. I provide a step-by-step guide on how to measure the things you can’t see using surveys and assessments below.
While it can be a bit more difficult to measure feelings, attitudes, and behavioral intentions than count people or sales, it can – and should – be done.
Determine what you are trying to measure. Once you have determined your research question or documented your strategic plan, you should create a list of the constructs you want to gain insight into. For example, if one of your goals is to improve the effectiveness of your leadership team, you will likely want to measure leader effectiveness or leader performance.
Research your constructs. Conduct a brief literature review on each of the constructs you are trying to measure. This will provide you with a basic overview of each construct, including how it has been defined and measured by other researchers or organizations. If you have access to an online research database such as EBSCO, this is an excellent avenue to find peer-reviewed articles. You may be able to access your alma mater’s database for a small alumni fee or donation. If you don’t have access to a database, you can utilize Google Scholar or find mainstream articles published by reputable consulting firms. I recommend searching a variety of terms to ensure you conduct a comprehensive search. For example, if you are researching leader effectiveness, you could search the terms leader, supervisor, or manager and effectiveness, performance, or capability.
Define and operationalize your constructs. Using your research from Step 2, define and operationalize each of your constructs. Your definition should be a clear and concise description of the construct. For example, leader effectiveness may be defined as the extent to which a direct supervisor motivates his/her employees to perform at a high level. Your operationalization of leader effectiveness would further clarify what this means or looks like by translating it into something that can be measured or quantified. For example, leader effectiveness may be operationalized as how well direct reports believe their supervisor: a) provides direction, b) provides feedback, c) provides recognition, and d) serves as a source of inspiration.
Write your items. Now that you have defined and operationalized your constructs, you can write items to measure each one. I recommend starting this process by reviewing your construct research or conducting an additional search to determine whether there are existing measures available for public use. Oftentimes, academic researchers will develop, validate, and publish measures that can be used by the public. This can provide a good foundation for measurement, but you may need to modify the measure to fit your needs and should review the article in its entirety to ensure you understand the sample and context in which it was developed. I recommend a general guideline of 3-5 items for each construct; however, you may need to use more items to ensure you assess the entire construct. For example, you may ask 1 or 2 items for each component of leader effectiveness (i.e., direction, feedback, recognition, inspiration). You also need to check that each item only assesses one construct by not double-barreling any items. For example, my direct supervisor provides direction and feedback at least once a week assesses two areas of a construct (i.e., direction and feedback) so you would need to separate this into two items. Finally, it is important to consider the reading level, language preferences, and computer proficiencies of your audience to ensure they will be able to complete your survey or assessment in the language and delivery mechanism (e.g., paper or online) you select.
Determine optimal response options. For each item, you need to determine the optimal response option that participants should use when submitting their answers. Measures often utilize 5- or 7-point response options with scale anchors associated with each option. For example, a 5-point agreement scale would have five response options (i.e., strongly disagree, disagree, neither disagree not agree, agree, and strongly agree). The direction of your scale should ensure that higher scores indicate a higher level of your construct (i.e., 5 = a highly effective leader). Although 5- and 7- point scales are commonly used by social science researchers, we at Category One Consulting often use a 10-point scale with anchors at each end. This is also a common practice in market and health research because 10-point scales are often more intuitive for participants completing the measure and stakeholders reviewing the results. If you are measuring several constructs across a wide variety of items, I recommend having as much scale consistency as possible. This will not only help you with analysis and reporting, but it will also help the participant understand the response options and complete the survey/assessment efficiently.
Develop instructions. Once you have your items written, you should develop a set of instructions that clearly explains the purpose of the measure, how participants should complete it, and how the results will be used. This will maximize response rates and across-participant consistency.
Pilot test your measure. The final step you should take before you start collecting official data is to pilot test your measure. You can do this by distributing the measure including the instructions, items, and response rates, to a small group of individuals who are similar to the population you intend to collect official data from. You should use the pilot test to ensure that all instructions, terms, response options, etc. are understandable and easy to complete. You can also conduct analyses such as reliability tests, exploratory factor analysis, and correlation analyses to test reliability, structure, and construct validity of your assessment.
I hope this post helps you track the metrics that are most central to your organization’s strategy. If you have questions or need additional support with research, analytics, or evidence-based practices, feel free to reach out!
