Competence: It’s All About Context!

Great test design for performance-based exams depends on a shared understanding of competence.

Competence is doing meaningful work to standard under normal conditions. Agreeing on what is normal is where the hang up is. Normal assumes agreement on the uniformity and diversity of both work tasks and the workplace environment. People with the same job title may or may not do the same tasks even in the same company. They certainly do not share the same workplace environment if they work in different locations or for different bosses.

Rob Foshay, Ph.D. et.al, in their research on emerging best practices in the private sector discusses the rising interest in performance-based tests to measure competence. Part of that interest is fueled by the evidence that performance-based tests have higher fidelity (they more fully reflect the workplace environment) and; therefore, can demonstrate greater external integrity (passing correlates with something important to the organization). For example, is there a correlation between the number of people who pass the test and those who do well on the job, stay in the job longer, or have fewer cost overruns or missed deadlines?

The Test Design Challenge

The challenge is to design a test that accommodates workplace variables so test results correlate with measures the organization cares about. Consider the following scenarios. Assume you want to measure the skills of:

• Mobile software developers who are writing code for mobile computing platforms for a vendor, such as Google,
• Network management professionals who use software to manage corporate networks for a vendor, such as CISCO, or
• System administrators who manage servers. for a vendor, such as SUSE

In each of these examples you want to identify what is different and similar about the tasks and the work environment. For example:

• Are tasks done as part of a team or individually?
• Is there a difference in the maturity of the technology used and the efficiency of the procedures?
• Do bosses have the same expectations?

You do this by asking 1) the right questions and 2) the right people. The process is called a practice analysis. Typically, the task analysis depends on the input of a few people and the questions are limited to what you do, and what do you have to know to do it.

A practice analysis solicits opinions from diverse perspectives such as the people upstream and downstream who are impacted by what is done and how it is done. The questions are about what information, systems, and support people can access, how they work, what feedback is available, the maturity of the systems, and what outputs and behaviors really get rewarded. This information allows the test designer to build in constraints that more accurately measure and predict on-the-job performance.

A knowledge test can answer the question do people know relevant rules, terms, and concepts. However, performance-based tests can answer questions about competence if they reflect the context in which the work is done, that is, emulate the environment.

Performance-based assessment was once only accessible to large organizations and only used for high-risk scenarios. Today, many of the economic and technical barriers to adoption have been largely removed. TrueAbility, for example, has shown that it can cost-effectively emulate software platforms used at the worksite and deliver performance-based assessment globally and at scale.

The challenges that remain relate to organizational alignment and the tried-and-true practice of test design. And fortunately for many practitioners, the resources available to help you on this journey are growing every day.

---

---