We continue with the fourth (of four) video excerpts from Dr. Leslie Thomas’ presentation at the Association of Test Publishers’ Conference 2019. That presentation was entitled The Four Immutable Laws of Certification Success: A Psychometrician Speaks. Today’s video focuses on Law #4: Follow the Proper Test-Development Process and Maximize the Use of Technology and Data.
You can find a summary of Law #4 below the video.
Always Follow Professional Testing Standards and Legal Guidelines
There are professional testing standards and legal guidelines for the test-development process. They help ensure that your exam scores are valid, reliable and legally defensible. Following standards and guidelines are especially important for licensure exams.
Testing Standards Help Ensure Scores Are Valid, Reliable and Legally Defensible
It’s not uncommon for a credentialing exam to launch, gain traction and become popular. So popular, in fact, that another organization requests to use the exam for a higher-stakes decision only to discover that standards and best practices weren’t followed during the original test-development process. Had professional testing standards been followed from the beginning, the proposed higher use of the exam might have been possible.
Following Standards Are Required For ANSI or NCCA Accreditation
If you’ve ever wanted your program to be accepted for licensure in certain states be advised that many of them will investigate whether or not professional testing standards, like those from ANSI or NCCA, were followed during test development. Leslie recounts getting calls from clients who were unable to take advantage of proposals from states offering reciprocity or similar benefits because professional testing standards weren’t followed.
Testing Standards Create a Higher Quality and More Credible Product
The most important reason for following testing standards is that they yield a higher credentialing product. There are sound reasons for all of the steps in the test-development process. Employers will be able to recognize an inferior credentialing exam. They will hire certificants who aren’t competent in their specific job role. Employers will connect the dots, and your program will lose credibility.
Streamline Your Test Development Process with Technology
Today you can use virtual workshops for item writing, standard setting, job-task analyses and more. There’s a real skill in facilitating them well because you’re dealing with people who are working at home and there are distractions. It’s relatively common that not enough people show up for scheduled tasks creating uncertainty about moving forward or rescheduling. Workshops require a real time commitment for SMEs and a high level of skill for the facilitator.
A Few Caveats about Virtual Workshops:
- If you have an aggressive timeline with a hard deadline that you cannot meet, do the workshop on site. We used to say that the way to handle an item-writing workshop was to pack the SMEs into a room, feed them and don’t let leave until they’re done with their assignments.
- As a facilitator working on the test-development side, Leslie noticed it’s easier to miss the nonverbal cues (e.g., looks of confusion or disagreement among SMEs) in a virtual workshop than in an on-site workshop.
- If the SMEs have not worked together before, it’s always a good idea to do the on-site workshop first because that really builds relationships and networking. Your biggest challenge in doing things virtually—and that’s when you really need to recruit more subject matter experts than you need—is that people just don’t show up. So if you want people to show up, the best strategy is to do the first workshop onsite and include some relationship-building activities so that SMEs feel more of an obligation to show up virtually for their peers.
Leverage Data to Create More Products and Constantly Improve Your Program
Often we’re collecting data throughout the test-development process, like for a job-task-analysis survey, and some of that data would make a good industry report. Many companies want to know where they fall within an industry and within a job role. Sometimes companies can use that information related to a task or job competencies to help them write job descriptions or to create benchmarking tools.
There are several ways to create products out of the information that you’re collecting and doing so helps you be seen as a thought leader in your industry. Things like job-task-analysis surveys will often include questions about the state of the industry and the answers are always valuable to share.
Use Item Statistics to Identify “Model Items” That Can Be Cloned
You can use your statistics on your items to improve items and to identify your “model item,” especially if you have scenario-based items. Scenario-based items often measure more applied knowledge, which is what we want, but they’re also more difficult to write. If you can identify scenario items that are performing well statistically, there are variables in there that you can switch out to create more items and do so in a more efficient manner because item writing is difficult.
Survey Your Test Takers and Your Customers
Ask them what they like, don’t like about your existing program about areas that could be improved. Sometimes exam feedback will let you know that you’re focusing on one area too much or maybe not enough. Maybe there’s new technology that should be a part of the exam.
A credentialing program is not a “one and done” type of thing. If the program is to stay competitive, you have keep improving the program over time, especially these days where technology is advancing at lightning speed. For our industry to stay relevant, we’ll have to do more and more to stay competitive.
The 4 Benefits of Obeying the 4 Immutable Laws of Certification Success
- Create a higher-quality credentialing program in less time with less cost.
- Better communicate the value of your program based on real market needs
- Maximize your investment in technology and people to streamline development
- Enhance your program over time based on valuable market feedback
Focus on what your market’s pain point is and get really clear about that. Look in the data and see if you can get information about what specifically you should cover. And use those specifics to help you market the credential to your target audience. Remember to speak their language.
Maximize your investment in technology and people to streamline development. Use technology when it makes sense and keep in mind the human factor. You’re working with people.
No credentialing program is ever one-and-done. Enhance your program over time based on valuable market feedback to make sure that you’re remaining competitive. You can offer more products to your industry and project yourself as a thought leader.
