We continue with the second of four excerpts from Dr. Leslie Thomas’ presentation at the Association of Test Publishers’ Conference 2019. That presentation was called The Four Immutable Laws of Certification Success: A Psychometrician Speaks. Today’s video focuses on Law #2: Document Program Goals and Major Decisions BEFORE You Begin. You can find the transcript below the video.
Transcript of Video
The Four Immutable Laws of Certification Success: A Psychometrician Speaks
Law #2: Document program goals and major decisions BEFORE you begin
Leslie Thomas, Ph.D.
Chief Psychometric Officer
Kryterion, Inc.
So, you’ve decided you’re going to develop a credential. You’ve concluded your market research. Documenting program goals and major decisions is something that I strongly recommend before you ever begin test development.
Slide #9: Law #2: Document program goals and major decisions BEFORE you begin
- Get stakeholder agreement up front on key program parameters to prevent “moving target syndrome
- Purpose of the exam
- Target audience (e.g. job roles)
- May have a primary and secondary audience
- List major job responsibilities since job roles vary across organizations
You want to get stakeholder agreement before you begin to avoid what I call “moving target syndrome”
The Elusive Moving Target
Many years ago, I was brought to Long Island, New York by a certification sponsor, a technology company. Twenty subject matter experts from around the world flew in.
I had just gotten involved with the credentialing program, which dealt with RFID technology.
At the time, the credential was being developed to be vendor-specific for the technology company’s software. Without notice, the CEO decided that he wanted the credential to be an industry-based certification and wanted it to be vendor-neutral. You can’t imagine how hectic those next three days were. We basically threw out everything that had been done prior to that point and started over.
This second law of certification success centers around goals and key decisions that your stakeholders must agree on before test development begins. By stakeholder, I mean anyone who can influence the direction of the program down the road, including reversing the direction of the program.
So you want to get these key things decided up-front before you start.
I always recommend making sure that you and all other stakeholders are clear about the purpose of the exam and the target audience for the exam.
Keep in mind that often when you develop an exam, you’re going to have a primary job role, which is the one that you’re really targeting. You might also have some secondary job roles in mind. People in these secondary roles can take the exam, but they’re not really the target market.
Know Your Market
Make sure that your target market is clearly defined before you start test development because that market will be the focus of the program.
Also, sometimes there’s confusion in the marketplace because job roles can vary quite a bit from company to company, as I’m sure you’re aware.
One of the things I always recommend is listing the major responsibilities of the job role. This is also going to help your marketing team immensely.
One of the worst things that can happen is to launch a credentialing program and then to have someone in the target job role take the exam and subsequently discover that the exam tested for a job role very different from the one they do every day.
Job roles do vary quite a bit from organization to organization—especially between smaller and larger organizations. Understanding these variations is essential to creating a better product.
Slide 10: Law #2: Document program goals and major decisions BEFORE you begin
- Description of the Minimally Qualified Candidate (MQC) – defines minimal competence and, therefore, the overall difficulty level of the exam
- Level of education/training, existing knowledge/coursework
- # of hours or years of on-the-job experience
- # of implementations/surgeries
- Which job activities can be performed with and without assistance
Know Your MQC
Another thing that’s really important is the description of the minimally qualified candidate (MQC). I don’t know how many of you use the concept of the MQC. It’s often used during the standards-setting process.
In our development methodology, we use it at the very beginning of the test-development process. This is basically the conceptual definition of the person who is just minimally qualified to earn the credential. It sets your standard of performance, the level of difficulty for the exam. It’s kind of like a persona that you’d use in marketing.
A lot of times when we talk about this person, we do it in terms of the level of education or training or the existing knowledge they might have. So we might say that this person has gone through certain college courses or maybe has one year of on-the-job experience. We might talk about the numbers of hours or years of on-the-job experience, especially in healthcare, where experience might be defined as 250 hours of supervised experience.
Sometimes in software or in healthcare, experience may be defined as the number of implementations or the number of surgeries…whatever makes sense. Everybody is a little bit different, you have to think these through. We’re trying to help subject matter experts level-set on who this person as our standard of performance. We’re trying to determine the level of difficulty that’s appropriate for them.
Also you need to decide on is which job activities can or cannot be performed with or without assistance or adult supervision.
So these are some of the key things are included in the description of the MQC. The description is important is because it sets the level of difficulty for the exam.
When you start the development process, you’re saying, OK, which types of job activities do they do on these jobs at one year of experience?
What types of knowledge and skills do they need to know at one year? So then, if a stakeholder later comes in and says, “Well, I really think that this should be for a five-year person and not a one-year person.” And if you’ve already started development, you’re going to have to go back and start the development process all over again.
So for anything that impacts that level of difficulty for your exam, especially when it comes to things like levels of experience, make sure that it’s very clear at the beginning of the process. Because an exam that’s developed for someone, say, with one year of experience (essentially an entry-level exam) is very different from one developed for someone with 10 years of experience.
Slide 11: Law #2: Document program goals and major decisions BEFORE you begin
Other potentially important program parameters
- Vendor neutral vs. vendor specific
- US vs International (scope of practice, laws, regs)
- Seeking accreditation (e.g., ANSI/NCCA) – understand what’s involved, costs and timeline
- Reference vs. working knowledge
Parameters
So other potentially important program parameters include vendor neutral vs. vendor specific. If the program is U.S. or international, that’s also going to have implications. If it’s international, there are often differences in scope of practice when you deal with healthcare, for example. Also, there are different regulatory issues, so if your credentialing program includes making sure that people comply with OSHA regulations, and you decide that you want to go international, you might want to think that through a little bit.
So I can’t give you an exhaustive list of parameters to think about, but these are just some of the things that, over the years, have been kind of gotchas, so to speak. So they’re just things to think through at the beginning because going back later to address them almost always involves substantional rework and considerable cost.
Accreditation
Also if you’re seeking accreditation, ANSI and NCCA are the most popular options in our profession. I can’t tell you how many calls I get—and they seem to come in waves—about ANSI or NCCA accreditation, and I say “Great! Have you done this, this and this? And they’re like, no. And I’m like, well, you at least have another year of development. You’re probably about one-and-a-half to two years out from accreditation.”
So you need to understand what’s involved with accreditation. That will also impact your structure in terms of how you set things up because these organizations, for very good reasons, want to make sure that there’s a firewall between training and your certification, your credentialing organization. Just make sure that you understand what’s involved with it.
Also when you’re doing your design work in the very beginning, make sure that you’re clear about what’s in-scope and what’s out-of-scope. Sometimes what you’re not measuring on an exam is just as important as what you are measuring. So it’s really good to be clear about that.
Slide 12: Law #2 Document program goals and major decisions BEFORE you begin
Continuing Education and Recertification Requirements
- Communicate what is required and have process to track/verify
- Test Maintenance/Updates
- Health checks to ensure exam is performing as expected
- Content updates to reflect changes in practice/technology
- Refreshing content to enhance test security
Other Considerations
And the other thing to really think about at the beginning is continuing education and recertification requirements. The reason that this is important is because once you have your first test taker, the clock starts ticking. All of a sudden it’s like, “OK, what am I going to do for recertification or continuing education?”
As you’re designing your program, think through what’s going to be required to have them keep their credential current and how you’d like to track that. You’re going to need to communicate that clearly to your certificants
Test-maintenance updates are important for anyone working in IT/software industry. For most programs, you’re going to want to do a health check every year, if not more often, depending on the volume of the program. You just want to make sure that the test is performing as you expect it to perform.
You’re also going to need to do content updates. This is a particular challenge within the IT/software industry. Their content is constantly going out of date. So it’s a real challenge especially for bigger IT/software providers to keep their exams current. It used to be that they would try to go on release cycles every three or four months depending on when releases came out, and if it was a major release. Now a lot of them are doing what we call continuous deployment where they’re continually pumping out releases. It’s a challenge for the industry as a whole and it’s something you need to think about because frequent test updates impact your budget, costs and timelines.
You’re also going to want to refresh content to enhance test security. If your tests contain the same content over and over and over again, people are just going to end up memorizing it. You may have the best proctoring technology in the world with people standing over test-takers’ shoulders, but they’re still going to be able to memorize. You can’t stop that.
Slide 13: Law #2 Document program goals and major decisions BEFORE you develop
- Document and share decisions. Manage expectations!
- Ensure stakeholder agreement on major program decisions – prevent the “moving target” syndrome
- Keep SMEs aligned and focused on MQC
- Inform marketing initiatives – clear as to purpose of the exam, intended meaning of the scores, how scores should and should not be used, targeted job roles, what is and what is not measured
Document all decisions. We use something called a test-definition document to record decisions. I always recommend that as decisions are made, that they be documented and sent out—even if, and especially if, key stakeholders aren’t present. I like to say, “Speak now or forever hold your peace.” Everyone must be on the same page for big design decisions.
Use the test-definition document to make sure that all of your subject matter experts are aligned on the meaning of the minimally qualified candidate (MQC). The document is also important for informing marketing activities. Marketers must understand the purpose of the exam and the intended interpretation of the scores, and how it’s going to be used. You want to make sure that when you market the exam that it’s clear how it’s to be used so that any misuse doesn’t come back to hurt you.
***
There are Four Immutable Laws of Certification Success. Find a video for each one here on the Kryterion blog (www. KryterionOnline.com/blog).
Contact Us! The Kryterion Psychometric Team would love to learn more about your first or next certification exam. You can start the conversation here.
