A Peek Behind the Curtain, How NCARB is Establishing the Cut Score for ARE 5.0

by: JENNIFER PENNER, AIA, NCARB

I attended an NCARB webinar earlier in the month and received wonderful insight in to the world of NCARB’s endeavor to reach a cut score for ARE 5.0. Based on their analytics they knew of at least 2,000 candidates that had positioned themselves to strategically transition and have only two remaining ARE 5.0 divisions on November 1, but the candidates didn’t transition. NCARB was puzzled. They offered $100 Visa gift cards to the first 600 candidates to sit for each exam and yet the sign ups remained stagnant. Using their resources to poll eligible candidates they arrived at a few conclusions:

  1. The $100 Visa gift card for the first 600 ARE 5.0 exam takers didn’t go over as well as they imagined, with candidates expecting that they may miss the cut off if they scheduled too far out. Expecting that they wouldn’t receive a gift card, they didn’t even bother looking into it.
  2. Candidates were asking, “how do I study for this exam?” There were limited resources and despite NCARB’s release of its own material, candidates still felt ill-prepared.
  3. They encountered the “I don’t want to be the first one” mentality that often times occurs with such a huge undertaking. Candidates responded, “I’m just going to wait until others take it and see what they say.”
  4. The appearance that some candidates were over studying. Meaning they are taking unusually longer to prepare for the exam in an attempt to be best prepared for a completely unknown experience.

NCARB didn’t see the sign ups it anticipated and knew that it would need to identify a new strategy to engage candidates if examinees who completed exams in 5.0 were ever going to get their test scores. Thus came guaranteed free retakes through February 28th plus the $100 Visa gift card. At that point February volumes shot through the roof and over *2,000 candidates scheduled and sat for many of the 5.0 exams.

Excited about the results of their new incentives and reaching their seat number goal, NCARB was able to begin establishing cut scores for three of its exams; PPD and then PCM & PDD.

In order to evaluate these exams, they set up a standard setting panel made up of 12-15 registered architects in different stages of their careers. These brave souls spent two full days taking the same exam candidates took and subsequently discussing the content and making recommendations.

Some things that NCARB wants you to know about the exam:

  1. Time management is more important in 5.0 than it was in 4.0. There is time to complete the exams but candidates need to understand that every question is worth just one point. Whether it is the standalone multiple choice or a part of a case study, be aware of your clock.
  2. Score reports will look mostly the same using levels to describe the scoring but reports will be a little more descriptive than before.
  3. Score reports will be released two days after the exam, for all exams taken after the cut score has been established. Score reports were previously slower to be released in ARE 4.0 due to batch scoring of the vignettes.

There was some concern by other webinar attendees that NCARB’s free retakes and gift cards led to a rush of unqualified candidates sitting for the exam who would then drag the cut score down and invalidate the results. NCARB did anticipate such an occurrence and is using a mechanism called an equator block to measure the caliber of the candidate. An equator block is a set of questions that appear in both ARE 4.0 and ARE 5.0 to judge a candidate’s basic knowledge of the exam content. If an unqualified examinee has sat for the exam, meaning they have not studied, then these equator blocks will identify low performance and the cut score will take this into account.

There you have it, the inside scoop on how NCARB is able to reach a cut score and how it almost didn’t! For more information on ARE 5.0 please go to NCARB’s website at www.NCARB.org and click on Architect Registration Examination.

*From the time I attended the webinar on February 9th to the time of this publication on February 28th, the number rose from 800 scheduled exams to over 2,000 per NCARB officials.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

Up ↑

%d bloggers like this: