Taking a Practice Exam isn’t studying

Every day hundreds of people come to HamStudy.org and take practice tests. Others go to other websites and take practice tests. There are differences in the user experience between those two groups of people, but they both have one thing in common — they aren’t actually studying for their exam.

I’m going to repeat that.  Taking a practice exam is not studying for an exam.

I’ll pause here while you think of ways to flame me on the comments section and I decide if I should turn it off or not.

tl;dr: For those too lazy to read the article, the conclusions are simple: Practice Exams are fabulous for benchmarking yourself to see how close you are, but when used to study result in a lot of wasted time and/or questions on your exam which you have never seen before. Reading through the questions and intelligent directed flash cards are by far the most effective study methods we have found. Whether you use our tools or not (and did you know HamStudy.org works in mobile Chrome and Safari?) make sure you are using something more than just practice exams for studying.

Read on for charts, justifications, pretense at math, and how we arrived at these conclusions.

Seriously, though, I realize that it is actually possible to get your license just by taking practice exams over and over again.  In fact, given the number of people who tell me they got their license “only because of your website”, and the number of practice exams vs flashcards that are taken on HamStudy.org, I have to conclude that a large number of people do actually manage that.

(as a side note, this claim is ridiculous; I appreciate the thought, and I’m glad our efforts are useful, but you would have gotten your license another way if we hadn’t been here; I’d like to think we made it easier, however).

HamStudy.org Questions answered per week

This is a graph of the number of questions answered per week by HamStudy.org users from April 28, 2013 to November 29, 2015. Red is practice exams, green is flashcards.

Before you heat up the proverbial oil, let me explain what I mean.  The current Technician exam has 426 questions in it.  It has 10 subelements, and 35 sections.  The typical way that questions are selected for an exam is to choose one at random from each of those 35 sections.  Let’s look at the question distribution in that pool:

Screenshot 2015-12-05 11.50.06

That’s not the most exciting graph in the world, but as you see there out of the 35 subelements, all have at least 11 questions and none have more than 14 questions; therefor the probability of any question being chosen for your exam (assuming it is randomly generated) is at least 1 chance in 14 but never more than 1 chance in 11 (roughly between a 7.14% chance and a 9% chance). So, we have a list of 426 questions, all of which are within 2% of having the same probability of appearing on the exam, and you need to make sure you pass that exam.

So, naturally, you do what any normal thinking person would do, right? You split up the questions with each section in its own box.  You pick one question from each of the 35 boxes and have a friend quiz you to see if you got it right or not. If you miss it, he tells you what the correct answer was!  Then, after doing one question from each box, you put them all back into the box, shake it up, and start over again as though you’d never done the first round.  Makes perfect sense, right?

Hmm.

Yes, this is exactly what practice exams are. They are random selections of questions from a question pool which you need to learn.  How many practice exams do you need to take before you see each question, do you suppose?  Let’s do a bit more math.

First, there are 14 questions in three of the sections (T1A, T1C, and T9A).  That means that if you want to be sure you have seen all the questions, you’ll need a *minimum* of 14 practice exams.  No problem, right?  But wait!

Each practice exam actually chooses all the questions at random, so if you took 14 exams, then each time you had a 7.14% chance of getting any given question.  The probability that you have seen each of the 14 questions is easy to calculate; the probability of the first one being new is easy: 1/1. The probability of the second giving you a unique question is 1-(13/14). Following this through, we get a 0.000027% chance of that occurring.  That’s just in one section!

The solution – actually study

One of our biggest secrets to success here at HamStudy.org (I’ll tell you, but don’t go posting it online or anything, k?) is that we have our own rotating focus group that we use to see how our software is doing for people. 9 months out of the year there is a one-day Technician class held at Brigham Young University in Provo, Utah, which is attended by 30-50 students each time.  After the “cram” portion of the class all of those applicants sit down and study for their exam using — yep — HamStudy.org.  You might even say that the website was created to help these people — you’d be at least half right.

Here are the methods we have tried over the years and the results:

1. Practice Exams

Taking practice exams has been considered by most hams to be studying for so long that it is naturally how we started. The result? The #1 complaint I heard from applicants who didn’t pass: “I don’t get it! I’ve been taking practice exams all day, and these questions never showed up on those tests!”

Practice Exams aren’t studying! They aren’t specific enough, and they aren’t directed!

We did find, though, that practice exams are the best way to get a feel for how close  you are — if you have been studying already.

2. Smart Practice Exams

The astute reader will have already hit on the same solution I first did — if practice exams not showing all the questions are the problem, then make the practice exams intelligent so they pick different questions each time!

This is actually a pretty good method, but there are a few problems with it.  The first is that while it’s very effective, the simplest implementation is *incredibly* frustrating.  I had a lady come up to me and say “I have just taking 10 practice exams in a row, and I am not getting any better.  I think I’m just going to give up and maybe come back next month”.

Anyone spot the problem? Yep, those 10 practice exams contained different questions each time.  Of *course* she didn’t improve any! She hadn’t gotten through the whole pool yet, so each time it was all new!

The biggest problem with smart practice exams is that the exam is too long to effectively repeat things across exams.  You can do it, but by the time you’ve done that, doesn’t it make a lot more sense just to use flashcards?

3. Flashcards

This is really the place where HamStudy.org currently shines; our flashcards are phenomenal. I say that with full (-ly absent) humility and knowing how much time we’ve spent on them. Where people before complained that the smart exams were exhausting and draining, many tell me that the flashcards are addictive.  Here are some of the plusses to flashcards if done correctly:

  • Flashcards are fast – they can easily adjust on the fly.
  • Flashcards are targeted – they know what you’ve done in the past, so they can choose well for future cards
  • Flashcards are hard but not too hard – when the questions are all hard, it can throw in easier ones.  When it’s getting boring, it can give you harder ones.
  • Flashcards repeat quickly as needed – you don’t depend on luck to review a question that you didn’t know well
  • Flashcards can be specific – You can target them to a specific group of questions. We are working on improving how we do this.

We have found that if you want to learn the question pool, the flashcards are the most effective way to do it.  Most of these things could be applied to smart practice exams as well, but that doesn’t really buy you anything.

4. Reading the questions

This is easily the most often overlooked study method, and yet it’s one of the most effective.  Just reading through the question will get you familiar with them faster than any other method. We’re working on finding ways to better tie our read section into the other study tools, but don’t underestimate the power of reading questions.

Conclusion

Don’t get me wrong here; Practice Exams are invaluable.  How else will you know how likely you are to pass, or how effective your studies are? If you want to actually study, though, use a tool that gives you something targeted — even if it’s just a printout.

73,

Richard
KD7BBC


 

For those interested, here are some of the features we’re focussing on right now. This is a project we do in our “free time”, and we all have families and children (I have four) so we don’t have fixed timelines for when these will be done:

  • Mobile app – Android and iOS to start with, it will probably cost $3.99 but will allow full offline use. Don’t worry, the website will have all the same features and will remain free for use on mobile devices for those who don’t want to pay for offline access.
  • Improved flashcards – the Aptitude thing? We have a far better plan. We’ll also give you more information about how you’re doing on the question, and we’ll let you choose which questions to study with much more control.
  • Integrated “read questions” area – we’ll tie it in with the stats and hopefully make it possible to build a card deck while reading questions which you can later use with studying flashcards.
  • Simplified user accounts – the one we have seems to annoy people, so we’re going to try to improve that.  We’ll make it possible to convert a guest account to a full account and possible to create an account without an email address.

What other things would you like to see us work on?

This entry was posted in Uncategorized. Bookmark the permalink.

39 Responses to "Taking a Practice Exam isn’t studying"

Leave a Reply

Your email address will not be published. Required fields are marked *


*