Growth Engine’s automatic curricula generation for the US Air Force was prudent and a time saver. It has also proven its potential as a generally useful tool. It certainly sounds useful to reduce the amount of time it takes a complex and high-tech organization to develop training protocol, especially when the training is necessary to prepare people to operate equipment that can be any combination of powerful, delicate, and expensive, but all of this is only true if the automatically generated curricula are as good as the manually-generated ones. Well, rest assured. They are.
In fact, when it comes to getting content approved, the manual process involved considerably more iterations than Growth Engine’s automated process. Let’s pretend for a moment that you know your way around Air Force equipment and its use. Imagine that you now need to take 1,000 pages of technical documentation and turn it into a series of what we might call — to simplify the curricula development process — “quizzes”. What would be the specific steps involved? Deciding what facts and logic to include in the quizzes would likely be some function of what is important and what is likely to trip up trainees. After all, there are plenty of important topics that are obvious and therefore do not need to be overly stressed, and there are some less important topics that may still be the cause of confusion and require some double checking.
How do you begin to decide between these options? Well, it’s a lot of work. You might pour through all the documentation enough to develop an intuitive sense of recurring issues and draft those into questions. You might draw on your experience and choose to emphasize certain things you believe would otherwise confuse trainees. You might draft many questions as you go and filter them afterwards. Ok. Say you do all three. Do you think what you create on the first pass will make it into the final curricula? Possibly, but most will likely not. You can already begin to see that, at this point, the quizzes resulting from the manual process are going to require a lot of revision.
But it doesn’t end there. The questions need to have compelling answer choices and reference documentation, too. It is not enough just to know what content to include; that content has to also be formatted into questions with tricky and believable incorrect answer options (called “distractors” in the world of education). Think back to the last time you took a multiple choice quiz. One or perhaps more of the answer options were true, but there were some false ones in there, too. That’s what makes the quiz a quiz. The process of creating compelling distractor options was as iterative as it was time consuming for the Subject Matter Experts (SMEs) who manually created curricula before Growth Engine got involved. Finally, the SMEs were required to explore the reference material to find the instances covering the material in the questions. Overall, there were a minority (~20%) of “Tier 1” questions requiring only quick and obvious revision or no iteration at all. The majority of questions on the first pass were either “Tier 2” questions (~55%), which required significant additional research and rewriting, or they were “Tier 3” questions (~25%, which were decided against and scrapped completely.
Wherever you have humans attempting to read and compile a lot of data, the situation is ripe for a programmatic approach to contribute. Growth Engine provides just such a solution. Over 90% of the questions generated by Growth Engine were Tier 1 questions. That’s right: less than 10% of the curricula Growth Engine created for the US Air Force needed to be reworked in any way amounting to more than a quick glance and on-the-spot amendment. Of the 90%, a significant proportion needed no adjustment whatsoever and could be pushed directly to training deployment.
Question relevance and accuracy is extremely important in the creation of any training program, especially when the stakes are high. We do not need to stretch our imaginations far to understand that adequately trained US Air Force personnel is not just a “nice-to-have”. It is absolutely essential that US Air Force personnel understand their domain. Technically complex and interdependent source documents makes this a harder goal to achieve. Nevertheless, Growth Engine got there, and it got there with flying colors.
