Level 2 Test Reliability & Evaluation

Level 2 (Achievement) Evaluation requires the use of testing to determine the extent to which trainees changed attitudes, improved knowledge or increased skills. This evaluation requires that procedures for test creation, validation, administration and scoring be in place, to assure that tests administered are in conformity with accepted principles of test validation, are administered in a consistent manner, and produce reliable results.

Tests developed using the PTG automated approach to job task analysis and test objective determination, and for review and analysis of the test items themselves, are assumed to have met the requirements for Face and Content validity. The system also makes available the appropriate coefficients to calculate and determine criterion-related validity, if appropriate for the planned use of the test.

If there is no measure of accomplishment, there is no way to document the worth of the training event. However, for worth to be measured properly the test must be valid but it must also produce reliable test results. The lack of these characteristics diminishes the value of testing, per se, and can even have legal ramifications. The method selected to determine reliability is based on the structure (tests with a range of scores vs. binary tests to determine mastery / non-mastery), and use (recruitment, promotion, separation, course improvement, trainee progress development) of the test.

PTG has developed an online system for test development and reliability verification. The system features a test repository, and Worksheet Wizards to perform job analyses, build and validate test instruments and estimate the reliability of both pilot and actual test results. The system is password protected, with 128-bit encryption security, and allows the printing of tests and answer templates by authorized personnel. Any tests developed in the system can also be administered and scored online.