News Fronts
/ The Galveston
Light / The Vasthead
James Thomson
/ Houston Retro Radio
/ Articles
/ Work Samples
PERFORMANCE OBJECTIVESAfter their completion of the Time Power program, the participants will…
|
| Indicator: | Criteria: | Measurement Standards: |
| When participants like the training. | How
did the participants feel about the…
|
Measurement
standard: An
average approval rating of 80% among at least 80%
of participants. Success for the program requires that at least 80% of the students give a "good" (or higher) rating for the program as a whole. |
| When participants have learned enough of the material to know how to use it on the job. | Learning
objectives:
|
Test
scores of 80% or better for 80% of the students will
be necessary to prove a transfer of learning.
. |
| When participants apply the key elements of the program to their jobs. | How
many times have the participants applied each of the
above learning objectives on the job?
Have any factors on the job impeded the application of the Time Power program? |
At least 95% of participants have applied learning objectives no. 1. at least one time. At least 95% of the participants have applied learning objectives nos. 2 and 3 on at least 90% of the work days since completion of training. |
| When participants contribute more to profits as a result of the program. | What
expenses were incurred purchasing and delivering the
training?
What dollar value has the company received as a result of the training? |
Return on investment. |
LINKAGE BETWEEN PROGRAM GOALS AND STRATEGIC GOALS
| Program Goals: | Strategic Goals: |
| Have increased awareness of the value of more effectively managing time. | Salespeople initiate contacts with more customers, resulting in more sales. |
| Have in writing clearly defined productivity goals. | Employees devote time on the job to activities which contribute to company profits. |
| Have a system for writing goals with the company and personal life goals. | Employees achieve goals which support the strategic goals of the company. |
| Have a system for planning and control in carrying out the goals. | Employees avoid revenue losses that occur when goals are not achieved. |
|
Carry
out the goals. |
Employees achieve goals which increase company profits. |
LINKAGE BETWEEN TIME POWER SKILLS
AND PERFORMANCE APPLICATIONS
| Skill: | Performance Application: |
| The ability to establish individual goals with the company that harmonize with company goals and mission. | Employees devote time on the job to actions that contribute to individual and company objectives. |
| The ability to write a prioritized daily action list during a morning planning period. | Employees devote most time on the job to high priority actions. |
| The ability to identify time management problems and to write productivity goals. | Employees include three productivity goals on their daily action list. |
| The ability to record and retrieve information. | Employees avoid delays caused when information is not recorded or can not be located. Customers are favorably impressed by command of detail. |
| The ability to schedule follow-up with current and potential customers. | Employees follow-up customer contacts at an appropriate time. Customers react favorably to follow-up on commitments. |
| The ability to write an agenda for a forthcoming meeting. | Employees devote meetings to items of highest priority. |
| To ability to delegate effectively. | Employees assume increasing levels of responsibility. Employees acquire increased knowledge and range of experience. |
| The ability to avoid procrastination. | High priority tasks are performed as soon as possible. Salespeople eliminate "call reluctance" when contacting potential customers. |
EVALUATION ROLES AND TIMELINE
| Stakeholder Roles: | Actions: |
| Sales manager | Sales manager will provide data for comparing sales before and after training |
| Customers | The evaluation team will study customer feedback before and after training |
| Development team | The development team will verify validity of knowledge tested during evaluation. |
| Participants | Participants will provide data on their knowledge and use of the system. |
| Evaluation Team Actions: | Deadline & Days Allowed for Completion: |
| Upon conclusion of first pilot class, collect level 1 and 2 evaluation data. | March 25. Three days. |
| Analyze data. | March 28. Three days. |
| Upon conclusion of second pilot class, collect level 1 and 2 evaluation data. | April 1. Three days. |
| Analyze data. | April 5. Three days. |
| Distribute evaluation report to stakeholders. | April 12. One day. |
| Collect level 3 data. | May 1. Three days. |
| Analyze and interpret level 3 data. | May 15. Five days. |
| Collect level 4 data. | May 21. Five days. |
| Analyze and interpret level 4 data. | May 27. Five days |
| Write and distribute final evaluation report. | June 1. Three days. |
Level One Evaluation
The Level One Evaluation form assesses the reaction of participants in Time Power, a time management program.
The content of the program is very similar to TimeQuest, a program offered by Franklin Quest of Salt Lake City, Utah. A TimeQuest reaction form was one of the samples that we received in class. Because of the similarity between the programs, I used the TimeQuest form as a point of reference in creating a Level One evaluation for Time Power.
The form tests participant reaction to the knowledge and presentation skills of the seminar leader. It goes on to evaluate the overall organization and sequence of the material. It also evaluates the explanation of the two pivotal elements: the relationship between "unifying principles" (values) and goals and the use of the Day-Timer as a time management tool. Because the program is a holistic approach to time management, the form asks the participants about the value of the material for both their business and personal lives.
The form tests participant reaction to the overall pace of the program and whether they thought the pace was appropriate.
The questionnaire asks two open-ended questions to determine what the participants liked the most and the least.
Level Two Evaluation
The Level Two Evaluation form assesses the learning of Time Power participants.
The test presents eleven multiple choice questions which cover the key elements of the program. It is designed for implementation at the end of the seminar.
I had originally planned to include a set of matching questions. I later ruled that out because the items were not sufficiently homogeneous, and many could have been selected through the process of elimination.
The test includes questions about such abstract elements as the definition of time management and the relationship between "urgent" and "important" when setting priorities.
It also tests participants on their understanding of some of the specific behaviors that the program teaches. Examples include the ways for scheduling appointments and for retrieving data.
Level Three Evaluation
The Level Three Evaluation form assesses tests how well Time Power participants have put the program into practice. It covers the behavior goals that were set in the evaluation plan.
I have ruled out performance contracts and observation. Either of these methods would put a participant in the position of being monitored while caring out the program. This would tend to contaminate the results. For example, Time Power recommends that a person spend fifteen minutes "in solitude" to plan each day. Likewise, the involvement of another party would affect other elements of the program such as the "unifying principles," personal life goals, and goals with the company.
The survey would be sent to each person by mail. Persons who have not responded within two weeks would be given the same survey over the telephone.
Level Four Evaluation
The Level Two Evaluation form assesses the effect of the program on business results.
The main focus is on any increase in sales generated by salespeople who received the training. Their increased sales profits are compared to a control group. The comparison is based on available accounting records. If no control group is available, the pre-training records of the salespeople would be used as a baseline.
The evaluation also assesses the value of reduced meeting time. It is quantified by computing the annual salary cost of the people who attend the meetings. Savings can be realized by having fewer meetings, by having shorter meetings, and by eliminating participants who do not need to be present. The cost of meetings led by training group and control group employees would be compared.
The evaluation also looks for any reduction in customer complaints. If the training group compares favorably to the control group, this can be attributed to the follow-up system included in the program. The cost of a customer complaint would require a value judgment by a knowledgeable person within the company.
This program is presented by a training company, rather than an in-house trainer. Therefore, there is no design or development cost on the part of training department. The cost varies according to the number of participants and includes all course materials.
DATA COLLECTION STRATEGY
Level One
A reaction evaluation will be conducted immediately before the "Summary for Action" that concludes the program. All participants will be asked to complete a Level One questionnaire. A member of the evaluation team will distribute the form while the instructor is not present.
The form has been created by the evaluation team. It will be used after the course development organization has certified it as acceptable.
The desired response rate is 90 %, but a response closer to 100% is likely under the circumstances.
Success
for the program requires that at least 80% of the students
give a "good" (or higher) rating for the program as a whole.
Level Two
A test of learning will follow the "Summary for Action" at the end of the last session. The test assesses the participants’ knowledge of the key learning objectives in the evaluation plan. The test instrument will be used after it has been certified as acceptable by the course development organization.
The entire population of the training group will be asked to complete the test.
Each copy of the instrument will have its own number in small type at the bottom of the sheet. The participants will be assured that the data will be reported as a whole and not be connected to any individual.
Test
scores of 80% or better for 80% of the students will be necessary
to prove a transfer of learning.
Level Three
A behavior questionnaire will ascertain how well the learning from the program has been put into practice. It measures behavior according to the application objectives of the evaluation plan.
The behavior evaluation instrument has been developed by the evaluation team. It will be certified as acceptable by the company which presents the program.
Six months after training, all participants will receive a Level Three questionnaire in the mail. Each person will also receive a stamped, pre-addressed envelope for returning the survey to the evaluation team.
The postage stamps on the envelopes will code the surveys for follow-up. The stamp on each envelope will be different from each of the others.
This is not as much trouble as it may seem. The U.S. Postal Service offers a wide variety of commemorative stamps. Many of these depict a number of different personalities or events on one low-cost sheet. In addition, a stamp could be coded by tearing off one of its four corners. This would allow each type of stamp to be used four times.
The evaluator will record which participant receives each stamp and will log the individual participants as they return their surveys.
Three weeks after the mailing, a member of the evaluation team will contact anyone who has not responded and will conduct the survey over the telephone. All interviewers will be uniformly trained.
A
response rate of 90% will be considered adequate for the evaluation.
Level Four
An evaluation of business results will also be conducted six months after training. The business results of the training group will be compared to a control group of equal size. The control group (salespeople who did not receive training) will be selected through simple random sampling.
Using available business records, the performance of the control group and training group will be compared for the same six month period. The evaluation team will obtain commitment from upper management for access to the records.
The evaluation will compare increased profits due to sales, reduced meeting time, and reduction in customer complaints. The value of training will be determined by subtracting the contribution of the control group from the contribution of the training group.
A 0.05 level of significant will be necessary to attribute any difference to training.
The cost of the training will be extrapolated from accounting records. It will include the cost of administering the program, the cost of participant time and expenses, the amount paid to the training provider, classroom and equipment costs, and the cost of evaluation.
The contribution to business results will be determined by subtracting the cost of training from the value of training.
REPORT STRATEGY
The following stakeholders will receive the evaluation report: The CEO and other company leadership, the Sales Manager, the Training Manager, the participants, and training provider.
The customers are the only stakeholders who will not receive the report. Although customer complaints are studied in the Level Four evaluation, the customers are not directly involved in either the training program or the evaluation process.
The report will include a front cover, an executive summary, the background of the program, a description of the evaluation procedures, and the evaluation results. The report will offer conclusions and recommendations.
The main report will be followed by appendices. These will include the tests and data collection instruments, statistical summaries, and raw data.
The CEO, the company leadership, and the participants will receive only the executive summaries with a cover letter.
The Training Manager, the Sales Manager, and the training company will receive the full report. The training company copy will be sent to the Training Resource Manager at Day-timers, Inc.
The main report will present data in frequency histograms. The executive summary will include frequency distribution charts for key evaluation questions.
ANALYSIS STRATEGY
Level One
The Level One evaluation includes nine closed-response questions. Each will be analyzed on a frequency histogram.
The
open-end questions will be coded according to their meaning
and grouped into categories. Each response will be converted
into a frequency histogram and included in the evaluation
report.
Level Two
Each
participant’s raw test score will be converted into
a percentage of correct items.
The following items will be included in the evaluation report:
- Mean score.
- Median score.
- High score.
- Low score.
- Range (high score-low score).
- The standard deviation.
- IDIF and IDIS item analyses.
No index of learning gain is included because the evaluation does not include a pre-test for learning.
The
post-test will be checked for reliability using the KR21 formula.
This analysis will be included in the evaluation report.
Level Three
The Level Three analysis will include mean scores, standard deviations, frequency distributions, and frequency histograms for each test item.
Several of the survey items contain more than one question. In some instances, a filter question determines whether or not a follow-up question should be answered. This data will be reported as a Use of Training Skills table, a format which allows for filter questions.
Open-ended
questions will be coded and converted into raw data. In the
evaluation report, each question will be presented as a table
of employee responses with the comment code on the left and
the number of responses on the right.
Level Four
The business results will be analyzed by comparing a training group to a control group of equal size for a six month period.
The evaluation will compare increased profits due to sales, reduced meeting time, and reduction in customer complaints. The value of training will be determined by subtracting the contribution of the control group from the contribution of the training group.
A 0.05 level of significance will be necessary to attribute any difference to training.
The cost of the training will be extrapolated from accounting records. It will include the cost of administering the program, the cost of participant time and expenses, the amount paid to the training provider, classroom and equipment costs, and the cost of evaluation.
The contribution to business results will be determined by subtracting the cost of training from the value of training.
The evaluation report will include a table which compares the business results for the training and control groups. A separate table will present the costs of training.
The Vasthead
is the professional web site of
Grady McAllister of Houston, Texas.
