The SCANS Skills and Competencies Checklist: An Assessment Tool for Youth Work Readiness Programs

Abstract: The SCANS Report for America, 2000 (1992) identified the skills youth need to compete in the workplace. The not-for-profit sector responded by implementing programs designed to give at-risk youth opportunities to learn the skills and competencies required to be work ready. Program evaluators, however, are challenged with the need to assess behavioral changes, which at best are difficult to document. In addition, at-risk youth often do not perform well with traditional paper and pencil assessments (for a variety of reasons). Improvements in SCANS attributes m ust be captured while the youth are engaged in learning and practicing the desired skills and behaviors. The SCANS Skills & Competencies Checklist , developed by evaluators of the RCRE Youth Farmstand Program, is a tool that can be customized easily for use with youth work readiness programs that include an experiential learning component.

Background: The SCANS Report

In 1990 the U.S. Secretary of Labor organized the Secretary’s Commission on Achieving Necessary Skills (SCANS). The commission engaged in a thorough study to document and understand the attributes youth should acquire before they enter the work force, and published the findings as Learning for Living: A Blueprint for High Performance (U.S. Department of Labor, 1992).

The “SCANS Report” clearly identifies the skills, competencies and personal qualities youth need to compete successfully in the workplace (Table 1). The commission also charged the U.S. education system with the responsibility of integrating those skills into students’ academic preparation.

Table 1

SCANS Report Competencies and Foundation Skills

Competencies. Effective workers can productively use:
ResourcesAllocating time, money, materials, space, and staff
Interpersonal SkillsWorking on teams Teaching others Serving customers Leading Negotiating Working well with people from culturally diverse backgrounds
InformationAcquiring and evaluating data Organizing and maintaining files Interpreting and communicating Using computers to process information
SystemsUnderstanding social organizational, and technological systems Monitoring and correcting performance Designing or improving systems
TechnologySelecting equipment and tools Applying technology to specific tasks Maintaining and troubleshooting technologies
Foundation Skills. Competence requires:
Basic SkillsReading Writing Arithmetic and mathematics Speaking Listening
Thinking SkillsThinking Creatively Making Decisions Solving problems Seeing things in the mind’s eye Knowing how to learn Reasoning
Personal QualitiesIndividual responsibility Self-esteem Sociability Self-management Integrity

The not-for-profit sector responded to the report as well, and many non-school organizations and agencies implemented programs designed to give youth opportunities to learn the skills and competencies required to be work ready – particularly within “at-risk” populations. Since publication of the SCANS Report, many work readiness programs have incorporated the Report’s findings into their program development and implementation activities (see for example Harkins, 2001; Kowalski, Weaver, Green, & Pfaller, 1993; Packer, 2001).

That was the easy part.

The Challenge

The greater challenge faced by work readiness programs is forging the link between assessment tools that adequately capture changes in youth’s skills and competencies and the recommendations found in the SCANS Report. Program administrators are called upon to more accurately document program effects, especially those purported to change behaviors or increase skills and competencies (Clements, 1999).

Changes in human behavior, an increase in a desirable (or decrease in an undesirable) skill, behavior, or quality are difficult at best to document, much less to attribute to any specific program (Bernard, 2000; Posavac & Carey, 1997). Evaluation professionals in the past relied upon some combination of end-of-program and/or follow-up knowledge tests, or individuals’ predictions of future behaviors. Longitudinal studies are often outside of organizational capabilities due to funding constraints (Bernard, 2000; Posavac & Carey, 1997).

Paper and pencil assessments, however, whether performed before, during or after program participation do not accurately measure skills or behavior – knowing is not equivalent to doing . Evaluators have tried to bridge this gap by administering post-post self-report questionnaires and conducting interviews in an effort to cobble together enough information to ascertain whether programs had the desired results (Bernard, 2000; Clements, 1999).

The challenge to document change in at-risk youth is further compounded. Research indicates that youth of low socioeconomic status are often diagnosed with learning disabilities, and many youth have below grade literacy levels (Blair & Scott, 2002; Olson & Jerald, 1998).

Children with these difficulties may also suffer from test anxiety, and may not perform well with traditional paper and pencil assessments (Goonan, 2003). This problem further complicates the notion that scores may not capture participant gains in knowledge and competencies.

Evaluators often face this scenario when they know youth exhibited gains as a result of program participation, but traditional assessment results do not reflect achievement. So, even if a paper and pencil instrument can (or theoretically could) adequately capture improvements in skills and/or behaviors, the difficulties many at-risk youth face would continue to confound assessment results.

The problem, then, is just how do youth work readiness programs measure gains in skills, competencies and personal qualities identified in the SCANS Report?

The Solution

The first part of the solution is to use the SCANS Report as a guide to develop a program specific data collection tool that emphasizes actions over knowledge. The second part is to capture changes in youth attributes while they are engaged in learning, practicing, and/or implementing the desired skills and behaviors.

Successful work readiness programs include opportunities for youth to acquire and practice specific skills identified in the SCANS Report through service learning, internships, simulations, entrepreneurship programs, etc. This facilitates data collection while youth are in the act of using the desired skills.

The following case study illustrates how a program-specific SCANS Skills and Competencies Check list can be an ideal tool to include as part of a comprehensive evaluation protocol.

Case Study

The Program

The Rutgers Cooperative Research & Extension (RCRE) Youth Farmstand Program is a statewide, interdisciplinary program designed to increase workforce readiness skills in at-risk youth, support local farmers, and build healthier, stronger communities. Specifically, y outh-operated farmstands provide at-risk and special needs youth (grades 9-12) with opportunities to develop and practice many of the work readiness skills defined in the SCANS Report. Participants make most decisions related to stand operations and receive a share of the profits at the end of the selling season.

The evaluation team needed to develop assessment tools that did not rely solely upon paper and pencil instruments, especially since program participants were at-risk youth and/or special needs youth.

Instrument Development

The team developed multiple methods and tools to triangulate data without increasing participant anxiety. The SCANS Skills & Competencies Checklist (the SCANS Checklist) was created, based upon the skills, competencies and personal qualities outlined in the SCANS Report. The team used the following process to develop the assessment:

  1. Identified tasks and skills youth could learn, develop and practice as program participants.
    • Interviewed program faculty and staff.
    • Observed the program in action.
  2. Matched identified program specific skills to relevant categories from the SCANS Report (see example Table 2).
  3. Created assessment tool incorporating SCANS Report terminology and program specific skills (revised version shown in Table 3).

Table 2

Example of Matching Program Skills to SCANS Skills and Competencies

Personal QualitiesRCRE Youth Farmstand Program Skill
ResponsibilityWorks hard at tasks (even if unpleasant)
Arrives to work on days scheduled
Dresses appropriately (uniform, badge)
Understands workplace expectation
Arrives to work on time
Positive attitude in completing tasks
Self-EsteemBelieves in own self-worth
Has knowledge of own skills and abilities
Is aware of his/her impact on others
SociabilityRelates well to others
Responds appropriately to situation
Takes interest in what others say and do
Self-ManagementAssumes responsibility for actions & decisions
Exhibits self-control
Responds to feedback non-defensively
A “self-starter”
Sets well-defined and realistic personal goals
Integrity & HonestyCan be trusted
Bases decisions on values and goals
Chooses an ethical course of action

The SCANS Checklist designed for the Youth Farmstead Program was piloted in the summer of 2004. Subsequent focus groups with program staff revealed:

  1. The instrument contained redundancies.
  2. Instructions were not clear.
  3. The subjective rating system lacked objective guidelines.
  4. Program personnel did not understand clearly how results would be utilized. Supervisors gave youth high ratings at the very beginning of the program because they did not want anyone chastised, or worse, fired, for substandard performance!

The revised checklist incorporated changes suggested by program personnel (Table 3). Staff were trained in assessment procedures during orientation and provided with a copy of the complete Youth Farmstand Program Evaluation Protocol. (Note, the Checklist uses only one page, front and back.)

Table 3

SCANS Skills and Competencies Checklist

Name ____________________________ Date___________
CategorySkillRateComments
ResponsibilityWorks hard at tasks (even if unpleasant)
Arrives to work on days scheduled
Dresses appropriately (uniform, badge, etc.)
Understands workplace expectations
Arrives to work on time
Positive attitude in completing tasks
Self-EsteemBelieves in own self-worth
Has knowledge of own skills and abilities
Is aware of his/her impact on others
SociabilityRelates well to others
Responds appropriately as situation requires
Takes an interest in what others say and do
Self-ManagementAssumes responsibility for own actions & decisions
Exhibits self-control
Responds to feedback non-defensively
A “self-starter”
Sets well-defined and realistic personal goals
Integrity & HonestyCan be trusted
Bases decisions on values and goals
Chooses ethical courses of action
Completes tasksUnloads & stocks produce
Set-up stand (or assist with)
Take stand down (or assist with)
Displays produce neatly and creatively
Follows directions
Cleans stand as appropriate
Completes tasks in a timely manner
Customer serviceAcknowledges and greets customers
Smiles at customer
Speaks clearly, audibly, and courteously
Uses eye contact
Listens to customers with concern
Uses “please,” “thank-you,” “excuse me”
Does not react negatively (or over-react)
Does not swear or curse
Asks supervisor for help when needed
Works with supervisor to resolve conflicts
TeamworkShows respect for others’ opinions
Helps others w/tasks (even if not “his/her job”)
Mentors other workers (teaches, advises)
Does not make rude comments (curse, racial, etc.)
Contributes equally to group efforts
Does not shun people different from him/her
Directs co-workers in a professional manner
Demonstrates leadership (versus “bossiness”)
Uses “please,” “thank-you,” “excuse me”
Information processingIdentifies produce correctly
Identifies over-ripe and spoiled produce
Identifies NJ grown products (Jersey Fresh)
Suggests appropriate cooking methods
Sprays greens when needed
Places heavy items in shopping bags first
Describes nutritional value of produce
Weights and + measuresDozen: 1/2, 1, and 1-½
Pound: ¼, ½, ¾, 1
Bushel: quarter, half, whole
Case: half, whole
Flat; half, whole
MoneyCoins: penny, nickel, dime, quarter
Cash: $1, $5, $10, $20
FMNP Vouchers: Senior and WIC
Change up to $20 bill
Government vouchers (no change, stamp)
Business systems (inventory, order, pricing, banking, etc.)Accomplishes tasks in proper order
Fills out bank deposit slips
Balances cash drawer
Prepares customer bill/receipt (if applicable)
Administers customer & WIC/Senior surveys
Price sheets
Employee manual
Prices produce correctly
Generates accurate produce order
Health and safetyCautious during stand set-up & take down
Sun and heat safe practices
Assists in stand security
Uses safe produce handling procedures
Weighs produce with accuracy to ¼ pound
Performs basic math functions accurately
Can use calculator without help
Sorts money into correct compartments
Rings up purchase accurately
Can use cash register without help
Additional Comments Supervisor___________________________________________________________

SCANS Checklist Protocol: Administration and Analyses

Adult supervisors and managers independently (and discreetly) rate each participant’s progress towards acquiring identified program skills. Supervisors may record additional comments if desired to expand upon their observations.

The checklist is completed three (3) times during the program and a total score is derived per assessment. A minimum of two (2) adults rate each youth at each site, and scores are averaged to minimize potential biases.

The following rating scale (Table 4) provides pseudo-objective parameters for scoring:

Table 4

SCANS Checklist Rating Scale

RatingCriteria
OutstandingDemonstrates skill (attribute) with no reminders. Helps others acquire skill.
SatisfactoryDemonstrates skill most of the time. Rarely needs reminders.
Shows ImprovementDemonstrates skill or attribute with some reminders.
Needs ImprovementNeeds more practice or constant reminders.
Not ApplicableSkill not observed or not required.

Paired t-test analyses determine whether youth demonstrated statistically significant overall gains between t 1 , t 2 , and t 3. Additional analyses can identify change across time between program sites, subgroups, individual youth, or even select skills, competencies and qualities.

Observational assessments are inherently fraught with threats to reliability and validity (see for example Bernard, 2000; Pullin, 1994), however, the SCANS Checklist can identify behavioral changes more accurately than can be captured using traditional paper and pencil methods in the target audience .

The Checklist is only one component of the RCRE Youth Farmstand Program evaluation protocol. Results are further analyzed in conjunction with other tools to determine whether youth demonstrate real gains in skills, competencies, and personal qualities.

Adapting the SCANS Checklist

Program evaluators must explore creative methods and tools to measure real change in skills and competencies, particularly with at-risk youth audiences. The SCANS Checklist is just such a tool. It can be customized easily for use with programs that include an experiential learning component by following the guidelines below:

  1. Clearly define program goals and objectives.
  2. Identify tasks and skills youth can learn, develop and practice as program participants.
    • Interview program staff.
    • Observe program in action.
  3. Match identified program specific skills to relevant categories from the SCANS Report (see Worksheet 1).
  4. Create assessment tool incorporating SCANS Report terminology and program specific skills (refer to Table 3).

Worksheet 1

Adapting the SCANS Checklist to other Youth Work Readiness Programs

SCANS Skill or CompetencyExample TasksProgram Skill
ResourcesAllocating time, money, materials, space, and staff
Interpersonal SkillsWorking on teams Teaching others Serving customers Leading Negotiating Working well with people from culturally diverse backgrounds
InformationAcquiring and evaluating data Organizing and maintaining files Interpreting and communicating Using computers to process information
SystemsUnderstanding social organizational, and technological systems Monitoring and correcting performance Designing or improving systems
TechnologySelecting equipment and tools Applying technology to specific tasks Maintaining and troubleshooting technologies
Basic SkillsReading Writing Arithmetic and mathematics Speaking Listening
Thinking SkillsThinking Creatively Making Decisions Solving problems Seeing things in the mind’s eye Knowing how to learn Reasoning
Personal QualitiesResponsibility Self-Esteem Sociability Self-Management Integrity & Honesty

References

Bernard, H.R. (2000). Social research methods: Qualitative and quantitative approaches. Thousand Oaks, CA: Sage Publications.

Blair, C., & Scott, K.G. (2002). Proportion of LD placements associated with low socioeconomic status: Evidence for a gradient? Journal of Special Education, 36 (1), 14-22.

Clements, J. (1999). Results: Behavior change. Journal of Extension [On-line], 37 (2).

Goonan, B. (2003) Overcoming test anxiety: Giving students the ability to show what they know. (ERIC Document Reproduction Service No. ED480053)

Harkins, M.A. (2001). Developmentally appropriate career guidance: Building concepts to last a lifetime. Early Childhood Education Journal, 28 (3), 169-174.

Kowalski, T.J., Weaver, R.A., Green, J.E., & Pfaller, J.E. (1993). Developing a world class work force: Business and industry, government, and schools respond to school reform. Contemporary Education, 64 (2), 94-98.

Olson, L., & Jerald, C.D. (1998). The achievement gap. Education Week, 17 (17), 10-13.

Packer, A.H. (2001). The career transcript system for lifelong learning. Community College Journal, 71 (5), 24-28.

Posavac, E.J., & Carey, R.G. (1997) Program evaluation: Methods and case studies. Upper Saddle River, NJ: Prentice Hall.

Pullin, D.C. (1994). Learning to work: The impact of curriculum and assessment standards on educational opportunity. Harvard Educational Review, 64 (1), 31-54.

U.S. Department of Labor (1992). Learning for living: A blueprint for high performance.