RESEARCH REPORT July 2001
RR-01-14
Statistics & Research Division Princeton, NJ 08541Preparing for the
Pre-Professional Skills Tests Lawrence J. Stricker
Gita Z. Wilder
Preparing for the Pre-Professional Skills Tests
Lawrence J. Stricker and Gita Z. Wilder Educational Testing Service, Princeton, New Jersey
July 2001
Abstract
This study investigated the extent and nature of preparation for the Pre-Professional Skills Tests (PPST?), the reasons for preparing or not preparing, and differences in these results for White and minority-group test takers and for middle-class and working-class test takers. Recent PPST test takers were surveyed. Preparation for the PPST was limited and mainly involved activities that were free or inexpensive, such as taking a sample test. The reported reasons for not preparing and the empirical correlates of measures of preparation were primarily attitudinal. Ethnic-group and social-class differences in the extent and nature of test preparation were minimal, but there were some differences in reported reasons and correlates of preparation, primarily less awareness of test preparation resources by White and middle-class test takers and few correlates of test preparation for Black test takers.
Key words: PPST, test preparation, attitudes, ethnicity, social class
Acknowledgements
Thanks are due to Gail S. Rys for arranging the pilot testing; Drew H. Gitomer for reviewing the questionnaire; Susan J. Miller for assisting in the pilot testing; Cheri L. Ashton and Bert Palencia for providing PPST data; Debra E. Friedman for coordinating the data collection; Gerry A. Kokolis, Ting Lu, Norma A. Norris, and Michelle Najarian for carrying out the computer analysis; and Daniel R. Eignor, Donald E. Powers, and Richard J. Tannenbaum for reviewing a draft of this report.
Introduction
In a recent focus-group study, a substantial number of students who had taken the Pre-Professional Skills Tests (PPST) said that they did little or no preparation for the test (Wilder & Stricker, 1998). Moreover, only one person acknowledged ever hearing of LearningPlus?, a computerized instructional program that covers the same skills evaluated by the test (reading, mathematics, and writing) and provides test preparation for the PPST and another Praxis I test, the computer-based Praxis I: Academic Skills Assessments. Similarly, in a 1996 survey of Praxis I test takers, 31% reported that they had not prepared for the test (Gordon S. Black Corporation, 1996). This massive lack of preparation is particularly surprising in view of the high-stakes nature of the PPST when it is used for admission to teacher education programs and for teacher certification, as well as in light of the high failure rates on this test, especially for members of minority groups (e.g., Garcia, 1985; Minnesota Board of Teaching, 1991; Smith, 1987; Snow, 1995). The students in the focus groups offered several explanations for this lack of preparation, notably (a) the belief that one simply cannot prepare for basic skills tests and (b) the cost in time and money for test preparation. Still, solid evidence is lacking. As a matter of fact, very little is known about why students prepare or do not prepare for any kind of test, including classroom tests (Van Etten, Freebern, & Pressley, 1997).
Clearly, it would be in everyone’s interest if test preparation were more universal. This would minimize an extraneous source of variation in test performance (e.g., Anastasi, 1981; Messick, 1981) and might increase the number of test takers passing the PPST and qualifying for teacher education and teacher certification. There appears to be a demand for test preparation courses provided by test publishers. In a 1997 survey of a national sample, 75% of the undergraduates reported that testing organizations should offer such courses (Rowan & Blewitt Incorporated, 1998). And there is dissatisfaction with ETS’s current efforts at test preparation. In surveys of Praxis I test takers in 1996 and in 1997–98, 19% in 1996 and 14% in 1997–98 reported that ETS materials did not provide good preparation on test taking strategies (Gordon S. Black Corporation, 1996, 1998). At the same time, though, there is also evidence that examinees do not use the test preparation resources that are readily available. In a 1997 study of six institutions that provide LearningPlus to their students, few students used it and those who did used it very little (Wilder & Cline, 1997).
The puzzling juxtaposition between the lack of preparation for the PPST and the high failure rate for this high-stakes test, as well as the social consequences of these failures, makes it evident that a better understanding is needed of the reasons many test takers do not prepare for the PPST. Accordingly, the purpose of this study was to survey the test preparation of recent PPST test takers to illuminate (a) the extent and nature of test preparation; (b) the reasons for preparing or not preparing; (c) interest in new kinds of test preparation material and services; and (d) differences in these results for subgroups of special interest: White and minority-group test takers, and middle-class and working-class test takers (as indexed by parental education).
Method
Sample
The sample was drawn from the 4,486 test takers taking the PPST at the June 12, 1999, administration: one-fifth of White test takers (N=708) and all of the remaining test takers, Black (N=591), Hispanic (N=96), Asian (N=132), other ethnicity (N=132), and unknown ethnicity
(N=231), a total of 1,890 test takers. A total of 951 (50.3%) test takers responded to the survey: 383 White, 285 Black, 45 Hispanic, 75 Asian, 68 other ethnicity, and 95 with unknown ethnicity.
Measures
Questionnaire
A questionnaire was devised to assess preparation for the PPST, reasons for preparing and possible correlates of preparation, and reactions to possible changes in test preparation material and services. (This questionnaire appears in the Appendix.) The questionnaire was pilot tested with a group of 10 teacher education students at the University of Delaware who had recently taken the PPST for teacher certification. The questionnaire took approximately 5 to 10 minutes to complete. The variables on the questionnaire follow: 1
1. Test preparation activities. This is a 13-item scale (Q. 2, subquestions 1 to 13), “Which of these things did you do to prepare for the recent PPST?” (e.g., “I took a sample PPST”), adapted from Powers (1988, 1998), with four alternatives (“Yes, I did this”; “No, too costly”;“No, too time-consuming”; “No, other reason”). The score was the number of “Yes, I did this”responses.
2. Test preparation time. This is a single item (Q. 3), “Altogether, about how many hours did you spend preparing for the recent PPST?” with seven alternatives (“None” to “More than 40 hours”).
3. Preparation relative to the SAT? or ACT. This is a single item (Q. 5), “Compared to the recent PPST, how much did you prepare for the SAT or ACT?” with seven alternatives (“Much more for the SAT/ACT” [1] to “Much less for the SAT/ACT” [7]).
4. Reasons for preparing more for the SAT or ACT. This is a single item (Q. 6), “If you prepared MORE for the SAT or ACT than the PPST: Why?” with nine alternatives (e.g., “I thought the PPST was not as important”).
5. Information received about the PPST—Timing. This is a five-item scale (Q. 8, subquestions 1 to 5), “When did you first find out these things about the PPST?” (e.g., “The PPST is required by your college or for hiring/licensing as a teacher”), with five alternatives (“1997 or earlier”; “1998”; “1999”; “Don’t recall”; “Never”). The score was the number of “Never” responses.
6. Own opinions about the PPST. This is a four-item scale (Q. 11, subquestions 1 to 4),“Before you took the PPST recently: What were your opinions about the test?” (e.g., “Difficulty of the test”), with five alternatives (e.g., “Very difficult” to “Not at all difficult,” plus “Don’t know”). The items assess attitudes about the PPST that may affect preparation for it. The score was the number of “Very…” and “Somewhat…” responses.
7. Own confidence about the PPST. This is a single item (Q. 11, subquestion 5), “Before you took the PPST recently: What were your opinions about the test?” (i.e., “Confidence that you would pass the test”), with five alternatives (“Very confident” [4] to “Not confident at all” [1], plus “Don’t know”).
8. Friends’ opinions about the PPST. This is a four-item scale (Q. 12, subquestions 1 to 4), “Before you took the PPST recently: What did you think were the general opinions of your college friends about the test?” paralleling Own Opinions about the PPST.
9. Friends’ confidence about the PPST. This is a single item (Q. 12, subquestion 5),“Before you took the PPST recently: What did you think were the general opinions of your college friends about the test?” (i.e., “Confidence that they would pass the test”), paralleling Own Confidence about the PPST.
10. Professors’ opinions about the PPST. This is a four-item scale (Q. 13, subquestions 1 to 4), “Before you took the PPST recently: What did you think were the general opinions of your professors about the test?” paralleling Own Opinions about the PPST.
11. Friends’ preparation. This is a single item (Q. 7), “How much did most of your college friends prepare for the recent PPST?” with five alternatives (“A lot” [4] to “Not at all”
[1], plus “Don’t know).”
12. Test anxiety. This is a single item (Q. 14), “How anxious do you usually get when you take important tests?” with six alternatives (“Extremely anxious” [5] to “Not anxious at all”
[1], plus “Don’t know”).
13. Academic self-esteem. This is a single item (Q. 15), “How do you think you compare to other college students in academic ability?” with eight alternatives (“Very much above average” [7] to “Very much below average” [1], plus “Don’t know”).
14. Academic locus of control. This is a single item (Q. 16), “How much of your success in school depends on your own effort and ability rather than luck or favoritism?” with five alternatives (“All of it” [4] to “None of it” [1], plus “Don’t know”).
15. Information received about the PPST—Source. This is a five-item scale (Q. 9, subquestions 1 to 5), “How did you first find out these things about the PPST?” paralleling Information Received about the PPST—Timing, with six alternatives (“ETS material”;“Professors or college staff”; “College friends”; “Another way”; “Don’t know”; and “Never did”). The four scores were the number of “ETS material” responses, number of “Professors or college staff” responses, number of “College friends” responses, and number of “Another way”responses.
16. Reasons for taking the PPST. This is a single item (Q. 1), “Why did you take the PPST recently?” with four alternatives (“To enter a teacher education program”; “To meet requirements of the teacher education program I am enrolled in”; “To be licensed or certified by
a state or city”; and “Other”).
17. SAT or ACT taken. This is a single item (Q. 4), “Did you take the SAT or ACT?”with four alternatives (“I took the SAT only”; “I took the ACT only”; “I took both the SAT and ACT”; and “I did not take the SAT or ACT”). The score was the number of tests taken.
18. Changes in test preparation material and services. This is a four-item scale (Q. 10, subquestions 1 to 4), “Based on your experience with the PPST, how much would these changes
help students taking the test?” (e.g., “More information about the PPST’s content and format”), with five alternatives (“Very helpful” to “Not helpful at all,” plus “Don’t know”).
Other Variables
Additional variables were derived from the registration form and Background Information Questionnaire completed by the test takers when they registered for the test, from the PPST itself, and from archival data from several sources for the test takers’ colleges (specifically, the schools where training relevant to the PPST was received). These variables consisted of demographic variables needed to categorize subsamples in the analyses and possible correlates of preparation for the PPST—characteristics of test takers themselves and their colleges. The variables follow:
1.Age (age in years at the test administration)
2.Sex
3.Ethnicity (White, Black, Hispanic, Asian, Other)2
4.Parental education (Both parents have high school education or less; one or both parents have college education.) (Note that for brevity these two categories are described in this report as high school educated and college educated parents.)
5.Years since school (number of years since college or graduate school; “Currently attending college or graduate school” to “More than 10 years”)
6.Enrolled in teacher education program (“Currently”; “Formerly”; “Never”)
7.Undergraduate GPA (“below 1.5” [1.25] to “3.5–4.0” [3.75])3
8.College: Control (Public, Independent, Catholic, Protestant and other religious; National Center for Education Statistics [NCES], 1998b)
9.College: Historically Black college (NCES, 1998a)
10.College: Carnegie Classification (Research Universities to Associate of Arts Colleges; Carnegie Foundation for the Advancement of Teaching, 1994)
11.College: Selectivity (entrance difficulty in 1997; Most Difficult to Noncompetitive; Peterson’s, 1997)
12.College: Number of PPST takers (current test administration)
13.Current PPST tests taken (number in current test administration)
14.Previous PPST tests taken (number in October 1997 to April 1999 test administrations)
15.PPST Reading score (This is a scaled score, ranging from 150 to 190.)
16.PPST Writing score (This is a scaled score, ranging from 150 to 190.)
17.PPST Mathematics score (This is a scaled score, ranging from 150 to 190.)
Procedure
A letter describing the purpose of the study, along with the questionnaire and a return envelope, was mailed to each test taker on June 10, 1999, to arrive immediately after the test administration on June 12. A follow-up letter, with another questionnaire and return envelope, was mailed to each test taker on July 1, 1999, three weeks after the initial letter. (Both letters appear in the Appendix.)
Analysis
The representativeness of the respondents was appraised by chi-square analyses of the categorical background variables, and t tests of the means for the continuous background variables and PPST scores for respondents and nonrespondents.
The overall extent of test preparation was evaluated by frequency distributions of these variables for the total sample.
Ethnic-group and social-class differences in Reasons for Preparing More for the SAT or ACT were assessed by frequency distributions computed separately for the ethnic groups and for test takers with high school educated and college educated parents. Multiple responses precluded significance tests of group differences for these variables. (Ethnic-group and parental education differences for the other test preparation variables were analyzed in the correlational analyses described below.) The data for White test takers were weighted (by a factor of five) to compensate for their undersampling.
The convergence between the four key test preparation variables (Number of Test Preparation Activities, Test Preparation Time, Preparation Relative to SAT or ACT, and Information Received about PPST—Timing: Number of “Never” Responses), their correlates, and differences in relationships across ethnic groups and social classes were evaluated from the product-moment intercorrelations of the questionnaire measures and other variables. The
intercorrelations were computed using a pair-wise missing-data program for the total sample, and separately for White and Black test takers and for test takers with high school educated and college educated parents. (Small sample sizes precluded separate analyses for test takers with Hispanic, Asian, or other ethnicity.) The data for White test takers were weighted in the analyses for the total sample and for the parental education subsamples. Dummy codes were used for Test Preparation Time, Sex, Ethnicity, Parental Education, Years Since School, Enrolled in Teacher Education Program, College: Control, College: Historically Black College, College: Carnegie Classification, College: Selectivity, and Reasons for Taking the PPST.4
The multiple correlations of the six sets of dummy variables were computed with each of the four key test preparation variables. The sets were Ethnicity, Years Since School, Enrolled in Teacher Education Program, College: Control, College: Carnegie Classification, and Reasons for Taking the PPST.5
The overall reactions to Changes in Test Preparation Material and Services were evaluated by frequency distributions for the total sample. Ethnic-group and social-class differences in this variable were appraised from frequency distributions computed separately for the ethnic groups and for test takers with high school educated and college educated parents, and accompanying chi-square analyses of the group differences in the distributions. The data for White test takers were weighted in the analyses for the total sample and for the parental education subsamples.
Both statistical and practical significance were considered in evaluating the results. For statistical significance, a .05 alpha level was used in all analyses. For practical significance, indexes that reflect a “small” effect size, accounting for 1% of the variance, was used: a W of .10 in the chi-square analyses, a d of .20 in the t test of means, and an r or R of .10 in the correlation analyses (Cohen, 1988). In analyses involving weighted data, the actual N, not the weighted N, was used in calculating chi-square and W, and in assessing the statistical significance of r and R. (In assessing the statistical significance of R when there was missing data, the smallest N for the constituent intercorrelations was used.)
Results
Comparisons of Respondents and Nonrespondents
The frequency distributions of the categorical background variables for the respondents and nonrespondents and the chi-square tests are summarized in Table 1. The means of the continuous background variables and the t tests are summarized in Table 2. Only two differences in these two sets of analyses were statistically and practically significant: Sex—more respondents (82%) than nonrespondents (74%) were women (χ2 = 18.46, p < .01, W = .10), and Current PPST Tests Taken—respondents took more PPST tests (M = 2.39) than nonrespondents (M = 2.10) in the current administration (t = 6.28, p < .01, d = .29).
Table 1
Frequency Distributions of Categorical Background Variables, for Respondents and Nonrespondents
______________________________________________________________________________
Percentage
Variable Respondents Nonrespondentsχ2W
______________________________________________________________________________ Sex(N = 948)(N = 935)18.46*.10a Male1826
Female8274
Ethnicity(N = 951)(N = 939).00.00 White4035
Black3033
Hispanic55
Asian86
Other77
Unknown1014
Parental education(N = 806)(N = 770) 1.01.03 High school or less3331
College or more6769
Years since school(N = 869)(N = 818).86.02 Currently attending7473
Under three years1616
Four or more years911
Enrolled in teacher education program(N = 866)(N = 813).48.02 Currently4445
Formerly1615
Never4040
College: Control(N = 741)(N = 703) 3.54.05 Public7073
Independent66
Catholic53
Protestant and other religious1917
College: Historically Black College(N = 751)(N = 721) 3.96.05 Yes1418
No8682
(Table continues)
Table 1 (continued)
_____________________________________________________________________________
Percentage
Variable Respondents Nonrespondentsχ2W
______________________________________________________________________________ College: Carnegie Classification(N = 743)(N = 717) 6.58.07 Research universities I and II1615
Doctoral universities I and II1413
Master’s colleges and 4248
universities I and II
Baccalaureate colleges I and II2520
Associate of arts colleges and44
specialized institutions
College: Selectivity(N = 719)(N = 695) 3.01.05 Most difficult or very difficult44
Moderately difficult7166
Minimally difficult or 2630
noncompetitive
______________________________________________________________________________ Note. Percentages may not total to 100% because of rounding error.
*p < .01. a W > .10.
Table 2
Means of Continuous Background Variables and PPST Scores, for Respondents and Nonrespondents
______________________________________________________________________________ Variable Respondents Nonrespondents t d
N Mean S.D.N Mean S.D.
______________________________________________________________________________ Age95028.8510.3493727.188.65 3.81*.18 GPA859 3.09 4.78809 3.02 4.67 2.98*.15 College—Number of 75117.7017.8872118.1018.77.41-.02
PPST test takers
Current PPST tests taken951 2.39.92939 2.10 1.14 6.28*.29a Previous PPST tests taken951 1.17 2.24939 1.20 2.27.28-.01 PPST Reading score732175.84 6.80620175.68 6.82.43.02 PPST Writing score749174.16 4.91647173.42 4.72 2.87*.15 PPST Mathematics score795173.759.23700172.118.88 3.49*.18
______________________________________________________________________________ * p < .01. a d > .20.
Frequency Distributions of Test Preparation Variables
Individual Test Preparation Activities
The frequency distributions for individual test preparation activities appear in Table 3, and the frequency distributions for the number of these activities appears in Table 4. The most common activities were “I read the free ETS booklet, Tests at a Glance—Praxis I: Academic Skills Assessment”(70%); “I took a sample PPST” (51%); “I used books on test preparation”(43%); “I studied books or material from college courses” (32%); and “I took an actual PPST at a previous administration” (21%). “Too costly” and “Too time-consuming” were equally common reasons for not doing these activities, but neither accounted for 20% or more of the
reasons for any activity (Table 3). The number of activities ranged from 0 (15%) to 10–13 (1%); the median was 2.3 activities (Table 4).
Test Preparation Time
The frequency distribution for the time spent preparing for the PPST appears in Table 5. Nineteen percent reported spending no time preparing, at one extreme, and five percent reported spending more than 40 hours, at the other extreme. The median was 4.9 hours.
Preparation Relative to the SAT or ACT
The frequency distribution for preparation time for the PPST relative to the SAT or ACT appears in Table 6. The percentage reporting that they prepared more for the SAT or ACT than for the PPST (43%) was greater than the percentage reporting that they prepared more for the PPST than for the SAT or ACT (30%).
Reasons for Preparing More for the SAT or ACT
The frequency distribution for the reasons for preparing more for the SAT or ACT than for the PPST appears in Table 7 for the total sample. The corresponding frequency distributions for the ethnic groups and for test takers with high school educated and college educated parents appear in Tables 8 and 9, respectively.
Total sample. The most common reasons given were “I thought my PPST scores would be good enough without preparation” (35%); “I had some other reason” (26%); “I plan to take PPST again if my scores are not good enough” (25%); “I did not know about resources for test preparation” (25%); and “I thought it was not possible to prepare for the PPST” (20%).
Ethnicity. The White and Black groups differed appreciably on four reasons. (Small sample sizes precluded comparisons for other ethnic groups.) “I thought it was not possible to prepare for the PPST” and “I did not know about resources for PPST preparation” were more frequent responses for White test takers (21% and 26%, respectively) than for Black test takers (13% for both reasons). “I plan to take PPST again if my scores are not good enough” and “I had some other reason” were more frequent responses for Black test takers (36% and 39%, respectively) than for White test takers (23% and 24%, respectively).
Parental education. The parental education groups differed on three reasons. “I did not know about resources for PPST preparation” was a more frequent reason for test takers with college educated parents than for test takers with high school educated parents (28% vs. 13%). “I plan to take PPST again if my scores are not good enough” and “I had some other reason” were more frequent responses for test takers with high school educated parents than for test takers with college educated parents (30% and 32% vs. 24% and 25%).
Information received about the PPST—Timing. The percentage of “Never” responses for individual kinds of information received about the PPST appears in Table 10 and the frequency distribution of the number of these responses appears in Table 11. The percentage of “Never”responses was consistently low, ranging from 2% for “The PPST is required by your college or for hiring/licensing as a teacher” to 12% for “Sample PPST questions on tests” (Table 10). Seventy-seven percent of the test takers had no “Never” responses, at one extreme, and 0% had five such responses; the median was 0 “Never” responses (Table 11).
Frequency Distributions for Individual Test Preparation Activities
______________________________________________________________________________
Percentage
Yes, No, No, No,
I did this too costly too time-other
Activity consuming reason N ______________________________________________________________________________ I read the free ETS booklet, 7037202,444 Tests at a Glance—Praxis I
Academic Skills Assessment
I took a sample PPST51611332,424 I took an actual PPST at a 2166662,304 previous administration
I studied books or materials 32214522,375 from college courses
I used flash cards, audio10619662,381 cassettes, or other study aids
I used books on test preparation43610402,447 I took a test preparation course51416662,416 I received private tutoring or 81512652,398 coaching
I studied with other students10211782,327 I used computer software on 9119722,402 test preparation
I watched video cassettes on1813782,351 test preparation
I watched television courses on1513812,330 test preparation
I used an Internet (on-line)8311782,341 computer information service
on test preparation
______________________________________________________________________________ Note. Percentages may not total 100% because of rounding error.
Frequency Distribution of Number of Test Preparation Activities
______________________________________
Number of Percentage Activities(N = 2,146) ______________________________________
013
120
220
318
412
57
63
72
81
91
10 to 131
______________________________________
Note. Percentage may not total 100% because of rounding error.