Cowboy ” and “ Cowgirl ” Programming : The Effects of Precollege Programming Experiences on Success in College Computer Science

This study examines the relationship between students' pre-college experience with computers and their later success in introductory computer science classes in college. Data were drawn from a nationally representative sample of 10,197 students enrolled in computer science at 118 colleges and universities in the United States. We found that students taking introductory college computer science classes who had programmed on their own prior to college had a more positive attitude toward computer science, lower odds of dropping out, and earned higher grades, compared with students who had learned to program in a pre-college class, but had never programmed on own, or those who had never learned programming before college. Moreover, nearly half of the effect on final grades was mediated by a positive attitude toward computing.


Introduction
"…it is often the case that students come into the institution with a self-taught 'cowboy' style of programming that in no way resembles the good programming practices which we are trying to convey.In teaching, we have to make sure that those students change their habits over time.Changing one's habits, however, is harder than learning new ones."-Kölling, 1999, p. 4 Preparing students for careers that require college degrees in science, technology, engineering, and mathematics (STEM) is a dominant concern of K-12 education in the United States (Handelsman & Smith, 2016).Computing lies at the epicenter of this concern (Smith, 2016), as labor force experts have issued dire warnings that the overall size of the American-trained future workforce in the computer science and IT fields is too small to maintain the nation's status as a leader in this area (National Research Council, 2012).
Students interested in majoring in computer science (CS) or other STEM fields want to do well in their introductory college CS course, because inferior performance may lead them to abandon career plans and change majors.However, learning to program is difficult-it requires thinking in a completely new way (Robins et al., 2003) known as computational thinking (Grover & Pea, 2013).It also requires that students be comfortable with learning by trial and error, (Anderson & Gegg-Harrison, 2013;Cooper, Dann, & Pausch, 2003), learning from open communities, and creating and participating in socially engaged practice ("the social turn"; Kafai & Burke, 2013).It has been found that it takes a novice programmer about 10 years to become an expert (Winslow 1996).Researchers have noticed a bimodal distribution in course grades (Lilja, 2001;Lister & Leaney, 2003;Carey, 2010), with failure rates commonly ranging between 30% to 60% (Bennedsen & Caspersen, 2007;Dehnadi & Bornat, 2006;Robins, 2010, Watson & Li, 2014).
Of all the factors tested in the CS education literature, previous programming experience stands out as the single factor most consistently predicting student success in these courses (Wiedenbeck, 2005 [n=120]; Byrne & Lyons, 2001 [n=110]; Wilson & Shrock, 2001 [n=105]; and Hagan & Markham, 2000 [n=75]).Furthermore, "prior self-initiated computer experience, [mostly gained outside of school through "hacking" and unguided exploration], was highly predictive of university-level introductory CS course performance" (Kersteen, Linn, Clancy, & Hardyck, 1988, p. 328).Other researchers in the 1980s had also demonstrated that students were able to acquire a considerable amount of computational thinking skills by learning in a purely self-discovery environment without any teaching intervention (Kurland & Pea, 1985;Papert, 1980).The underlying philosophy is consistent with learning theories such as constructivism (Piaget & Inhelder, 1969), active learning (Harmin & Toth, 2006), and learning by design (Goldman, Eguchi, & Sklar, 2004), and quickly became the cornerstone of educational interventions using LEGO-Mindstorm (Kabatova & Pekarova, 2010) and robot design (Mubin et al., 2013).This philosophy, however, stands in contradiction to Kölling's (1999) assertion of the drawbacks of "cowboy programming" (in the title of this article, we expanded the original term adding "cowgirl programming" to steer clear of what some might consider gender-biased language, We have combined both, for convenience in places, with "cowhand" programming )-if we define cowhand programming as self-initiated programming practice, unguided and outside of school.In any case, just how much a self-initiated/self-discovery cowhand programming experience improves, or hampers students' long-term programming efficacy and performance is yet to be examined.Researchers who took a middle ground advocated that self-discovery-based learning should be incorporated with a guided and supportive curriculum (Fay & Mayer, 1994;Lee & Thompson, 1997;Mayer, 2004).Nevertheless, the comparative efficacy of cowhand-oriented and instruction-oriented approaches, the interplay of the two approaches, and the optimal sequence of the two approaches, remain hotly debated (Chase & Klahr, 2017;Dean & Kuhn, 2007;Furtak et al., 2012;Klahr, 2010;Roll et al., 2017;Tobias & Duffy, 2009;Weaver et al., 2017).
A clearer understanding of the specific ways of attaining programming experience that helps students succeed in later CS coursework will benefit practitioners who operate or create programs, students who must choose in what fashion to expend their time and energy, and college CS instructors who seek to understand their students' learning styles and to create the best learning opportunities for their students.This is more important today than ever because of the rising investment in K-12 CS offerings, as well as the proliferation of free, interactive, responsive, webbased programming education platforms like CodeAcademy, code.org, and Khan Academy, where one can learn programming on one's own.
While CS education at the K-12 level has expanded significantly in the United States (e.g., CS 10k, AP Computer Science: Principles), across the European Union (e.g., European Coding Initiative), in the United Kingdom (e.g., Computing at School), Australia (e.g., Digital Technologies), and Mexico (Escherle, Ramirez-Ramirez, Basawapatna, Assaf, Repenning, Maiello, Endo & Nolazco-Florez, 2016), little is known about the degree to which such initiatives are successful beyond the immediate measurement of student enjoyment.A more rigorous approach to evaluating the effectiveness of precollege experiences is to measure their impact on performance in later introductory college coursework (Tai, Sadler, & Mintzes, 2006).In this way, many different types of in-school and out-of-school initiatives and experiences can be compared, and resources can be better allocated to those programs that are the most effective, rather than relying on anecdotal reports to make policy decisions.In the case of computer science, examining the degree to which precollege experiences predict performance in introductory college CS courses nationwide, provides a metric that is both fair and meaningful.
In this study, we ask if pre-college cowhand programming experience is associated with different college level CS attitude and performance, compared with a) students who had never had any programming experience, b) students who had learned programming only in a previous class, but never cowhand, and c) students who had both in-class and cowhand programming experience (in each possible sequence).

Data collection
The "Factors Influencing College Success in Information Technology" (FICSIT) study, which was conducted at Harvard University with funding from the National Science Foundation (grant number 1339200), supplied the data for this article.The FICSIT study sent out a 52-item survey to examine the pre-college computer experiences of students.The study's investigators obtained responses from a stratified random sample of 10,197 students enrolled in 118 2-and 4-year colleges and universities across the United States.The Integrated Postsecondary Education Data System of the National Center for Educational Statistics provided a listing of post-secondary institutions that was used to build a stratified random sample reflecting the proportion of students in the U.S. enrolled in 2-and 4-year colleges in three different size bins (small, medium, and large).We contacted 1080 different institutions (279 2-year and 801 4-year) that offered introductory courses in CS in the fall semester of 2015; 138 of them agreed to participate (30 2-year and 108 4-year), and 118 of them (23 2-year and 95 4-year) actually returned the survey.Thus, our final sample consisted of 118 institutions.Within the first two weeks of class, 10,197 students filled out a 52-item, paper-and-pencil, in-class survey about their prior CS experiences and current attitudes about computing, along with background questions about a wide variety of their educational experiences, as well as family background and demographic characteristics.Instructors provided students' final grade in the college introductory CS course or indicated that the student had dropped the course.We excluded respondents who had missing values in more than 50% of the questionnaire or in the key predictor variables or outcome variables.As a result, the final sample size in our analysis was 8,891.

Variables and modeling
This section explains the variables and analysis methods.In a nutshell, the predictor of interest was participants' programming experience, but we also controlled for a wide range of background variables.The predictor variables and controlled variables were collected retrospectively.The dependent or outcome variables were 1) the grade in computer science that they eventually received in the introductory CS course by the end of the semester (filled in by their instructors); 2) participants' attitude toward computer science at the time they answered the survey; and 3) course completion.Students in the sample were clustered in the models by their course instructor.To account for the significant autonomy instructors often have in grading, a random intercept, two-level hierarchical approach was used in all models.We first built hierarchical regression models to predict each of the dependent variables separately.We further constructed a path model to predict multiple dependent variables simultaneously.

Predictor of interest
While this study considered many student pre-college experiences with computing as possible predictors of performance, the final models examined programming experience in terms of "cowhand programming" and in-class computing experience.Our definition of cowhand programming was that a student had programmed by him/herself without the supervision of classroom teachers.It could happen prior or after taking computing classes, or not in combination with any in-class experience.The type of programming experience was determined based on participants' answers to questions about "taking a class about computer programming," "have programmed on my own," and the order of the two.There were five groups: 1) control (n=6824), students who had never taken any class about computer programming, nor had practiced programming outside of class; 2) cowboy-or-cowgirl-only (479), students who had programmed on their own, but had never taken any class about computer programming; 3) cowhand→class (n=250), students who had first programmed on their own prior to taking a class about computer programming; 4) class→cowhand (n=422), students who had started programming on their own during or after taking a class; 5) Class-only (n=916), students who had taken classes about computer programming, but had never programmed on their own.We did not know the particular programming languages that the participants learned because such information was not specified in the questionnaire.We discuss this limitation by the end of the article.

Control variables
Several variables were used in the model to account for significant differences in student backgrounds.Gender was coded as a dichotomous variable, zero for female and one for male.Race/ethnicity was divided into five categories coded as a set of dummy variables-Asian, Hispanic, black, and white, with white as the baseline.
Socioeconomic status (SES) entered the model through the proxy of parents' education.The parents' highest level of education was indicated by students on a scale from zero through five: zero representing "did not finish high school," one-"high school graduate," two-"some college," three-"4 years of college," and four-"graduate level education."Mathematics preparation has often been found to be associated with CS course performance (Byrne & Lyons, 2001;Werth, 1986;Wilson & Shrock, 2001).The SAT/ACT mathematics score was used as a proxy for the students' academic preparation and aptitude.If students reported ACT scores, but no SAT scores, their ACT scores were converted to the SAT scale according to a concordance published by the College Board (1999).In our sample, the correlation between SAT/ACT mathematics score and overall SAT/ACT score was 0.25.
We also controlled for whether the student was born in the United States, whether the first language was English, whether the student went to a public high school, whether the student had access to computer at home, and whether the job of any of the parents was related to CS.In all statistical models, we have controlled for the covariates listed above.

Dependent variables
This study estimated with three distinct dependent variables.The first was a hierarchical linear regression model, for which the dependent variable was the numerical grade in the course in introductory college CS at the end the semester.Institutions in the sample used different measures in awarding grades, with 11% using single letter grades (i.e., A, B, C, D, F), 40% using "+" and "-" to augment letter grades (e.g., A+, B-), and 49% using a 100-point scale.All letter grades reported were converted to the same 0 to 100-point scale (e.g., B+=88.5, B=85, B-=81.5).For uniformity, all grades below 60 were coded as 55, representing an "F." The second was another hierarchical linear regression model.The dependent variable was positive attitude toward CS, which was represented by the first component score based on a principal component analysis of 23 items.Overall, this is a measurement of the level of interest, efficacy, and comfort in regard to computing.The scale was developed by the authors, Inspiration for some attitude items was drawn from the Computing Research Association (CRA) Undergraduate Survey and the Scientific Attitude Inventory (Moore & Hill Foy, 1997).The scale had a very good internal consistency with a Cronbach alpha value of 0.91.We used the first component because it has an eigenvalue of 9.12 and successfully explained 38% of the variance, whereas the second component only has an eigenvalue of 1.60 explaining 6.4% of the variance.Table 1 presents the wording and loading of each item.The third model was a hierarchical logistic regression model, with a binary dependent variable indicating whether students either completed their college CS course or dropped out of the course-5.54% of sampled students.
Lastly, we specified a path model to test the relationship among cowhand programming experience, positive attitude towards CS, and final grade.Specifically, we examined whether the effect of prior experience on the final grade was mediated by positive attitude.

Descriptive statistics
The mean grade for those who completed the course was 84.34 (sd=13.83)on a 100-point scale.
The mean mathematics SAT/ACT score was 639.27 (sd=126.53),approximately 1.39 standard deviation (in standard deviation units of our sample) higher than the national average of 500 (College Board, 2014).Two-thirds of the sample had at least one parent with either 4 years of college or a post-graduate education.With just a 5.5% dropout rate, this U.S. study's findings are at odds with Bennedsen and Caspersen's (2007) and Watson and Li's (2014) global studies, which estimated a roughly 33% dropout rate for introductory programming courses.Other descriptive summaries, broken down by programming experience groups, are shown in Table 2.

Hierarchical models
Our first hierarchical regression model (M1) showed that all three groups that had cowhand programming experience did not statistically differ from each other in the final grade.All three of them outperformed the control group.They also outperformed the Class-Only group based on our post-hoc test (for Cowhand-Only versus Class-Only,  2 =17.70, p<0.001; for Cowhand→Class versus Class-Only,  2 =13.11, p<0.001; for Class→Cowhand,  2 =9.59, p=0.002).Figure 1 shows the predicted score by programming experience groups, controlling for other covariates at the mean.This result also translates to a significant main effect of cowhand programming (=2.13,se=0.47,p<0001) if we aggregate the five groups into two groups (Cowhand versus non-Cowhand).On average, students who had programmed on their own scored 2.13 points higher than students who had not, an effect size of 0.15.
Similarly, in M2, the three groups that had cowhand programming experience did not differ from each other on the scale of positive attitude towards CS, whereas they scored higher than the control and the Class-Only group.
In the hierarchical logistic regression predicting student dropout (M3), we found that the Cowhand-Only group was the only group that to yield significantly lower odds of dropping out than the control group.The parameters for all three models are shown in Table 3. Lastly, we used the path analysis to examine a possible mediation of the effect from cowhand programming experience to final grade via positive attitude.Because M1 and M2 have both shown that the three groups with cowhand programming experience did not differ from each other, and that Class Only and the control group did not differ from each other, we decided to simplify our model to merge the five groups into two groups (Cowhand versus Non-Cowhand) to model the cowhand programming effect in general.We ran a path analysis that simultaneously tested for the effect of cowhand programming on positive attitude, the effect of cowhand programming on final grade, and the effect of positive attitude on final grade, while controlling for all covariates and accounting for the two-level clustering.
As shown in Figure 2 (for presentation purpose this figure does not show the covariates and clustering; they were nevertheless accounted for in the model), when modeling the effects simultaneously, cowhand programming had significant positive effect on positive attitude (std.est=1.735,robust.se=0.113,p<0.001) and final grade (std.est=0.086,robust.se=0.035,p=0.015), and the positive attitude also had a positive effect on final grade (std.est=0.041,robust.se=0.005,p<0.001).There was a significant mediation effect from cowhand programming to final grade via positive attitude (std.est=0.070,robust.se=0.009,p<0.001), whereas the total effect from cowhand programming to final grade was 0.157.In other words, 44.58% of the total effect from cowhand programming to final grade was mediated by positive attitude.

Discussion and conclusion
In Paradoxes of Education in a Republic, philosopher Eva Brann (1979) wrote, in 1979, "Those who lack a good grounding must learn laboriously and lengthily in schools what the well-educated learn quickly by themselves."She mentioned computer programming explicitly as an example of a field in which this is the case.We have found little to refute this nearly half-century-old claim.
At the pre-college level, self-directed learning still may be the best way to learn to program.
Why might it be the case that cowhand programmers outperformed the other students on all fronts?It is reasonable to suspect that cowhand programmers have a greater perceived self-efficacy and comfort level with programming.Moreover, by programming alone, cowhand programmers have expressed greater interest in, and commitment to, the subject, which may help them push through the difficult portions of their college CS course when many of their peers do not.This interpretation is supported by the mediation effect of positive attitude, which measures the level of interest, efficacy, and comfort.Previous research has also shown such a mentality to be associated with success in CS (Ramalingam, LaBelle, & Wiedenbeck, 2004;Ventura, 2005;Wiedenbeck, 2005).In addition, because college requires a more independent working style than does high school, it is possible that cowhand programmers are more suited to college success in general.College work may require more independence from classroom instructors, but it is not independent of the outside world.This is especially true for CS, which enables, and heavily relies upon, the open source online community.The development of search engines, as well as the ease with which learners can access the help of experts through message boards and websites like Stack Exchange, means that one is hardly learning alone anymore when one is programming "on one's own."Students who have had the experience of programming independent of a classroom instructor may be more familiar and comfortable using a variety of tools to "geek out and mess around," an attribute that is crucial for new generation programmers and innovators (Ito et al., 2009;Liggett, 2014;McKenna & Bergie, 2016).
Compared with the cowhand-only group, our study did not find any distinctive advantage for cowhand programming in combination with classroom learning in whichever sequence.This might be due to the lack of information about the pedagogy adopted in the prior computing classes.Nevertheless, our finding strongly suggests that to learn in class without any independent handson experience is a rather ineffective strategy.
The major limitation of this study was that we could not narrow down the language, environment and pedagogy in which the cowhand and in-class groups were introduced to computer programming.Because of the varying complexity of underlying data structures and algorithms, the specific programming languages the students are exposed to could make a significant difference in their attitude toward CS subjects in general.For example, students who had experience with Alice, which uses stories and games to teach logic and primitive data structures, would have very different understanding and expectations of CS subjects from those who had exposure to a modern high-level programming language like C++ or Java.It is not uncommon for CS majors to drop out or change majors simply because they are frustrated by the very first programming language they encounter.As a matter of fact, many schools have moved from the strongly typed (stricter typing rules) languages, such as C++ or Java, to scripting languages like Python in their CS1 curriculum, to improve retention.Future studies should carefully observe the context in which student learn their first programming language by themselves, with family, online or in class.
Cowhand programmers' stronger college CS performance suggests that, whatever bad habits they may have developed by lacking a teacher to guide them, as Kölling (1999) suggests, those are more than compensated for by what they learn on their own.However, the claim that cowhand programmers have a tendency pick up negative habits may be far less tenable today than it may have been in 1999, at the time of his writing.Recent advances in graphic based languages, such as Scratch, as introductory languages, have greatly reduced the demand for syntactic memory.Accompanied with well-designed tutorials, challenges and forums online, these introductory languages have empowered beginners to self-explore and self-teach at a young age.Many scholars once expressed concerns, similar to Kölling's, that non-professional languages may introduce bad syntactic habits, such as missing semicolons or cluttered global variables (Meerbaun, Armoni & Ben-Ari, 2011;Powers, Ecott & Hirshfield, 2007;Techapalokul, 2017).An increasing number of studies, however, has shown that such an introductory language environment both promotes interest in programming (Bers, 2010;Brennan et al., 2011;Fessakis, Gouli & Mavroudi, 2013;Sáez-López, Román-González & Vázquez-Cano, 2016;Weintrop & Wilensky, 2017;Wilson & Moffat, 2012) and has long-term performance benefits in programming (Chen, Haduong, Brennan, Sonnert & Sadler, 2018).Based on our findings, and the recent development of beginner friendly programming environments, we recommend that 1) introduction to computer programming should not wait until taking a formal course; students should be encouraged to explore by themselves.2) Getting one's hands "dirty" is more important than keeping the code "clean" and well formatted.Only following instruction in a pre-college classroom without more independent hand-on experience may place students at a disadvantage compared with those who have hand-on experience, but are not class-trained.3) To program in cowhand style does not only lead to acquisition of knowledge and skills, but more importantly appears to cultivate a positive attitude towards computer science, which translates to better performance in programming in the long term.
The demand for professional programmers, the interest in majoring in CS in college, and the general push by parents and policy makers for K-12 offerings in computer science have all increased (Yadav, 2016;Strickland, 2014;Taylor & Miller, 2015).In the meantime, many secondary CS teachers are currently undertrained, with limited teacher preparation for teaching pre-college computer science in the United States (Yadav, 2016).Despite the scarcity of resources, students have pursued self-teaching through innovative channels such as Code Academy, Dev Bootcamp, and open resources (McKenna & Bergie, 2016).We believe such a self-initiated approach should be supported wherever and whenever possible by pointing students towards the best available learning resources and encouraging them to work on their own with the help of responsive tools, online community support, and well-designed instruction.

Figure 2 .
Figure 2. Path diagram of the effect from Cowhand programming to final grade partially mediated by positive attitude towards computer science.Other covariates and two-level clustering were accounted for in the estimation but not shown in the diagram.

Table 3 .
Hierarchical regression model predicting the final grade and the odds of dropout