What is the Relationship between Students’ Computational Thinking Performance and School Achievement?

This study investigates the relationship between computational thinking performance and general school achievement and explores to see if computational thinking performance can be predicted by algebra and informatics achievement. The sample group of 775 grade 8 students was drawn from 28 secondary schools across Kazakhstan. The students responded to a Computational Thinking Performance test of 50 multiple-choice questions and Computational Thinking Scale questionnaire. The test covers the concepts: logical thinking, generalisation and abstraction. The validity and reliability of the multiple-choice questions are tested using the Item Response Theory. The Likert type questionnaire covers five factors: creativity, algorithmic thinking, cooperation, critical thinking and problem solving. School achievement results (secondary data) include scores for a number of school subjects. The results of the study showed that the multiple-choice questions are valid and a reliable tool to measure computational thinking performance of students. Algebra, general school achievement and students’ perception of their computational thinking skills were significant predictors of computational thinking performance. The results revealed no gender difference in computational thinking performance and perceptions of computational thinking. The findings regarding the relationship between computational thinking performance, the students’ general school achievement and perceptions of computational thinking skills are compared and discussed.


Introduction
The idea of integrating computational thinking into the school curriculum as one of the essential abilities of children was first introduced in the 1980s by Seymour Papert (1980), and has now become popular with the widespread development of STEM education and 21st century skill sets. There is much debate regarding the nature of the teaching of computational thinking; questions include definition, teaching and measuring, universal values and the transferability of skills of computational thinking to other areas of learning. As computer programming enhances computational thinking (Brennan & Resnick, 2012;Selby, 2014;Werner, Denner, & Campe, 2015) and since 8th grade students (aged 12 to 14 years) are familiar with basic programming skills such as variables, conditional statements, logical operators and loops in informatics lessons, then it might be expected that informatics scores could be a predictor of computational thinking performance. Likewise, as computational thinking is related to problem solving skills (Román-González, Pérez-González, & Jiménez-Fernández, 2016) and academic success (Ambrosio, Almeida, Franco, & Macedo, 2014;Durak & Saritepeci, 2017;Gouws, Bradshaw, & Wentworth, 2013), then students' general school achievement and especially algebra performance could be predictors.  developed the Computational Thinking Scale, a self-reported questionnaire as a measurement of computational thinking skills. It is used to measure the perception of computational thinking in this study.
The main aims of this study are to validate a computational thinking performance test, measure the computational thinking performance of the 8th grade students and thus to determine whether general school achievement, algebra achievement, informatics achievement and perception of computational thinking are predictors of computational thinking. The following questions are addressed by this research: RQ1: Is the computational thinking performance test valid and reliable? RQ2: Is algebra performance a predictor of students' computational thinking performance? RQ3: Is informatics performance a predictor of students' computational thinking performance? RQ4: Is perception of computational thinking skills a predictor of students' computational thinking performance?

Computational Thinking
The idea of computational thinking is not novel, but it is a current trend in education. Computational thinking is a cognitive process, which reflects the ability to think in abstractions, algorithmically and in terms of decomposition, generalisation and evaluation (Selby, 2014, p.38). Algorithmic thinking, logical reasoning, decomposition and abstraction are, if taken separately, individually established and have a long history. However, when combined to construct computational thinking, it is not yet agreed what to expect from mastering it. The multi-definition of computational thinking, the lack of evidence supporting the transfer of computational thinking skills to other fields and difficulty of measuring computational thinking level bring a complication in description and hardship in teaching (Guzdial, 2015;Denning, 2017). Big claims on the universal value of computational thinking are exaggerated and they should be softened (Denning, 2017), as there is no evidence of the transfer of knowledge from computer science to daily lives (Guzdial, 2015). The following concepts of computational thinking are introduced by several organisations: algorithmic thinking, logic, abstraction, decomposition, generalisation and evaluation (Barefoot, 2014;CS Unplugged, 2016;Google for Education, 2015). Computational thinking can be found in a semi-resident state, which can only be effectively used when it is taught and developed. Computational thinking can be described as a focused approach to real-life problems that is applied by transforming these problems into computable chunks and applying solutions in an efficient way (Barefoot, 2014;ISTE & CSTA, 2011). Although there is no single accepted definition to computational thinking, there are common concepts, such as, logic, abstraction, generalisation, decomposition and evaluation. Any person, not just computer scientists, can utilise either one of these concepts or the set of computational thinking skills. Since computational thinking is a thought process (Wing, 2011), it can be found in a semi-resident state in people, which can be activated and effectively used when it is taught and developed.
The idea of teaching computational thinking and ways of delivering it are still in the early stage of development. The educational reform in Kazakhstan, along with shifting from 11-year to 12-year school and three-language education has also integrated the computational thinking into the updated informatics curriculum. In the Kazakhstani updated informatics curriculum for 5 th -9 th graders, computational thinking is presented as a separate section among computer systems, information processes and health & safety with the following subsets: modelling, algorithms, and programming (National Academy of Education, 2016).
Among the main objectives of this updated curriculum two points are closely related to computational thinking (National Academy of Education, 2016): a) Teaching students to tackle a variety of tasks by means of analysis, abstraction, modelling, and programming.

a)
Enabling students to gain the abilities, such as, thinking logically and algorithmically, finding patterns, thinking in terms of generalisation, decomposition and evaluation.
By comparing items a and b above with the UK programme of study (Csizmadia et al., 2015), it is possible to see the similarities between the UK and Kazakhstani computational thinking concepts. Having analysed this updated curriculum (National Academy of Education, 2016), annual plans and the new informatics textbook (Shaniyev et al., 2017), it was concluded that 8 th grade students in Kazakhstan are familiar with concepts, such as, algorithm, generalisation, logics, and abstraction, as expected by Chuang et al. (2015). For this reason, the study narrowed its focus to the following concepts of computational thinking: logical thinking, abstraction and generalisation.

Evaluation studies of computational thinking
Without considering evaluation and assessment, computational thinking is unlikely to successfully advance in any curriculum. In addition, in order to judge the effectiveness of any curriculum integrating computational thinking, measures must be approved that would allow teachers to assess what children learn (Grover & Pea, 2013). Many studies on computational thinking have a small sample size between 7-30 participants (Ambrosio et al., 2014;Grover, 2011Grover, , 2015Moreno-Leon, Robles, & Román-González, 2016;Oluk & Korkmaz, 2016;Werner, Denner, Campe, & Kawamoto, 2012) that cannot be generalised or they are self-reported studies (Durak & Saritepeci, 2017;Korkmaz, Çakir, & Özden, 2017;Korucu, Gencturk, & Gundogdu, 2017), which do not measure performance. There are a lack of large-scale studies (Kallia, 2017) that could uncover the questions around how computational thinking skills are related to other disciplines and academic achievement. Studies on computational thinking differ on their findings regarding the gender factor. Therefore, gender variable is included this study. Since computational thinking is a multi-dimensional and complex phenomenon, it is hard to measure. Multiple-choice items are considered the most suitable format for assessment of higher order cognitive skills and abilities, such as problem solving, synthesis, and evaluation (Downing & Haladyna, 2006) and more efficient for a large sample size (Becker & Johnston, 1999;Dufresne, Leonard, & Gerace, 2002).

Research Design
This study has a quantitative research design. For the primary data collection, the multiple-choice questions, delivered through an online quiz, were designed to measure the computational thinking performance of the secondary school students. In addition to multiple-choice questions, the computational thinking scale questionnaire (Korkmaz et al., 2017; is used to measure perceptions of computational thinking skills in a standardised online form. As for the secondary data, the results of the General Achievement Test were taken. The secondary data relating to the individual respondent's general ability is drawn from the General Achievement Test results taken by all students of the Bilim Innovation Lyceums in Kazakhstan. The schools have integrated curricula of the natural-mathematical subjects and the test is taken termly. The validity and reliability of the multiple-choice questions are tested using the Item Response Theory (IRT). The item difficulty and discrimination coefficients are calculated. In addition, item characteristic curves for each question and test information functions for each quiz are generated.
The research seeks to identify, through the analysis of variables, if a relationship exists between the prior measures and the computational thinking levels of 8 th grade students. A regression analysis is used to predict computational thinking performance based on the predictor variables (Field, 2013). 50 multiple-choice items were carefully constructed and validated to measure computational thinking performance of the participants. The sample group respondents are 775 (549 boys, 226 girls) 8 th grade students aged 13-14 years from 28 selective Bilim Innovation schools located in different parts of Kazakhstan. The national curriculum, Informatics textbooks (Shaniyev et al. 2017), annual lesson plans of informatics for 7 th and 8 th grades of Bilim Innovation schools have been reviewed and the following topics related to computational thinking were discovered: logics, algorithm, abstraction and generalisation.

Instruments
The first instrument is a multiple-choice test for measuring the computational thinking performance of the participants. The multiple-choice questions have been carefully developed in line with the context relevant recommendations on writing good multiple-choice items provided by the authors Downing & Haladyna (2006), Frey et al. (2005), Gierl et al. (2017) and Reynolds et al. (2009). Then, these test questions were approved by two reviewers with experience in assessing computational thinking. The concepts of computational thinking included in this test are abstraction, generalisation and pattern, algorithmic thinking and logic. The test questions are designed to measure the computational thinking performance of 8th grade students taking into consideration the national curriculum topics and students' experience with problem solving.
Each item in this multiple-choice test has four response options, with one correct answer and three distractors. The test has 50-multiple-choice questions (A set of 5 quizzes with 10 questions each) with a maximum score of 50, and is conducted online. There was a time duration of 100 minutes. Sample questions from the computational thinking performance test are presented in appendices A to E.
The second instrument used in this study is the Computational Thinking Levels Scale originally developed by Korkmaz et al. (2017) in 2015 for university students, which later was adapted to secondary school level . The scale consists of 22 items with a 5-pont Likert type scale and has five factors each with the following number of items: "Creativity" -4 items, "Algorithmic thinking" -4 items, "Cooperation" -4 items, "Critical thinking" -4 items and "Problem solving" -6 items. Each one of the items in the factors has been scaled as never (1), rarely (2), sometimes (3), generally (4), always (5), in which the maximum total score is 110.
The General Achievement Test is used as secondary data in this study. It is a multiple-choice test taken four times a year (once per term) in Kazakhstan from the following subjects: algebra, geometry, physics, chemistry, biology, computer science, English language, Kazakh language, Kazakh literature, Russian language, world history, history of Kazakhstan and geography. There are 160 multiple-choice questions in total in this test with maximum score of 320. The scores for algebra and informatics have a maximum score of 20.

Variables
The following variables are used in this study: CTP, CTS, GAT, ALG, INF and G. CTP (Computational thinking performance) is a sum of scores measured by 50 items of multiple-choice questions covering the following concepts: logic, algorithmic thinking, generalisation, and abstraction. CTS (Computational thinking scale) is a sum of scores measured by 22 items questionnaire covering five areas of computational thinking: creativity, algorithmic thinking, cooperation, critical thinking, and problem solving. GAT (General Achievement Test) is a sum of average scores of 4 tests that cover the following subjects: Physics, Chemistry, Biology, English language, Kazakh language, Kazakh literature, Russian language, Algebra, Geometry, Computer Science, General history, History of Kazakhstan and Geography. ALG (Algebra) is a 10-item subscale of the General Achievement Test that measures algebra performance. ALG (Algebra) is a 10-item subscale of the General Achievement Test that measures algebra performance. INF (Informatics) is a 10-item subscale of the General Achievement Test that measures informatics performance. G (Gender) is a binary variable.

Data analysis
The Mean values and Standard Deviation of each variable are calculated for all participants as well as for each gender group. In the multiple regression analysis, CTP is a dependant variable and GAT, ALG, INF and CTS are independent variables. The Cronbach alpha for CTS and CTP are calculated. The coefficients of item difficulty and discrimination, the item characteristic curve plots and the test information plots using a 2-parameter IRT model are presented for each quiz separately.

Results
Descriptive statistics, reliability and multiple regression analysis are presented in this section of the article.

Descriptive statistics
775 8 th grade students from 28 schools in Kazakhstan participated in this research study. The descriptive statistics are presented in Table 1.

Computational Thinking Performance
The multiple-choice questions are tested for item difficulty and discrimination using a 2-parameter IRT model. All item characteristic curves for items fit well. The difficulty coefficients of items are between the range of -0.7 and 1.3. Three outliner items are item #1 and item #6 in the Abstraction quiz and item #7 in the Pattern figures quiz with the difficulty coefficients of 3.0, 1.8 and 2.0 respectively. The discrimination coefficients are close to 1 for each item in all quizzes. The test information for each quiz show that the average ability is tested the best. The item characteristic curves for each quiz can be found in appendix F-J. The Cronbach Alpha (Field, 2013) coefficient for all 50 items is 0.87. Therefore, it can be concluded that the test is a valid and reliable tool to measure computational thinking performance.

Computational Thinking Scale
The adapted version of the Computational Thinking Scale (CTS) instrument was tested for validity and reliability with 241 7 th and 8 th grade school students in Turkey .  have conducted exploratory factor analysis, confirmatory factor analysis, item distinctiveness analyses, internal consistency coefficients and constancy analyses and concluded that the CTS questionnaire is a valid and reliable measurement tool to measure computational thinking skills of students. In this study to see the reliability of this instrument with 775 participants, the Cronbach alpha coefficients for each subscale are listed in Table 2. The result of the reliability test of the CTS questionnaire Cronbach alpha for all 22 items is 0.838, which shows good reliability of the instrument. When each five factors are tested, it was found that the subscales' coefficient are greater than 0.7, except for the creativity subscale with Cronbach Alpha of 0.599. Students' perception on computational thinking differs between 2.6 and 4.2, and the mean is 3.7. The mean values of subscales are greater than 3.0, creativity 3.8, algorithmic thinking 3.5, cooperation 4.0, critical thinking 3.7. Except for the lowest one; the problem-solving subscale with 2.5. The authors of the instrument,  found the Cronbach alpha for the total items as 0.809, where 241 students from 7 th and 8 th grade participated. Therefore, the CTS questionnaire can be considered as a reliable tool.

Regression
A multiple linear regression was calculated to predict the computational thinking performance based on general school achievement, algebra achievement, informatics achievement and perception of computational thinking skills in Table 3. The result of the multiple linear regression indicated that the general school achievement, algebra achievement and perception of computational thinking skills are significant predictors for computational thinking performance; informatics achievement is not a significant predictor for computational thinking performance. The Table 3 contains the final step of a 4-step regression analysis. Step 4 (p<.05).

Discussion and Conclusion
In this study, multiple-choice questions mainly focused on logics, abstraction and generalisation were constructed to measure students' computational thinking performance. Additionally, as a perception of computational thinking skills, a CTS questionnaire with a five-point Likert type scale with 22 items was used. As a general school achievement, the results of the General Achievement Test were obtained. The study reveals that the computational thinking performance test is a valid and reliable tool to measure the computational thinking performance of students (RQ1). The results also show that students' general school achievement, algebra achievement (RQ2) and perception of computational thinking skills (RQ4) can be predictors of their computational thinking performance. One finding is that the students' problem-solving subscales in a CTS questionnaire are lower than other subscales, which is similar to what , the authors of the CTS instrument found. This might indicate that the participants were less confident in their problem-solving skills. However, self-reported measures do not actually show that they can affect their problem-solving abilities (Guzdial, 2015). The results show that the informatics score is not significant in being a predictor of students' computational thinking performance (RQ3). As we explore the informatics annual plan for 8 th graders, we see that there is not only programming but also other topics, such as measuring data, hardware network and spreadsheets. Although students are familiar with programming skills, perhaps they were simply not engaged in programming activities during the data collection in this study. The finding of this study that the general school achievement and algebra scores are significant predictors of computational thinking corresponds to the idea that computational thinking skills are closely related to problem-solving  and academic success (Ambrosio et al., 2014;Durak & Saritepeci, 2017;Gouws et al., 2013).
All the multiple-choice questions were carefully designed to measure higher-order thinking, specifically the level of computational thinking of 8th grade students, as multiple-choice items are an efficient method for large-scale studies (Becker & Johnston, 1999;Dufresne et al., 2002). However, applying computing ideas in daily problems by transferring the knowledge of computing into real life problems is hard to prove (Guzdial, 2015).
The limitation of multiple-choice questions might be in their diversity and that they may not fully assess and reflect complex performance (Gayef, Oner, & Telatar, 2014;Hancock, 1994;Martinez, 1999;Paxton, 2000;Simkin & Kuechler, 2005). Another limitation might be that the Kazakhstan context makes a difference. All participant students speak English, their science lessons at school are conducted in English and every single question of the computational thinking performance test was carefully written using simple English. Nevertheless, the language barrier might also play a role in measuring computational thinking performance as all the test questions were in English for Kazakhstani 8 th grade students. The sample groups are all from selective schools, meaning that the students' abilities and performance on average are higher than those of the general school population. The implications from this study might differ to other sample groups. As Weintrop et al. (2016) claim that there are other disciplines that explicitly link with computational thinking concepts, it is important that students understand these concepts and terms (Denning, 2009).
The future direction of this investigation will be, refinement and replication of the computational thinking test, obtaining more secondary data on general school achievement and determining whether promoting computational thinking skills increases students' general achievement.