Article Type: Research Article

 

Article Citation: Tiffany Sutter, and Mingyuan Zhang. (2020). EXAMINING THE RELATIONSHIP BETWEEN STUDENTS’ CREATION OF SPECIFIC DIGITAL PRODUCTS AND THEIR TECHNOLOGY ASSESSMENT SCORES. International Journal of Research -GRANTHAALAYAH, 8(9), 153-175. https://doi.org/10.29121/granthaalayah.v8.i9.2020.1342

 

Received Date: 27 August 2020

 

Accepted Date: 30 September 2020

 

Keywords:

Technology

Digital Media

Technology & Engineering Literacy (Tel)

NAEP

Student Computer Use
ABSTRACT

This study presented a secondary analysis of the National Assessment of Educational Progress (NAEP) dataset. The paper examined the impact of eighth grade students’ specific digital product creation on their Technology & Engineering Literacy ICT scores. In order to gain a better understanding of the impact of computer use technology achievement of eighth-grade students, this study used a quantitative descriptive research design to analyze secondary data extracted from the 2018 NAEP data set. The findings include (1) using computers to create, edit, or organize digital media both for school work and activities beyond school increased overall ICT scores. (2) The average score of students who used computers to create presentations in school increased while those who did this activity after school time saw a decrease in scores. (3) Students who reported that they create spreadsheets mostly saw a decrease in their average ICT score no matter the frequency or purpose for the activity. These findings may indicate that there is an ideal frequency for digital product creation in school, but that these types of activities may not be indicative of real-world use which is how students are assessed.

1.      INTRODUCTION

 

With the advent of the term digital native by Prensky (2001), many are led to believe that students today are actively using technology in a meaningful way simply because of the era into which they were born. While generation is certainly a factor in a user’s comfort with technology, it is not the only factor that decides if someone should be considered a digital native nor should age automatically mean proficiency in technology (Helsper & Eynon, 2010). According to Kirschner and De Bruyckere (2017), understanding that digital natives are not necessarily sitting in our classrooms will help teachers “avoid the pitfall of assuming that their students possess talents and abilities that they do not actually have. The skills and competences attributed to this generation of students…need to be properly taught and acquired before they can be applied” (p. 137). It is through purposeful instruction and inclusion of technology in curricula that students can increase their abilities. 

 

In 2001, the Enhancing Education Through Technology Act required districts to not only demonstrate technological integration into curriculum but to prove that all students had met a specific level of mastery by the completion of eighth grade (Barron et al., 2003; No Child Left Behind Act of 2001, Title II, Part D, § 2402, 2002). This incentive-based program and its rewards and consequences directed some of the technology curricula of K12 schools for numerous years. However, once the federal aspects of No Child Left Behind were eliminated and the states were given authority regarding curriculum and governing standards, many states reverted back to a time when they did not assess students’ overall technological abilities since it was not required (Davis, 2015; Layton, 2015).

In order to gauge students’ abilities regarding technology, the National Assessment of Educational Progress (NAEP, 2014) designed and administered a national assessment to eighth-graders involving technology and engineering challenges. In 2014, only 43% of tested students performed at or above the proficient level (NAEP, 2014). While the number of proficient students grew significantly in 2018, to 46%, there is still a clear majority of students who are not meeting basic levels of abilities (NAEP, 2018). In order to discern characteristics that may positively impact scores, this research is concerned with whether a relationship exists between students’ achievement in the Technology and Engineering Literacy (TEL) assessment and how often and in what ways they use a computer for school work and activities unrelated to school. In particular, computer use will constitute creating and organizing digital media and creating presentations and spreadsheets.

Despite a lack of basic proficiency, our digital native students are utilizing technology both in and out of school more than previous generations. Teachers are integrating the creation of digital media, presentations, and spreadsheets into their classroom instruction which not only gives students hands-on experience but provides a variety of ways to utilize technology in a constructive way to meet standards. In fact, Helsper and Eynon (2010) found that experience with technology helps to create a competent digital user as does the breadth of that use. Therefore, it is these factors that should be looked at when evaluating users’ abilities with technology. This study seeks to bridge the gap between the scores and some of the factors which may positively contribute to overall proficiency. The data of computer use in and out of school and its relationship to TEL scores may be a helpful reference for teachers to understand what kind of computer-created products to include in their lessons. This information may also serve as a catalyst for encouraging students to create their own products beyond the classroom.

The purpose of this study is to explore the relationship between eighth-grade students’ 2018 Technology and Engineering Literacy (TEL) scores and specific computer-created products for school work and outside of school activities. In order to explore this relationship, this study focused on these research questions:

Does using computers to create media, presentations, and spreadsheets for school work have an impact on eighth-grade TEL scores?

Does using computers to create media, presentations, and spreadsheets outside of school for activities unrelated to their school work have an impact on eighth-grade TEL scores?

Our theoretical framework for this research adopts a scientific inquiry-based approach. The framework was described in great details in The Impact of Conversations on Fourth Grade Reading Performance - What NAEP Data Explorer Tells? (Bond & Zhang, 2017). In summary, the research methods combined the inquiry process with scientific knowledge, reasoning, and critical thinking. We started with an extensive exploration of the dataset, and that led to the designing of the research questions. The research questions further guided us to mine the data with great in-depth.

 

2.      LITERATURE REVIEW

 

Digital literacy is no longer a way to differentiate oneself from others who do not have that skill; it is imperative that all graduating students reach at least a basic, proficient level in their abilities with technology. In fact, the ability of students to be digitally literate is no longer optional but has become a necessary survival skill for all regardless of curricular requirements (Eshet-Alkalai, 2004; Eshet-Alkalai, 2012). Even though technology will constantly change, and current tools will shift and evolve, the concepts of digital citizenship and literacies will not only sustain but expand (National Research Council, 1999). Through direct instruction, independent discovery, and hands-on experiences, students need opportunities to learn about and with technologies (National Research Council, 1999, p. 55). Often, students can be introduced to digital tools in the classroom which may prove useful for them beyond the walls of the school as well.

 

2.1. TECHNOLOGY USE IN SCHOOL

 

Since the choice to use technology or not within a lesson is a personal one, it is important to understand that teachers’ beliefs and perceptions impact the overall design of their classrooms. The decisions of what types of activities occur in a classroom each day are largely dependent on teachers’ pedagogical choices and knowledge of their content. Additionally, the correlation between teachers’ beliefs about technology and how and when they choose to integrate it into their lessons has been established by several researchers (Albion & Ertmer, 2002; Badia et al., 2014; Chen, 2008; Ertmer, 2005; Taimalu & Luik, 2019). Taking it one step further, Ritzhaupt et al. (2012) noted that how teachers choose to integrate technology into the classroom directly impacted the frequency with which students used technology for learning.

More than a decade ago, findings were already showing an encouraging trend where teachers had students use more advanced, creation software rather than basic answer-input technologies (Niederhauser et al., 2007). For example, students can create digital presentations, spreadsheets, and concept maps and 76% of the activities Niederhauser et al. (2007) witnessed included these types of activities (pp. 495-496). Furthermore, in the case study by Ezquerra et al. (2014), the researchers found that by working with digital media in class, students not only reported an increase in their overall abilities, but also an increase in their potential to replicate the activity on their own. More recently, Schmid and Petko (2019) found that a clear majority of students used computers in their classes regularly. Combined, these data show that teachers are choosing to increase technological experiences for students in school, and they are utilizing higher-level computer use and creation rather than using the technology solely as a productivity tool or for instructional purposes.

 

2.2. TECHNOLOGY USE OUT OF SCHOOL

 

Today’s students not only use technology in school environments, but most have better access to technology outside of their formal learning as well. Wang et al. (2014) found that a vast majority of students have regular use of smartphones, laptops, and desktops at home and tend to use technology more at home than in school. Research has found, however, that students are more likely to be creators of content in school whereas outside of school they may be more of a consumer; the difference in technology use may be attributed to students’ personal entertainment purposes during their free time as well as requirements from their instructors to produce rather than share others’ work (Lu, Hao, & Jing, 2016). In school, students were much more likely to engage in the use and creation of spreadsheets and presentations than outside of school where other media engagement surpassed the in-school use of the same tools (Wang et al., 2014). According to DeBell and Chapman (2006), a majority of students use home computers to play games and be entertained and 47% use the home computer to complete school work with word processors being the most common tool. Therefore, while students have more home access to technology and computers than ever before, they do not appear to be using it productively most of the time.

 

2.3. STUDENTS’ TECHNOLOGY ABILITIES

 

With the prevalence of technology today, students regularly use digital and social media tools both within and outside of the classroom. However, Erstad (2015) noted that this “can easily be misleading and give the impression that all young people today are super-users and highly competent in their use of different media” (p. 85). In fact, often, the opposite is true as teachers report that students do not always know how to use new and emerging technologies without devoting school instructional time to navigating the tool alone (Wang et al., 2014). Furthermore, the types of assessment necessary to judge students’ abilities cannot be done with typical standardized testing scenarios (Hohlfeld et al., 2010). This means that most of the data about students’ abilities are tied to self-reported surveys and observations done by teachers and researchers.

A survey by Kaminski et al., (2003) found that students had relatively low software proficiencies despite having technology access and only about half of the respondents, or less, reported having some basic skills in hardware and information gathering. Additionally, they found that students preferred to increase their understanding through classroom instruction designed at improving their skills rather than seeking out to advance on their own (Kaminski et al., 2003). This was in contrast to the survey findings of McEuen (2001) where a majority of students stated that they troubleshoot problems and learn new technologies on their own. The McEuen (2001) survey also found that a majority of students self-reported average to expert capabilities in word processing, media creation, spreadsheeting, and using the internet.

Overall, students who use integrated technology in school will be better tech users than those who are not exposed to digital production and creation mandated by teachers. Moreover, students who have technologies available at home and choose to use them beyond consumerism will continue their learning in authentic ways. Just as classroom use of technology can impact home computer use, students who increase their abilities beyond the classroom may try to incorporate that knowledge into their school work. Whether students are interacting with digital tools formally or informally, there should be an overall impact on their general capabilities when it comes to technology.

 

3.      METHODOLOGY

 

The National Center for Educational Statistics (NCES) designed the National Assessment of Educational Progress (NAEP), also known as The Nation’s Report Card, in 1969 to measure what U.S. students knew and could do within various subject areas. NAEP is a congressionally mandated project administered by the NCES within the U.S. Department of Education and the Institute of Education Sciences (NCES, 2018). The present study will examine a subcomponent of the eighth-grade Technology & Engineering Literacy (TEL) scores of students at the national level. Of the three content classifications within the TEL score, the data used in this study is from the Information and Communication Technology (ICT) segment. This score ranges from 0-300 and was chosen because it “measures students' knowledge of the software and systems used for accessing, creating, and communicating information, and for facilitating creative expression” (The Nation’s Report Card, 2018a, para. 5). This definition was most closely related to the variables needed for the research questions and aligned with the categorization of the same variables in research completed by Park et al. (2017).

 

3.1. PARTICIPANTS AND SCHOOL SELECTION

 

Using a probability sample design, participants for the assessments are selected randomly to ensure accurate, proportional representation of the greater population. According to a NAEP sampling infographic, NAEP uses a primary sampling unit (PSU) to identify single counties or groups of contiguous counties which are a representative sample of the nation at large (NCES, 2019). The national sample tests, such as the TEL, include public and non-public schools. Then, schools are chosen within PSUs to ensure that there is an equal representation of location type, race or ethnicity, and student achievement. Finally, students are then selected randomly from the appropriate grade level and students are assigned to a test. NAEP provides accommodations for special education students, students with disabilities, and English-language learners (NCES, 2019). Given every four years, the most recent TEL assessment was administered between January and March of 2018 to approximately 15,400 students from 600 schools (The Nation’s Report Card, 2018c). 

 

3.2. NAEP SAMPLING AND DATA COLLECTION

 

The TEL assessment is given completely via computer and does not break down collected data by state. According to The Nation’s Report Card (2018a), the complete TEL assessment contains 15 scenario-based tasks and 77 standalone questions; however, students do not answer each question. Students test for a total of 60 minutes which is broken into two 30-minute sessions that vary based on the participant; the assessment is accompanied by a student questionnaire where participants have an opportunity to respond about their education and experiences both in and out of school (The Nation’s Report Card, 2018a). The present study uses the average ICT scaled scores for the 2018 TEL assessment and there were six variables selected for analysis. Results were restricted to national public schools which exclude scores from the Bureau of Indian Education schools and Department of Defense Education Activity schools but include charter schools.

3.3. DATA ANALYSIS

 

The NAEP website includes a Data Explorer which allows users to access state and national results from all the assessments given. By inputting selected criteria, researchers can access data tables and charts and run basic statistical analyses. The Data Explorer was used to choose the appropriate TEL sub-score for eighth-graders and the desired variables before running statistical tests.

The six variables chosen for this study were all student-reported. Three of the variables asked students to consider the instruction received within the classroom while the other three focused on their time outside of school. The specific variables are presented in Table 1.

 

Table 1: Student-Reported NAEP Variables Chosen for Analysis Against TEL Scores

 

The possible answers to all six of these student questions were a 5-point Likert-type scale indicating frequency from “never/almost never” to “almost every day.” After selecting the proper sub-score and these variables, statistical significance tests were run to determine differences between groups.

 

4.      RESULTS AND DISCUSSION

 

The NAEP Data Explorer provided summary means and standard deviations of TEL scores in relation to each category answer for the variables. The average ICT scaled score for all eighth-grade participants was 151 (SD=37) in 2018. Presented below are the mean scores for each variable related to the research questions as well as significance testing results. When significance was found, Cohen’s d was calculated by hand using an effect size calculator from the University of Colorado (https://www.uccs.edu/lbecker/).

 

4.1. RESEARCH QUESTION #1 - COMPUTER USE IN CLASS

 

During the student questionnaire, students were asked about their computer use in the classroom and the ways in which they interact with specific technologies during instruction and assessment. To address the first research question, three variables were chosen for how students work during classroom activities. 

 

Table 2: Students’ ICT Scores by Frequency of Specific Computer Use for School Work

 

The average ICT scaled score, as well as the standard deviations, by question and response are shown in Table 2. Each variable is then broken down into its significance in the sections that follow.

 

4.2. CREATE, EDIT, OR ORGANIZE DIGITAL MEDIA

 

The highest overall scores for students using computers to create, edit, or organize digital media for school work were from the students who used technology for this purpose once or twice a week or almost every day. The overall lowest score earned in the ICT portion was from the students who never or almost never used a computer for this purpose. The results of multiple t-tests for the first variable, taken from Data Explorer, are shown in Table 3. 

 

Table 3: Difference in ICT Scale Scores Between Variables for Use Computer to Create, Edit, or Organize Digital Media for School Work

 

Students reporting using a computer to create, edit, or organize digital media for school work a few times a year (M=153, SD=36), 1-2 times a month (M=152, SD=37), once or twice a week (M=156, SD=37), and almost every day (M=155, SD=38) had significantly higher average scale scores than those in the never or almost never group (p<0.01). The mean of the once or twice a week group was also significantly higher (M=156, SD=37) than the mean of the 1-2 times a month group (M=152, SD=37).

 

Table 4: Effect sizes of Significant Mean Score Differences when Using a Computer to Create, Edit, or Organize Digital Media for School Work

 

To report the effect sizes of the significant variable responses, Cohen’s d was calculated and is presented in Table 4. According to Cohen (1992), an effect size of 0.2 is considered small, a 0.5 is medium, and a 0.8 is large. For this variable, the effect sizes ranged from 0.11 to 0.24 indicating relatively small significance.

 

4.3. CREATE PRESENTATIONS

 

In looking at how often students create presentations on the computer for their school work, the highest average scores for the ICT portion was earned by students who did these activities 1-2 times a month or once or twice a week. The lowest overall scaled score was earned by those students who never or almost never use a computer in this way. The results of multiple t-tests for the second variable, taken from NAEP, are shown in Table 5. 

 

Table 5: Difference in ICT Scale Scores Between Variables for Use Computer to Create Presentations for School Work

 

Students reporting using a computer to create presentations for school work a few times a year (M=152, SD=36), 1-2 times a month (M=157, SD=36), once or twice a week (M=154, SD=37), and almost every day (M=146, SD=38) had significantly higher average scale scores than those in the never or almost never group (p<0.001). The mean of the 1-2 times a month group was also significantly higher (M=157, SD=36) than the mean score of 152 (SD=36) of the few times a year group (p<0.001). Significantly lower scaled scores were found between the group who responded almost every day (M=146, SD=38) and those in the few times a year (M=152, SD=36), 1-2 times a month (M=157, SD=36), and once or twice a week (M=154, SD=37) groups (p<0.001). Finally, the 3-point scaled score difference between the group who created presentations 1-2 times a month (M=157, SD=36) and those who did it once or twice a week (M=154, SD=37) was also a significant difference (p<0.05).

 

 

 

Table 6: Effect sizes of Significant Mean Score Differences when Using a Computer to Create Presentations for School Work

 

To report the effect sizes of the variable responses, Cohen’s d was again calculated and is presented in Table 6. The range for the effect size from this variable was from 0.14 to 0.70. Five of the effect size comparisons were on the small end while the largest significance was found between the students who use a computer to create presentations for school 1-2 times a month and those who never do it (0.702). A medium effect size was found between once or twice a week and never or almost never (0.613) and a few times a year and never or almost never (0.567).

 

4.4. CREATE SPREADSHEETS

 

Students who used a computer to create spreadsheets for school a few times a year yielded the highest scaled score on the ICT portion of the TEL while those who did it almost every day had the lowest scores. The results of multiple t-tests for the third variable, taken from the NAEP Data Explorer, are shown in Table 7. 

 

Table 7: Difference in ICT Scale Scores Between Variables for Use Computer to Create Spreadsheets for School Work

 

Students reporting using a computer to create spreadsheets for school work a few times a year (M=158, SD=36) had significantly higher average scale scores than those in the never or almost never group (p<0.05). The mean of the 1-2 times a month group was significantly lower (M=148, SD=36) than the mean score of the never or almost never (M=155, SD=37) and the few times a year (M=158, SD=36) groups (p<0.001). Significantly lower scaled scores were found between the group who responded once or twice a week (M=145, SD=36) and those in the never or almost never (M=155, SD=37), few times a year (M=158, SD=36), and 1-2 times a month (M=148, SD=36) groups (p<0.05). Finally, students who use computers in class for spreadsheet creation almost every day (M=139, SD=38) had significantly lower scaled scores than those in the never or almost never (M=155, SD=37), a few times a year (M=158, SD=36), 1-2 times a month (M=148, SD=36), and once or twice a week (M=145, SD=36) groups (p<0.01).

 

Table 8: Effect sizes of Significant Mean Score Differences when Using a Computer to Create Spreadsheets for School Work

 

To report the effect sizes of the variable responses, Cohen’s d was calculated and is presented in Table 8. The overall range on these calculations was from 0.08 to 0.51 with the majority of the effect sizes at the small level. A medium effect size was found between the responses of a few times a year and almost every day (0.513) noting that the lower creation frequency yielded the higher score.

 

4.5. RESEARCH QUESTION #2 - COMPUTER USE BEYOND SCHOOL

 

During the student questionnaire, students were asked about their computer use outside of the classroom and the ways in which they interact with specific technologies for activities unrelated to their school work. To address the second research question, three variables were chosen for how students use their personal computer time beyond school.

 

Table 9: Students’ ICT Scores by Frequency of Specific Computer Use for Activities Beyond School

 

The average ICT scaled score, as well as the standard deviations, by question and response are shown in Table 9. Each variable is then broken down into its significance in the sections that follow.

 

4.6. CREATE, EDIT, OR ORGANIZE DIGITAL MEDIA

 

When looking at product creation on the computer for non-school activities, the highest scores on the ICT portion came from students who created, edited, or organized digital media almost every day. The lowest scaled score came from the group of students who did this activity a few times a year. The results of multiple t-tests for the first variable, taken from the NAEP Data Explorer, are shown in Table 10.  

 

Table 10: Difference in Scale Scores Between Variables for Use Computer to Create, Edit, or Organize Digital Media for Activities Beyond School

 

Students reporting using a computer to create, edit, or organize digital media for activities beyond school almost every day (M=157, SD=38) had a significantly higher average scale score than those in the never or almost never (M=152, SD=36), a few times a year (M=150, SD=37), and 1-2 times a month (M=151, SD=38) groups (p<0.01).

 

Table 11: Effect sizes of Significant Mean Score Differences when Using a Computer to Create, Edit, or Organize Digital Media for Activities Beyond School

 

To report the effect sizes of the variable responses, Cohen’s d was calculated and is presented in Table 11. They range from 0.14 to 0.19, indicating a small significance.

 

4.7. CREATE PRESENTATIONS

 

When evaluating the variable which asked about students’ use of computers to make presentations unrelated to school, the lowest ICT scores came from those students who did this almost every day while the highest scores were students who never or almost never did this activity. The results of multiple t-tests for the second variable, taken from the NAEP Data Explorer, are shown in Table 12. 

 

Table 12: Difference in ICT Scale Scores Between Variables for Use Computer to Create Presentations for Activities Beyond School

 

Students reporting using a computer to create presentations for activities beyond school almost every day (M=136, SD=38) had a significantly lower average scale score than those in the never or almost never (M=160, SD=35), a few times a year (M=153, SD=38), and 1-2 times a month (M=144, SD=36) groups (p<0.001). The means of the once or twice a week group (M=141, SD=38) and the 1-2 times a month group (M=144, SD=36) were both significantly lower than the means of the few times a year (M=153, SD=38) and the never or almost never (M=160, SD=35) groups (p<0.001). Finally, the 153 mean scaled score (SD=38) for the students who created presentations outside of school work a few times a year was significantly lower than those who never or almost never (M=160, SD=35) did this activity (p<0.001).

 

Table 13: Effect sizes of Significant Mean Score Differences when Using a Computer to Create Presentations for Activities Beyond School

 

To report the effect sizes of the variable responses, Cohen’s d was calculated and is presented in Table 13. For this variable, mostly small effect sizes were found, ranging from 0.19 to 0.45; however, a medium effect was found for responses of never or almost never and once or twice a week (0.520) as well as never or almost never and almost every day (0.657) with the never or almost never group having the highest score over the other two groups of students.

 

4.8. CREATE SPREADSHEETS

 

The final variable looked at how often students use computers beyond school to create spreadsheets. The highest scaled score was earned by students who never or almost never did this activity while those who did it almost every day had the lowest scores on the ICT portion of the assessment. The results of multiple t-tests for the third variable, taken from the NAEP Data Explorer, are shown in Table 14. 

 

Table 14: Difference in ICT Scale Scores Between Variables for Use Computer to Create Spreadsheets for Activities Beyond School

 

Students reporting using a computer to create spreadsheets for activities beyond school almost every day (M=134, SD=39) had a significantly lower average scale score than those in the never or almost never (M=160, SD=34), a few times a year (M=149, SD=37), and 1-2 times a month (M=139, SD=37) groups (p<0.05). The means of the once or twice a week group (M=136, SD=36) and the 1-2 times a month group (M=139, SD=37) were both significantly lower than the means of the few times a year (M=149, SD=37) and the never or almost never (M=160, SD=34) groups (p<0.001). Finally, the 149 mean scaled score (SD=37) for the students who created spreadsheets outside of school work a few times a year was significantly lower than those who never or almost never (M=160, SD=34) did this activity (p<0.001).

 

Table 15: Effect sizes of Significant Mean Score Differences when Using a Computer to Create Spreadsheets for Activities Beyond School

 

To report the effect sizes of the variable responses, Cohen’s d was calculated and is presented in Table 15. The overall range of effect sizes was from 0.13 to 0.71. While many of the significance were on the small side, three medium effect sizes were found between never or almost never and 1-2 times a month (0.591), never or almost never and once or twice a week (0.685), and never or almost never and almost every day (0.711). The strength in these frequency changes is negative as scores decreased with more use rather than increased.

 

5.      DISCUSSION

 

This study explored students’ scores from the TEL assessment on the NAEP website as they related to specific uses of digital technologies both in and out of the classroom. O’Dwyer et al. (2005) noted the importance of using a subscale test score when researching standardized assessments in order to more closely align the data to specific instrument variables; therefore only the ICT scaled score was used as it included “...computers and software learning tools...and other technologies for accessing, creating, and communicating information and for facilitating creative expression” (Firman et al., 2015). The selected variables measured the frequency of students’ ICT use of digital media, presentations, and spreadsheets for school work and personal projects.

 

5.1. INTEGRATION OF TECHNOLOGY INTO THE CLASSROOM

 

One challenge to this research is that students are not in control of their use of technology in the classroom. Involvement with and use of digital technologies such as media, presentations, and spreadsheets requires teachers to integrate them into their pedagogy which is a deeply personal instructional choice affected by many factors (Ertmer, 2005; Taimalu & Luik, 2019). Teachers’ knowledge and implementation of specific technologies can hinge on many circumstances including their personal experience and comfort level, their teacher training program, and if any professional development has continued (Li et al., 2019).

This study found that students who responded that they used computers in school to create, edit, or organize digital media at any frequency greater than never achieved better scores on the TEL assessment. Additionally, a use for this purpose once or twice a week yielded the highest score and a significant (albeit small) gain over those who only did it 1-2 times a month. This is similar to the findings of Wang et al. (2018) who noted a positive association between in-class creation of digital media and higher ICT assessment scores.

Furthermore, this research found that students who responded that they used computers in school to create presentations at any frequency greater than never achieved better scores on the TEL assessment. The work by Lei & Zhao (2007) found that creating presentations was one of the most popular technology uses in school. The best frequency, according to the NAEP data, to create presentations for school work was 1-2 times a month as it had a medium to large effect size over other frequencies and students in this category demonstrated the highest ICT score on the TEL assessment. This means that teachers do not need to integrate this type of activity regularly to realize its benefit. This contradicts the work by Park et al. (2017) who found almost no relationship between students’ TEL scores and their ICT use for school-related work and Carter et al. (2017) who noted a negative association between school computer use and content assessment scores.

One paradox was the use of spreadsheets which generally found that the more often a student used this tool, the lower their ICT scores. This is despite the fact that spreadsheet knowledge may be considered an important vocational tool and a necessary component of ICT instruction and knowledge (Tort et al., 2008). The highest score, however, was earned by students who did this type of activity a few times a year, but there was only a small effect size over those who never created spreadsheets. Without knowing specific numbers of students in each category, it is hard to know just how many students created spreadsheets in the classroom at all meaning that a small number of participants could have dictated the scaled score. Regardless, these findings mimic those of Wang et al. (2018). This type of activity may not really be integrated into teaching and learning opportunities or may have very specific uses in math, science, or STEM courses. Moreover, students and teachers may not fully understand the capabilities and options offered with the use of a spreadsheet which may impact the frequency and depth of their use for classwork (Tort et al., 2008).

Overall, the increased use of computers at school does not necessarily translate into higher abilities with computers as the frequency of ICT use itself is not enough to develop the higher-level quality addressed on technology assessments (Hatlevik et al., 2018; Wastiau et al., 2013). Teachers cannot control the technology students utilize at home and must integrate important ICT components into the classroom in order to best impact achievement (Hatlevik et al., 2015). This study showed that occasional use of these digital tools and media may impact students’ technology assessment scores but there is other research that has not found the same correlations (Beland & Murphy, 2016; Biagi & Loi, 2013; Claro et al., 2012; Spiezia, 2011). Therefore, as with most in-school pedagogical choices, moderation in a breadth of areas may be the key to helping students slowly and steadily improve their technological abilities.

 

5.2. UTILIZING TECHNOLOGY BEYOND SCHOOL

 

When asking students to self-report their use of technology for projects unrelated to school, surveys rely on them to be forthcoming. Researchers have documented that perceptions do not always reflect reality, especially for sensitive topics, and may be biased or dishonest (Northrup, 1996; Wilson & Zietz, 2004). It is not likely that students taking the TEL assessment seek to be misleading with their answers to the questionnaire; however, they may not consider some of what they do independently as creation of digital media, presentations, or spreadsheets if it does not appear identical to in-class uses. It is with this in mind that some interesting correlations were found between students’ ICT scores and their use frequency of these types of products.

Students who responded that they used home computers to create, edit, or organize digital media once or twice a week or almost every day saw higher scaled scores than those who never did these activities beyond school. Significantly higher scores, albeit a small effect size, for those who did it almost every day are similar to findings by Wang et al. (2018) regarding ICT experiences out of school. This also correlates to research which found that students who use a home computer in addition to having regular access in school had greater confidence in their ICT competencies which can account for higher overall scores (Wastiau et al., 2013).

In looking at the scores for students who create presentations or spreadsheets beyond school, the data showed that those who never do these two activities scored significantly higher than those who completed these tasks at any frequency. In fact, the participants who did these two tasks almost every day yielded the smallest ICT score of all the frequencies. Wang et al. (2018) also found a negative association between ICT score and more frequent creation of presentations and spreadsheets out of school, and this is also in alignment with the findings of Zhang et al. (2016) when looking at a previous administration of the TEL assessment. These types of tasks are less challenging and may not adequately match scenarios offered by the TEL assessment for real-life problems. Further, the repetition of basic tasks such as creation of presentations and spreadsheets may distract students by taking time away from other TEL-related learning which could better impact scores (Zhang et al., 2016).

Wang et al. (2018) found that many of the out of school variables negatively impacted TEL scores. It is possible that the items on the TEL assessment which contribute to the students’ ICT score are not strongly aligned with the types of activities mentioned in the questionnaire especially considering the ways in which students engage with technology at home. Wastiau et al. (2013) found students to be more competent with social media if they had online access at home. However, Uzun & Kilis (2019) found a significant relationship between heavy media usage at home and lower levels of academic performance. This association may explain some of the lower ICT scores on the TEL assessment as students do not practice media, presentation, and spreadsheet product creation at home, yet they spent an immense amount of social time in the digital world.

According to a study conducted by the Pew Research Center, at least 92% of students are accessing the Internet daily with approximately 24% who responded that they are online almost constantly (Lenhart, 2015). Students are regularly accessing social media, video games, and text-based communication; in other words, they are using the Internet for entertainment and communication (Lenhart, 2015; Wang et al., 2011). This work confirms that it is not a meaningful occurrence for students to be creating, editing, or organizing digital media, presentations, or spreadsheets in their personal time. However, negative relationships between students’ use of home technology and their standardized achievement scores have been documented in areas of and beyond technology (O’Dwyer et al., 2005). Therefore, it is important to recognize how students’ behaviors outside of the classroom will impact their achievement on standardized assessments such as the TEL in order to combat it with proper classroom instruction involving technology.

 

6.      CONCLUSIONS AND RECOMMENDATIONS

 

Students today are generally not proficient in technology, as evidenced by the TEL scores overall and the low scores exhibited in the ICT subset; some students are “independent and critical users of ICT,” but there are many who are not (Fraillon et al., 2014, p. 24). O’Dwyer et al. (2005) found a positive relationship between technology use in school and at home and student achievement on assessments. However, Zhang et al. (2016) posited that students do not need daily access to these technologies and activities but can gain proficiency through other instructional methods. This is further confirmed by Carter et al. (2017) who noted higher content assessment scores from students without regular use of technology. The main consideration, therefore, is that technology must be an appropriate pedagogical choice to enhance content and not distract from it or be taught in isolation as it is the quality and not the quantity that impacts scores (Lei & Zhao, 2007). With this in mind, there are three principles to pull from the findings.

 

6.1. IDEAL FREQUENCIES FOR IN-CLASS PRODUCT CREATION

 

This study showed that it is best to have students create, edit, or organize digital media once or twice a week to help improve their ICT scores. Additionally, having them create presentations once or twice a month can increase standardized assessment achievement. It is these two classroom activities which increased scores on the ICT portion of the TEL a minimum of 5 points but up to an astounding 26 points. The implication for teachers is to find ways to meaningfully incorporate the creation of digital media and presentations into all curricular areas so students can use their higher-order processing abilities to see a better connection to real-life applications of these digital tools. After all, students who are highly engaged in technology-based lessons and passionate about the learning at hand are more likely to learn faster and retain it better (Bandekar et al., 2018).

 

6.2. OUT OF SCHOOL TECHNOLOGY IMPACTS SCORES

 

The results of this research indicated that what students do outside of school mostly has a negative impact on their standardized assessment ICT scores. Therefore, students need to be aware of how their media usage can impact their academic achievement and the risks of being too involved with technology (Sert et al., 2019; Uzun & Kilis, 2019). Since the only way to positively impact scores is to create, edit, or organize digital media, students should be encouraged to do this at home although research shows they do this type of behavior already (Lenhart, 2015). This can help students to see transference between the topics and tools learned in school and how those types of products can have authentic, real-world applications and uses.

 

6.3. INCORPORATION OF REAL-LIFE APPLICATIONS FOR TECHNOLOGY

 

There are marked differences in how technology is used in class, out of class, and what is asked on an assessment. Several of the technology assessments available both nationally and internationally are boiled down to multiple choice questions or may ask for constructed responses about highly abstract concepts, and often find that a vast majority of students are not proficient (Fraillon et al., 2014; Fraillon et al., 2013; Krupczak & Disney, 2013; The Nation’s Report Card, 2018b). These types of assessments may not match how instruction is usually delivered in class and how students utilize technology outside of school which can account for the lower proficiency scores overall and within the ICT subset. Therefore, when considering technology integration, it is important to go beyond simple uses of software programs and encourage scenario- and simulation-based tasks which better encourage problem-solving and allow for better transference of skills to an assessment.

Overall, the results of this study imply that teachers need to be utilizing technology in the classroom in order to see an increase in students’ technology scores. Since student scores can improve when they create, edit, or organize digital media within the classroom, then teachers need to be provided proper professional development to know what this looks like in their specific content area. Specifically, teachers need to know how to include more authentic uses of technology for their subject area rather than having students artificially working with technology. With this information, administrators can make sure their staff are knowledgeable about some uses of authentic learning and assessment as well as problem-based learning. Additionally, presentation creation is one way to see scores rise, so teachers should include this type of digital activity in their classes although it does not need to be done daily. In this way, teachers can better equip students with proper technical knowledge for the 21st century.

 

6.4. LIMITATIONS

 

This research had several limitations. The first, already mentioned, is that students provided responses on questionnaires after completing the TEL assessment and may provide skewed answers that do not match reality (Northrup, 1996). The ambiguity of the questions asked, and thus used to analyze results, may also contribute to this problem and may not best match how students accurately use technology both in and out of school. A second limitation is the proficiency bands created by NAEP for use with their assessments to determine student achievement; many of the scores presented here show a lack of student abilities with regard to ICT when the cut scores set by NAEP may simply be too high (U.S. Department of Education, Office of Planning, Evaluation and Policy Development, 2009). This could mean that evaluating scores provided by NAEP is artificial and does not provide adequate information for stakeholders to draw meaningful conclusions. Finally, the NAEP website allows for some limited statistical analyses with the data but does not allow researchers to download original datasets or know N values (NAEP, 2014). This limited scope of use restricts findings to simple correlations and does not account for any regressions or predictions to make impactful changes.

 

6.5. RECOMMENDATIONS FOR FUTURE STUDY

 

Since a student’s home use of computers cannot be regulated or monitored, it is necessary for administrators and educators to focus on what is being done in classrooms regarding technology use and instruction. Future research with ICT scores from the TEL assessment should look at some of the other variables related to in-class uses of technology such as the frequency of students designing computer programs or using digital tools to test hypotheses. These other variables may provide greater insight into what specifically helps to improve student scores. Additionally, some of the areas which found higher effect sizes may direct future research. For example, looking at the frequency of students creating presentations for the optimal success in addition to looking at what constitutes presentation-making with technology tools available. Ideally, researchers should strive to find classroom technology uses that significantly contribute to higher scores overall as the nation continues to look for ways to increase the technology abilities of our students.

 

SOURCES OF FUNDING

 

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

 

 

CONFLICT OF INTEREST

 

The author have declared that no competing interests exist.

 

ACKNOWLEDGMENT

 

None.

 

REFERENCES

 

     [1]        Albion, P. R., & Ertmer, P. A. (2002). Beyond the foundations: The role of vision and belief in teachers' preparation for integration of technology. TechTrends, 46(5), 34-38.

     [2]        Badia, A., Meneses, J., Sigalés, C., & Fàbregues, S. (2014). Factors affecting school teachers’ perceptions of the instructional benefits of digital technology. Procedia - Social and Behavioral Sciences, 141(C), 357-362.

     [3]        Bandekar, B., Yitbarek, E. M., Lopez, K., Tran, C., & Momoh, E. (2018). Social Media: Effect, Affect, Teens, and Possible Addiction. Collin College Undergraduate Interdisciplinary Student Research Conference. 2. https://digitalcommons.collin.edu/ccuisrc/2018/thursday/2

     [4]        Barron, A. E., Kemker, K., Harmes, C., & Kalaydjian, K. (2003). Large-scale research study on technology in K–12 schools. Journal of Research on Technology in Education, 35(4), 489-507. https://doi.org/10.1080/15391523.2003.10782398

     [5]        Beland, L. P., & Murphy, R. (2016). Ill communication: Technology, distraction & student performance. Labour Economics, 41, 61-76.

     [6]        Biagi, F., & Loi, M. (2013). Measuring ICT use and learning outcomes: Evidence from recent econometric studies. European Journal of Education, 48(1), 28-42.

     [7]        Bond, J., & Zhang, M. (2017). The Impact of Conversations on Fourth Grade Reading Performance - What NAEP Data Explorer Tells?. European Journal of Educational Research, 6(4), 407-417. doi:10.12973/eu-jer.6.4.407

     [8]        Carter, S. P., Greenberg, K., & Walker, M. S. (2017). The impact of computer usage on academic performance: Evidence from a randomized trial at the United States Military Academy. Economics of Education Review, 56, 118-132.

     [9]        Chen, C. (2008). Why do teachers not practice what they believe regarding technology integration? Journal of Educational Research, 102(1), 65-75.

   [10]     Claro, M., Preiss, D. D., San Martín, E., Jara, I., Hinostroza, J. E., Valenzuela, S., Cortes, F., & Nussbaum, M. (2012). Assessment of 21st century ICT skills in Chile: Test design and results from high school level students. Computers & Education, 59(3), 1042-1053.

   [11]     Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155-159.

   [12]     Davis, J. H. (2015, December 10). President Obama signs into law a rewrite of No Child Left Behind. The New York Times. Retrieved January 28, 2019, from https://www.nytimes.com/ 2015/12/11/us/politics/president-obama-signs-into-law-a-rewrite-of-no-child-left-behind.html.

   [13]     DeBell, M., & Chapman, C. (2006). Computer and Internet Use by Students in 2003. Statistical Analysis Report. NCES 2006-065. National Center for Education Statistics.

   [14]     Erstad, O. (2015). Educating the digital generation: Exploring media literacy for the 21st century. Nordic Journal of Digital Literacy, 10, 85-102.

   [15]     Ertmer, P. (2005). Teacher pedagogical beliefs: The final frontier in our quest for technology integration? Educational Technology Research and Development, 53(4), 25-39.

   [16]     Eshet-Alkalai, Y. (2004). Digital literacy: A conceptual framework for survival skills in the digital era. Journal of Educational Multimedia and Hypermedia, 13(1), 93-106.

   [17]     Eshet-Alkalai, Y. (2012). Thinking in the digital era: A revised model for digital literacy. Issues in Informing Science and Information Technology, 9, 267–276.

   [18]     Ezquerra, A., Manso, J., Burgos, M., & Hallabrin, C. (2014). Creation of audiovisual presentations as a tool to develop key competencies in secondary-school students: A case study in science class. International Journal of Education and Development Using Information and Communication Technology, 10(4), 155-170.

   [19]     Firman, H., Rustaman, N. Y., & Suwarma, I. R. (2015, November). The development of technology and engineering literacy through STEM-based education. In 2015 International Conference on Innovation in Engineering and Vocational Education. Atlantis Press.

   [20]     Fraillon, J., Ainley, J., Schulz, W., Friedman, T., & Gebhardt, E. (2014). Preparing for life in a digital age: The IEA international computer and information literacy study international report. Springer.

   [21]     Fraillon, J., Schulz, W., & Ainley, J. (2013). International computer and information literacy study: Assessment framework. Springer.

   [22]     Hatlevik, O. E., Guðmundsdóttir, G. B., & Loi, M. (2015). Digital diversity among upper secondary students: A multilevel analysis of the relationship between cultural capital, self-efficacy, strategic use of information and digital competence. Computers & Education, 81, 345-353.

   [23]     Hatlevik, O. E., Throndsen, I., Loi, M., & Guðmundsdóttir, G. B. (2018). Students’ ICT self-efficacy and computer and information literacy: Determinants and relationships. Computers & Education, 118, 107-119.

   [24]     Helsper, E., & Eynon, R. (2010). Digital natives: Where is the evidence? British Educational Research Journal, 36(3), 503-520.

   [25]     Hohlfeld, T., Ritzhaupt, A., & Barron, A. (2010). Development and validation of the student tool for technology literacy (ST2L). Journal of Research on Technology in Education, 42(4), 361-389.

   [26]     Kaminski, K., Seel, P., & Cullen, K. (2003). Technology literate students. Educause Quarterly, 3, 34-40.

   [27]     Kirschner, P., & De Bruyckere, P. (2017). The myths of the digital native and the multitasker. Teaching and Teacher Education, 67(C), 135-142.

   [28]     Krupczak, J., & Disney, K. A. (2013). Technological literacy: assessment and measurement of learning gains. In Makalah disajikan dalam at the 120th American Society for Engineering Education Annual Conference & Exposition.

   [29]     Layton, L. (2015, December 10). Obama signs new K-12 education law that ends No Child Left Behind. The Washington Post. Retrieved January 28, 2019, from https:// www.washingtonpost.com/local/education/obama-signs-new-k-12-education-law-that-ends-no-child-left-behind/2015/12/10/c9e58d7c-9f51-11e5-a3c5-c77f2cc5a43c_story.html?noredirect=on&utm_term=.9c2e0eb231c0.

   [30]     Lei, J., & Zhao, Y. (2007). Technology uses and student achievement: A longitudinal study. Computers & Education, 49(2), 284-296.

   [31]     Lenhart, A. (2015, April 9). Teens, social media & technology overview 2015. Pew Research Center. https://www.pewresearch.org/internet/2015/04/09/teens-social-media-technology-2015/

   [32]     Li, Y., Garza, V., Keicher, A., & Popov, V. (2019). Predicting high school teacher use of technology: Pedagogical beliefs, technological beliefs and attitudes, and teacher training. Technology, Knowledge and Learning, 24(3), 501-518.

   [33]     Lu, J., Hao, Q., & Jing, M. (2016). Consuming, sharing, and creating content: How young students use new social media in and outside school. Computers in Human Behavior, 64(C), 55-64.

   [34]     McEuen, S. (2001). How fluent with information technology (FIT) are our students? Educause Quarterly, 24(4), 8-17.

   [35]     The Nation’s Report Card. (2018a). About the TEL assessment: Assessment framework and design. Retrieved February 18, 2020, from https://www.nationsreportcard.gov/tel/about/ assessment-framework-design/

   [36]     The Nation’s Report Card. (2018b). About the TEL assessment: Sample scenario-based tasks and discrete questions. Retrieved March 20, 2020, from https://www.nationsreportcard.gov/tel/tasks/

   [37]     The Nation’s Report Card. (2018c). About the TEL assessment: Samples, inclusion, and participation. Retrieved February 18, 2020, from https://www.nationsreportcard.gov/tel/about/ samples-inclusion-participation/

   [38]     National Assessment of Educational Progress. (2014). 2014 Technology & Engineering Literacy (TEL). The Nation’s Report Card. https://www.nationsreportcard.gov/tel_2014/#

   [39]     National Assessment of Educational Progress. (2018). NAEP Technology & Engineering Literacy (TEL) Report Card. The Nation’s Report Card. https:// www.nationsreportcard.gov/tel/

   [40]     National Center for Education Statistics. (2019). Focus on NAEP: Sampling. The Nation’s Report Card. Retrieved February 18, 2020, from https:// www.nationsreportcard.gov/focus_on_naep/files/ sampling_infographic.pdf

   [41]     National Center for Educational Statistics. (2018, May 24). About NAEP. Retrieved February 18, 2020, from https://nces.ed.gov/nationsreportcard/about/

   [42]     National Research Council. (1999). Being fluent with information technology. Washington, DC: The National Academies Press. https://doi.org/10.17226/6482

   [43]     Niederhauser, D., Lindstrom, D., & Strobel, J. (2007). Evidence of the NETS*S in K-12 classrooms: Implications for teacher education. Journal of Technology and Teacher Education, 15(4), 483-512.

   [44]     No Child Left Behind Act of 2001, PL 107-110, 107th Cong., § 2401-2404, 115 Stat. 1425 (2002)

   [45]     Northrup, D. A. (Fall 1996). The problem of the self-report in survey research. Institute for Social Research, 11(3). Retrieved from http://www.math.yorku.ca/ISR/self.htm

   [46]     O’Dwyer, L. M., Russell, M., Bebell, D., & Tucker-Seeley, K. R. (2005). Examining the relationship between home and school computer use and students’ English/language arts test scores. Journal of Technology, Learning, and Assessment, 3(3).

   [47]     Park, B. J., Broer, M., & Bohrnstedt, G. W. (2017, April 28). The relationship between students' contextual factors related to technology and Technology and Engineering Literacy performance. Paper presented at the 2017 annual meeting of the American Educational Research Association. Retrieved March 2, 2020, from the AERA Online Paper Repository.

   [48]     Prensky, M. (2001). Digital natives, digital immigrants part 1. On the Horizon, 9(5), 1-6. https:// doi.org/10.1108/10748120110424816

   [49]     Ritzhaupt, A., Dawson, K., & Cavanaugh, C. (2012). An investigation of factors influencing student use of technology in K-12 classrooms Using Path Analysis. Journal of Educational Computing Research, 46(3), 229-254.

   [50]     Schmid, R., & Petko, D. (2019). Does the use of educational technology in personalized learning environments correlate with self-reported digital skills and beliefs of secondary-school students? Computers & Education, 136, 75-86.

   [51]     Sert, H., Taskin Yilmaz, F., Karakoc Kumsar, A., & Aygin, D. (2019). Effect of technology addiction on academic success and fatigue among Turkish university students. Fatigue: Biomedicine, Health & Behavior, 7(1), 41-51.

   [52]     Spiezia, V. (2011). Does computer use increase educational achievements? Student-level evidence from PISA. OECD Journal: Economic Studies, 2010(1), 1-22.

   [53]     Taimalu, M., & Luik, P. (2019). The impact of beliefs and knowledge on the integration of technology among teacher educators: A path analysis. Teaching and Teacher Education, 79, 101-110.

   [54]     Tort, F., Blondel, F. M., & Bruillard, É. (2008, July). Spreadsheet knowledge and skills of French secondary school students. In International Conference on Informatics in Secondary Schools-Evolution and Perspectives (pp. 305-316). Springer, Berlin, Heidelberg.

   [55]     U.S. Department of Education, Office of Planning, Evaluation and Policy Development. (2009). Evaluation of the National Assessment of Educational Progress, final report. https://www2.ed.gov/about/offices/list/opepd/ppss/reports.html

   [56]     Uzun, A. M., & Kilis, S. (2019). Does persistent involvement in media and technology lead to lower academic performance? Evaluating media and technology use in relation to multitasking, self-regulation and academic performance. Computers in Human Behavior, 90, 196-203.

   [57]     Wang, H., Zhou, X., Lu, C., Wu, J., Deng, X., & Hong, L. (2011). Problematic Internet use in high school students in Guangdong Province, China. PloS one, 6(5).

   [58]     Wang, K., Wang, X., & Li, J. (2018, April 16) Relationship between students' experiences and Technology and Engineering Literacy performance. Paper presented at the 2018 annual meeting of the American Educational Research Association. Retrieved March 2, 2020, from the AERA Online Paper Repository.

   [59]     Wang, S., Hsu, H., Campbell, T., Coster, D., & Longhurst, C. (2014). An investigation of middle school science teachers and students use of technology inside and outside of classrooms: Considering whether digital natives are more technology savvy than their teachers. Educational Technology Research and Development, 62(6), 637-662.

   [60]     Wastiau, P., Blamire, R., Kearney, C., Quittre, V., Van de Gaer, E., & Monseur, C. (2013). The Use of ICT in Education: a survey of schools in Europe. European Journal of Education, 48(1), 11-27.

   [61]     Wilson, M. L., & Zietz, J. (2004). Systematic bias in student self-reported data. Journal for Economic Educators, 4(4), 13-19.

   [62]     Zhang, T., Xie, Q., Park, B. J., Kim, Y. Y., Broer, M., & Bohrnstedt, G. (2016). Computer familiarity and its relationship to performance in three NAEP digital-based assessments. Washington, DC: American Institutes for Research.