Диссертация (1136614), страница 20
Текст из файла (страница 20)
In this paper, we show that PISA scores of individual students can be predicted from their digital traces. Weuse data from the nationwide Russian panel study that tracks4,400 participants of PISA and includes information abouttheir activity on a popular social networking site. We builda simple model that predicts PISA scores based on students’subscriptions to various public pages on the social network.The resulting model can successfully discriminate betweenlow- and high-performing students (AUC = 0.9).
We find thattop-performing students are interested in pages related to science and art, while pages preferred by low-performing students typically concern humor and horoscopes. The difference in academic performance between subscribers to suchpublic pages could be equivalent to several years of formalschooling, indicating the presence of a strong digital divide.The ability to predict academic outcomes of students fromtheir digital traces might unlock the potential of social mediadata for large-scale education research.IntroductionMeasuring educational outcomes of students is cruciallyimportant for education research and policy-making.
Suchmeasurements are usually performed using standardizedtests that are typically expensive and time-consuming intheir development and administration. The ability to infer academic outcomes of students from their digital tracescould unlock the potential of social media data for educationresearch, and provide a way to conduct large-scale studies asbecame recently possible elsewhere in social science (Lazeret al. 2009).It has already been shown that various demographic characteristics of the population such as ethnicity, gender, andincome level could be inferred from tweets (Preoţiuc-Pietroet al. 2015), profile images (An and Weber 2016), user posts(Rao et al. 2011) or photographs of neighborhoods (Gebru etal.
2017). It was also shown that a wide range of personalitytraits including intelligence could be predicted from users’behavior on a social networking site (Kosinski, Stillwell, andc 2018, Association for the Advancement of ArtificialCopyright Intelligence (www.aaai.org). All rights reserved.Graepel 2013). As academic achievements are known to beat least as highly heritable as intelligence (Krapohl et al.2014), one might expect that they should be predictable too.The gold-standard instrument for evaluating educationaloutcomes is the Programme for International Student Assessment (PISA) (Breakspear 2014).
PISA is a triennial international comparative study of student learning outcomesin reading, mathematics and science across 72 countriesand economies (OECD 2016). It is arguably the most influential educational study to the point of affecting policymaking in participating countries (Egelund 2008; Ertl 2006;Rautalin and Alasuutari 2009).
In this paper, we predictPISA scores of individual students from information abouttheir activity on a popular social networking site.We use data from the Russian panel study “Trajectories inEducation and Career” that tracks 4,400 students who participated in PISA in 2012. In addition to survey data, participants’ online activity information was collected in 2016.We assume that academic performance is a relatively stable characteristic and that the time interval of four years between two measurements should not prevent our ability tomake a prediction.The information about the online behavior of students wascollected from VK (a Russian analogue of Facebook). VK isubiquitous among young Russians: more than 90% of 18–24years old use it regularly (Public Opinion Foundation 2016).Users of this social network are subscribed to various publicpages that might be dedicated to anything from a local bar toa theater and from handcraft to quantum physics.
There aremore than 28 million public pages on VK. In our sample,the total number of different public pages is 73,000, and themedian number of users’ subscriptions is 54. We use a dimensionality reduction technique to extract ten componentsthat represent interests of each user. We then build a linearregression model to predict their PISA scores.ResultsPredicting PISA scoresPISA scores are scaled so that the OECD average in eachdomain (mathematics, reading, and science) is 500 and thestandard deviation is 100, while 40 score points roughly correspond to the equivalent of one year of formal schooling(OECD 2014a). Instead of a single estimate of students’Correlation coefficient0.5Table 1: Names of public pages that contribute most to gender component of users’ interests.
If the name of a publicpage is not self-descriptive, then the main topic of its content is described in parentheses. Translated from Russian.0.40.30.20.1readingsciencemath0.0 1 2 3 4 5 6 7 8 9 10Number of componentsFigure 1: Pearson correlation coefficient between predictedand real PISA scores as a function of the number of components used in the linear regression. The results are shownfor three PISA subjects (reading, mathematics, and science).The 10-fold cross-validation was used to control for overfitting.
The largest increase in model performance is providedby the third (academic) component of users’ interests. Thesecond component is correlated with users’ gender and provides a substantial increase in model performance for thereading score.abilities, PISA provides five plausible values drawn froman estimated distribution of their outcome level (see (OECD2014b) for details about PISA’s plausible values and Methods for their use in our analysis).We use singular value decomposition to extract ten maincomponents containing information about users’ subscriptions to public pages (see Methods for details). We then usea linear regression model to predict PISA scores and compute the Pearson correlation coefficient between predictedand real outcomes as a measure of model performance.
The10-fold cross-validation is used to control for potential overfitting of the model. The average correlation coefficient as afunction of a number of components used in the regressionis shown in Fig. 1.The largest increase in performance is provided by thethird component that is correlated with PISA scores for allthree subjects (r = 0.35 for reading, r = 0.33 for science, andr = 0.30 for mathematics). The third component, therefore,might be considered as an academic component of users’interests.The substantial increase in model performance for reading scores is also provided by the second component.
Thenames of public pages that contribute most to this component (i.e. the pages with highest absolute values of weights)suggest that it might reflect users’ gender (see Table 1). Indeed, the second component is strongly correlated with gender (r = 0.66). It is no surprise that gender component provides a substantial increase in model performance for reading scores, because there is a large gender gap in reading:girls outperform boys in this subject in every country andeconomy where PISA is administered (OECD 2014a).Negative contribution40 KG90-60-90 Sport girlsCharm SchoolBeauty SchoolModern GirlPositive contributionOrlyonok (funny pictures,obscene language)MDK (funny pictures,obscene language)IGM (computer games)Academy of Decent GuysAUTODigital divideWith almost universal Internet penetration in developedcountries, concerns about the digital divide have shiftedfrom the simple question of access to the broader notionof inequalities in the usage of the Internet (DiMaggio andHargittai 2001).
It was observed that higher-educated people more often use the Internet for educational purposes,while less-educated people more often use it for entertainment (Pearce and Rice 2017; Büchi, Just, and Latzer 2016;Van Deursen and Van Dijk 2014). We find a similar patternin subscriptions to VK public pages.We select public pages that contribute most to the identified academic component and compute average PISA scoresfor its participants (Table 2). Names of the pages with thepositive contribution to the component are related to scienceand art, while pages with highest negative weights concernhumor and horoscopes.These pages were not selected to provide the highestdifference in PISA scores, instead they arise exclusivelyfrom the structure of users’ interests.
Despite this fact, theobserved gap in performance of subscribers to these public pages could be dramatic. The subscribers to the WorldArts and Culture (WAC) page demonstrate results on parwith top-performing countries and outperform subscribersto Love Horoscope by 79–88 score points that are roughlyequivalent to two years of formal schooling. Note that theseare not marginal public pages on VK, with almost two million subscribers to the WAC page and more than four millionsubscribers to Love Horoscope.Predicting proficiency levelsTo help users interpret what student scores mean in substantive terms, the PISA scale is divided into six proficiencylevels (OECD 2014a). In Table 3, we report the ability ofour model to distinguish between students of different proficiency levels in reading.
The performance is measured asthe area under the ROC curve (AUC). The results for readingare slightly better than results for science and mathematics.According to the OECD, Level 2 is a baseline proficiencythat is required to participate fully in modern society (OECD2014a). Students who do not meet this baseline are considered as low-performing. High-performing students are thosewho achieve proficiency Level 5 or higher.
If this proficiencyTable 2: Names of public pages that contribute most to the academic component of users’ interests. If the name of a public pageis not self-descriptive, then the main topic of its content is described in parentheses. Names are translated from Russian. Meanvalues of subscribers’ scores with standard errors (in parentheses) are provided for each of three PISA subjects.Positive contributionWAC (World Arts and Culture)ScienceBest poems of great poetsScience and TechnologyFive Best MoviesNegative contributionF*CK (funny pictures often related to sex)Killing humorCool GagsUnorthodox HoroscopeLove Horoscopeis achieved in all three subjects simultaneously, then students are able to draw on and use information from multipleand indirect sources to solve complex problems, and theywill be at the forefront of a competitive, knowledge-basedglobal economy (OECD 2014a).The ability for our model to distinguish between low- andhigh-performing students is 0.90 for mathematics, 0.92 forscience, and 0.94 for reading.DiscussionWe show that there is a relatively strong signal with respect to academic performance in data on students’ subscriptions to various public pages on the VK social networking site.
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
