University Rankings: For Who, for What, and Why?

By Alex Chapple

Photo by Vasily Koloda on Unsplash

In year 13, many students are faced with an important decision of their life: which university should they go to? You may have seen videos online called “college decision videos”, where high

school students in other countries record their live reaction of finding out if they got into the university of their dreams. Some jump up for joy and hug their families, and some crumble into tears as reality hits them. Perhaps not so much in New Zealand but in much of the world (for example Japan), society is built around an academic meritocracy where which university you study at determines much of your life after graduation.

There are huge industries built around university admissions and entrance exams around the world, like tuition services for specific universities and private college counsellors. Often parents will do whatever they can to get their kids into prestigious universities because for them it’s a status symbol. This was even adapted into a Netflix Original called “Operation Varsity Blues: College Admission Scandals”, where parents would bribe athletic coaches through a middleman to get a recommendation to the admissions office for their kids.

Universities are a place for research and higher education, but they can be just as much about prestige and status. Often this status is driven by University rankings. There are many different rankings around the world, but the three global rankings that are most prominent are: The Quacquarelli Symonds (QS) World University Rankings, the Times Higher Education World University Ranking, and the Academic Ranking of World Universities (also known as the Shanghai Ranking).

In New Zealand, all eight universities are ranked in the top 500 globally [1]. At the time of writing, the University of Auckland is ranked at 81st on the QS World Ranking, 147th on the Times Higher Education World University Ranking , and somewhere between 201st-300th on the Academic Ranking of World Universities. From this, we can already see that there is great variation between the rankings. There are over 20,000 universities globally [2], so a university ranked in the top 500 is already in the top 3% of universities worldwide (You probably have seen the buses around Auckland with this sort of statistic about AUT). Despite many universities being in the top 3%, we often look at only the universities ranked in the top 50 or 100 as the “worthy” universities. This is concerning because many other universities not ranked in the top 100 worldwide have excellent courses, world-leading research being done, and a great learning environment. These rankings shine the light only on the “top” universities, driving resources and attention to them and them only, leaving other universities in the dust. This isn’t beneficial for society as a whole at all, because the few get it all. Additionally, we’ve already seen that there can be large variations between the rankings. What does it really mean to be a “top” university? Which rankings, if any, can we trust?

Each of the three main rankings has its ranking methodology public, so let’s look at how they rank universities and how they differ from one another.

Firstly the QS World University Rankings [3]. The ranking is broken down into 6 categories with each weighted as follows:

• Academic Reputation - 40%

• Citations per Faculty - 20%

• Student to Faculty ratio - 20%

• Employer Reputation - 10%

• International Faculty ratio - 5%

• International Student ratio - 5%

Academic reputation is calculated from a survey sent out to over 100,000 academics around the world. It’s now the largest survey conducted of its kind. Citations per faculty are calculated based on the number of citations the publications from the university has gained in the past 5 years, divided by the number of academics at the institution. It’s a measure of the research output of the institution, suggesting that the number of citations that the research gets is the indicator of how important and valuable the research is. The student to faculty ratio is the ratio between the number of students and the number of staff at the university. The QS World Ranking suggests that this is a measure of the quality of teaching done at the university. The international faculty/student ratio is a measure of how good the university is at attracting talent from overseas. Much of the QS World University Ranking is based on institution reputation, student to staff ratio, and citations per faculty. It can be a good indicator of how the university is perceived by others, but it is not very indicative of the experience students can expect at the university.

Next, the Times Higher Education World University Rankings [4]. Their methodology is broken into 5 main categories, which are further broken down into smaller categories. The main 5 categories are as follows:

• Teaching - 30%

  • Reputation survey - 15%

  • Staff to student ratio - 4.5%

  • Doctorate to Bachelor’s ratio - 6%

  • Institutional income - 2.25%

• Research - 30%

  • Reputation survey - 18%

  • Research income - 6%

  • Research productivity - 6%

• Citations - 30%

• International Outlook - 7.5%

• Industry Income - 2.5%

The teaching is further broken down into another 5 categories, with 15% a reputation survey, 4.5% staff to student ratio, 2.25% Doctorate to bachelor’s ratio, 6% doctorates awarded to academic staff ratio, and 2.25% institutional income.

The research category is also further broken down into 3 categories. 18% is attributed to a reputation survey, 6% to research income (which they say describes the importance of the research being done), and 6% to research productivity. Interestingly, Times Higher Education admits that research income is a controversial metric because it can heavily depend on national policy and economic circumstances [5]. Nevertheless, they argue that it’s an important metric because research income is vital to conduct world-class research. Research productivity is measured by the number of papers published per academic. The Times Higher Education says that this is a measure of how good the university is at getting its publications published in a high-quality peer-reviewed journal.

The industry income is funding received from commercial sources, for example, businesses commissioning research. This category describes the commercial impact the university's research has, which is a unique category not found in the other two rankings. The international outlook category is a measure of how good the university is at attracting students and academics from around the world. This is calculated by looking at the international staff and student ratio, as well as the number of research publications with international co-authors.

The Times Higher Education has a stronger focus on teaching and education than the other two rankings, but this can be quite subjective and hard to understand from a student’s perspective. Despite the strong teaching focus, the ranking is still more than 60% determined by the research output of the institution.

Finally, we have the Academic Rankings of Universities (Shanghai Ranking) [6]. This ranking originally was created as a way for Chinese universities to see how they stack up against the global competition. As the name suggests, it’s a ranking based almost solely on the academic performance and prestige of the institute. The methodology is as follows:

• Number of alumni winning Nobel Prizes and Fields Medals - 10%

• Number of staff winning Nobel prizes and Fields Medals - 20%

• Number of highly cited researchers in 21 broad subject categories - 20%

• Number of articles published in Nature and Science - 20%

• Number of articles indexed in Science Citation Index - 20%

• Per capita academic performance of an institution - 10%

The graph shows the percentage of the 15+ year old population with a tertiary degree over time. Source: https://ourworldindata.org/tertiary-education.

The Academic Rankings of World Universities is heavily influenced by whether or not academic staff/alumni at the institution have won these prestigious awards and medals. When you look at the actual rankings, you see that it’s dominated by institutions that have been around for centuries. By using Nature, Science, the Nobel Prize and Fields Medal as indicators of high academic performance, the Shanghai Ranking itself amplifies the prestige and status of these prizes and journals.

Note: the Fields Medal is sort of like the Nobel Prize equivalent for mathematics.

You may disagree with the methodology of these rankings, or the general philosophy behind ranking institutions in this way. However, there is one undeniable fact, university rankings matter. They matter to the students who go to or want to go to these universities, they matter to the universities themselves, as they heavily determine the future of an institution, and they matter to governments too as they often use these metrics to decide on the allocation of funding. Although the three rankings are transparent about how the total score is calculated, how they decide the weighting of each category is quite arbitrary. Who is to say that citations per faculty should be 20% of the overall score instead of 15%?

In a survey conducted by professor Ellen Hazelkorn at Technological University Dublin, it was found that more than half of the universities had taken strategic actions because of the rankings [7]. Sometimes the strategic decisions that universities make can have a negative influence on the teaching or learning experience for students at the university, but it is justified for the university because being highly ranked can lead to more students, more funding, and more prestige. For many smaller and lesser-known universities, the rankings can mean life or death as it’s crucial in attracting students, and getting funding from the government. It should also be noted that most of the data used for these rankings are self-reported by the universities. As you can imagine, this has caused some issues in the past, and there have been numerous reports/scandals of universities bending the statistics or outright cheating to climb up the rankings. In 2020, Temple University was fined $700,000 by the US Department of Education for sending fraudulent data about its online MBA program which helped the course to be ranked at the top in the country for several years. This year, the course is ranked 88th, tied with six other universities [8]. Back in 2015, Trinity College Dublin was accused of trying to influence academics that are part of the annual reputation surveys. Trinity College Dublin issued a statement of regret but said their intentions were in good faith [9].

There are other concerns around the legitimacy of the rankings as well. Recently a paper was published by researchers at the Centre for Studies in Higher Education at the University of California, Berkeley, raising concerns about conflict of interest between QS World University Rankings and some universities. QS has a consulting business that helps universities in various aspects of their business [10]. Igor Chirikov, a senior researcher at the Centre for Studies in Higher Education argues that this consulting business is inappropriately influencing the rankings of universities [11]. The study looked at 28 universities, of which 22 of them had spent collectively nearly three million US dollars in QS consulting services. These universities that had used this consulting service frequently, rose approximately 140 positions than they would have otherwise if they hadn’t used the service. Though other ranking institutions hold events for universities, their revenue is not nearly as reliant on universities as QS is. For example, Times Higher Education makes money from advertising and through its subscription-based content. Igor says that this type of conflict of interest is similar to those that are seen in other sectors of the economy where consultation leads to a biased evaluation of the rankings.

So should you trust university rankings, and are they useful? The fact of the matter is that rankings aside, there are plenty of great universities for us all to get a great education at. The differences between universities can be quite arbitrary and whatever methodology these rankings employ, they will never be able to encapsulate the entire experience and the entire calibre of a university down to a number. It’s important for us to be aware of how universities are “ranked” and what this actually says, or doesn’t say, about the quality of a university. A university is a place with vast responsibilities, from educating the next generation, to doing cutting edge research that changes the world and the way we think, to helping other sectors of the economy to grow. Universities play a huge role in our society, and for better or for worse, universities will continue to make strategic decisions around the rankings. One thing is for certain, these rankings aren’t going anywhere anytime soon. So maybe it’s time to not take these rankings too seriously ourselves.

References:

[1] Universities NZ, 2021, Introducing NZ’s eight universities, https://www.universitiesnz.ac.nz/universities

[2] TruOwl, 2018, How many universities exist in the world?,

https://truowl.com/university/how-many-universities-exist-in-the-world/

[3] QS Quacquarelli Symonds Limited, April 20 2021, Ranking Methodology, https://www.topuniversities.com/qs-world-university-rankings/methodology

[4] Duncan Ross, Times Higher Education, September 2020, Times Higher Education Ranking Methodology, https://www.timeshighereducation.com/sites/default/files/breaking_news_files/the_2021_world_university_rankings_methodology_24082020final.pdf

[5] Times Higher Education, September 2019, Times Higher Education Ranking Methodology, https://www.timeshighereducation.com/world-university-rankings/world-university-rankings-2020-methodology

[6] Shanghai Ranking, 2020, Shanghai Ranking Methodology, http://www.shanghairanking.com/ARWU-Methodology-2020.html

[7] Hazelkorn, Ellen. (2019). University Rankings: there is room for error and "malpractice". http://doi.org/10.5281/zenodo.2592196

[8] Scott Jaschik, December 7 2020, Education department fines Temple $700000, https://www.insidehighered.com/admissions/article/2020/12/07/education-department-fines-temple-700000-rankings-scandal

[9] Carl O’Brien, March 22 2016, Trinity College Dublin accused of trying to sway world university rankings

https://www.irishtimes.com/news/education/trinity-college-dublin-accused-of-trying-to-sway-world-university-rankings-1.2582286

[10] Scott Jaschik, April 27 2021, Buying Progress in Rankings?, https://www.insidehighered.com/admissions/article/2021/04/27/study-charges-qs-conflicts-interest-international-rankings

[11] Chirikov, I. (2021). Does Conflict of Interest Distort Global University Rankings? . UC Berkeley: Center for Studies in Higher Education. Retrieved from https://escholarship.org/uc/item/8hk672nh