When ChatGPT Attempted UPSC Exam

ChatGPT seems to have all the answers in the world. Since its launch in November last year, it has passed some of the toughest exams in the world. In the US, ChatGPT cleared the final exam for the Master of Business Administration (MBA) programme at the University of Pennsylvania’s Wharton School. It also scored better than most students in the US medical exam. ChatGPT also managed to clear Google Coding Interview for Level 3 Engineers.

It was emerging as the world’s brightest kid until it was asked to sit for the Indian Civil Services Examination conducted by the Union Public Service Commission (UPSC)- one of the toughest examinations in the world.

We at Analytics India Magazine have also been experimenting with ChatGPT for a while and were intrigued by its ability to crack tough exams and interviews. In that spirit, we wondered, ‘Can ChatGPT clear UPSC?’

The chatbot sensed the tough ordeal it was supposed to go through even before appearing for the preliminary paper of the UPSC.

UPSC Breaks ChatGPT

We asked ChatGPT all the 100 questions from Question Paper 1 (Set A) from UPSC Prelims 2022, the answers to which are available on the web. To our surprise, only 54 of them were correctly answered by ChatGPT. 

In 2021, the cut off was 87.54 for general category candidates, so going by the data, ‘ChatGPT did not clear USPC’.

The questions ranged from topics such as Geography, Economy, History, Ecology, General Science to current events of national and international importance, social development, and polity.

ChatGPT’s knowledge is limited to September 2021, hence, it won’t be able to answer questions on current events. However, ChatGPT provided wrong answers for topics such as Economy and Geography that are not necessarily time-specific.

It even got a simple question on History wrong.

In some instances, ChatGPT created its own alternatives. As an illustration, in the following queries, the chatbot presented Option ‘e’ as an answer choice, despite it not being a part of the originally provided options.

The Civil Service Examination is one of the toughest examinations in the world. Nearly, 11.52 lakh candidates appeared for UPSC last year alone. However, only around 5% makes it to the mains. Among those 5%, only a handful managed to crack the mains and get placed as an Indian Administrative Officer (IAS). 

Hallucinatory in nature

Here, it is important to note that ChatGPT is hallucinatory in nature. The same has been acknowledged by Sam Altman, chief executive at OpenAI.

“ChatGPT is incredibly limited, but good enough at some things to create a misleading impression of greatness,” he said. 

Altman believes it would be a mistake to be relying on it for anything important right now. “It’s a preview of progress; we have lots of work to do on robustness and truthfulness.”

Even though ChatGPT has managed to crack some top examinations, it failed terribly in an examination designed for sixth graders in Singapore.  

Researcher Christian Terwiesch, who published the white paper titled, ‘Would Chat GPT Get a Wharton MBA?’ also notes that ChatGPT performed exceptionally well in answering fundamental queries within its area of expertise. However, it occasionally made unexpected errors in middle school level calculations.

Hence, it’s imperative that we consider ChatGPT taking different exams, including USPC, merely as fun experiments. 

In the same inquiry, we also discovered that we could prompt ChatGPT to arrive at correct responses by modifying the inquiry. This means that in certain cases, asking the same question differently could lead ChatGPT to give the right responses, and vice versa. But, during the experiment, we only took into account ChatGPT’s first responses.

Moreover, since it failed on Question Paper 1, we did not go ahead with the experiment with Question Paper 2 and further on. With all the question papers and answers available on the web, anyone can redo the experiment or further it and test ChatGPTs reasoning and analytical skills.