On a sunny morning in the Stone Room on the top floor of the Faculty, Melvyn and I took the opportunity to have a cup of tea, and take a step back from AI developments and look at the state of machine learning. After all, AI seemed to go mainstream in 2024, dominating news headlines, linkedIn feeds, and business conversations. And yet, despite witnessing significant progress over the past year, it looks as if we are just at the beginning of AI’s potential. I initially ask him why machine learning models is so important right now.
He explains that AI can provide insights that that would have been impossible just a few years ago. “Machine learning models can be used for prediction problems like predicting house prices. They can also be used for a causal problem where you want to know the effect of prices on consumption, such as food consumption or cigarettes. It isn’t a case of predicting cigarette demand, but you're interested on the causal effect of price. You can now do that without pre-specifying the function.”
So why, I ask, has there been a sudden interest in AI now?
“Oh, AI has existed for quite a while. What is different is that in November 2022 Open AI brought out ChatGPT. They never would never have anticipated that within eight months they had 100 million users,” he says. “The chat bit on top of GPT refers to the fact that you can just write a question in a chat interface, which is very powerful. Now we have Generative AI, underpinned by a large language model. The model has been trained on a vast amount of data - in effect a lot of text, and that provides a vastly imporved response."
He explains that one of the fundamental differences between AI and Generative AI now is that the latter allows people access to advanced AI tools simply by writing natural language. GPT stands for Generative Pre-trained Transformer. In effect it allows a computer to parse written English, pick up some key words using the Transformer architecture, and then decipher what you are asking. It sounds simple but it takes a vast amount of very high-speed computation, which until a few years ago was prohibitively expensive. It is also depends on the nature of the question or prompt. Melvyn explains; “If the query is too general you may end up with a response that is not informative; similar in a sense to errors in computer code”.
It is important to understand the distinction between generative AI and a search engine. Google functions as a search engine, retrieving documents that are most similar to the query, it has limited understanding of the broader context or the intent behind the query. Generative systems, such as ChatGPT, might be thought of as a reason engine, given the ability to capture and understand the context and nuances of a given query, thereby generating responses that are contextually relevant. If you use Google, it takes your query and offers a selection of ‘best matches’ by retrieving information. However, you have to work out what is relevant to you.”
He says Generative AI is different. “It has learned the foundations of language by being trained on huge amounts of textual information; in effect the computer reads a lot of text. People label it ‘everything that's ever been written’ which is a perhaps a bit too far-fetched, but can offer specific information, from travel information or cookery or even economic principles. In this sense it generates but does not retrieve information”
Melvyn explains that a new tool under the name of Perplexity.ai, combines retrieval and generative functions. For example, a query on the impact of education on economic growth would retrieve a number of references on this issue, provides a full set of references and then uses a Large Language Model to generate a response. One of the characteristics of more recent machine learning models is that they have moved on from providing predictions across a whole sample, to predictions for specific types of individuals. We can also think of Generative AI in this way, in that it has a capacity to provide outputs that are targeted to specific use cases, or contextualised, using a prompt.
Melvyn says the AI appears to effectively converse with you. “What we had before could address specific questions for which you had data on; the sort of data you would store in a spreadsheet. For example, people's electricity consumption over many years. A model to predict consumption in the next quarter or similar is now relatively easy to code,” he says, explaining it is ‘AI’, but domain specific.
He sums it up by saying “Generative AI is different, in the sense that it has been trained on a vast corpus of textual informatio,n it can appear that it has significant knowledge across multiple domains.”
For the Business, Banking and Financial sector, AI isn’t just a new tech trend, it’s a powerful tool that will have a wide range of impacts, from risk management to operational efficiency, and customer experience. However, I suggest it sounds almost as if people have a perception that we’ve suddenly ‘got AI and we didn't have AI before’. “That’s incorrect; it’s just developed further but with a very significant development: with a model trained on lots of data processed by a much more sophisticated transformer algorithm,” he says, adding it is still derived from the same concepts of understanding a query and offering information. “We’ve been able to process basic language for decades." So-called natural language processing (or NLP) has been around for a long time. "There have been relatively crude ways of making sense of language; seeking to understand and compare documents or speeches, for example."
Understanding even the sentiment of what is written has been performed before, but in a relatively primitive way. “If you're trying to make sense of, say, a Twitter feed of Elon Musk or someone, you can get to the main point,” he says, adding that primitive forms of sentiment analysis was based on the frequency of words that were positive or negative. Pure frequencies, but no context.
“Now, with the advent of new large language models, such as ChatGPT, Llama 3, or Claude3, we have supercharged the ability to query and understand. For data we can use Twitter feeds, income reports, corporate documents, vast amounts of information, which can be queried, extracting sentiment and meaning,” he says.
He emphasises that seeking to understand unstructured data in the form of text is a real challenge. If you're looking at time series data, although there's a natural order to the data, it's still very challenging to extract meaning and make predictions. The problems then multiply if you're trying to do that with text.
“How do use models to convey the subtle nuances of an essay? You might start by converting words into numbers - so-called word embeddings - which then provides the basis for comparing words: for example, how close they are to one another.”
A classic example is countries. “If you just had a database of country names, you could use a large language model to create a two dimensional plot of the closeness of these countries, based on an analysis of lots of textual information providing instances and contexts where individual and collections of countries were discussed," he says "Based on this you might start grouping countries because they're in the same trading block or they had the same colonial power. So, you can understand countries not just through numerical data, but using data that's more vast: text. However, there is a long way to go to create real understanding.”
I agree, and use the example of say text based computer games in the 1980s. These in theory understood basic English – but in reality, it was just pattern matching nouns and verbs. AI could be seen as extrapolating on that earlier work, but even now, with some of the advanced AI systems, you can tell there's still a computer behind it. It doesn't have that natural fluency that you would expect from somebody who is totally eloquent in the subject. You can still spot AI.
I pose the question of asking ‘will we ever get past that stage?’ He replies that it is a good question. Alan Turing developed a test to determine whether a computer is answering your questions, or a human. According to Ethereum co-founder Vitalik Buterin, OpenAI’s GPT-4, has now passed the test.
He uses the example of students who mix AI generated answers with their own work. “Pre-Generative AI, if a student writes an essay, the essay can be passed through something called TurnItIn, which basically takes the text and then seeks to match it with existing text; if it’s too similar, that's plagiarism. That problem has been reasonably well solved,” he says. “However, generative AI is not retrieving anything. It's writing it. Often answers will be subtly different, and essays might have a significant component coming from ChatGPT or other LLMs, mixed with their writing. Can we now determine it is plagiarism, if only some parts are generated by AI via a computer? But AI is getting better all the time. It is an ongoing issue.”
I ask him about other challenges, and there are a lot. “There are many areas where regulators such as the UK's Competition and Market Authority is looking at the implications of the fact that these LLMs are incredibly expensive to train and develop. Although the costs are coming down, there's still a question at the moment of potential monopoly power; the potential abuse of power and not as much choice for the consumer.”
Although it's fine to think about regulating these models, can you regulate what you don't understand, I ask? “You know, when you seek to regulate companies in the water sector or the energy sector, there's often a world where the companies know more than the regulator; they may have more resources than the regulator. Here we are in a very, very different situation. When you speak to some of the developers of the LLMs at Microsoft and Amazon, as I have done, they come back to me and say that only a few people have a deep understanding of these models.
It's so vast that the understanding is not there yet,” he says. “And this is where there's been lots of questions about do we understand what we're developing, what are the associated risks? When you talk about regulating these things, its important to commit time and resources to understanding the thing which you want to regulate. There are really interesting, developing, dimensions to this. It’s an ongoing issue.”
So how did he started becoming interested in AI and machine learning models? “The interest came automatically, as I had a focus on econometrics and I wanted to understand and make inferences from data, the models, and the estimators I was using. I had traditional models, which were parametric. We had some prior information of what the model would be, and lots of data.”
That data is sometimes referred to as big data.
“Many people understand big data as ‘5 million rows in your data set’, but actually big data relates to the relationship between the rows and the columns. For example, if you've got a small data set, you could have say 60 countries, but if you've got a vast number of measured variables, you can generate a cross country growth model; you don't have a lot of observations on which those variables are measured. In this instance you shouldn't build a model with many variables; you need to be more parsimonious with the variables, and theory will not always generally be useful in helping you decide which variables you use. If you want to be agnostic on that, machine learning can really help you set up the model, without going too far down the route of looking at all the variables.
Machine learning helps by examining the data to estimate parameters or the effects you're interested in. In this context, the difference is it's used to locate the model and the key variables based on some function that you're using.”
It is clearly an ongoing issue, that will occupy Melvyn for a long time to come.
The Faculty of Economics has launched an MPhil in Economics and Data Science
This master’s degree is intended to equip you with the skills and knowledge necessary to apply advanced data science techniques to economic analysis and decision-making. The degree will give the technical training required to pursue a career in a variety of fields such as consulting, government, academia, and the private sector. The course will provide a deep understanding of economic theory and principles and applied economic research. You will develop enhanced analytical skills and be exposed to a variety of cutting-edge tools and technologies used in data analysis and modelling, such as machine learning algorithms, data visualisation tools, and cloud computing platforms. Further details: https://www.econ.cam.ac.uk/postgraduate-studies/mphil-data
