In this AI research series, our team at Lotus Labs will test ChatGPT on its ability to perform in roles typically fulfilled by humans. With the rise of artificial intelligence, much discourse has emerged at the prospect of this advanced technology potentially replacing human jobs. While these concerns once seemed relevant only in the distant future, the groundbreaking rise of ChatGPT indicates that the time for AI to start carrying out professional services might be now. By putting ChatGPT through a series of experiments in different industries and scoring its performance, our team intends to discover how well AI can replicate the services of humans.

The inspiration for this series came from a fundamental truth: An individual’s quality of life depends on their ability to access knowledge, most notably medical and financial information. While it is true that the internet has democratized access to most types of information, the sheer amount of knowledge available can sometimes be counterintuitive to finding answers. By using machine learning algorithms to analyze and filter from vast amounts of data, ChatGPT can generate natural and conversational responses to questions or prompts. As a result, our curiosity to see whether ChatGPT could fulfill the role of a doctor and financial advisor resulted in the first two articles in this series — ChatGPT: Your Next Doctor and ChatGPT: Your Next Financial Advisor?

However, after researching these first two topics, our team realized that there was potential to test ChatGPT in even more roles and industries — from music and sales to retail. As a result, we created a general experimental framework to apply to ChatGPT as we tested it in different contexts. The format of each experiment will be relatively flexible as we want to communicate with ChatGPT like we would with another person. After all, if ChatGPT is capable of filling human roles, it needs to be capable of supporting natural conversation.

We will begin by creating a hypothetical scenario to input into ChatGPT such as a young boy struggling with a cough on a particular morning or a recent college graduate looking to revaluate her financials. After getting ChatGPT’s initial diagnosis of the situation, we will ask it a series of follow up questions simulating the interaction of patient/client and a service provider. The conversation about the scenario will end once we ask ChatGPT for a final recommendation such as a diagnosis for the little boy’s cough or a laid out budget for the recent college graduate. In each research article, we will walk ChatGPT through several scenarios relevant to the topic and note down all of its responses.

ChatGPT’s performance will always be scored based on five metrics: accuracy, completeness, clarity, relevance, and efficiency. Since, we will be putting ChatGPT through tests where the answer is already known, to score accuracy, we will compare ChatGPT’s response to the correct answer. For completeness, we will analyze whether ChatGPT’s answer is comprehensive or if it leaves out important details. To measure clarity, we will look for a clear, concise, and humanistic response. For relevance, we want to see whether ChatGPT’s answer is direct and personalized or if it seems to be regurgitating information. Finally, our efficiency metric measures if the answer is provided in a timely manner and if ChatGPT can understand our directions right away or if it needs additional prompting. After scoring the Chatbot’s performance in each test, we will determine an overall score reflecting ChatGPT’s capability in its experimental role.

While ChatGPT is an excellent baseline test subject for our initial tests, as this series continues and AI continues to evolve, it is conceivably that we could change our test subject as we wish to utilize the most advanced AI free to the public. Ultimately, we believe this series will help shine a light on both the tremendous potential and limitations of artificial intelligence as well as illuminating how humans and AI can best work together.

Next Thursday, stay tuned for the first article in the series — ChatGPT: Your Next Doctor?