February 24, 2023

By: Debashish Har, Principal Data Scientist, Consulting

Introduction

Since its launch, ChatGPT has created a lot of buzz and attracted multibillion-dollar investments. The various social media channels have come up with interesting stories on how the conversational AI tool has helped in developing websites and generating boilerplate code. There have been discussions on whether open-book exams will have any meaning with tools like ChatGPT available in the market. There have been discussions on how this would disrupt the whole search engine business and how Google would lose a large market share in the digital ad business. Some  experts have gone on to state that this is perhaps the beginning of the end for Google as a search engine business. There are hundreds of thousands of articles and blogs written to highlight all the amazing possibilities that the tool brings, but very few to throw light on the gaps.

The author has attempted to find gaps in the tool, while being a fan of the tool himself. This is a humble critique and invites comments and discussions on the topic. The post represents a personal point of view and doesn’t represent those of any organization.

What is ChatGPT?

It is a large language model-based conversational AI engine. It provides search results in the form of a piece of text specific to the search question. Conventional search engines provide indexed, ranked pages relevant to the search criteria. The chatbot uses the OpenAI GPT 3.5 engine. The model was trained on data with 499 billion tokens, and ultimately 175 billion parameters were used to train the model. It is a model of conversational AI; a semi-supervised and unsupervised algorithm used in generative AI models allows for the creation of synthetic material, such as text, audio and video files, graphics, code, and more. The created openAI content could be regarded as original because it is not a perfect replica of the training data.

A generative classifier assumes a functional form of P(Y) and P(X|Y), then generates parameter estimates using Bayes’ theorem to calculate the posterior probability P(Y|X). A discriminative classifier, on the other hand, assumes a functional form of P(Y|X) and directly calculates parameter estimates from the data.

Language models are tools to predict the next word or set of words in a sequence of text. They calculate the probability distribution over a sequence of words.

Some key applications include:

• Machine Translation
• PoS Tagging
• Text Classification
• Sentiment Analyses
• Speech Recognition
• Information Retrieval
• News Article Generation

ChatGPT in Logical and Analytical Problem-Solving

Basic logical reasoning is an integral part of comprehending human language. The author has tried to test the logical reasoning and more nuanced requirements to compare ChatGPT with established search engines like Google and Bing.

The author’s humble view is that the chatbot could fail the Turing test in solving logical reasoning problems, but that needs to be established as a separate exercise with the strict conditions of a Turing test.

The author also noted the results for a few popular puzzle-based questions, e.g., the probability of person A and person B meeting at a given location within a given time window and the famous Mr. P and Mr. S puzzle concerning the product and sum of two numbers that could be uniquely identified. The results were not satisfactory.

Solution: Smith and John’s arrivals are independent, so the total sample size could be represented by a square with sides of a 1-hour time window. The cross-hatched region in figure 4 represents the potential events when they meet. The probability is given by P = 2[(1/2) – (1/2) (5/6) (5/6)] = 11/36.

The famous Mr. S and Mr. P puzzle didn’t yield the desired results. It is a puzzle that demonstrates the properties of numbers and a logical method for eliminating incorrect outcomes.

The author investigated whether ChatGPT could correctly answer the sum of a well-known series that requires knowledge of complex variables, the Reimann-Zeta function, and analytical continuation. The correct answer is -1/2.

Top-ranked search results using Google and Bing could yield correct results for the puzzles and the series sum.

ChatGPT in Content Creation

Furthermore, the author attempted to investigate the quality of new content generated by ChatGPT. The answering machine yielded a Tweedledum-Tweedledee copy of the famous Elizabeth Bishop villanelle, named ‘One Art’ when asked to write a new villanelle in Elizabeth Bishop’s style.

A villanelle, like a sonnet, is a form of poetry that follows a certain rhyme scheme, rhythm, and pattern of strophes.

When the answering machine was given the first line of the same villanelle, it yielded results that deviated from the original poem. It wrote a villanelle without being asked! This may be due to the limitations of the training data.

When asked to compose a song in Leonard Cohen’s style, it didn’t seem to resemble anything from his oeuvre.

When asked to write in Paolo Coelho’s style, the chatbot could yield some text that did represent some original content, but I’m not sure if it could remotely represent the famous author’s content.

The answering machine could write the correct Python code to generate prime numbers between 1 and 1000.

However, when asked to generate the Python script without using loops, it couldn’t yield satisfactory results.

Whereas a single line of python script could yield the same result that doesn’t require any loop.
print(list(filter(lambda x: not list(filter(lambda y:x%y == 0, range(2,x))), range(2, 1000))))

Conclusion

The conversational AI chatbot must evolve to emulate human intelligence in solving logical reasoning problems and may fail the Turing test. The answering machine could generate new content when asked to do so and generate pieces of code. However, how much of an impact it would have on search engine and ad tech businesses remains to be seen. Surely there will be copious studies estimating how much search engines would be impacted by ChatGPT. What fraction of search results could be replaced by an answering machine like ChatGPT?

Traditional search engines provide ranked, indexed search results in the form of pages. This makes the process of searching transparent, and the user can authenticate the information, whereas an answering machine-generated result doesn’t have the authentication process. Besides, the ranked and indexed pages provide the space for ad Techs to monetize search engines, while an answering machine could perhaps be packaged with cloud technologies as an add-on and become part of an eclectic list of bells and whistles. Perhaps it must evolve before it can obliterate search engines, and additional algorithms are needed other than the Generative Adversarial Network (GAN). With a few additional LLM platforms getting launched, and several AI tools spouting out, and several in the pipeline at incubators like Y Combinator and large tech organizations, we are at the cusp of an AI revolution. GAN models may have additional challenges to contend with, e.g., non-convergence and mode collapse, which could be attributed to an imbalance between the discriminator and generator. Such models may be more suitable for sequential problems. Reasoning-based problem-solving requires a different set of toolkits.