Using ChatGPT to Train an Interview Model¶
In my first article on the impressive skillset of Open AIs new model ChatGPT I demonstrated ChatGPT's ability to take in a prompt identifying a psychological construct (i.e. Extraversion) and an item type and output fairly well written Likert style personality items.
In this article I'll discuss a few new topics and bring them all together to train a model capable of accurately differentiating poor from good responses.
First I want to discuss a few new, but important concepts. Namely, the ideas behind transfer learning and few shot learning. Then I will use ChatGPT to create a unique set of interview responses to a fairly standard behavioral interview question on customer experience. I will then show readers how to use those responses to train a classification model using the few shot learning technique.
Transfer Learning¶
Large language models and even methods like word2vec have brought about the idea of transfer learning. Transfer learning is a technique that enables a model to leverage knowledge acquired from one task to improve performance on a different but related task. It is based on the idea that many tasks share common features and that by training a model on one task, it can learn these common features, which can then be applied to other related tasks. Large language models can readily be leveraged using this technique for many downstream supervised NLP tasks. One of the biggest problems with language early on was the idea of equifinality, or the concept that you can use many terms or phrases to describe a similar concept. Large language models help us solve that problem because they are trained on billions of examples of prose in the English language (and with some models over a hundred languages), which allows it to understand that terms like customer and client are very similar to one another without explicitly learning it through your training dataset. We can see that with a quick example below.
Let's examine the cosine similarity of the following 3 sentences. We should expect A and B to be very similar even though they are using different language to describe a similar situation. C, should be the least related.
a = "That client was difficult to work with"
b = "That customer was a problem to deal with."
c = "I enjoy working with all of my associates."
%%capture
!pip install -qU transformers sentence-transformers
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('bert-base-nli-mean-tokens')
sentence_embeddings = model.encode([a, b, c])
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
# calculate similarities (will store in array)
scores = np.zeros((sentence_embeddings.shape[0], sentence_embeddings.shape[0]))
for i in range(sentence_embeddings.shape[0]):
scores[i, :] = cosine_similarity(
[sentence_embeddings[i]],
sentence_embeddings
)[0]
import matplotlib.pyplot as plt
import seaborn as sns
plt.figure(figsize=(10,9))
labels = ['a', 'b', 'c']
sns.heatmap(scores, xticklabels=labels, yticklabels=labels, annot=True);
As we expected the cosine similarity between A and B is extremely high and C is not very close to them.
The idea of transfer learning helps us understand that these large language models already have a deep understanding of language and consider sentences using similar, but different, language as still similar the way a human would.
Few Shot Learning¶
Few-shot learning is a technique that allows a model to perform well on a new task with only a small number of examples. The model is trained to learn from a few examples and generalize to unseen data. We can use this to our advantage and reduce the number of training samples we need to generate a reasonably good model. We can provide a pre-trained large language model (LLM) with a handful of examples of what good and bad look like on a new task. This is in contrast to traditional supervised machine learning where often times thousands of examples of good and bad are needed in order to train an accurate model for making predictions on out of sample examples.
For those interested in learning a lot more about few shot learning, I'd recommend this article or for a comprehensive overview this paper.
ChatGPT¶
This brings us to the point where ChatGPT gets involved. We showed many of the capabilities of ChatGPT in the previous article.
Typically one of the main problems with building a model is access to quality training data. How do you get people to respond to an interview question? You can provide it with a few of your own examples, but chances are your personal examples are limited. Using ChatGPT and prompt engineering may help us solve that problem. You can provide ChatGPT with a request and context and it can use that to generate multiple realistic responses for you to train your model on. There are also many other ways to create synthetic data that I may write about later, like translation-back translation using NMTs, random synonym replacement, etc.
Let's ask ChatGPT to give us some training examples of poor responses to the interview question
Describe time when you had to deal with a challenging customer. What was the situation and what was the outcome?
So you can see they gave us the STAR we asked for, but not in a paragraph format, so we might need to make some adjustments to that before we use it.
Now let's ask the same for good responses.
Now let's go ahead and turn these into data and examine them.
x_train = ["I was working as a customer service representative at a large retail store when a customer became extremely upset with the quality of a product they had purchased. I listened carefully to their concerns and apologized for the inconvenience. I then offered to replace the product at no cost and arranged for a manager to follow up with them to ensure their satisfaction. The customer calmed down and was satisfied with the resolution. They even thanked me for going above and beyond to resolve their issue.",
"While working as a server at a restaurant, I had a customer who was dissatisfied with their meal. I politely asked if there was something I could do to improve their experience and offered to have the kitchen prepare a new dish for them. I also checked in with them throughout their meal to ensure they were happy with their replacement dish. The customer was satisfied with their new dish and left a positive review on our restaurant's website.",
"As a sales associate at a clothing store, a customer became angry and aggressive after realizing they had been overcharged for an item. I calmly explained the mistake and offered to refund the difference immediately. I also apologized profusely and assured the customer that it was an honest mistake. The customer calmed down and accepted the refund. They even thanked me for handling the situation professionally.",
"I was working as a server at a local bar when a customer became upset because they were overcharged for their meal.I listened carefully to their concerns and apologized for the mistake. I immediately refunded the difference and offered to prepare a complimentary dessert for them. The customer calmed down and was satisfied with the resolution. They left a positive review on our bar's website and continued to visit regularly.",
"While working as a server at a local bar, I had a customer who was dissatisfied with the quality of their meal. I politely asked if there was something I could do to improve their experience and offered to have the kitchen prepare a new dish for them. I also checked in with them throughout their meal to ensure they were happy with their replacement dish. The customer was satisfied with their new dish and left a generous tip. They continued to visit the bar and became a regular customer.",
"I had a customer who was upset about the quality of a product they had purchased. I didn't really do anything to help them because I didn't want to deal with it. The customer was still upset and left the store without purchasing anything else.",
"I had a customer who was angry because their meal was not cooked to their liking. I just told them to deal with it and that's how the kitchen cooked it. The customer left the restaurant without leaving a tip.",
"I had a customer who was overcharged for an item. I told them it wasn't my problem and they would have to take it up with the manager. The customer left the store without purchasing anything else and left a negative review online.",
"I was working as a server at an Italian restaurant when a customer became upset because they were overcharged for their meal. I told them it was their own fault and that they should have paid attention to the menu prices. The customer was still upset and left the restaurant without leaving a tip.",
"While working as a server at an Italian restaurant, I had a customer who was dissatisfied with the quality of their meal. I just shrugged and told them that's how the kitchen cooked it. The customer left the restaurant without leaving a tip and left a negative review online."]
y_train = [1,1,1,1,1,0,0,0,0,0]
x_val = ["While working as a receptionist at a medical office, a patient became upset because their appointment was running behind schedule. I apologized for the delay and explained that their doctor was running late due to an emergency. I offered them a complimentary beverage and asked if there was anything else I could do to make their wait more comfortable. The patient calmed down and appreciated the gesture. They were understanding of the delay and had a positive experience overall.",
"As a customer service representative at a call center, a caller became angry and yelled at me because their order had not been delivered on time. I apologized for the inconvenience and offered to expedite the delivery at no extra cost. I also offered to provide a discount on their next order as a gesture of goodwill. The caller calmed down and accepted the resolution. They even thanked me for my help and apologized for their behavior.",
"As a server at a local bar, a customer became angry and aggressive after being asked to follow the bar's dress code. I calmly explained the dress code policy and offered to help them find an appropriate outfit in the bar's merchandise store. I also apologized for any inconvenience and assured the customer that the dress code was in place for their safety and comfort. The customer calmed down and appreciated the gesture. They complied with the dress code and had a positive experience at the bar.",
"As a server at a local bar, a customer became upset because they were not happy with the music selection. I politely asked if they had any specific music preferences and offered to change the music to something they would enjoy. I also apologized for any inconvenience and assured the customer that their feedback was valuable to us. The customer calmed down and appreciated the gesture. They had a positive experience at the bar and left a positive review on our website.",
"I had a patient who was upset because their appointment was running behind schedule. I told them it wasn't my fault and they would have to wait. The patient left the office and never returned.",
"I had a caller who was angry because their order was not delivered on time. I just hung up on them. The caller never called back and I had to deal with the fallout from my supervisor.",
"As a server at an Italian restaurant, a group of customers became upset because their table was not ready when they arrived. I told them they should have made a reservation and that they would have to wait. The customers left the restaurant and never returned."]
y_val = [1,1,1,1,0,0,0]
x_test = ["While working as a receptionist at a medical office, a patient became upset because their appointment was running behind schedule. I apologized for the delay and explained that their doctor was running late due to an emergency. I offered them a complimentary beverage and asked if there was anything else I could do to make their wait more comfortable. The patient calmed down and appreciated the gesture. They were understanding of the delay and had a positive experience overall.",
"As a customer service representative at a call center, a caller became angry and yelled at me because their order had not been delivered on time. I apologized for the inconvenience and offered to expedite the delivery at no extra cost. I also offered to provide a discount on their next order as a gesture of goodwill. The caller calmed down and accepted the resolution. They even thanked me for my help and apologized for their behavior.",
"As a server at a local bar, a customer became angry and aggressive after being asked to follow the bar's dress code. I calmly explained the dress code policy and offered to help them find an appropriate outfit in the bar's merchandise store. I also apologized for any inconvenience and assured the customer that the dress code was in place for their safety and comfort. The customer calmed down and appreciated the gesture. They complied with the dress code and had a positive experience at the bar.",
"As a server at a local bar, a customer became upset because they were not happy with the music selection. I politely asked if they had any specific music preferences and offered to change the music to something they would enjoy. I also apologized for any inconvenience and assured the customer that their feedback was valuable to us. The customer calmed down and appreciated the gesture. They had a positive experience at the bar and left a positive review on our website.",
"I had a patient who was upset because their appointment was running behind schedule. I told them it wasn't my fault and they would have to wait. The patient left the office and never returned.",
"I had a caller who was angry because their order was not delivered on time. I just hung up on them. The caller never called back and I had to deal with the fallout from my supervisor.",
"As a server at an Italian restaurant, a group of customers became upset because their table was not ready when they arrived. I told them they should have made a reservation and that they would have to wait. The customers left the restaurant and never returned."]
y_test = [1,1,1,1,0,0,0]
x_train[0:2] # 2 good responses
x_train[4:6] # 2 poor responses
%%capture
!python -m pip install setfit
# turn the examples and labels into a DatasetDict
from datasets.dataset_dict import DatasetDict
from datasets import Dataset
d = {'train':Dataset.from_dict({'label':y_train,'text':x_train}),
'val':Dataset.from_dict({'label':y_val,'text':x_val}),
'test':Dataset.from_dict({'label':y_test,'text':x_test})
}
test_dataset = DatasetDict(d)
test_dataset
We can see that our dataset dictionary has a train, validation, and test and each one has labels and text. Our training set has 10 rows/examples and our validation and test sets for this purpose are identical, but we'll create some different responses to evaluate on later as well.
train_dataset = test_dataset["train"]
eval_dataset = test_dataset["test"]
from setfit import SetFitModel, SetFitTrainer, sample_dataset
from sentence_transformers.losses import CosineSimilarityLoss
# Load a SetFit model from Hub
model = SetFitModel.from_pretrained("sentence-transformers/paraphrase-mpnet-base-v2")
# Create trainer
trainer = SetFitTrainer(
model=model,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
loss_class=CosineSimilarityLoss,
metric="accuracy",
batch_size=5,
num_iterations=20, # The number of text pairs to generate for contrastive learning
num_epochs=1, # The number of epochs to use for constrastive learning
column_mapping={"text": "text", "label": "label"} # Map dataset columns to text/label expected by trainer
)
# Train and evaluate
trainer.train()
metrics = trainer.evaluate()
metrics
So you can see here that our model had 100% accuracy on the examples that were part of the test set, but let's ask ChatGPT to create some more examples for us to test on the completed model.
I asked ChatGPT for 2 new good examples and 2 new poor examples. 2 with the context of being a bartender and 2 as a customer service representative in a call center.
new_examples = ["One time, I had to deal with a challenging customer who was angry because their food was not cooked to their liking. The situation was tense because the customer was shouting and causing a scene. I remained calm and apologized for the mistake. I then offered to have the kitchen re-cook the dish to their specifications or to provide a different dish. The customer eventually calmed down and accepted the offer to have the dish re-cooked. The outcome was that the customer's experience was salvaged and they left the restaurant satisfied.",
"One time I had to deal with a challenging customer. They were demanding a full refund for a meal they did not enjoy. I didn't know what to do, so I just told them that the restaurant's policy didn't allow for refunds on consumed meals and that there was nothing I could do. The outcome was that the customer became even more upset and the situation was not resolved.",
"As a customer service representative at a call center, a caller became angry and aggressive because they were unable to access their account. I calmly explained the situation and offered to reset their password. I also apologized for any inconvenience and assured the caller that their account information was secure.The caller calmed down and appreciated the help. They were able to access their account and had a positive experience overall.",
"I was working as a customer service representative at a call center when a caller became upset because their order was not delivered on time. After a while the call disconnected and I made no attempt to call them back. The caller never called back and I had to deal with the fallout from my supervisor."]
new_labels = [1,0,1,0]
new_examples[2] # good response
new_examples[3] # poor response
# Run inference
preds = model(new_examples)
preds
from sklearn.metrics import accuracy_score
accuracy_score(new_labels, preds)
Again, we see that the model achieved 100% accuracy. Pretty good for training a model on 10 examples total.
Wrap Up¶
I'm sure we could give some mediocre responses that the model would have trouble classifying, but this was meant to be a rather simple example of demonstrating how you can use ChatGPT to create realistic responses to interview questions to train a model on and how you can use a few examples to train a reasonably good model with a few shot learning approach.
There are a number of ways we can leverage ChatGPT and LLMs like it in our work as I/O professionals. Can you think of any others? Feel free to leave them in the comments below and perhaps I can use the idea to write a future article.
