I thought ChatGPT could perform this straightforward task, but I was wrong
There are some things that ChatGPT cannot do well on its own
The power of generative AI was unleashed recently with the public release of ChatGPT. Many people are still trying to figure out what can and cannot be done with this jaw-dropping technology. As part of my own efforts on this front, I recently ran another simple experiment to determine if ChatGPT was able to perform a straightforward task.
Conclusion: I was disappointed by the results, but not surprised. The outcome helped improve my understanding of ChatGPT's (current) strengths and weaknesses, and how it could eventually be applied successfully in a small business setting.
The experiment
Rather than running an artificial experiment, I prefer to run these tests in an actual business situation when possible.
As part of my regular business activities, I needed to set up a meeting with a potential client located in the Eastern time zone. To protect the innocent, I will change the name of my correspondent. Let's call him "George".
The usual process, whether done manually or automated, goes something like this:
- Confirm which time zone George is located in (I need to know if I have to be available in the morning or the evening)
- Inform George, who in this case happens to be in the Eastern time zone, that there is currently a 14-hour difference with Japan
- Inform George that no, just because it is the next day in Japan, I still don't know the future in the US (I say this as a joke, and once in a while people find it funny 🙄)
- Ask for some times that could work for George
- Check against my own schedule, and propose one (sometimes more) slots that match
- Confirm the agreed time
- Determine the means by which we will meet (Zoom, or whatever)
- Send a calendar invite (usually by attaching an iCal file)
This task cannot be labelled as "simple" because there are a number of steps involved, but each step is easy to understand, and easy to execute. I therefore consider the overall task of "setting up a meeting" to be quite straightforward.
Now, performing this task manually is boring and repetitive. If it occurs often enough, it is a perfect candidate for automation. There are many possible ways to automate this task, but I thought I would see if ChatGPT was up to the task.
The context
First, allow me to set up the context of my interaction with George. Although the exchange occurred via email, I will write it out more like a chat, and remove irrelevant information.
I already started my interaction with George, and it went something like this (with irrelevant parts of the exchange removed):
George: Let me know if you have any time this week and we can discuss next steps.
David: Ok, sounds good. What time zone are you in?
George: We are the Eastern US Timezone.
David: I am in Japan, so it is the worst combination.
One of us will have to do very early morning, while the other does very late evening.
Do you have a preference?
George: I can do either. Evening is more flexible and I can do anytime after 8pm or in the
mornings I can do 5-6am and after 7:30am.
Let me know what days/times might work best for you, and we can confirm.
Normally at this point, I would have to figure out how to match times. Usually I use a simple application that helps plan international meetings, but this time I thought I would give ChatGPT a try.
The interaction
My interaction with ChatGPT was a bit painful, so I will cut out most of the exchange and only show just enough to get the picture.
Help me plan for a meeting. I am in the JST time zone,
while the person I want to meet with (George) has written this to me:
> I can do anytime in the mornings 5-6am and after 7:30am ET.
> Let me know what days/times might work best for you, and we can confirm.
I am available on Tuesday (March 7) after 8:30pm, or Wednesday after 8pm.
Write an email reply that clearly indicates the times for both parties.
----
Dear George,
Thank you for letting me know your availability. I am in the JST
time zone, which is 14 hours ahead of Eastern Time (ET).
Based on your availability, I propose the following meeting times:
- Wednesday, March 8, 8:30 PM ET (which is Thursday, March 9, 10:30 AM JST)
- Thursday, March 9, 9:00 PM ET (which is Friday, March 10, 11:00 AM JST)
Please let me know which of these options works best for you,
and we can confirm our meeting time.
Thank you for your time, and I look forward to our conversation.
Best regards,
[Your name]
I thought the response looked pretty decent, so I copied it into my email client and was just about to hit the "send button".
Then I noticed that ChatGPT got the times backwards. Instead of using my 8:30pm slot, it assigned 8:30pm to George, which is 10:30am for me. Not at all what I asked!! I had asked for 8:30pm my time, which would be 6:00am for George.
I thought that maybe a simple follow-up instruction could easily fix the problem. While I was at it, I thought that I may as well give it my name to replace the [Your name] placeholder.
My name is David, and you got the time zones mixed up.
George is available in the morning, I am available in the evening.
----
I apologize for the mistake in my previous response.
Here's a corrected version of the email you can send:
Dear George,
Thank you for letting me know your availability.
I apologize for the confusion in my previous message.
As you are available in the morning and I am available in the evening,
I propose the following meeting times:
- Tuesday, March 7, 8:30 PM JST (which is 6:30 AM ET)
- Wednesday, March 8, 8:00 PM JST (which is 6:00 AM ET)
Please let me know which of these options works best for you, and we can confirm our meeting time.
Thank you for your time, and I look forward to our conversation.
Best regards,
David
Meh.
It added my name and corrected the dates, but it is too "me" centric and adds unnecessary text. The dates should be written with George in mind.
I try again...
Write the dates like this: 8 March.
Also, make the data more apparent for George by putting the JST time in parentheses.
----
Sure, here's the updated email:
Dear George,
Thank you for letting me know your availability. I apologize for the confusion
in my previous message. As you are available in the morning and I am available
in the evening, I propose the following meeting times:
- Tuesday, 8 March at 8:30 PM JST (which is 6:30 AM ET)
- Wednesday, 9 March at 8:00 PM JST (which is 6:00 AM ET)
Please let me know which of these options works best for you, and we can confirm our meeting time.
Thank you for your time, and I look forward to our conversation.
Best regards,
David
ChatGPT did not understand my prompt.
I had to start over and retry a number of times. I will spare you the pain.
In the end, I gave up and just wrote the text by hand.
Once ChatGPT finally understood the actual proposed times, I got it to write out the data in iCal format. It did this pretty well, but this also required a few tries to get it right. I will skip the intermediary results and just show the final result:
Write out these times in iCal format so I can send an invitation via email.
----
Sure! Here are the proposed meeting times in iCal format:
----
BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//OpenAI//ChatGPT//EN
CALSCALE:GREGORIAN
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20230307T063000
DTEND;TZID=America/New_York:20230307T073000
SUMMARY:Meeting with George
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20230308T060000
DTEND;TZID=America/New_York:20230308T070000
SUMMARY:Meeting with George
END:VEVENT
END:VCALENDAR
----
You can copy and paste this text into a new text file and save it with the ".ics" file extension.
Then you can attach the file to your email message and send it to George.
I hope this meets your requirements! Let me know if you have any further
questions or if there's anything else I can assist you with.
Ok, this part was pretty nice!
The problems
ChatGPT appears to be quite weak with logic and computations. It had some trouble understanding simple instructions that a normal human would have understood.
It is possible to provide just the right prompt to get the desired result, but it is not easy. To make ChatGPT provide desired answers, the prompts have to be oh, so right. This requires a lot of trial and error, which opens up a new set of problems. Learning how to prompt becomes a chore and a field reserved for experts. I can easily imagine having to stock up on hundreds, or even thousands of prompts, and getting lots in all the mess. It will be necessary to manage prompts somehow, perhaps by keeping a prompt library or something similar.
Even once the perfect prompt is discovered, each language model will likely react to prompts in a different way, including the next version of the GPT model upon which ChatGPT is based. So, even if efforts are made to build up some kind of prompt library, the fix will only be temporary. The library will have to be completely rebuilt for each new language model.
In any case, it would have been much quicker for me to do the task manually than the time it took to get a decent answer from ChatGPT.
The conclusion
ChatGPT is amazing with language, but lousy at logic and calculations. Much of the grunt work should probably be done with the tried and true techniques of programming and simple templating. ChatGPT could call upon the programs and non-AI template engines to do its thing. In other words, mixing ChatGPT with other AI and non-AI techniques would likely produce a much better result than relying uniquely on ChatGPT.
So, to make the most of ChatGPT, we need to figure out how to embed it with the right mix of tools to relieve it of the tasks that it does not do well, and leave it to shine in areas it does do well: langauge generation.
Appendix
When I described the experiment in this article, I broke the task down into a series of successive steps. This way of designing the process is based on a logical, methodical approach.
Just for the fun of it, I asked ChatGPT to design a similar process to see if it could come up with something similar:
Write out the steps of a process for negotiating a time and setting
up a remote meeting with a person who is located in a different time zone.
----
Negotiating a time and setting up a remote meeting with a person
who is located in a different time zone can be challenging.
Here are some steps you can follow to make the process smoother:
1. Identify the time zones: Determine the time zones for both parties.
This will help you avoid confusion and ensure that you don't schedule
the meeting for a time that is inconvenient for either party.
2. Propose a few potential meeting times: Look at your schedule and identify
a few potential meeting times. Make sure to consider the other person's
availability and try to propose times that work for both of you.
3. Communicate via email or message: Send an email or message to the person
you want to meet with, explaining who you are, why you want to meet, and
what times you have available. Be sure to clearly state the time zone for
each potential meeting time to avoid any confusion.
4. Consider alternative communication methods: If the time difference is
significant, consider alternative communication methods that might be
more convenient, such as a pre-recorded video or audio message.
5. Confirm the meeting time: Once you have agreed on a meeting time,
confirm it with the other person. Repeat the time and time zone to
avoid any misunderstandings.
6. Prepare for the meeting: Make sure you are prepared for the meeting
by testing your technology, setting up the meeting platform, and
having any necessary materials ready.
7. Attend the meeting on time: Attend the meeting on time, and be mindful
of the other person's time zone. If you need to reschedule or cancel,
be sure to communicate this as soon as possible.
8. Follow up after the meeting: After the meeting, send a follow-up
message thanking the other person for their time and summarizing
any action items or next steps.
The results are interesting, but in my opinion not practical. It is truly fascinating that such human-like text can be produced by a machine. It is surely entertaining. However, it is not useful in a business context. At least, not yet.