GPT-2 experiments with AI-generated copy

Do you remember hearing about GPT-2, this AI so powerful OpenAI didn’t release it to the public? It made for great headlines but they were really worried that this “large-scale unsupervised language model” would be used to generate fake news at scale or to impersonate other people online. They put out a comprehensive release strategy and only shared a very small version of GPT-2 (that’s how this thing is called).

Six months later, and in spite of some research showing that “humans can be convinced by synthetic text,” larger models were released. It was not until our Giving Day was approaching that I thought of giving it a try.

What if?

Cornell has been having a Giving Day since 2015, so this year was our sixth. Each year we try new things and learn from past mistakes. This year I wondered: What if we used GPT-2 to generate email copy? We write and send hundreds of emails for Giving Day. All links are UTM tagged and we can track performance very easily. Wouldn’t it be a great way to find new ways to talk about Giving Day? We’d have direct access to conversion data to compare the effectiveness of our AI-generated messaging.

I found a simpler Python wrapper for GPT-2 and started playing with it a bit. Angie Giammarino, the brilliant mind who writes all central copy for our Giving Day shared all the emails she wrote for 2018, 2019, and 2020 to fine-tune the model (around 4,500 words). So I started feeding the beast.

Workflow

I love Python and one of the reasons is virtual environments. I love being able to spin something in an enclosed environment and trash it if it doesn’t work, without adding any garbage to my machine. So here are the few steps to get up and running with GPT-2 Simple (not pictured here, mk your own dir and cd into it):

I skipped the GPU stuff for Tensorflow because I don’t really have one on my machine. You might want to look into it (it should make things much faster).

The next step was fine-tuning on all the Giving Day emails following the GPT-2 Simple instructions.

Getting warmer… Getting warmer…

The fun part was tweaking the temperature. I can’t find a good definition of temperature in the context of text-generating models but here’s the important thing: The higher the temperature, the crazier the text gets. To prove it, let me share some gems:

temperature=1.7, prefix=Cornellian

Cornellians can make an additional $.00 in stock on every rotation!

temperature=1.7, prefix=Dude

Dude.. please don’t just join receiving boxes full of favors and flowers— you’ll make a big difference for both teams.

temperature=1.7, prefix=Touchdown

Touchdown default option
Competite For Giving Cards on This Day? Let us know at [email protected]

Touchdown is Cornell’s mascot. This email simply doesn’t exist. I’m not sure where the cards come from but I love the verb competite. Compete with appetite?

temperature=1.7, prefix=Today

Today, iba’s real iba’s better than you know it will be. For iba’s No. 11, your empire’s of trees, hats, and poppies is growing!

That’s when the edibles hit for GPT-2. Iba? Empire of trees? Tell me more, GPT-2…

So I kicked it up a notch, pushing the temperature to 2.

temperate=2, prefix=”Far above”

Far above pointing bomber rail miles from revolutionary VievosPhone.connection variables pierced grape solar ale college regalia. Rather than Rainbow Bolivia Shippingargeʼ­lbs, Motion Expresses your rebellionbay confidence by signing up to be a Crisis Scholarslayer

Returns and factory confirmed!

Far above are the opening words to Cornell’s Alma Mater. This has to be the best CTA ever: signing up to be a Crisis Scholarslayer.

OK, this is fun but what do we do with this?

After spending a lot of time fine-tuning the model and generating text at different temperatures, declaring start and end of text for each email, trying different prefixes, etc. I was not getting anything of quality. The first two lines were usually non-sensical and the rest was pretty much copied verbatim from the emails.

What was happening? Was it working? I had to try something else to make sure I was doing it right. This is when I turned to Bon Jovi.

Highway in which these young girls were born

I decided to use song lyrics to fine-tune the model and generate some. And I chose Bon Jovi, of course. I found a notebook containing lyrics for +55,000 songs from which I extracted all those of the New Jersey nightingale, for a total of 19,000 words.

And it worked. I am proud to present this GPT-2 generated Bon Jovi hit.

Highway in which these young girls were born

(Abi Dawood, “Who Could Have Took these Girls?“)

It was one of the most beautiful and least heartwarming experiences that you could ever have.
But I can’t watch you.
When you have to scream out in pain?
I can’t close my eyes.
It’s all so black in my head.
I’m drowning in my own blood.
It’s all so too real.
I can’t close my eyes.
I can’t close my eyes.

I got there with my heart, I got there with my words, I got there with my prayers.
I could sit, sit, and sit.
I could feel you on the other side of the line.
But you’re not there, you’re just there.
I got there with my soul, I got there with my prayers just the same.
Now I don’t want to be alone, I don’t want to be alone.
Now I don’t want you to be alone, I don’t want to be alone.
That’s just me.
I get in the car at midnight, drive me home, drive me home.

I got there with my heart, I got there with my words and my prayers.
I could sit, sit, and just sit and let myself go.
I could feel you on the other side of the line.
But you’re not there, you’re just there.
I got there with my soul
I got there with my soul.
Now I don’t want to be alone, I don’t want to be alone.
I got there with my soul, I got there with my prayers.
I could sit, sit, and just sit and let myself go.
I could feel you on the other side of the line.
But you’re not there, you’re just there.
I got there with my soul
I got here through the cracks, deep in the heart of the matter.

I got there with my heart, I got here with my words and my prayers.
I could sit, sit, and just sit and let myself go.
I could feel you on the other side of the line.
And you’re not there, you’re just there.
I got here with my soul, I got here with my holy words.
Now I don’t want you to be alone, I don’t want you to be alone.
You’re the one we used to love.
I get there with my soul, I got there with my holy words.
I tried, I tried, I tried.
I tried, I tried and I still got just the same thing.

If this isn’t epic, I don’t know what is. I didn’t touch a single character. This is what GPT-2 spat out, verbatim.

So it works, right?

Yes, GPT-2 is an incredible tool but it just didn’t work for the Giving Day email copy. Why? My theory is that:

• there was not enough entropy: the language used in these messages was similar email over email. Small variations in segments didn’t create enough change and a lot of expressions were too salient.
• these emails were individually too short to have enough of an arc for GPT-2 to build on

What do you think? Have you tried AI for marketing copy? If not, give it a shot and tell me how it worked for you.