Michael Levitz (00:00)
Hello and welcome to episode two of the forecasting podcast. How are you doing, Robin?

‍

Robin (00:05)
I'm doing well, how are you?

‍

Michael Levitz (00:06)
Good, I'm glad we are trying to do these weekly on Fridays so that we can actually build out loud and remember all the things that were super important this week that will be a distant past by next Friday.

‍

Robin (00:20)
Yes.

‍

Michael Levitz (00:20)
Think of all the genius that we've lost over the last year.

‍

Robin (00:24)
Yep, it's all gone.

‍

Michael Levitz (00:26)
Yeah. So we did promise that we'd have a name this week for our listeners, but we don't have any listeners because I didn't publish the one from last week. So I guess we're in like an interesting loophole.

‍

Robin (00:36)
We are Nobody will know we're committed to Fridays yet. I feel never say what the actual day is.

‍

Michael Levitz (00:42)
That's a trick, all right, so we have to learn that. So today, for episode two, we are talking about basically why we decided to connect game theory to content marketing.

Do you want to kick it off?

‍

Robin (00:55)
Sure.

ultimately, what game theory is, is the idea that intelligent actors can make actions based off of information that they have access to. Actor there, right, is just any entity capable of making a decision of impacting a game and sure the

term

is game theory, but this isn't just, you know, settlers of Catan. This is just any environment in which you are influencing the outcome of that environment.

through your actions. So a marketer is influencing the marketplace through their actions. They are sending emails, they're writing blog posts, they are influencing all these things based off of the information that they have. And that information can be the KPIs that they're getting out of their email service provider. It's information about the brand's current positioning, about the industry overall, and the marketers that making actions based off of those that have.

payouts. know, will receive more open rates, you will receive more clicks to your site and all of that. So we're not here to go on a long rant about the prisoner's dilemma and the trolley problem and all these other kind of fun aspects of game theory, but we're here to just talk about how marketers can optimize the expected value of their campaigns through some of this kind of bread and butter.

allocation of resources and kind of strategic decision making based off of the information that they have.

‍

Michael Levitz (02:14)
But can we reserve the right to rant about those on future episodes?

Robin (02:18)

‍
Yes.

We will have one episode dedicated to each of them. To all the philosophical questions. We're have a Thesias' ship episode.

‍

Michael Levitz (02:30)
Actually, something I never told you until right now is that Peter Nikolai, and I went to grad school together and we built a multiplayer game based on Prisoner's Dilemma.

were a temp worker robot.

everybody's goal was to slack off and do the least work possible without getting caught by the boss. And the way you got caught by the boss was by doing too little work. And there was some collaboration and some kind of, gaming out, how much or how little did you do in any given moment to let the boss look at someone else.

‍

Robin (03:01)
My quick prisoner's dilemma anecdote is I went to Chapman University for undergrad, which has this large economics school, and they run these experiments where you can go in as a student, right, and you'll be paid for your time to be a lab rat for the economics department.

So I think, like, you know, the thing they told me was, come in, you'll be here for three hours to do this experiment. And I go in and I'm sitting at the computer and they do like 45 minutes of training about how to use this thing. And then, you know, we go into the room to do this experiment and I'm like presented with the screen of like, just this massive flow chart of payouts and, know, I'm going to make a decision. And then my anonymous, just the exact prisoner's dilemma, my partner is going to make an anonymous decision and we're going to navigate.

this

tree together. So I go in, you know, and I'm faced with like the first decision is, okay, both of you will be paid $7 Or I can just, you know, take $9 right now. So I, you know, make the decision to play to not rat out my partner.

then immediately they just take the $11 payout. So it's like, the whole experiment was 15 seconds, you know, after 45 minutes of training and all of this to just, all right, I'm 19 or whatever, learning the hard lesson that, yeah, the prisoner's dilemma. I lost my $4 or whatever.

‍

Michael Levitz (04:17)
And did you get to see the person after that?

‍

Robin (04:19)
No,

But you walk out of the room, you know, evil eyeing everybody there because it was like everybody that got eliminated after the first round walks out together, you know.

‍

Michael Levitz (04:29)
That is classic. the question is, you know, how did we decide to connect us to content marketing?

I spent two years content marketing for Pampers with P &G. Their research was phenomenal. They basically had people traveling the world, talking to moms and dads And they, know, package those insights up, they operationalize them, all that stuff was happening. But then as me, the person who was at that time in charge of

the email workstream, like accessing or activating those insights almost didn't happen. You can imagine the size of the list, the kind of segmentation, the amount of pressure, to actually

activate those insights and kind of like make that fresh week over week and work within kind of the limitations of you know the way the brand wants to present itself was incredibly a tough needle or thread and for me ended up with me you know often playing it very safe.

There's not a lot of risk reward being in that position. here to not damage this very premium, most trusted brand in the world for parents.

So what got me interested in game theory was basically how do we get kind of analytics and like data that we can actually use to make that kind of work stream more interesting. That's not so abstract.

You know, and when you really started introducing me to game theory on this level, it was the first time I started realizing that there was this gap of like, it's not that there aren't great insights and great research, it's that it didn't map to my reality as a content creator with like a ton of deadlines and a ton of channels and a ton of different audience perspectives.

‍

Robin (06:05)
one of the things that stands out there to me and one of the reasons that I got into game theory overall is just this Bayesian view of the world where...

you as the marketer are sitting there and you have these kind of different actions you can take and you have different amounts of information available to you about each of these different things. And every one of these little decisions that you're making can be kind of viewed as this like intelligent gamble, this intelligent play.

Bayesian inferences all about priors and posteriors. You have some prior belief about the likelihood of something happening, and then you sample some event, and then you update your priors So, you know, you're going to send a campaign and you believe that this is going to be somewhat likely to succeed. And then you send it out to kind of a canary test, a limited sample, and see that it's starting to work. And then that can kind of update the posterior.

to then roll that out to the full list. So that kind of workflow can be like, it can be hard to figure out how to kind of balance that across a bunch of different mechanisms and channels and to know at what point you're capable of making the most strategic decision and how often you need to kind of gain more information rather than start executing some of these campaigns.

But I do think one of the real core things we're trying to do at Forecasting is simplify some of that decision making and just make it more front and center about, hey, here's what your priors can be about the relevance of an individual marketing theme to your audience. People care about marketing topic A because we're seeing an increase in

know, search volume for it, we're seeing an increase in kind of external signals that are relevant to that campaign, and thus this is kind of worth sampling for the next campaign.

‍

Michael Levitz (07:43)
Yeah.

remember when the tech was execution and you didn't start, right? Like a line of code until everything was figured out. And then you're in this waterfall pit of doom where anything you learned or realized during that process was a penalty because that meant changing the commitment that had been like planned a year ago. So

tech, think, was freed from that kind of constraint, and it became much more of this iterative process where we're gonna learn and we're gonna adjust along the way. We're not gonna just put these blinders on and execute a requirements document. In a way, content may be trapped in that kind of blinder mode because...

the editorial calendar gets defined, you know, by, senior people gets agreed upon. And now, six months later, it's accessory Tuesday at this e-commerce brand. And you have to send that accessory email, regardless of, what's actually happening with your audience and the products, or it's a thing where like, Hey,

had

an awesome month now that's the baseline of expectation I have to deliver that performance and if I use a couple emails to experiment with something I may miss my targets and

there's a lot of pressure to deliver on that.

‍

Robin (08:56)
yeah, I think that's very interesting. I mean, that's kind of that idea of like, you have new prior information, you have seen much observations about Accessory Tuesday for the last six months, but you're never actually kind of revisiting the true value of Accessory Tuesday, so.

like in this kind Bayesian view, right? You'd always be updating and kind of reprioritizing and realizing, all right, you know, either Accessory Tuesday can kind of drift into something else or be revisited. And, you know, that does, think, tie to that kind of like...

waterfall to agile engineering transition where, you know, this classic agile metaphor of first you build the scooter and then you build the skateboard and you end up with a motorcycle rather than just building the steering wheel for the Ferrari first is, is applicable there where the marketer could often feel like, yeah, you know, I have all this new information about accessories, but I'm still just following the roadmap that we created nine months ago.

And I think that can be kind of, it leads to, I would say, worse campaigns, but it also leads to just kind of this frustrating situation where you just know that these systems aren't changing and you're bound to them.

And you have to play by the playbook that you created nine months ago and you're just stuck there staring at the blinking cursor, What do I even write that's aligned with the unflinching playbook that we created X amount of time ago? And, I don't think this campaign will be as relevant or useful as we think it will be because all this information has been

Michael Levitz (10:28)
Yeah, so I think maybe a big part of what we're doing in this game theory is like first and foremost, just getting people to think about their content workstream as a dynamic changing landscape and not one that's fixed, not one where, okay, we're gonna do these three things and we're just gonna redo them until the next annual planning meeting.

‍

Robin (10:50)
Yeah, and I think the other introduction there too is just that your content marketing decisions are a game that you have information about. Like in game theory, there's the idea of full information games versus partial information games. A full information game is...

Chess both agents taking actions in that game have full knowledge of every action available to the up to the opponent partial information game poker you have some amount of information about the board state about the position that you're in and Generally speaking if you can get more information, know If a game approaches a full information game from being a partial information game You are capable of taking more data driven decisions You are capable of calculating the expected value of your actions

more. You are capable of making kind of more intelligent decisions in this kind of zero-sum game. So like...

Yeah, know, part of this just, hey, if you start thinking about this like a, you know, a game theory example, and you start thinking about this as a game of partial information, you can start gathering data intelligently and you can start allocating resources accordingly. And really like behind the scenes, you know, that should give confidence. It does give confidence. I mean, it is like the data is correct, but also just, allows you to kind of...

you know, make correct decisions today based off of the position that you're in.

‍

Michael Levitz (12:11)
And that's, I think that's super interesting because, you know, as I started working with you, you know, I thought about like blinking cursor. thought about, you know, the pains of, my, you know, writing process of procrastination and, you know, research, you know, over researching stuff, always thinking I'm going to find the perfect stat and then four hours later.

You know, maybe I find it, it's not, it doesn't do the thing I wanted or I never find it. Now I'm out four hours of, you know, kind of wasted time. And I think that Tangent just went on as a perfect example. I'm now illustrating my research process, but to bring it back, you you started helping me really just break down, like, what are those decisions? What are those actions? When we think about content marketing, you know,

Publishing just kind of a lot of stuff gets bundled into that, especially on very lean teams. How do we then kind of start to unbundle those actions and decisions so that you can get some data around them?

‍

Robin (13:07)
I mean, think one of the most relevant exercises in the beginning is just starting to figure out what are the levers that you have in play and what are the hierarchy of those levers. And, one of these initial levers can just be that decision between like,

What is the marketing stream that is most relevant? Right? Are we getting more returns from the blog post than the landing page? And then you can kind of dial it down deeper there and say, all right, what is the topic that is relevant for the landing page? And this is, you know, one of the kind of core areas what we're trying to do with forecasting is help you answer that question. And then once, you know, like, all right, here's kind of the things in play for the medium with which I'm marketing and here's the things in play for the topics that I'm advertising to, then you get

into

that next hierarchy of, right, what is the actual content or what is the structure of the content? What are the levers that I have to interact with the structure of the content? And then perhaps finally, what is the actual copy that executes this campaign? And if you start thinking about each of those as distinct steps and prioritizing which of these steps can have the highest return there,

you can make better decisions overall. think one of the things that we've kind of learned too is like, people can find themselves.

feeling that there's only one lever that they have access to. you know, they could just a hypothetical, right? Like, oh, I'm only ever going to think about tweaking the subject line of my emails that are going out, but everything else is going to kind of remain static. And really, if you do that, like you can get some gains, but you're only ever playing in that field of what's actually the maximum gains you can get from tweaking the subject line, where five hours of tweaking a subject line might not be as valuable as...

20 minutes of kind of prioritizing which topics are most relevant. So really just trying to understand in this kind of information game, what do you have access to? What are the actions you can take?

‍

Michael Levitz (14:59)
I think like my

my mode as a content marketer is to rush through exactly what you just said. And I think this happens a lot where, you know, there's either you receive a, you receive direction and you just kind of jump to execution or you, what I do is panic. And, know, I don't like being in the unknown. So I like jump out of that. I jumped through the first like four steps and then, I kind of like this subject line hook, you know, now let me get to the intro and.

I end up going back and back and this isn't working and I'm frustrated. Or like, it's awesome and I publish it but the audience isn't interested because they didn't actually start with kind of what they were thinking about in the moment. So I think like going through, if there's a way to kind of lead a person through those kind of phases, then you arrive at a place where your time is actually being invested in the right thing.

‍

Robin (15:51)
Yeah, I mean, think like the game theory terms for a lot of this in my mind are this trade-off between exploration and exploitation. Exploration is your ability to start taking different actions, exploring different games. Exploitation is kind of how profitable each of these individual endeavors are. like, in the kind of example I was giving earlier, right, if you're just like only ever kind of dialing in on

lines you are kind of further exploiting topic A's subject line B where there's often a position where you actually want to start exploring some of these other options. You want to start exploring either new mediums to market in or new topics or new subjects, just any of these other kind of levers that you have available to you. if you are, you know, just like under

intense deadlines, it will often seem easier to just kind of pursue that path of exploitation, where with forecasting, you know, we want to surface the exploration side of it and make that a little bit more, you know, make that more streamlined, make that more accessible, make them more visible to you, the marketer, and start allowing you to then kind of, you know, at a lower cost, because we will do that kind of research for you, start getting some of that.

ability to start doing the exploration side of the equation.

‍

Michael Levitz (17:05)
Yeah, and I think one of the addictive or kind of addictive slash trapping pieces is an interesting thing with email is you will hit a point where you have this critical mass of people.

if you're talking about a certain topic that will generally just open your emails, you know, so you're, you hit a point where, you know, you've kind of lost some people that aren't interested in that stuff, but you've got like sometimes like a really big base of people, that are just going to kind of.

follow you along that path. But as a result, there's an opportunity cost where you're a, managing decline, because a certain amount will unsubscribe or just stop reading every week. And you're not re-evaluating what the space is and what the opportunity cost is of not going into these new areas.

‍

Robin (17:53)
Yeah.

And I mean, think that same pattern exists not just in email, but in the other kind of channels people have available too, right? Like kind of jokingly, but like if you go to your daily active users and your site traffic, you'll most startups will see that they have a daily active user base of people trying to go to their site slash WP.admin, Slash admin slash passwords, all these things. It's just bots trying to like, you know.

mine poor security practices. But yeah, like, and that's, you have some of these just kind of static things where sure you will have audience members that will always open your thing. You will have.

bots that are always trying to scrape your thing. Your mother will always say your product is a good idea, right? There's like all these constants, but being able to kind of underscore, you what are the constants versus what are the actual like realms of opportunity and realms of exploration becomes this that that is what this game that we're talking about ultimately is.

‍

Michael Levitz (18:52)
And then, okay, this has been amazing and we are coming up on time. I think kind of maybe the last topic here is...

I feel like we promised not to wax away on Prisoner's Dilemma, but let's touch a little on Multi-Armed Bandit. Because I think from the very beginning, that's been kind of core to how we think about content marketing. Do you want to just talk a little bit about your experience on Yang Campaign and then at forecasting?

‍

Robin (19:21)
Yeah, so multi-armed bandit is a favorite term of art of mine. It's just a fun thing to say and a good way to impress people in a meeting. But ultimately what multi-armed bandit is, is it comes from the world of gambling, many of these game theory terms do. Bandit is a word for a slot machine.

and the arm, you you have the arm on the side that you pull to spin the wheels. So multi-armed bandit is you walk into a casino and you see 10 different machines and behind the scenes you are unaware but each of these machines has a different expected payout. And I have a hundred dollars walking into that casino.

And I want to maximize the amount I walk out with. So I will put a dollar into each of these different machines and I'll see what happens. And then I win on machine number three. That's nice. And I lost on machines, all the other machines. Okay, what do I want to do now? I have, you know, another $10 that I want to spend now. So let's put two of those dollars into machine number three because it just had payout. And let's still put a dollar in every other machine. And you know, you were, now machine number eight goes. So think machine number three and machine number eight,

those

are looking pretty good. I keep doing that and then ultimately, know, machines three loses, machine eight pays, machine six, you know, so on and so on. But I'm gaining information every step about the expected payout of any of these machines. And that's, I'm exploring the machines by giving a dollar to them conditionally based off of my perception of their payout. And I am exploiting them by putting more and more money into them as this game goes on.

So that's like the kind of textbook example of what a multi-armored bandit is. When I was the director of data analytics for Angie Yang's presidential campaign, the way that plays out is my...

Casino here is the American populace and I'm trying to explore which districts I'm most likely to succeed in to raise funds to find volunteers, to find supporters, all the actions I want. Which districts are they in? What's the socioeconomic backgrounds of these people? These are all these kinds of things that I'm exploring. And the mechanism, I don't have slot machines, right? But I have email campaigns, text campaigns, landing pages, volunteers I can send. I have all these different ways

that I can interact with the kind of constituency that I'm trying to engage. And I can see the payout by seeing, all right, well, now I'm getting five new volunteer signups in Nebraska, whatever it may be. So that's kind of how this game plays out, where I'm constantly kind of like...

to use the Bayesian term too, I'm sampling some distribution. There's some real number of how many people in Nebraska are likely to be Yang supporters and by putting more and more money, putting more more campaign resources into Nebraska, I'm increasingly unlocking knowledge of what that kind of true distribution is. well, maybe it'll turn out that Nebraska is a better place for us to kind of allocate our resources than South Carolina, whatever it may be.

So that's kind of the quick tour of Multi-Armed Bandit. But yeah, the relevance here is just kind of what we've been talking about through this podcast already, where the marketer, sure, you might already know what your audience segment is, but you have all these different kind of slot machines that you are.

putting the money into, both in terms of the topics and the copy and all these different actions you can take. And you do want to be kind of balancing that, like, at what points am I exploring new themes? At what points am I exploring new audience segments? And at what point am I kind of playing with the knowledge that I have where I know that this kind of campaign will work? Because sometimes Accessory Tuesday is a good idea, and sometimes it's not. And you want to be pretty prudent about, well,

The expected value of Accessory Tuesday this week is X amount and we should take action based off of that. And next week, the expected value of Accessory Tuesday is X minus one, but the expected payout of...

Membership loyalty reminder email is X plus one and thus you know membership loyalty program is the highest expected value and it's kind of the action that should be taken that week.

‍

Michael Levitz (23:23)
if we can get content marketing to think in expected value, I think that is huge and music to a lot of people's ears. I did lie. There was one other thing we wanted to talk about before we go, which is why now? So like this stuff's been out there, but it's kind of, I think we feel that it's particularly accessible now when it wasn't before. What are your thoughts on that?

‍

Robin (23:44)
Well, I think there's like technical enablements that allow this to happen now. mean, I think one of the.

byproducts of kind of the GenAI movement is the improvement of the state of the art for both encoder and decoder models. And just a very quick explanation of what those are. Encoders are a thing that can take text and put it into, think called a vector space, math, ultimately that computers can read. And the decoder is a thing that takes math and turns it into words. So ChatGVT is a decoder. It can take kind of math and turn it into words. And that's why you can

a chat interface with it. But encoders, the inverse of that that takes text and turns that into numbers allows us to do a lot of this kind of quantitative analysis that we're doing at forecasting where we say, all right, this is why we think this topic is actually most relevant this week. This is why, you know, we think this landing page is appropriate for this week. So there are like kind of

know, innovations in the industry that are allowing this kind of analysis to be done at higher scales than we've ever seen before. There's also a bit of that kind of thing of like,

computation is getting cheaper and cheaper over time. So we have it's cheaper and cheaper to create the into into fatigable beast that can crunch all those numbers for you and kind of give you the just in time topics where you just say like, all right, you know, here's last night, we crunched all the numbers for you. And today you can sip your coffee at 8am and have the most recent analysis about which topic is going to be the most successful.

So I think that's all the kind of like technical side of it. But the other reason I think is relevant for the why now is we are seeing, you know, a constriction in marketing overall where it's, know, people are, there are, there are layoffs. It's getting harder to do the work. So enablement of allowing.

marketers to be able to do more work is relevant. And also we are seeing kind of an increase in the quantity of content that's being produced by like, Gen.ai writers. So I think for us, we have the belief that, you know, as that kind of sea rises, it'll be more and more relevant over time to have relevant content for your audience and being able to do this kind of, to calculate the EV of the campaign that you're writing.

will allow you to outperform somebody that is not doing that. And then you will rise, you will always be the ship that sails across the ocean rather than the ship that just kind of gets lost as the tides rise around you.

‍

Michael Levitz (26:12)
All right, well, this has been fantastic. Any last thoughts?

‍

Robin (26:15)
No, start thinking about some of these themes. I think for lot of people, you don't need to necessarily formalize all of these things in pure EV. You don't need to exactly calculate, well, the expected revenue from this action will be this amount. A lot of this stuff is intended to be relational, where like...

the perceived, you know, EV of this action is viewed to be more like,

If 60 % of the time the EV of this action will be likely higher than the EV of another action, like that's what you're looking for rather than just like, can I perfectly dial in, you know, the calculation of what the exact EV for an action is. This gets back to some of that Bayesian stuff of just, well, in most realizations of what the EV of this action is, it'll outperform another action.

‍

Michael Levitz (27:01)
Yeah, and you just brought up another thing I'm jotting down notes for future episodes, which is the riff on tanking. in poker, you know, you spend too much time thinking about that last bit

Don't sit there just trying to get it absolutely perfect because you're only allowing emotions and second guessing to creep in that's going to reduce the quality of the final decision.

Alright, lot of good stuff coming up. Thank you so much. You're on your way to Malibu. Well, I'm on my way to downtown Jersey City. You won.

Have a good weekend and we'll talk to everybody next week.

‍

Robin (27:31)
Yep.

talk to you later. Bye.