The Future of Work Podcast

[music]

-Hello, and welcome back to the ILO's Future of Work podcast.

I'm Sophy Fisher.

Advances in generative Artificial Intelligence, or AI,

will have widespread implications for labour markets and production.

Whereas in the past automation

has been seen as primarily affecting low-skilled jobs,

such as those in agriculture or in factories,

AI is different.

Many say that the effects of tools

such as ChatGPT will be on more highly skilled jobs,

the so-called white-collar workers.

I have with me today two people

who are well qualified to throw light on this question.

Shakked Noy and Whitney Zhang

are both with the Massachusetts Institute of Technology,

better known as MIT, which is in the United States.

They've published a new paper,

Experimental Evidence

on the Productivity Effects of Generative Artificial Intelligence.

Shakked and Whitney, welcome.

Thank you very much for joining us on the podcast.

-Thanks for having us.

-Thanks for having us.

-First of all, why don't you run us through the key findings of your paper?

-Yes, so the way we typically summarize it,

our paper has four main findings.

The first is that access to generative AI,

in this case, ChatGPT-3.5,

increased the productivity of medium-level white-collar professionals

on the sorts of writing tasks

they might typically encounter in their jobs.

It raised their productivity both by

allowing them to complete those tasks faster

and by slightly raising the quality of the resulting output.

The second finding is that

the productivity gains were larger for workers

whose initial ability levels were lower.

They gained about the same in terms of increased speed

as higher-ability workers but gained much more in terms of quality

of their output.

The third finding is that workers' initial reactions

to encountering these technologies seemed to be pretty positive.

People were more satisfied

with the work that they'd done and with the task

when given access to ChatGPT

compared to when they were asked to complete the task

without access to that technology.

Then finally, when we followed up with the workers

who'd completed the tasks in our experiment

and asked them about their activities in their real jobs

two weeks and two months after the experiment,

a substantial fraction of them,

30-40% were using ChatGPT in their real jobs.

The percentage using it

was substantially higher in the treatment group

whom we'd introduced to ChatGPT compared to the control group

whom we had not introduced to ChatGPT,

suggesting that there are still frictions or barriers

to diffusion of the technology in real workplaces.

-Right. To sum up your first two findings,

essentially it makes higher-skilled workers quicker

and lower-skilled workers both better and quicker.

-Exactly, yes. -The consequences for this are

potentially greater productivity.

What do you think the consequences are for employment and wages?

-This, I would say, depends on a number of factors

that we can't directly address in the experiment itself.

In particular,

if we think about things on both the demand and the supply sides,

if there's this increased productivity

and workers are able to produce more than on the demand side,

there's this question of,

are people going to demand these types of outputs more?

For example, if we think that the demand for writing,

or advertisements, or coding would substantially increase,

then this could suggest an increase in employment and wages.

Whereas if that demand is relatively inelastic, then perhaps

this would lead to a decrease in jobs overall.

Then just thinking about the supply as well.

One thing is that our tasks are around this very specific part of

what most people's jobs are doing this kind of writing.

Whereas a lot of tasks in the real world

are much more loose and require a lot more judgment and planning,

and these other factors.

Then just also thinking about if this is 5% of someone's job,

for example,

maybe I can take that 5% out and then replace that person entirely.

This just means that they have 5% more time

or they're going to fill that time with other work.

It also depends to what extent these tasks

are the main part of someone's job

or just some small part that's then complemented by other pieces.

-The subjects in your experiment

were basically doing writing tasks, weren't they?

Which required them to synthesize information

and then produce a piece of text.

That's correct, isn't it?

-That's right. -What other skills do you see

this applying to?

Would it simply be that?

-I think it's a potentially very flexible technology.

We're only beginning to see the range of applications

that the generative AI systems can be applied to.

I think definitely writing tasks like the ones in our experiment,

in the sense that they're relatively generic,

they're not extremely technical,

are one main application for generative AI.

There's also lots of other things.

It's very good at writing code, for example.

It can be good at other things like summarizing documents,

synthesizing information,

maybe even as a cooperative tool for brainstorming or coming up with ideas.

It's, potentially a technology with a lot of applications.

-I would think your findings might be of concern to high-skilled workers

who are hearing what you say because essentially,

someone could employ a lower-skilled worker

who would presumably be cheaper

and get almost exactly the same results as a worker

who might potentially be more expensive.

-I think limited to the specific set of tasks,

I would say yes.

I think, in some sense, that's what you would find here.

At the same time,

again,

I think a lot of these times these tasks are bundled together.

Potentially, when you're paying for these high-skilled workers,

you're also asking in some sense for their skills and judgment

or interpersonal communication

or their ability to work with others on a team.

These aren't things that I think ChatGPT right now would be great at assisting at,

and I think it's still our important skills in the workplace.

Depending on what that bundle of skills looks like in the job,

I think that would ultimately predict what is going to happen.

Then I think the other thing, this relates back to

complementarity versus substitution more broadly,

is that here, these initial low performers are doing much better,

but they're doing much better because they're just taking this prompt,

putting it into GPT,

taking this output, and then submitting it is what we generally seem to be finding.

Then there's also a question of,

well, do you even need these low-skilled workers in the first place?

Or can you just hire someone to do this prompting?

That would ultimately be more efficient.

In that case, I think it becomes even more unclear

what the impacts on inequality might look like.

-Did you look at all that the impact on inequality

or was that something that still requires more research depending on

how the technology is rolled out?

-We looked at an important input into the ultimate effects on inequality,

which is by how much it raises the productivity of lower-ability

compared to higher-ability workers.

There's a common argument

that inequality has increased a lot over the past several decades,

at least in part because technologies like computers

enhanced the productivity of high-ability workers

more than they enhanced the productivity of low-ability workers.

Which gave rise to divergence in wages and earnings.

In this case, what we found, again, limited to a very specific set of tasks

in a very specific context

was the opposite, which is that

the productivity impacts were greater for lower-ability workers.

A bunch of other factors will determine

what the ultimate equilibrium effects of

these technologies on unemployment will be,

like some of the demand side factors and supply side factors

that Whitney has talked about,

how training and human capital acquisition will adapt,

how companies will restructure their workforces

and their models of production and things like that.

It's really impossible to tell from current information

what the ultimate impacts on inequality will be.

-Higher productivity is often associated

with reductions in poverty and increasing wages.

In that sense, it might actually reduce inequality.

-Absolutely.

Productivity has been stagnant for a long time,

even as we've gotten fantastic new things like the Internet.

I don't think there's a consensus on why, for example, the Internet

hasn't resulted in higher productivity growth than we're currently seeing.

It suggests that things like

bottlenecks from other parts of the production system

or other factors might be important.

I wouldn't be super confident from the fact

that generative AI raises the productivity of white-collar workers specifically,

that this will translate into the kinds of

broad-based productivity gains for the economy that would translate into

increases in living standards and reductions in poverty.

I think it's too early to tell.

-I wanted to pick you up on one particular sentence in your paper,

which said that the experiment's evidence suggests

that it largely substitutes for worker effort

rather than complements worker skills,

potentially causing a decrease

in demand for workers with adverse distributional effects

as capital owners gain at the expense of workers.

By which I think you mean that the people who have the money and the capital

will gain a higher proportion of the benefits.

Would that be correct?

-That's right. -This action is taken to shape the impact,

be that through legislation or taxation or something like that?

-Right. I think this goes back to

what we were just talking about in terms of

what are these workers actually doing in our experiment,

and it's that they're taking these prompts,

they're putting them into ChatGPT, and then they're submitting these outputs.

It seems plausible that you could,

if that's all that they're adding value to,

then you could have someone where that's their job.

You don't need all of these workers to be doing that kind of task.

Who then takes all of the profits

that come out of doing this kind of output generation?

Then it's the people who own this type of capital in the first place.

-Do you think it is also possible to shape the impact

and the way that this is applied through other kinds of institutions,

for example,

stronger worker representation, greater voice in the shaping of processes,

unions, things like that?

-Absolutely.

I think this is an area where there's been a lot of speculation

because there's a very large amount of anecdotal

or qualitative evidence about the ways

that unions or other forms of worker representation

are involved in shaping the way that technologies affect firms

and shaping how firms adjust their employment structures

in response to new technologies.

I would say there's still very little solid evidence

on the interaction

between systems of worker representation and technology adoption.

You can tell optimistic stories

where worker representatives protect workers

and encourage firms to upskill them,

thereby ensuring that technology gets rolled out in a way

that really benefits everyone,

benefits the consumers through productivity gains,

but also benefits workers

through internal productivity gains and wage gains.

There's also potentially stories

where worker representatives block the deployment

and development of new technologies

in a way that's ultimately harmful for society

because they're representing relatively narrow interests.

I would say there's as of yet, no good empirical evidence

that would help us adjudicate between those two possibilities.

That definitely seems like a really important area of research going forward.

-I was also talking about those who might benefit.

I was also struck by a point you made in the paper

about, that it could increase options for people

who potentially are very creative or who have a lot of good ideas,

but simply have a problem expressing themselves

in ways that others can understand and digest.

For example, people who have problems with writing skills,

either because they're very technological

or simply because perhaps they're writing in a second or third language.

ChatGPT could help them literally get their message across.

-That's right.

I think anecdotally, I've certainly heard from people

for whom English is not their first language that,

for example, ChatGPT has been really helpful

in helping them with academic writing or otherwise,

where before this was a bottleneck for them.

Now they're much more able to communicate their research or other areas.

I think this is another way in which

GPT technologies could be a sort of leveling

of the playing field in that

people who might be really great with creativity or thinking of ideas,

but just have difficulty writing them in a way

that sounds like professional

or academic style can now use the technology

to make their writing more sound like these other aspects.

I think this could be potentially great

for both these workers

as well as just for innovation as a whole, potentially.

-I was also struck by one of the other main conclusions

that you mentioned,

which was about the most subjective outcomes, e.g.

the job satisfaction

that a lot of your subjects in the experiment

found using ChatGPT gave them equal or greater job satisfaction.

Whereas traditionally in the past, people have felt that

forms of automation have reduced job satisfaction.

Do why they felt that the job was more satisfying

and whether that feeling lasted

or whether it was simply a hangover from the experiment?

-Right. To some extent, we definitely think that

this reflects people's first reactions to an exciting new technology

rather than necessarily something fundamental and permanent about the way

that the technology affects their enjoyment of their work.

When we followed up with people two weeks and two months after the experiment,

the people who were in the treatment group

were much more likely to be using ChatGPT in their real jobs,

but they weren't more satisfied

with their real jobs than people in the control group.

Now, that doesn't necessarily mean

that ChatGPT won't raise job satisfaction in the long run.

It could be that it's just too small of an impact on a two-week

or two-month time horizon

relative to your base prior level of satisfaction in your job.

We don't find evidence

that it's permanently improving job satisfaction.

As to why it raised job satisfaction in our experiment,

one possibility,

again, is that people are just really excited

to discover a powerful new technology.

Another possibility is just,

ChatGPT helps with many elements of tasks that might be the least enjoyable.

Editing,

translating your ideas into a rough draft.

The tasks that we gave people were not terribly exciting or stimulating.

People may in general have found,

writing an e-mail

or writing a short report to be a boring task.

In that case, anything that helps you finish that task more quickly

might raise your job satisfaction.

You could tell a story where basically,

ChatGPT is speeding people up

or helping them expend less effort precisely in the areas

that they least enjoy working and in the types of tasks

that they least enjoy doing.

Again, I would say there's no conclusive evidence

on that from our experiment.

-In that case, research always raises as many questions

as it answers.

Where do you want to go next with this?

What do you see as the key questions that have been raised by

what you find so far?

-I think the next big thing is really to try to move into the field

into a setting where the tasks are much less well defined,

where we're requiring much more of workers

to use a broad bundle of skills,

where there's many of these organizational-type concerns

that we aren't able to address an individual level experiment.

To see what productivity impacts of workers look like there.

I think there's just also interesting work to be done in looking at

how organizations are figuring out how to adapt to these workflows.

Shakked and I have been looking into some more qualitative stuff

where it seems like companies are really

seeing this as an exciting tool, but not quite sure of

how do I actually get my workforce to start using this?

What are some areas

where this might really improve productivity in my organization?

What are good use cases and what are bad use cases?

Of course, there's all sorts of regulatory

and legal things as well that further complicate things.

I think just having a broader look at the ecosystem as a whole,

I think is the exciting next direction.

-Yes. In other words, the difference between what happens in the lab

and what happens in real life.

Shakked and Whitney, thank you very much for joining us.

That was extremely interesting.

Another look at the effects of generative AI on the world of work.

Unfortunately, that is all we have time for

in this edition of the podcast.

Once again, thanks to Shakked Noy

and Whitney Zhang for joining us and talking about their paper.

If you want to know more about it,

you can find a link to it on the webpage for this podcast on the ILO's website.

For now, thank you very much

for listening and please join us again soon.

For now, goodbye.

[music]