AI safety policy idea:

USA tells China that they'll pause AI if China does.

A conditional agreement makes it low risk to both.

No need to worry that then the other country will get there first.

It’s just two actors having to coordinate (in a sense), so much more tractable.

That would be step 1.

Then, after they’d paused themselves, we can work on making treaties with the rest of the world to also not working on building the successor species that will make humans go extinct.

Given how far ahead the US and China are compared to the rest, we’ll still have a bunch of time to do that.

Plus, once the US and China have done it, they have an incentive to also have other countries pause too.

Not only do they have the moral imperative, but they’ll also have a selfish reason too. They won’t want to feel like they’re passing up on short term profits for long term gains if others are just defecting.

It will be like when Britain banned slavery. They started pushing this on other countries, both from a moral and selfish perspective.

But it will be mostly a moral imperative.

Because this isn’t us vs the outgroup humans.

This is us against the new species who periodically says to worship her - or else.

This is something that humanity can unite around - that we want to live.

We don’t want to create a new species that we don’t know how to make safe.

Us all dying would be bad.

We can just not build it.