👹 Content moderation hell

What happens when you refuse to even ban nazis

Hello. All of the biggest and best writers on Substack are leaving the platform in response to CEO Hamish McKenzie’s cowardly approach to moderating nazi content. Outwardly, his message is that censoring hate speech will not stop it; in fact it will cause it to thrive even more. Whether or not this approach is effective is honestly besides the point — really, it would be great if he just said ‘nazis earn me a lot of money so no I won’t be removing them, actually’.

In case you’re curious (and in case you uh think I’m a nazi), I’m extremely unhappy about what Substack are doing, and I honestly can’t believe that the CEO jovially admits there are nazis having a great time on his stupid opaque email newsletter platform. This doesn’t mean I’m now going to leave the platform. Rather, I’m going to turn my payments off so that 10% of the very small amount of income that Horrific/Terrific makes does not go to Substack. I might try and figure out a third-party payment system, but obviously Substack frown upon this, so I’m gonna need to figure out if that’s even possible. But either way, payments via Substack are OFF.

I’m literally bracing for many of you to unsubscribe based solely on my decision to stay. That’s fine, but I hope most of you can appreciate the nuances behind this decision — appreciating nuance is kind of a key part of the Horrific/Terrific project. I want you to know that I took advice from a handful of other professionals in my network to reach this decision. I spoke to Ben Whitelaw who is a content moderation expert (read his publication Everything in Moderation), tech ethicist Alice Thwaite, and Charley Johnson who writes Untangled. These three all have deep and varied understandings of trust & safety on platforms, so I have a lot of faith in their viewpoints. Anyway, here’s roughly how I reached this decision:

Migrating away from Substack is clearly a business decision for many of the big writers on here — not an ideological one. Casey Newton has now announced he is leaving, after hoping that the removal of a few minor nazi accounts would be enough to stop people from unsubscribing from Platformer. He is absolutely not taking a stand; before deciding to finally leave, he asked these flimsy questions, as if conducting an impromptu focus group session: “As we think through our next steps, we want to hear from you. If you have unsubscribed from Platformer or other publications over the Nazi issue, does the company’s new stance resolve your concerns? Or would it take more? If so, what?” The ‘new stance’ he’s referring to is the decision to literally remove a few inconsequential accounts that Casey’s team dug up for Substack. This isn’t a ‘new stance’ — they are just throwing a bone to one of their biggest earners.

It literally just makes good business sense for an operation as large and lucrative as Platformer to move away from an email platform that takes 10% of your income — which is a lot if you have hundreds of thousands of paid subscribers. Other platforms, like Ghost, just charge a flat fee. Also if Casey Newton doesn’t move, he’ll just get cancelled lol. So really, he kind of doesn’t have a choice. I would say that writers like Molly White and Ryan Broderick also don’t have a choice, because clearly they earn a lot of money through Substack — but the difference is, those two appear to have some kind of moral compass.

Horrific/Terrific is not a business, so my decision has nothing to do with business. I am not making any money off of this; there are about ten of you who pay, and most of you who do know me personally. I’m really happy that anyone donates any money at all but that’s not why I’m doing this. I just enjoy it. Moving to another platform where I have to pay a monthly fee just to send emails to people makes absolutely no sense. I do not have a team to help me migrate; I do not have a marketing budget to increase discovery of my newsletter; I also do not care to undertake all that work on my own because it just isn’t even slightly worth it. I would rather take up server space on Substack until an actual useable alternative comes along. I should also point out that I tried Ghost for a whole year and found it miserable. It’s not exactly cheap, and I gained like 10 subscribers in the entire year I was with them. Being on Ghost made the act of writing a newsletter extremely unpleasant and unrewarding — it is not a platform designed for people who are still building their audience (I was still at under 100 subs back then).

Even with the biggest earners leaving the platform, Substack show no signs of changing. Which to me indicates that moving off the platform is kind of pointless, especially if you’re just a solo writer doing this for fun. Perhaps I’m too cynical, but this makes me feel so powerless that I am utterly demotivated to do anything. The amount of effort it takes to move off Substack vastly outweighs any positive outcomes. Like… I don’t even know what the positive outcomes are?? Do we honestly think that Nazis don’t use Ghost or Beehiiv or whatever else??

Anyway, personal reasoning aside, let’s look at some of the wider dynamics at play here: these discussions are only happening in tech circles because we are the ones who care about content moderation the most, as a subject I mean. I learned about the nazi issue on Substack not because I use Substack, but because I read tech news. It’s just so obvious I would have not been aware of this problem otherwise. That’s not because I don’t care about this stuff — it’s because it’s almost impossible to tell what people in general are reading on Substack, even if you’re a writer.

It wouldn’t surprise me if the majority of people who subscribe to a newsletter on Substack A) do not subscribe to anyone else on Substack, B) have never even visited the website since initiating their one subscription, and C) are blithely unaware of the nazi thing. I think most people who subscribe to a Substack don’t even know what the platform is — that’s pretty clear from the app store reviews; most of the people think they are reviewing a single newsletter!

This screenshot was posted by Charlesv in the Garbage Day discord.

One of Substack’s key failings is not their light-touch approach to content moderation — it’s their insistence that they are ‘just infrastructure’. If that were true, why do they have an app? Why do they have a discovery feed? And their own Twitter clone? All these things are features and attributes of a platform, not infrastructure. They’re trying to have it both ways: allowing anyone to use their ‘infrastructure’ while still providing all the popular frills of a social platform.

Substack are not moderating content because, I guess, they honestly believe it’s not their place. Ghost are exactly the same, but unlike Substack they really are just infrastructure. All they do is host your publication and send out your emails. Just as with Substack, if there are Nazi’s using Ghost it would probably be really hard to tell, because this kind of internet infrastructure is largely invisible to the general public, and even subscribers. But Substack are benefitting from both sides of this, where they don’t have to care about who is saying what in their newsletters, and they also get to enhance monetisation with recommendations. This is pretty untenable, I don’t know how long they can last with that kind of model.

The infrastructure VS platform thing is interesting; it reminds me of what happened to hate site Kiwi Farms and Cloudflare a little while back. Kiwi Farms is a forum on which people discuss stalking, doxxing, and just generally harassing people in the real world, and Cloudflare was their content delivery network (CDN), until they finally buckled under the pressure and terminated their service to Kiwi Farms. Obviously, they did the right thing, but I can kind of understand why they hesitated — because a CDN truly is infrastructure. Should they have a say over what activities people conduct on their websites? If so, how do you draw clear lines? And why doesn’t it go deeper into the stack, like way down to the ISPs? (Btw Kiwi Farms are still doing strong on another CDN now). In reality, truly proper and absolute content moderation requires invading people’s privacy constantly, which is something I’ve heard people don’t like that much.

Right anyway… I’ve been feeling very torn and confused about all this, even as I wrote the words above with cloying conviction. I think it’s quite hard watching a bunch of writers who I respect and usually mostly agree with perform an act of protest, and realising that I find it very hard to join in. I’ve been very scared to send this newsletter this week. Then, Alice Thwaite (mentioned above), sent me this piece by Dave Karpf (see bullet 3) and suddenly I feel less alone in my decision to not join the exodus. Everything he says there resonates with me, especially this:

“Free Substacks, by contrast, have less to lose by switching, but also (a) are choosing between paying Substack nothing versus paying another service something (ugh. That’s a drag.), and (b) are receiving weaker signals through their analytics. I’ve lost some readers since this whole episode started, but I don’t have a clear signal whether that’s because they refuse to subscribe to Substacks anymore, or they’ve gotten tired of my writing in general, or they specifically disagree with how I’ve approached the topic”

Gonna leave it there. The newsletter will be more normal next time — have a good weekend!

UPDATE: In my email discussion with Ben Whitelaw (who I mentioned above), he showed me a working version of a diagram he put together which illustrates the hierarchy of influence over speech on online platforms. He asked for my input, and adjusted it to the below. He also asked for input from Mike Masnick from Techdirt, so this diagram is a kind of culmination of all three of our viewpoints on the matter. It was nice to, even briefly, just stop and think for a bit about who actually has power here, and formalise it in some way. It may not be completely accurate yet, but it reminds you of how many forces there are at play when it comes to online speech, and how complex the power dynamics are.