Don't share links on Mastodon! What?

Linux Server
1

I just stumbled upon the News article

Please Don’t Share Our Links on Mastodon¹: Here’s Why! by @Ankush_Das

The article has created quite some resonance in the Fediverse², not of the best kind.

Here’s a short summary of the alleged problem:

Every time, a Mastodon user shares a link to a website on the Fediverse, it gets shared further to the instances (servers) of this person’s followers. These can be quite a lot, depending on the user’s account’s number of followers.

These instances want to show the link nicely (i.e. a preview) to their users. In order to do so, they access the website, where the link points to, load its content and prepare the preview. This can generate quite a lot of traffic (hundreds or maybe thousands of http requests in a relatively short time span.

Apparently, the ItsFoss web server has difficulties handling this amount of traffic.
As I said before, this news article has created a lot of astonishment within the Fediverse, like this was a problem of an extremely poor server setup, as if this kind of problem was one of times long gone by. A common Apache server and some server side caching should be able to handle many thousands of http requests per minute.

I am not a sysadmin, so I really can’t tell, but some of you are, and might be able to give some insight.

Let’s be honest: I find it rather strange to see somebody asking for not sharing a link to their website. It’s definitely the first time, I have seen such a request.

¹Mastodon is an open source social media platform, similar to X/Twitter, BlueSky and Threads, but unlike those not owned by any corporation or a billionaire. Mastodon is part of the Fediverse².

²Fediverse: A distributed network of instances (servers), of varying sizes, ranging from Single Board Computers at somebody’s home to huge professional servers, providing social interaction via the Activitypub protocol

3 Likes
2

I have noticed that itsFOSS has occasional long delays recently.
Is it related to this?

3

At least, this is, what Ankush presumes.

2 Likes
4

As an aspiring DevOps engineer, I would love a write up how ItsFOSS and all the other websites under that umbrella are set up. When you have time @Ankush_Das @abhishek. I know its not the most exciting website in this regard, but almost all the write ups are on large, massive websites like AirBnB, Facebook, Figma, ect. Seeing something more down to earth in this regard would be very interesting.

I think there must be something else going on here. In the article they presented evidence that ItsFOSS is not affected by say, links on GoogleNews or Twitter, but are on Mastodon. This suggests that they are using some amount of caching.

I think they could increase the time to live (TTL) of the data in the cache/CDN. ItsFOSS articles are sometimes updated, but they can probably get away with a long TTL of maybe even several hours. The only places where they might have some trouble with this would be comments on a new article and ads. Both of these could be cached seperately with lower TTLs.

That wouldn’t completely fix the problem, but it should make it better.

3 Likes
5

Hello Jimmy,

I could brief it a little. I am not a full-fledged sysadmin or experienced devops person. I am more of a learn-on-the-go without getting too technical kind of person in this field.

For years, It’s FOSS used WordPress. It started with shared hosting on Hostgator and then moved to some other managed WordPress hosting.

Then, in 2018, I decided to host them “on my own”. So, I used Server Pilot to deploy WordPress on DigitalOcean/Linode cloud servers. That was for all the websites we had.

This forum is self-hosted on DigitalOcean, thanks to their marketplace script. Which means that I deployed a preconfigured server with Discourse on it.

The setup worked fine. But I felt WordPress was getting bloated and I wanted something lean with focus on content. Ghost CMS provides a lot better writing experience than WordPress (something all of us in the team agree on).

Around 2021 or so, I started moving the websites from WordPress to Ghost gradually. Again, I use DigitalOcean’s marketplace to deploy preconfigured Ghost servers.

Ghost in general is quite fast but to make it even faster for our readers, we use Cloudflare as the CDN.

We have a love and hate relationship with Cloudflare. On one hand, it makes the site faster but on the other hand, some users complain that they see Error 520 or so from Cloudflare.

That’s why on It’s FOSS News, Cloudflare was disabled (reenabled again yesterday).

We use Hyvor Talk for comments, which is loaded from its own server. The same is true for ads, which are served from its own server. It’s my understanding that caching won’t impact this.

6 Likes
6

Hey Abhishek, thanks for that! Nothing wrong with using preconfigured stuff. I do it sometimes, and even my company does it sometimes and we have a DevOps team. Sometimes you just don’t want to have to manage it yourself. That does mean it might not be as good if you did do it yourself, provided you know what you are doing.

I thought they might come from a seperate server, but I wasn’t sure. You are right that the caching of the articles would not affect the comments and ads, or at least they should not. Those servers could have their own caching. I am not too sure how ads work exactly in that regard, as sometimes they pay per click and sometimes pay per view, so caching makes it more complicated. Do you manage the Hyvor Talk and ads servers through Digital Ocean as well, or are they some sort of seperate service?

2 Likes
7

I tried to explain the reason for the difference in the initial post.

A post that gets shared on another social network, due to their centralised structure, only needs to generate a single preview. The Fediverse has about 18,000 instances, and it is fair to assume that a popular link might find its way to about 10-20% of them.

This makes a difference.

Considering that ActivityPub (the protocol that powers the Fediverse) seems to be the way to go for social networks in general, it might be worth to think about the idea of implementing news and forum directly upon an ActivityPub platform (say: Friendica for news and NodeBB for the community forum). This is certainly not a short term solution.

However, let me share here one of the comments, I just found on the initial news post:

People are currently dunking on this post that complains about Mastodon link previews DDoSing their servers - and they deserve every bit of it.

Pulling up this site in a browser with no privacy/sanity plugins installed, it made a total of 3740 requests within 4 minutes, which amounted to 267.22 MB transferred. It has ads left, right, bottom and center, and it has one of those annoying autoplaying videos, which ALSO has a video ad in front of it. The website would not scroll even close to fluently, and I’m doing this on a 24-core M2 Ultra Mac Studio with 128 GB RAM. I had to use the debugger command in the JS console in order to make the network panel in dev tools stop lagging.

In addition, they quote someone complaining about 114.7 MB being requested from their server over the span of 5 minutes. Not only is that less than half as much as you make a single user download if they don’t actively cut into your bloatware crap, it also amounts to a whopping 3.06 Mbit/s that you’re complaining about. THREE MEGABITS PER SECOND!

Your website is an insult to the internet.

This also might explain something.

3 Likes
8

Forgive me Mina, but I know very little about servers.
Are you asking us to not include links in our posting on this forum.

3 Likes
9

No, dear Howard!

It seems, I expressed myself extremely badly.

The news post asked us, not to share links to ItsFoss pages on the social network Mastodon.
Links to external resources in this forum are 100% fine.

Sorry for the confusion. I shall try to use clearer language in the future.

PS: As it probably was to be expected (check: Streisand effect), the original news post has gone pretty viral.

3 Likes
10

Not true. A single preview per cache or CDN, then depending on how long the time to live is of the data in the cache / CDN.

These “centralized” social networks are only centralized in the sense of control, but the actual architecture of their website is distributed across hundreds or thousands or hundred of thousands of servers¹, which are also distributed across the earth.

¹ Most of these giant websites are not using a bare metal or even virtual machine servers, but instead many containers. These containers still are servers though, but I thought it was important to make a distinction.

I don’t think that is the issue, or at least I don’t think that’s the full story. Google News and Twitter might both have more than 18,000 CDNs each, depending on how local the CDN is. The whole point of a Content Delivery Network is for it to be local to users to speed up response times, but I am not sure local they actually are.

Certainly something with how these Mastodon instances are set up is incorrect, but the scale does not seem to be bad for what Its FOSS has set up given the explanations on the article. That is what I meant when I said there is something else going on.

11

Seriously?

I was giving a simplifying explanation about the difference between a decentralised network and a centralised one, and you feel the need to lecture me on technical details? And explain that big corporations use distributed servers?

Here lays a crucial difference: The different nodes of Google’s or Twitter’s delivery network share the link preview with each other: They don’t each pull the ItsFoss website and create their own.

Certainly?

I don’t know. As far as I see it, most other websites don’t have any issues.

Is there room for optimisation within the Mastodon software. Sure!

But is a problem, which affects only some specific websites with some specific setups but not the Fediverse itself, a high priority? I’m not so sure any more.

In the news post, there is a sentence:

Here is a fundamental misunderstanding:

Mastodon is not aiming at replacing anything existing. There is no corporation with an agenda. It is just one, rather popular, implementation of the W3C’s preferred protocol (ActicityPub) to exchange messages within a decentralised network.

It provides a place for people to meet, chat, discuss, share information and cat pictures.

If you want to get a feel for what is going on there, check out

or feel free to check my public feed at

12

Hey Mina, I’ve thought about what you said here, and I think you are right. It was not my intention to come off like I was talking down to you or lecturing you, but I can totally see how you would interpret it that way. I want to apologize to you for this, and I promise I will try to be more careful in the future. This topic excites me, and I think I was just so excited to get to the bottom of this that I was started spewing a bunch of obvious technical facts. I should’ve stopped myself and thought about what I was really contributing to the conversation. Once again, I apologize.

I don’t have much more to comment on your post. I pretty much agree with what you said in regards to the technical aspects. I am not very familiar with Mastodon myself.

4 Likes
13

No problem, my dear. :kissing_heart:
Water under the bridge.

This happens to the best of us.

I am also quite excited about the topic, and I’d love to get to the bottom of this.

As I am convinced that decentralised structures in people’s hands are the best way to improve social networking in this age of corporate enshittification, I want it to thrive.

However: If it is causing damage to the traditional web or has an intrinsic growth problem, there’s something that needs to be done.

On the other hand: If the problem is entirely caused by bloated and badly setup websites, the ball is entirely in their field.

To get more insight into the issue, was the purpose of starting this discussion.

I thought so, and also that you’re not the only one.

That’s why I shared the two Mastodon links in the previous post.

Whilst Mastodon may be the most commonly known part of it, the Fediverse is a lot more, and it is all connected and allows sharing and following between all the different branches of the tree:

image
image1200×1201 98.7 KB

2 Likes