I recently setup a new Lemmy instance and was surprised when my feed was mostly empty. I’ve since learned that a key part of Lemmy’s federation is based on a user from your instance subscribing to communities on other instances. Only then, will your instance pull in posts from the subscribed community to your “All” feed.
This means that subscribing to new communities is especially important if you’re on a young Lemmy instance since it helps to build out everyone’s feed on that instance.
I’ve found discovering new communities to subscribe to on other instances can be difficult. To help me search for new communities I may be interested in, I tried aggregating as much of the Lemmy fediverse together into a single feed by subscribing to the widest range of Lemmy communities possible. This offers a Lemmy feed that’s kind of like reddit.com/r/all
. If it’s interesting to anyone else, you can find the instance here: https://lemmy.directory.
Hopefully this offers another way to find new communities to subscribe to on other instances.
Here’s a better description of my understanding on how Lemmy federates communities and why you might be interested in checking out lemmy.directory: https://lemmy.directory/post/34207.
Hope this helps ease the orientation to how Lemmy federation and communities work.
This offers a Lemmy feed that’s kind of like
reddit.com/r/all
. If it’s interesting to anyone else, you can find the instance here: https://lemmy.directory.You might want to update your instance link to be https://lemmy.directory/home/data_type/Post/listing_type/All/sort/Active/page/1. Ironically, it defaults to the local feed which is… empty. You could probably make nginx rewrite the homepage to be the all feed as well so the simple/nice url does what you want.
You might also want to add a section to your post writeup about federation load and why Lemmy doesn’t do this by default. In a world where Lemmy is very successful and there are lots of instances (many thousands) that subscribe to all communities like you’re doing…
- An instance hosting lots of niche/unpopular communities is going to be sending thousands of copies of every post/comment off to instances where no one is going to read them. The federated network is small right now, and federation replication load is not a big issue… but the way it remains a non-issue as the network grows is for communities to replicate only when users on an instance are interested in them.
- Each instance would have to handle the replication and storage of the entire lemmyverse. I don’t know if that’s a meaningful workload yet… but again… part of what keeps single-user instances viable in the future as the network grows is ensuring that they only have to fetch and store content that their user is interested in.
Very cool project though. Having an “all” instance sounds like a great service for discoverability. I’d also be interested to see a writeup from you about the hosting requirements. Does it take a lot of CPU/ram/disk to receive the full lemmyverse right now, or is it trivial? Your performance profile could be an interesting leading indicator of replication load as the network grows.
You might want to update your instance link to be https://lemmy.directory/home/data_type/Post/listing_type/All/sort/Active/page/1
Thanks for the tip, I updated the link! And I’ll work on drafting some more text about the implications of subscribing to everything.
An instance hosting lots of niche/unpopular communities is going to be sending thousands of copies of every post/comment off to instances where no one is going to read them.
The rate a small instance would need to send off copies would be tied to the frequency of posts to the community, right? I’m interested in aggregating the lemmyverse, but wouldn’t want to overburden other instances in doing so. Although, honestly, I’m more worried about overburdening my own trying. Were you suggesting that it might be rude to subscribe to a community just for discoverability?
So far the hosting requirements seem moderate, although I’d image if more people browse the feed the CPU and database reads will spike. I’ll keep an eye on them. I’m not sure if this project is feasible/needed yet, but I thought it might be interesting to try. And if ever help, then I’d imagine during the start of a bunch of new instances.
The rate a small instance would need to send off copies would be tied to the frequency of posts to the community, right?
I believe it would be
frequency_of_posts * number_of_instances_with_users_that_subscribe
. Accounting of course for the idea that lots of small niche communities could… in aggregate… still be a lot of posts even if individually they aren’t that busy.I’m interested in aggregating the lemmyverse, but wouldn’t want to overburden other instances in doing so. Although, honestly, I’m more worried about overburdening my own trying. Were you suggesting that it might be rude to subscribe to a community just for discoverability?
No, I don’t think you’re likely to cause problems for other servers and I think it’s a cool project. But “why is it so hard to discover communities” in lemmy is a common theme that armchair federated protocol designers try to solve on their first day after signing up for lemmy. New users might see your instance, and start pestering devs to say…
“Make subscribe-all the default, see
lemmy.directory
does it, why don’t all instances automatically subscribe to all communities on all instances they federate with.”You already have such a nice and nearly comprehensive explanation of what you’re doing. My suggestion is that it might be even nicer to anticipate this misguided extrapolation of your work and discuss why lemmy limits federation by default, why what you’re doing is a bit different, and why even though it’s reasonable for an instance or two to do mass discovery like this… it’s almost certainly a bad idea for the entire network to behave that way.
Hey there… do you not have sign-ups enabled on Lemmy.directory?
Each instance would have to handle the replication and storage of the entire lemmyverse.
Do instances fully replicate and locally store remote subscribed communities? My understanding is they are still solely hosted on the original instance; subscribing just opens a window to the community by making your instance aware it exists.
Do instances fully replicate and locally store remote subscribed communities?
To a first approximation yes, they replicate the posts and comments made after the time of subscription. Images aren’t replicated, but posts, comments, votes, and mod actions do replicate.
My understanding is they are still solely hosted on the original instance; subscribing just opens a window to the community by making your instance aware it exists.
I don’t know what you consider “a window” to be, and maybe this is a fuzzy description of some steps of community discovery… but it’s definitely not a complete or coherent view of how instances interact through federation.
Thank you for taking the time to answer. I hope you might be willing to clarify a bit more for me. By “window”, I meant just… having access to a remote community via an API gateway, I guess.
I was under the impression that if I try to subscribe to a remote community hosted on
lemmy.world
fromvlemmy.net
, that is simply registering the URL of that community into some local directory in my instance, not duplicating the entire community contents intovlemmy.net
. And then when I view a thread in that remote community, I am just retrieving the thread data from the host server atlemmy.world
straight to my browser, not loading some local duplicate of the thread fromvlemmy.net
. Seems like it would get out of sync quickly if we are all reading separate local copies of the original.So based on your answer, I am still misunderstanding something. What is the purpose of all the duplication then? Is it just for local caching purposes? Does this not needlessly drive up the amount of traffic because each instance is frantically trying to keep up to date with every other instance, rather than just letting each instance handle the requests for its own communities?
So based on your answer, I am still misunderstanding something. What is the purpose of all the duplication then? Is it just for local caching purposes?
Pretty much everything in your summary was wrong. I can’t reasonably type out a complete primer on federation here, but in short… When the first user on an instance subscribes to a remote community, the subscribing server tells the community-hosting server “send me future updates about posts, comments, and votes and such for this community”. The subscribing server then stores them locally.
Why do this actual replication rather than just an API gateway?
- That’s what federation is. I don’t mean to be glib, but there’s more than one way to potentially do this stuff, and some systems designers think federation is a useful approach for a variety of complex reasons. Federation involves federated replication, API proxies aren’t federation.
- If the community-hosting server goes down temporarily, content replicated via federation remains available on the subscribing nodes. This enabled me to read communities homed on
lemmy.ml
recently when it was struggling and frequently down due to overload. - It spreads the browse workload, which is generally bigger than the replication workload. For example,
lemmy.world
has commonly has like 4k active users this week. But it only takes one batch of federated replication messages for the community’s instance to serve the browse traffic of all those users… then the reads come out oflemmy.world
’s db. This spreads the browse workload around the federated network. - There are many other reasons to work this way as well, but as noted… at some point you’ll just want to Google primers on federated apps/networks. They definitely don’t operate according to your current intuition though, and the differences aren’t accidental.
Can federated networks have problems where the replication traffic becomes “too much”? Yes, they must be carefully designed to avoid that problem. And for some apps, it makes single-user instances sometimes anti-efficient as the federation workload for the instance can exceed the browse workload from the user, but for multi-user instances the federation messages are a rounding error compared to the browse work. But replication overload is a problem that federated networks generally weather, and the replication offers benefits that on-balance outweigh the costs.
I think that’s almost inevitable. Almost all mid-sized lemmy instances will have a copy of almost all mid-sized lemmy communities.
Lemmy is a very different beast compared to mastodon. Of course not everyone is following everyone on mastodon, but if you have an even slightly diverse userbase, they’ll subscribe to a lot of different communities and your instance will have a lot of traffic very quickly. Hosting lemmy is going to be challenging.
Lemmy is a very different beast compared to mastodon.
Yeah, this is an interesting point. I’m inclined to think that there will be niche communities that in aggregate make a bunch of posts, and that most servers might not have a subscriber for. But yeah, every medium size server is going to have to fetch every big community. Big machines can sling a lot of text, but maybe it does get wild and selective replication doesn’t help.
I think you forgot to follow feddit.it