Another user posted the blog where they discuss their speedup techniques: https://tailscale.com/blog/more-throughput/

It’s likely that the kernel version can use similar techniques to surpass the performance of the userspace version that tailscale uses, but no one has put in the work to to make the kernel implementation as sophisticated as the userspace one.

I had a look through the comments on this HN thread the other day and came away more intrigued by https://github.com/openobserve/openobserve than hyperdx. Hyperdx is built on top of clickhouse whereas open observe has it’s own storage engines based on parquet files that can be accessed from local disk, S3, or a few other protocols.

I haven’t tried either option yet… I’m, currently using netdata for metrics and don’t do anything special for logs or tracing, but at tiny self-hosting scale I often find software with it’s own storage engines (often sqlite) to be extra hassle-free. I’m curious to kick the tires on openobserve for that reason.

For the latest version of lemmy, hot sort works in the new fashion. There is a pull request with further implementation details linked in the GitHub issue.

Ah, fair enough. My response doesn’t apply then.

You misunderstand what the Hot rank is doing. It’s not balancing newness vs hotness, it’s scaling hotness according to community size. This might feel like newness if you’re focused on vote counts as a proxy for post age, but it’s a different approach. See https://github.com/LemmyNet/lemmy/issues/3622 for details.

There’s a couple ways to think about this:

1. There are a handful of Lemmy communities that are just WAY more active than everything else. The main feeds are kind of lame if you have to scroll 300 posts it to find anything other than a shit post from the same 3 communities. Scaled Hot rank shows a greater variety of communities by making it easier small communities to get ranked hotly.
2. Or you can consider Hotness to be a rough measure of what percentage of people who have seen the post interacted with it. A post with 500 upvotes in a community with 10,000 active users is kind of popular, but only 5% of the people likely to have scrolled passed it cared about it. A post with 50 upvotes in a community with 200 active members is much MORE popular relatively even though the absolute numbers are smaller.

At any rate, this preference toward smaller communities in hot is a recent change and deliberate. While they might further tweak the scaling factors, I wouldn’t expect it to be drastically different. It sounds to me like what you want is Top, Active, or Most Comments. All these are unscaled according to community size and will get you top posts by their absolute metric rather than posts that are doing well relative to their community size.

This is a very strong explanation of what’s going on. And as a follow-up, I believe that ZeroTier present a single Ethernet broadcast domain, and so WoL tricks are more likely to work naturally there than with Wireguard. I haven’t used ZeroTier, and I do use Wireguard via Tailscale/Headscale. I’ve never missed the Ethernet features of ZeroTier and they CAN result in a very chatty wan if you’re not careful. But I think ZT would make this straightforward.

Though as other people note… the simplest/least-disruptive change is probably to expose some scripty thing on the rpi that can be triggered via be triggered over a routed protocol and then have the rpi emit the Ethernet broadcast packets from the physical network.

I dunno how to hotlink, but if you scroll to the active users graph at https://fedidb.org/software/lemmy you can see there’s been like a 25% dropoff in active users since the peak in July. Lemmy has still grown 50x since May, and it’s much MUCH more active than it was then. But we’ve definitely crested a peak and not everyone who gave Lemmy a shot then is sticking around in a monthly basis.

This isn’t necessarily bad. Lemmy is still young and has many rough edges, it wasn’t realistic to win all the users that tried it on ease-of-use in a head to head with reddit. And Mastodon has had multiple growth waves interspersed with periods of declining usage, but with the spikes has grown ie remained stable overall. Early-stage commercial social media have big ups and downs in engagement and growth as well, and just like lemmy those ups and downs are often externally driven… when competitors mess up, when a big global news story hits, when a major sporting event happens… these can all be catalysts for one-time growth. It’s not a straight line.

Time will tell what user level we stabilize at in the short-term and what events spur new growth, but it’s normal to have a big expansion be followed by some degree of contraction.

Have you emailed [email protected] to try to get reinstated? This all seems like a pretty reasonable explanation if it isn’t repeated behavior.

That’s an interesting report but it’s possible to “work” at different latencies. And unless you have specialized audio capture/playback hardware and have done some tuning and testing to determine the lowest stable latency that your system is capable of achieving… “works” for you is likely to mean something very different than it does to someone who does a lot of music production.

It remains an interesting question to some users whether Wayland changes the minimum stable latency relative to X and if so whether it does so for better or worse.

I’d consider asking in a Linux audio or music production community (I’m not aware of any on Lemmy that are big enough to have a likely answer though). If music production is a primary use case and audio latency matters to you, almost no general users are going to be able to comment on the difference between X and Wayland from a latency perspective. There may not be a difference, but there might and you won’t be likely to learn about it outside of an audio-focused discussion.

Yeah, snapshots sent to a separate and often remote pool is an extremely common backup strategy for folks who have long-term settled on ZFS. There’s very nice tooling for this that presents a more traditional schedule/retention based interface to save you scripting snapshots and sends directly.

• Sanoid is an old standby in that space.
• Zrepl is getting a lot of traction lately and seems to be an up-and-coming option.
• I use pyznap, but I don’t recommend it to others as as the maintainer is on a multi-year hiatus which makes it undermaintained. It works great, but isn’t getting active development which makes it a poor bet in a crowded space with many great options. I plan to eval Zrepl when I get around to it.

I don’t know if what you’re suggesting is possible, which as I read it is to split your “live” raid-1 in half and use one drive to rebuild the “live” pool and the other drive to rebuild the “backups” pool. It might be, but I can’t think of any advantage to that approach and it’s not something I would have thought to attempt.

I’d do one of:

• Ship the data over the network using ZFS send or something like syncoid/sanoid (which use ZFS send under the hood). It might be slow, but is that an issue? Waiting a week for the initial sync might be fine.
• But syncing by sneakernet is a good strategy too, and can be faster if your backup site is close or your connectivity is slow. In this case, I’d build the backup pool at the live site… ideally in an external drive bay… but one could plug it in internally as well. Then sync them with a local ZFS send, export the backup pool, detach and transport the backup pool to the backup site, them reattach the backup pool at the backup site and import it. Et Voila, the backup pool is running at the remote site fully populated with data and subsequent ZFS sends will be incremental.

Splitting and rebuilding your live pool might be possible, but I can imagine a lot of that might go wrong and I can’t see any reason to do it that way over export/import.

It may seem kinda stupid to consider that an accomplishment, but I feel quite genuinely proud of myself for actually succeeding at this instead of just throwing in the towel…

Way to go. I’ve been at this a decent while and do some pretty esoteric stuff at work and at home… but this loop of feeling stupid, doing the work, and feeling good about a success has been a constant throughout. I spent a week struggling to port some advanced container setups to podman a month or so ago, same feeling of pride when I got them humming.

It’s not stupid to be proud of an accomplishment even if it’s a fundamental one that’s early in a bigger learning curve. Soak it in, then on to the next high. Good luck.

My money is also on IO. Outside of CPU and RAM, it’s the most likely resource to get saturated (especially if using rotational magnetic disks rather than an SSD, magnetic disks are going to be the performance limiter by a lot for many workloads), and also the one that OP said nothing about, suggesting it’s a blind spot for them.

In addition to the excellent command-line approaches suggested above, I recommend installing netdata on the box as it will show you a very comprehensive set of performance metrics without having to learn to collect each one on the CLI. A downside is that it will use RAM proportional to the data retention period, which if you’re swapping hard will be an issue. But even a few hours of data can be very useful and with 16gb of ram I feel like any swapping is likely to be a gross misconfiguration rather than true memory demand… and once that’s sorted dedicating a gig or two to observability will be a good investment.

Tailscale is out, unfortunately. Because the server also runs Plex and I need to use it with Chromecast on remote access…

I rather suspect you already understand this, but for anyone following along… Tailscale can be combined with other networking techniques as well. So one could:

• Access Plex from a Chromecast on your home network using your physical IP, and on your tailnet using the overlay IP.
• Or one could have some services exposed publicly and others exposed on the tailnet. So Immich could be on the tailnet while Plex is exposed differently.

It’s not an all or nothing proposition, but of course the more networking components you have the more complicated everything gets. If one can simplify, it’s often well worth doing so.

Good luck, however you approach it.

So for something like Jellyfin that you are sharing to multiple people you would suggest a VPS running a reverse proxy instead of using DDNS and port forwarding to expose your home IP?

I run my Jellyfin on Tailscale and don’t expose it directly to the internet. This limits remote access to my own devices, or the devices of those I’m willing to help install and configure tailscale on. I don’t really trust Jellyfin on the public internet though. It’s both a bit buggy, which doesn’t bode well for security posture… and also a misconfiguration that exposes your content could generate a lot of copyright liability even if it’s all legitimately licensed since you’re not allowed to redistribute it.

But if you do want it publicly accessible there isn’t a hoge difference between a VPS proxying and a dynamic DNS setup. I have a VPS and like it, but there’s nothing I do with it that couldn’t be done with Cloudflare tunnel or dyndns.

What VPS would you recommend? I would prefer to self host, but if that is too large of a security concern I think there is a real argument for a VPS.

I use linode, or what used to be linode before it was acquired by Akamai. Vultr and Digitalocean are probably what I’d look to if I got dissatisfied. There’s a lot of good options available. I don’t see a VPS proxy as a security improvement over Cloudflare tunnel or dyndns though. Tailscale is the security improvement that matters to me, by removing public internet access to a service entirely, while lettinge continue to use it from my devices.

Do I need to set up NGINX on a VPS (or similar cloud based server) to send the queries to my home box?

A proxy on a VPS is one way to do this, but not the only way and not necessarily the best one… depending on your goals.

• You can also use port-forwarding and dyndns to just expose the port off your home-ip. If your ISP is sucky, this may not work though.
• You can also use Cloudflare’s free tunneling product, which is basically a hosted proxy that acts like a super port-forward that bypasses sucky ISP restrictions.
• If you want to access Immich yourself from your own devices but don’t need to make it available to (many) others on devices you don’t control, I like and use tailscale the best. The advantage of tailscale is that Immich remains on a private network, not directly scannable from the internet. If there’s a preauth exploit published and you don’t pay attention to update promptly, scanners WILL exploit your Immich instance with internet-exposed techniques… whereas tailscale allows you to access services that internet scanners cannot connect to, which is a nice safety net.

Do I need to purchase a domain (randomblahblah.xyz) to use as the main access route from outside my house?

Not for tailscale, and I don’t think for Cloudflare tunnel. Yes for a VPS proxy.

I’ve run a VPS for a long while and use multiple techniques for different services.

• Some services I run directly on the VPS because it’s simple and I want them to be truly publicly accessible.
• Other services I run on a bigger server at home and proxy through the VPS because although I want them to be publicly accessible, they require more resources than my VPS has available. When I get around to installing Immich, there’s a decent chance it will go into this category.
• Still other services, I run wherever and attach them to my tailnet. These I access myself on my own devices (or maybe invite a handful of trusted people into my tailnet), but aren’t visible to the public internet. If I decide not to use immich’s shared gallery features (and so don’t need it publicly accessible) or decide I don’t trust it security-wise… it will go here instead of the proxy-by-vps category.

I use k8s at work and have built a k8s cluster in my homelab… but I did not like it. I tore it down, and currently using podman, and don’t think I would go back to k8s (though I would definitely use docker as an alternative to podman and would probably even recommend it over podman for beginners even though I’ve settled on podman for myself).

1. K8s itself is quite resource-consuming, especially on ram. My homelab is built on old/junk hardware from retired workstations. I don’t want the kubelet itself sucking up half my ram. Things like k3s help with this considerably, but that’s not quite precisely k8s either. If I’m going to start trimming off the parts of k8s I don’t need, I end up going all the way to single-node podman/docker… not the halfway point that is k3s.
2. If you don’t use hostNetworking, the k8s model of traffic routes only with the cluster except for egress is all pure overhead. It’s totally necessary with you have a thousand engineers slinging services around your cluster, but there’s no benefit to this level fo rigor in service management in a homelab. Here again, the networking in podman/docker is more straightforward and maps better to the stuff I want to do in my homelab.
3. Podman accepts a subset of k8s resource-yaml as a docker-compose-like config interface. This lets me use my familiarity with k8s configs iny podman setup.

Overall, the simplicity and lightweight resource consumption of podman/docker are are what I value at home. The extra layers of abstraction and constraints k8s employs are valuable at work, where we have a lot of machines and alot of people that must coordinate effectively… but I don’t have those problems at home and the overhead (compute overhead, conceptual overhead, and config-overhesd) of k8s’ solutions to them is annoying there.

Is there an issue for this in the GitHub project? It sounds like you’ve done the hard work of diagnosing the issue and an upstream fix seems likely a modest effort given this info.

I wanted to plug one of them over USB, but it seems that docker just doesn’t like to have volumes on external drives. AFAIK docker starts before the drive is fully mounted, preventing it from doing so. I couldn’t find any reliable way to work around this (but I’m open to suggestions!).

You haven’t said what operating-system you’re using, how your mount was configured, or how you’re starting docker or your containers. An external drive is the normal way to do this, though, and I do it on Linux with ZFS drives and docker-compose auto-starting the containers and it works fine.