I recently came across a proposal to create a marketplace of sorting algorithms for Lemmy, allowing users to browse, select, and even create their own custom sorting algorithms. This would provide more advanced users with additional ways to customize and sort content, and give the community the ability to share their creations. I’m curious to know if this is actually possible within the Fediverse and what your thoughts are on the idea. Would this be a valuable addition to Lemmy, or do you see any potential drawbacks?
Sorting algorithms run on the server.
No way a server can allow arbitrary remote code execution.
You could implement something that uses a server sort and has the client do custom sorting on the results the server told it about.
How you do it is a hybrid of how you described.
One of the first optimisations the lemmy devs did with the Reddit exodus was to precompute the “hot score” of a post and store it in a field rather than trying to compute it “live” in every query. So the score gets updated every hour and also whenever there is some comment or change to the post.
And the server simply sorts by this value.
There is no reason similar sorting fields can’t be exposed via the API allowing the client (or some client eg with mod access or admin access) to store some arbitrary value to a column and then the server can happily sort by this column.
If the server sends posts, and the client computes the ordering, the client has to do the sorting.
The “hot” thing is just server side caching of scores for a server side sort.
What you seem to be suggesting is that the server support storing post scores sent from clients so it can feed that sort back to the client. I don’t think that would scale for millions of users storing their preferred post sorting order on the server.
I was the guy who checked the performance issues on those queries and suggested the caching algorithm to the Devs so I know how it works and have a fairly reasonable understanding of the Lemmy schema.
Every single vote you make is already stored. Adding some “ranking vote” is not a big deal. It only has to scale on your own home instance. It’s not a matter of calling for millions of users.
If you run your own instance it will have to scale for 1 user.
The difference is the server doesn’t need to scan the giant vote table for ordinary polling or ranked posts.
Unless there was significant optimization work done on the DB schema, this approach would require joining the large posts table to the very large user-post-ranking table for each request for sorted posts.
This will kill DB performance without some sort of clustering or sharding.
I guess that answers it. It is a neat idea.
Bluesky has custom feeds with arbitrary selection and sorting algorithms, although it’s more Twitterish than forumish
The feeds don’t run on the host server, they run on separate feed generation servers
With fixed sorting algorithms, the server can do caching and indexing to speed it up. (Emphasis on “can”. IDK if Lemmy servers actually do this.)
With custom sorts, that goes out the window.
A plugin system that allows for custom scoring methods is a good idea, IMO.
This would allow instance admins to install whatever plugins fit their users’ needs. See how many instances are deploying multiple clients like default Lemmy, old Lemmy, alexandrite, and many more.
Arbitrary code execution is not safe, so it would probably require an admin permission to install plugins.
Some ideas for interesting scoring/sorting methods or feed generation in general:
- personalized AI scoring, most of us agree it’s bad when a big corp controls AI, but maybe it’s better if the user could fine tune it
- instance/community/other AI scoring, instead of super individualized scoring sometimes you may value the feedback of others including the communities you follow or your instance
- the explore feed that was recently suggested (or any another mixer), i.e. a mix of home, local and all based on factors the user can configure
- random
- Top/Hot/Active/New but only counting interactions from people you “trust” (i.e. you follow the same communities, or have at least X% compatibility) (again big corps bad, self hosting good)
Some of these services are not cheap to host, but the fediverse allows for multiple solutions to coexist. Some servers will charge for premium features, some servers will not provide them, some people will self host over engineered solutions, and big corps will show/sell ads and track user data (never forget big corps bad).
I literally just got down voted to hell for suggesting a personalized AI. I do really want it though. Obviously without the corporate greed, but that’s why we’re here.
I am guessing you didn’t add enough “big corps bad”.
PS big corps bad
And where are you going to find an A"I" that’s not the product of corporate greed literally stealing from the rest of humanity?
The modern approach to artificial “intelligence” is basically n-dimension mad-libs where
nis five figures or more. It can only work with truly awesome (in the original, literal sense of the word) amounts of training data which is first a) stolen, then b) categorized by biased reviewers (often exploited people doing that last job). There is no ethical A"I" and only the greedy corporate world has the resources to do modern A"I" at all.
Not particularly. I’ve said it before, but “Active” being the default sort is one of the big reason Lemmy is doing well right now, because it concentrates the conversations so that the front page is a lot busier relative to the userbase size.
You are free to submit a pull request to the Lemmy repository if you have a custom sort that you feel would be useful. It’s open source software, no need to have everyone to download a different sort from a third party marketplace.
I can’t be the only one who’s tired of seeing two-day-old posts on the frontpage with active, and less-than-an-hour-old posts with hot
I’m definitely there with you. I actually forgot that hot existed because it’s basically a ‘new’ sort.
On the one hand it’s handy because it makes me browse Lemmy less and do more productive things instead, but it isn’t good for encouraging growth of niche communities and topics if everyone is just looking at the same few posts.
The main problem I see with that approach is that the dropdown with the algorithms is getting ridiculously long. Even if the algorithms are only added through GitHub there should be a way to customize which ones to show.
Use a combobox? Seems easy to solve.
