Recent post re: AI as utility
Myself, I’m a fan of local LLM / self hosted ML… but if you ever needed a clarion call that a hard pivot is coming (soon) for online/ cloud based AI…Altman et al are making some concerning mouth noises (to say nothing of broader concerns with OAI, Anthropic etc).
Right now, I’m sketching out a plan where my Raspberry Pi (always on, 2-3w) uses a magic packet to wake up my modest AI server (Lenovo P330 with Tesla P4) if/when needed (Qwen 3.6-35B-A3B); no point in chugging down 80-100w, 24/7 for no good reason.
If the trend continues the direction it appears to be (increasing costs, environmental impacts etc) then I’d feel a lot better hosting my own as port of first call and replacing simpler tasks with more traditional programs. YMMV.
You must log in or # to comment.
Yeah.
It’s not even about efficiency, really, but independence from corporations, privacy, and principle. Kind of like Lemmy.
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
Fewer Letters More Letters ARP Address Resolution Protocol, translates IPs to MAC addresses IP Internet Protocol RPi Raspberry Pi brand of SBC SBC Single-Board Computer
[Thread #321 for this comm, first seen 30th May 2026, 09:50] [FAQ] [Full list] [Contact] [Source code]
I started working toward self hosting LLM for my small company using ollama and opencode as agent But I realized a good model like GLM 5 requures 250GB of RAM and 24GB vram with a 4080?? I dont know, this is what the LLM told me itself.
I ended up using qwen-code2.7-7b-16k.
Currently the best thing I have is my laptop, 16GB ram, i7 9750H gtx1650
How do you guys selfhost? What models do you use that are actually good?
I mean…that entirely depends on your use case - and I hate saying that. For me and what I do, Qwen SLM (esp Qwen3-4B 2507 instruct and Qwen3.5-2B) are exceptional. But I’m not trying to do Claude at home.
Best bet? Spend $10 on OpenRouter and try different models. In a head to head with ChatGPT 5.4 mini (excellent for coding BTW), I’ve found Qwen 3.5 27B more than able to hold its own for coding tasks…IF you narrowly gate it/confine it. The last batch of Qwen’s really are something. Dunno about the 3.7 series.
Having said ALL that, I’m really tempted to go back in time and code myself a deterministic expert system, with user updatable knowledge cascade, tool calling and a minimal amount of Markov chain word garnish for flavour. I think we use to just call that “a program” lol.
Really tempted actually, because if 50% of llm use case is basically Super Google but not shit…well, I can make that myself. I just need to point my autism at it.
PS: this might help
Qwen 3.5 24B is way too large for my specs. I’m barely running qwen2.5 7B
Hmm…it runs on a 1060…it’s a MoE not a dense. 24B is even lighter. Worth a shot.
https://www.youtube.com/watch?v=8F_5pdcD3HY
Else, if youre looking for a coding model (??) something like Sara or fara might suit
People will buy intelligence from us on a meter’
We have governmental surveillance and we have surveillance capitalism. Surveillance capitalism works so well that governments are now very interested in the data they collect, which is alarming. Unfounded conspiracy theory: It’s probably one of the reasons that governments don’t seem interested in AI’s regulation. If I had the proper equipment to run AI entirely local and efficiently so that the expenditure would justify it, I would.
You probably could. A Tesla P4 or P40 (old data centre cards) are more than up to the job. My Lenovo tiny hosts a P4 (card cost $100 on eBay; the lenovo itself was $200ish) and runs Qwen3.5-35B-A3B at about 20 tok/s. Smaller models are even faster.
https://www.youtube.com/watch?v=8F_5pdcD3HY
If you’re not bound by the one liter shoebox design, then the P40 is still a great and inexpensive card.
I think I mentioned elsewhere but right now I’m trying to figure out if I can use a magic packet from the Raspberry Pi to wake up the Lenovo as needed rather than leaving it on all the time.
if you’re selfhosting AI, make sure you at least firewall it off from the internet. many providers still send metrics back home that includes usage and content.
Respectfully, that’s not really how local LLMs work.
A GGUF model sitting on my hard drive has no ability to “send content back home” any more than a PDF or a JPEG does. If you’re running something like llama.cpp or Ollama entirely locally, the model weights are just data files.
The real privacy concerns are cloud APIs, telemetry in front-ends, browser extensions, analytics, update services, or accidentally exposing a service to the public internet.
“Self-hosted AI” isn’t one thing. There’s a huge difference between:
- Running ChatGPT through an API
- Running a commercial AI appliance
- Running a local Qwen/Mistral/Llama model on your own hardware
Firewalling internet-facing services is good advice. Assuming every local model is secretly uploading prompts is not.
EDIT: for the record, I didn’t down vote you - that was someone else.
not gonna self host bullshit that wastes resources and makes me dumber.
Me, looking at my Jellyfin server…
Oh. Ok.
Sure but all these self hosted ais are still done by companies who used massive amounts of power and water to train it.
Which is an interesting dilemma: Those AIs are already trained. That power and water was used. If you use them, you will not pollute anything. But you may encourage those companies to train another AI
No.
Even the biggest open weights models are trained on pennies compared to OpenAI and Claude. They just don’t have the hardware to be so wasteful.
In fact, the Nvidia GPU ban was the best thing to ever happen to “small” AI devs. It made them thrifty.
You’re still paying for electricity and a big part of the world is in a electricity crisis. “AI” has few real uses and LLMs are not one of them.
