sigh I just wanna upgrade my video card, man
cant wait for their debt calls and they have to sell their inventory to make payment.
y’all realize this is what a hostile takeover looks like, right?
Are you talking about a stocks takeover? Because I fail to see how buying up hardware has anything to do with it.
Intel must be kicking themselves for not using their own fabs for Arc, AND divesting from Micron, Optane, and HMC.
Apple and others must be too, for snubbing Intel and Samsung.
They would be making money hand over fist, and relieving the market at the same time.
My new gaming PC that I bought about 18 months ago went up in value by 75% based on its RAM and GPU price and while I’m relieved I got it when I did, that’s pure insanity that a PC would appreciate in value
I overbuilt built my NAS/container server for $1262 excluding the price of the rack chassis, PSU, cooler and two 1TB NVMe SSDs I cannot recall the price I paid for. B550 board, Ryzen 5, four 10TB Red Plus HDDs, Intel Arc A380, 64GB of DDR 5 ECC. Those listed components are now $3381. When or if you can even find them. The Arc is unobtanium as far as I can see. Of course the RAM and HDDs went up the most in cost, but it’s still like $65 and $75 more for the mobo and chip.
Whoop, decided to check and I found the SSDs: I paid $60 a pop for Samsung 980 Pros and they’re $240 now. So $1382 then vs $3861 now. Almost triple. The bubble can’t pop soon enough.
Tell me about it, my rig went up by ~30% and to think I was worried it wasn’t future proof enough
I’m convinced that OpenAI buisness model is to create AI bubble, to sell its “reserved memory wafers” and now Macs, at the top of ram extortion pricing moment.
That same thought just occurred to me: hype a McGuffin and get investors to lend you insane amounts of money which you then use to buy up a marker, causing artificial scarcity, then sell at massive profits.
Well, this may explain why my work computer has been on back order for the 4 months that I have had my job. Still have no idea when I’m actually going to get it.
Smells like 'Open’AI is digging another moat where there is none, like it did with the memory wafer deal that started RAMageddon. Heaven forbid people actually being able to run local models, can’t run local models if 'Open’AI has bought all the capable smaller machines, taps head. Fuck SamA and his market cornering with a rusty chainsaw. I guess they think all they have to do is keep that moat somehow until IPO and then it’s somebody else’s problem, they and their investors cash out. Dear Goddess I hope it blows up in their faces.
Ah, I see you’ve stumbled across the real reason for the “AI boom”.
Hint: it’s not because the models are SO incredible. It’s the fascists taking computers away from us so we can’t stifle them. So we can’t talk shit on billionaires anymore. So we can’t fundraise for a better world.
Just like our cars and our traffic lights watch our every move. Just like our phones have 100 sensors that profile us from the moment we wake up.
Demons took the wheel a long time ago.
Yeah, I’ve had the thought. ‘What are we going to do with all that compute now that AI as worker replacement is a bust? Why not run panopticon and get rid of that pesky general compute in the hands of the masses?’
Centralizing compute has been tried again and again, initially mainframes, then the home PC, thin client / server, cloud compute, arguably phones (as apps processing in the cloud), Geforce Now, AI. So far it’s always gone back to the edge for most of the population, except for niche cases (or not in the case of phones, but they turned into general, if locked down, compute), good chance it won’t stick this time either.
The niche case here is model training.
OpenAI buying tens of thousands of Macs for AI is the funniest possible plot twist: spend years convincing everyone the cloud is the future, then discover the hottest AI infrastructure is basically a warehouse full of Mac minis. Somewhere, an IT admin is whispering: »I told you so.«
What exactly do you think a cloud is? It’s built exactly like that, computers connected and orchestrated to run a client’s arbitrary services.
Now the interesting thing here is to choose many different types of smallscale hardware. Must be some fun to administrate.
Exactly. It’s super uncommon to use standard consumer hardware to build a data center.
I’d be interested in buying one as a headless server, but a lot of the FLOSS software I’d like to run seem to prefer x86 processors.
Truly? Even with their translation layer? I forgot what’s it called.
I was thinking more about things that are written for Linux and not necessarily ported to macOS. When I’ve tried things in an ARM VM there seems to be a lot less software compiled for ARM.
The shared memory for cpu and gpu was a cost saving thing, now it’s a massive help as nvidia limit consumer gpus to 24gb
The translation later is called Rosetta 2 (Rosetta 1 having been for the PowerPC to Intel transition), and it looks like it’s mostly going away next year, so buying a Mac with the intention of using it would probably be a bad idea at this point.
Which really really really sucks, because Rosetta is amazing. it’s extremely performant at running x64 code on Apple Silicon, and I’ll be sad to see it go.
To Apple’s credit, though, they did extend its end of life a full year. Rosetta 2 Support was supposed to end this year with the release of macOS 27 Golden Gate, but due to enough backlash, they extended it to next year. (which they also did with Rosetta 1, back in the day.)
The real thing I’m pissed about is they won’t open-source it, which Apple sometimes does (or, used to do) with its tools.
Fwiw they claim they will support it as a framework into the future, so things Linux VMs can still take advantage of it. The only thing you’re losing is desktop Mac apps which have largely all converted to arm64 at this point, except games which you probably aren’t going to play natively anyway
Well— Apple wants devs to support Apple silicon. Cyberpunk 2077, for example, did very well as a native release. But a lot of that was tied to Steam’s (and GOG’s) “works anywhere“ purchase model.
And as far as “framework for the future“ are concerned, yeah, we’ll get there independently eventually, but nothing compares to Rosetta. It’s just fucking amazing, and it works now.
That’s true Linuxaand other tools, etc works better on x86 than arm to this day. Might slowly change if arm gets more popular. But that might take another decade.
Seems like Apple‘s little gambit to be immune to AI price fixing savagery didn’t work as well as they thought it would…
I guess Rosetta 2 was just too good…
Rosetta 2 has literally nothing to do with this.
Rosetta 2 is irrelevant here because they’d just compile their shit for ARM for better performance.
At least unlike B300s, these can one day be used if OpenAI dumps them on the used market. IF they do that instead of trashing them to make sure everything stays expensive.
True, but getting there probably involves using Rosetta 2 to arrive at a solution for taking full advantage of the Apple Silicon SoC memory exchanges and other Apple proprietary stuff. Those chips aren’t bog-standard ARM chips and the Asahi Project (bless their hearts) have barely broken into the M3 chips— so… if OpenAI are leveraging Apple Silicon from the last year or 2, they’re building out their own solution using Rosetta as some sort of bridge.
Edit: which also means they must be getting help from Apple directly, ew. And that really pisses me off, because I have long maintained that Rosetta should be open-sourced.
I don’t understand why Rosetta has to be involved in any way at all.
Apple provides APIs for programming low level GPU instructions, including AI workloads, on their chips, using the Metal API. Anyone interested in using Apple hardware to its full potential can just write Mac-native software, same as any other MacOS native software directly compiled as binaries to run on Mac M series chips. No Rosetta required for translating x86 instructions to ARM, when the whole thing compiles for Apple’s native instructions in the first place.
Uhh, you’re skipping a few steps
Steps that either would not work between architectures or that Rosetta would handle (thereby answering your own question)
And I’m not here to teach a computer science class
Steps that either would not work between architectures or that Rosetta would handle (thereby answering your own question)
What? They would compile the code that they control for the M-series architectures. The target architecture is M-series ARM chips running on MacOS.
Rosetta is for translating x86 to ARM. Which wouldn’t be necessary because there are no x86 binaries involved at all.
Oh, I guess you’re right about that
My mistake
No, @GamingChairModel@lemmy.world is right.
What does Rosetta have anything to do with what OpenAI needs? Machine learning on Macs is done in MLX or something else with Metal, running OSX. Most agenic frameworks or other ML-related infrastructure are just Python, or open code that can compile on OSX/Macs just fine. In fact, OSX is often the first target platform.
This even includes inter-node frameworks for clustering Macs together.
No one doing “AI” on Macs is using Rosetta. There’s no point, and its a bad idea. They’re doing everything in OSX, which works fine, and I can’t imagine what OpenAI would be doing with them that wouldn’t work in OSX.
But maybe I’m missing what you’re implying here. And I’m sorry to come off as thorny, but condescending comments like “I’m not here to teach a computer science class” rub me the wrong way.
If you don’t know, then you are really not in a position to say that I’m wrong LMAO
Have fun with your little “my ignorance is proof that I’m right!” concept, lol
Oh, and the fact that you even brought “AI” into the conversation is hilarious
The two of you should get together and do a stand-up routine called “I and O my God I have no idea what I’m talking about”
You didn’t answer my question.
What does rosetta have to do with any of this?
Apple‘s little gambit to be immune to AI price fixing savagery
I don’t think it was ever a strategy. If it was they would’ve stopped mac’s from ever reaching OpenAI’s datacenters. Or atleast they’d have prevented them from hoarding so much of the M5 production.
But they did. For a while…
At this point it’s pretty blatant that OpenAI are engaging in price fixing more than they are actually delivering AI datacentres.
Local 27b models are good enough for most tasks.
Can’t wait to buy one of these from Ebay for 10% of the price next year.
https://github.com/AtomicBot-ai/atomic-llama-cpp-turboquant
I’m running gwen 3.6 with 131k context window on a 3090, it’s fast enough and about as good as pay to play Claude at work.
Upgrade that to 3.8 as soon as your hardware allows (and your use case makes sense). 3.8 is quite a bit more rational.
I plan to once there is a version with turboquant and MTP as that huge context window is key.
A used 3090 is like 2-3k though, IF you can find one :| A month of claude is like 20 EUR. A month of opencode go is half that, but you get less usage.
Yeah no way they will allow any of this hardware to go back onto the market. Anything they dont use anymore will be destroyed.
Buy it, destroy it. Just like buying bunch of old books, train their LLM’s and burn it. Humanity has gone a long way to be that stupid.
And that’s really why they’re hoarding them.
No, they’re hoarding them so you have to pay for cloud services they control from now on. With your little Fire tablet. No more pirating movies or political organizing for you, piggy
For what it’s worth, piracy is the primary thing I use my older 10" fire tablet for.
Any 27b Model you can currently recommend for a 16gb AMD ? Mostly coding tasks but not exclusively.
There is a way. There was a post yesterday on exactly this, let me find it… https://lemmy.world/post/51283416
Thx I’ll give this a try!
9060xt 16gb is the most cost effective new GPU, but if you’re going used look for a V620 on eBay. It’s a 6800xt chip but in server form factor GPU with 32GB vram. Can be a bit of a pain to set up but by far the most cost effective option IMO
V620s were a good deal when you could get them for $350, now they’re $700+ and no longer a good deal.
There just aren’t any good deals any more. Prices for everything have gone crazy in the last few months. For coding LLMs the cloud services may now be the least worst value, by design, until they hike the prices.
That said, I still just paid way too much for a used graphics card so I could do many things locally, because I just don’t want to give the likes of Sam Altman a single penny.
Oh wow you aren’t kidding. The dude I bought from on eBay @ $350 in February is sold out now. Guess I retract my statement.
This AI pricing is wrecking every deal on the market lol
Thanks! actually have a 6800xt already and was asking for model tips. I saw my question was easily read as asking for GFX card tips, edited.
For your hardware, the VRam is not enough to run 27b but, I’d recommend Qwen 3.5 9b for image / text to text.
And I’m planning to experiment with Qwen 3.8 9b for text to text.
4_k_m quantization is the sweet spot for performance and ram usage.
Also, I find Llama cpp is better than Ollama in terms of performance.
Thx for the tips!
Has the computers for local AI become that good?
Macs, in particular, are likely to be the best option at the moment, unless you absolutely need the largest Frontier models.
Just wait for Mac m5 ultra
The Mac Studio already has it. And from October, it’ll also be available with 512GB. It does cost around 18K, though.
That’s pretty crazy. I’ve been reading that models around 14-32B is plenty for daily tasks and then some.
The bigger the model the bigger the memory requirements.
M5 ultra as indeed up to 512gb of unified memory. So you can run very large models. At the same time it’s much much faster memory speed and bandwidth, almost simular to the nvidia rtx pro 6000.
Sure the price is also high. But you get much better value for this setup compared to what nvidia is delivering you today. I still want to see even more competition… Not just from Apple.
What I’ve been reading around lately, that’s too much for the daily/major of users.
No, no, they have not.
I did the math the other day, and the cost of the mac versus the token rate that you get off of it for a decent model would take you over 70 years of 100% 24/7/365 usage to pay for itself versus just using hosted inference.
They are wildly poor options for local AI models, at least on a cost basis. If you don’t care about spending $10,000 to run a halfway okay model at 100x the cost, then go for it.
Does that include energy usage (which I would think Apple being in the top 5 of being energy efficient) and just for AI? I’m sure most would be doing more than just hosting and running a single model.
You could include energy usage in the equation, but if you’re buying a $10,000 Mac Studio to edit your PDFs and watch YouTube, then that’s your own decision. That’s also not what we’re talking about here.
The Mac is still going to be incredibly energy efficient.
Let’s break it down for Qwen 3.5 MoE.
- M4 Ultra Mac Studio:
- 22t/s
- ~ 210Wh
- 9.55 watts per token/s
- B300 GPU:
- ~ 2160t/s per GPU
- ~ 1750Wh (14Kwh 8 GPU System)
- 0.88 watts per tokens/s
The Mac Studio comes out using ~ 11-12x more energy per token. Making it extremely energy inefficient at this task in comparison.
Not saying it wasn’t implied but didn’t offer it in my original comment but yeah, that’s what I’m talking about. It’s a computer, PC, desktop, workstation…… mostly like someone would be doing other things instead of just AI. You might be talking about just AI on the Mac Studio. To be fair, neither of us clarified our standings, till I ask further questions based on on my original comment. So you’re not wrong, just not what I was getting at. Thanks for sharing though.
- M4 Ultra Mac Studio:
Well local models are getting better. Open weight models are a real thread to their business. OpenAI’s prices certainly wouldn’t stay that low if they were the only player. Their prices are not profitable at the moment anyway.
They really aren’t.
Have you used open weight models for serious jobs? Ones that you can actually run effectively on a Mac studio? (Not deepseek v4 pro, not Kimi K3, not GLM 5.3)
They work alright, at best, sometimes.
And the ones you can’t run effectively, locally, like Kimi K3 are considerably better at demanding tasks like software engineering. But even then they still suck at that job compared to frontier Anthropic and OpenAI models. I’ve been building with all of the above, and the open weight models just overall suck at serious high demand software workloads.
I just tried some local llm on my gaming rig, sufficient for chat and images.
But buying the rig just for this is most likely not worth it.
Madoff Altman is at it again.
They’re the worst kind of scalpers
