I feel like people have been misunderstanding the new AMD Ryzen AI Max chips (and relatedly, the Framework Desktop). It’s not a “laptop CPU with a GPU tacked on”, it’s a laptop GPU with a CPU tacked on. By all accounts, it’s a GPU-first system - silicon area, memory bandwidth, etc. When comparing it with other AMD laptop chips, it becomes obvious that this is a GPU first, and a CPU second.

Which is why I dislike the criticism of Framework Desktop’s soldered memory. Nobody bats an eye if GPU’s memory is soldered (when was the last time that it wasn’t?), but because people see the chip as a CPU, and thus see the soldered down ram as a regression.

You should in all regards look at the chip as a GPU first, if you’re considering to buy something with it. Sure, it has CPU chops, but you can get those way cheaper, and without the limitations that being attached to a GPU brings - if you’re not buying it for the GPU, then get something else. Comparisons with other laptop CPU-centric chips are IMO misguided, the comparison should be done with other GPU’s first, and CPU’s attached to them second.

@ignaloidas so this is kinda more like a budget Nvidia grace hopper thing I guess

Yeah honestly given that they apparently did try to make camm2 work but couldn't get it stable makes it more understandable why they resorted to soldered ram. Makes sense for that sort of application.

@ignaloidas or rather it's basically AMD's answer to the Apple M series chips, which people seem more willing to forgive for using soldered ram

@ignaloidas afaict those also dedicate more die space to GPUs, NPUs, and other accelerator ASICs over the CPU

@mikoto Yeah, much more into that direction.

It looks cool, but I’m absolutely not in the market for one even if I have the budget, because I do jack shit with GPU’s, and the CPU side is way more important to me. I don’t game, I don’t do LLMs, I don’t do any kind of HPC that could go on a GPU, so I have zero reason to buy a system that’s GPU first, and a CPU second.

@mikoto more to M Max/Pro ones, but yes

@ignaloidas I personally would like to see other ML stuff used with these things, LLMs kinda feel like the most boring application of this sort of tech. Things like better image recognition, upscaling, more accurate simulations, etc.

@mikoto yup

@mikoto you don’t really need a huge GPU for things other than LLMs tbh, and I think most NPU’s on SoCs are better and more efficiently used for that other stuff like image recognition (at least on inference side, for training you still want a GPU). Such GPUs are good for LLMs because of decently high compute power, combined with high memory capacity and bandwidth - 128GB of memory in direct access to the GPU allows to use larger models more efficiently, as the weights don’t have to be streamed into from “main memory” through a PCIe connection, which is a bottleneck for traditional GPUs, and it’s exactly why a lot of the LLM crowd is excited for this - it’s exactly what they need for relatively cheap (in comparison to dedicated GPUs with a ton of memory)

@ignaloidas @mikoto
Yeah, it's "GPU first" but who needs a GPU with 32 GB of RAM? Mostly people who do ML.

@wolf480pl @mikoto I mean, consider that the ram is shared - IIRC you talked about 16GB being bare minimum on the CPU side, and for GPU’s, on the higher end at least 10GB for that level of GPU makes sense.

And that the same chip is intended for “mobile workstations”, where you do CAD and stuff, where all that RAM is very valuable. It makes sense beyond ML.

@ignaloidas @mikoto and I think Framework marketing it as a small gaming PC you can take to LAN parties is disingenous. If that's your usecase, get a gaming laptop, which comes with a screen.

@wolf480pl @ignaloidas @mikoto Also it could be a better gaming rig as well with differently architected memory. Pretty sure AMD will sell you a kit with the APU and RAM already. So thats why its soldered I guess. Kinda weird for framework to have such a device but okay.

But tbh if its a GPU-centric device then why use LPDDR5X (and soldered one at that?). I mean yeah the soldering does give you slight latency advantage but certainly here it is done to save space? Or for better clocks. I mean 8000 seems high. But GDDR will certainly run circles around and be way better for the GPU-portion of the APU. These things bandwidth starved for sure on DDR limiting performance. I don't know if bandwidth is important for other GPU-heavy tasks like rendering, editing and simulation though.

But yeah even the 395 AI Pro Max+ was a TDP of 45-120W. Prett, sure with 120W power envelope you could have done better memory wise even if GDDR used more Power. But thats on AMD not Framework.

It is kinda cool to see a 40-CU APU-Monster though. From the architecture I would not see this as "GPU-First" though. It has a strong 16-Core CPU on it and still uses DDR instead of GDDR. Consoles are more GPU-oriented in that regard.

It just is hella overpriced to be a desktop tbh. Yeah its small and uses less power but thats it. If you want to do serious tasks an AM5-Desktop with an Upcoming AMD-GPU will just be better. That means if power draw is of no concern. Also would game better for less money

@stefan @wolf480pl @mikoto AMD doesn’t directly sell that chip - it’s a laptop chip after all. Framework’s desktop mobo is the first one that’s publicly available in a standard format AFAICT

On the memory, I think it’s purely because of the attached CPU - while there’s no real problems with using GDDR with them, it’s quite different target to optimize, and the performance of many regular programs would suffer. You can allow that if you’re building a console, but for a more “general purpose” chip, it’s not great optics to loose a bunch of performance on CPU-focused benchmarks, even if the product is primarily GPU focused.

I’d say that it is as much GPU focused as possible for what it is - I mean, just look at the silicon area and IO space for memory that’s been used up for GPU side of things. Sure, consoles can be even more GPU focused, but those are not expected to run general purpose tasks with good performance, so latency hit on memory accesses can be eaten up.

And yeah, the value for it as a desktop really depends if you can utilize what’s unique about or not. I don’t see the value for majority of people.

@ignaloidas @wolf480pl @mikoto Lots of good arguments I can agree on tbh.

Though the argument on size is I think a bit flawed. GPUs will almost always use more die-space than the CPU thats just how they are. Even on the shitty Intel graphics it was and is like that.

I have seen the framework is a small box and GPUs are huge these days, but you can still build small very powerful rigs for the money they asking and will be better still. Also you likely wont be limited on power as much, rather cooling.

It has a somewhat AI-focus as the name suggests. You can have a very big memory Pool for that GPU allowing for loading big, big models. It should do well in its designed workload.

But tbh framework wanzs a lot of money for convienence alone, also with no screen, no battery or keyboard making it a bad deal. for 2300€ you should be normally be able to get quite the decent standard-sized desktop. But I guess like with ITX you pay a hefty premium for size alone. It may br cute but the value is not there for me certainly.

@wolf480pl @mikoto 🤷 It is a fairly good small gaming PC that’s easy to take wherever. Whether it makes sense for the price is another question.

@ignaloidas @mikoto you can use a Ford Transit to drive to work everyday, but you don't see it advertised for that usecase

@wolf480pl @mikoto I don’t think there’s anything inherently disadvantageous with putting that chip on a gaming desktop. Whereas there would be disadvantages to driving a van purely for commute. I’d see it more as driving something like a Porche 911 for commute - it’s perfectly fine for that, but it doesn’t necessarily make sense financially.

@stefan @wolf480pl @mikoto yeah, the pricing is the most questionable part. I guess it miiight make sense if you’re just buying the mobo, as it’s kinda the same as just getting a high-end CPU/GPU combo, but you should really think if you’re going to utilize the unique advantages or not, because the performance doesn’t fully match the price, even if for some workloads it is very good.

@ignaloidas @wolf480pl @stefan Yeah pricing is definitely the part I have the most concern about