Not gonna lie, I didn’t make a lot of headway with my local inference experiments with my beefy gaming PC. A beefy gaming PC with one GPU, even the best one I could buy, is not an inference workstation.
The models I could run ran lightning fast, but they were no match for the frontiers in coding and that’s most of what I use AI for.
Really curious (and tempted) to go down this rabbit hole with the new Apple hardware. Wanna see some data from people actually using it.