Here is a detailed breakdown of the video titled "I Tested Apple’s Secret macOS AI… The Results Made No Sense" by Alex Ziskind:
Overview
The video explores a hidden, built-in Large Language Model (LLM) command-line interface (CLI) tool called FM that quietly ships with macOS (specifically tested on macOS beta version 27). The creator investigates how fast this built-in AI tool runs, compares performance across different Apple Silicon hardware (an M4 Pro Mac Mini ver
sus an M3 Ultra Mac Studio), and solves a performance mystery regarding which hardware components actually handle the workload.
Key Sections & Details
1. Introduction to the Hidden macOS AI (FM)
What it is: macOS includes a built-in CLI tool called fm that lets users run large language models directly inside the terminal without installing third-party runners like Ollama, LM Studio, or MLX [01:48].
Cloud vs. Local: The tool features two distinct modes: a local on-de
vice mode and an Apple cloud-based mode via "Private Cloud Compute" [04:05]. The cloud version runs roughly three times faster than the local model [04:30].
Sponsor Interlude: The video briefly highlights Super Nori, a proactive family AI agent designed to run in the background and anticipate daily scheduling or family coordination issues [00:25].
2. The Benchmarking Challenge
Standard third-party benchmarking tools (like Llama Bencher) failed to read Apple's endpoi
nt correctly because Apple's server does not fully comply with standard OpenAI API timing outputs, returning erroneous results like 600,000 tokens per second [02:54].
To resolve this, the creator built a custom benchmark tool called Apple FM bench to measure real on-device metrics [03:41].
3. Hardware Comparison: Mac Mini vs. Mac Studio
The Contenders:
Mac Mini (M4 Pro): 273 GB/s memory bandwidth [04:52].
Mac Studio (M3 Ultra): 819 GB/s memory bandwidth and a much m
ore powerful GPU [05:25].
The Mystery: Despite the Mac Studio being vastly more expensive and boasting significantly higher memory bandwidth, running the local FM model yielded nearly identical benchmark results on both machines [05:50].
4. Investigating the Bottleneck & Hardware Usage
Meter Inaccuracies: Standard monitoring utilities like Mactop and Azytop (which rely on Apple's powermetrics under the hood) incorrectly reported 0% utilization on the Apple Neural En
gine (ANE) during tests [07:05].
Stress-Testing Strategy: Since meter software could not be trusted, the creator designed an experiment forcing the engines to compete:
Hammering the Neural Engine caused the decode stage of FM to slow down [09:08].
Hammering the GPU caused the prompt processing (prefill) stage of FM to slow down [09:12].
Conclusion:
Prompt processing runs on the GPU [09:27].
Token generation (decode) runs on the Neural Engine [09:26].
Summary Diagram
of Hardware Utilization
Plaintext
+-------------------------------------------------------------+
| macOS FM AI Tool |
+------------------------------+------------------------------+
|
+-----------------------+-----------------------+
| |
v v
+-----------------------------+
+-----------------------------+
| Prompt Processing Stage | | Token Generation Stage |
| (Prefill) | | (Decode) |
+-----------------------------+ +-----------------------------+
| |
v v
+-----------------------------+ +--------------------------
---+
| Runs on GPU | | Runs on Neural Engine |
+-----------------------------+ +-----------------------------+
| |
v (Scales with hardware) v (Identical across M2-M5)
+-----------------------------+ +-----------------------------+
| Benefited by Mac Studio's | | Limited by 16-core Neural |
| mass
ive memory bandwidth | | Engine present in both Mini |
+-----------------------------+ +-----------------------------+
Final Takeaways
Purchasing Advice: Because the decode stage relies heavily on the Neural Engine—which features a standard 16-core layout across Apple Silicon chips—a base Mac Mini delivers essentially the exact same performance running Apple's built-in AI tool as a maxed-out, much more expensive Mac Studio [09:32].
Alternative Frameworks: Unlike Apple's built-in tool, frameworks that rely entirely on large local models (such as MLX or Llama.cpp) actively scale with the massive GPU and high memory bandwidth of higher-end hardware like the Mac Studio [10:09].
Beta Status: The implementation is subject to change by Apple since the feature remains in beta [10:43].
http://www.youtube.com/watch?v=8vDuIVlfeV0
I Tested Apple’s Secret macOS AI… The Results Made No Sense
Alex Ziskind ·
Viewer Discussion & Comments