"My job is to make the tokens as cheap as humanly possible... I will use every chip, I'll use every source of power, and I will use every piece of land in the United States that's suitable for this." - Neil Movva [00:00:00]
"We still treat the agent as a person that is expensive to consult and you should ask them when you have a hard question... that's not the way to think about intelligence." - Neil Movva [00:00:11]
Disclaimer: Orignal content owned by or sourced from third parties. It does not represent the views of 'Nuggets' platform or it's team. AI is used extensively across this platform including for summaries. Accuracy is not guaranteed, there can be mistakes. Any info or content on this platform is not a financial, legal, or investment advice. Do your own research. Refer for complete disclosures:- Terms of Use · Full Disclaimer
"Whenever you make something 10 times cheaper, it's a new product category, and we aspire to do that for tokens." - Neil Movva [00:01:41]
"Security has become proof of work. When you want secure software, it's really a question of how many dollars did you spend on Anthropic's APIs trying to break into your software." - Neil Movva [00:07:59]
"I like the phrase that internet was a one-time subsidy on data... we've basically looked at it all already." - Neil Movva [00:36:19]
"I don't think there's a point in specializing to say we work on making the model better for just kernel engineering... we're all downstream of the frontier." - Neil Movva [00:41:54]
"There's no bad chips, there's really bad pricing, and I will make any chip work at the right price." - Neil Movva [00:46:15]
Speakers & Credentials
Patrick O'Shaughnessy - Host of Invest Like The Best, Founder and Managing Partner at Positive Sum.
Neil Movva - Founder and CEO of Sal Research; former NVIDIA GPU kernel and hardware engineer (joined in 2016) who worked on early Tensor Core performance optimization and speed-of-light GPU acceleration.
1. Executive Summary
The core premise of Sal Research is to turn raw compute into an abundant commodity by building a low-cost "token factory" optimized specifically for long-horizon background inference rather than low-latency real-time chat [00:00:53].
The AI paradigm is shifting from short, real-time human-in-the-loop interactions to autonomous background workloads that execute for hours, days, or weeks, driving non-real-time agentic workloads from a 50/50 split today to 90% of total token volume [00:06:25].
While the broader market optimizes for low latency via expensive interconnects like NVLink, long-horizon agents can easily tolerate sub-linear trade-offs, enabling the use of diverse, non-NVIDIA accelerators (AMD, TPUs, Trainium, Etched) at drastically higher FLOPs per dollar [00:22:26].
To bypass power grid and data center bottlenecks (where 100MW+ locations are functionally unavailable in the US), Sal Research adopts a "scavenger strategy" by purchasing smaller, 1MW distributed power pools without redundant infrastructure or backup diesel generators [00:54:19].
The team tolerates data center uptime as low as 95% by managing single-point failures via a resilient control plane that dynamically moves background agent execution across distributed sites without affecting end users [00:57:31].
Transformer architectures are fundamentally constrained by memory-bound attention layers sitting adjacent to compute-bound MLP layers, creating massive inefficiency in Key-Value (KV) cache storage that requires radical architectural compression or hybrid chip configurations (e.g., SRAM + DRAM/HBM) [00:28:49].
Human data sets are exhausted after consuming ~30 trillion tokens of clean internet text (or 300 trillion broad tokens), pivoting the future of model capabilities toward self-grading Reinforcement Learning (RL) environments that test models against verifiable tasks like cybersecurity or formal math [00:36:25].
Hardware efficiency gains across chip generations (Hopper to Blackwell to Rubin) have plateaued at the transistor level, meaning compute scaling won't suffer massively if fab access shifts, as Western foundry tech is at worst 2x behind TSMC in performance-per-watt [01:16:55].
2. Chronological Table of Contents
Introduction & What Sal Research Builds [00:00:00]
The Shift to Long-Horizon Agents & Background Workloads [00:03:09]
Use Cases: Deep Research, Cybersecurity, and Proactive AI [00:06:35]
Verifiable Tasks vs. Non-Verifiable Taste [00:09:59]
NVIDIA's History, Tensor Cores, and Speed-of-Light Culture [00:13:02]
Latency vs. Throughput: The Foundational Hardware Trade-off [00:18:10]
NVLink, Interconnects, and Heterogeneous Hardware [00:20:53]
SRAM vs. DRAM/HBM & Cerebras Wafer-Scale Architecture [00:23:44]
The Mechanics of Transformers & The KV Cache Problem [00:28:49]
The Exhaustion of Human Data & The Shift to RL Gyms [00:36:19]
Software Stack, Kernel Engineering, and GPU Utilization [00:39:16]
Secondary Chip Markets, AMD, and Heterogeneous Fleet Arbitrage [00:44:39]
Data Center Bottlenecks & The 1MW Distributed Strategy [00:53:14]
Intermittent Energy: Solar, Wind, and 80-95% Uptime Models [00:58:53]
Open Source vs. Closed Source, Distillation, and Data Subsidies [01:10:17]
Fab Economics, TSMC Dependencies, and Transistor Limits [01:16:32]
Company Culture, Hiring for Curiosity, and Closing Thoughts [01:08:03]
3. Detailed Thematic Summary
The Abundance Paradigm & Long-Horizon Inference
Sal Research positions itself as a specialized "token factory" built to deliver open-source model inference at the lowest dollar cost in the industry [00:00:53].
The company hosts cloud-native sandboxes—long-running agentic virtual machines designed to execute tasks autonomously over hours, days, or weeks without human-in-the-loop intervention [00:01:11].
Making tokens 10x cheaper fundamentally creates new product categories, shifting user behavior from asking synchronous, high-cost questions to letting agents self-administer dynamic token budgets [00:01:41].
The market previously over-indexed on ultra-low latency inference (100+ tokens per second), pushed by early coding tools like Cursor [00:03:51].
Long-horizon agentic workloads do not require high token generation speed; spitting out 10 tokens per second is sufficient if the agent works proactively in the background while the user sleeps [00:04:14].
Test-time compute scaling proves that giving an agent more reasoning time yields higher-quality outcomes, a concept validated by models running continuously for over an hour on complex tasks [00:05:15].
Workload distributions will shift from 50% real-time and 50% background today to a 90% background / 10% real-time split by the end of the decade [00:06:25].
Parallel Web Systems uses long-horizon tokens to continuously index and monitor changes across the entire internet at exabyte scale [00:06:54].
In cybersecurity, generating code is easy, but discovering software vulnerabilities requires running specialized agents across thousands of distinct vectors [00:07:15].
Cybersecurity has effectively transformed into "proof of work," where application security is directly correlated to the raw dollar amount spent hitting Anthropic/OpenAI APIs to attempt autonomous exploitation [00:07:59].
Frontier model capabilities are "jagged," meaning smaller models like Haiku or specialized open-source checkpoints frequently uncover logic bugs or memory vulnerabilities missed by massive frontier models [00:08:14].
Long-horizon agents excel at verifiable problems—domain areas where an outcome can be objectively confirmed, such as software compilation, mathematical proofs, and scientific discovery [00:10:08].
Complex, multi-step scientific investigations that historically required millions of dollars in compute are rapidly compressing down to hundreds or tens of dollars per definitive result [00:10:34].
Non-verifiable tasks—such as creative writing, artistic aesthetics, and human taste—remain outside this quantitative optimization push [00:12:31].
Matrix multiplication serves as the core atomic unit of computation in machine learning because linear algebra compactly represents complex data relationships [00:14:00].
NVIDIA introduced early Tensor Cores in 2016 by reallocating 5% to 10% of precious gaming GPU die area to accelerate matrix operations after executives noticed researchers buying consumer GPUs at conferences like ICML and NeurIPS [00:15:08].
GPUs are throughput-oriented parallel machines that maximize efficiency when fully saturated, contrasting sharply with real-time interactive chatbots that force sub-optimal compute utilization for low latency [00:18:10].
High-speed interconnects like NVIDIA NVLink cut minimum operation latency by sharding matrix multiplications across up to 8 GPUs, but yield sub-linear scaling (e.g., 8x hardware for 4x–5x speedup) due to communication overhead [00:21:04].
SRAM (Static RAM, built using 6-transistor cells directly on the logic die) offers extreme bandwidth (e.g., Cerebras delivering 21 petabytes/sec), but suffers from terrible spatial density [00:24:24].
DRAM/HBM (Dynamic RAM, built using capacitors that refresh state every ~50ms) provides massive capacity (e.g., 288GB on NVIDIA Blackwell vs ~500MB on-die SRAM), but exhibits orders of magnitude lower bandwidth [00:25:17].
Wafer-scale architectures (like Cerebras Engine 3) link entire silicon wafers to gain ~50GB of SRAM per wafer, enabling 1,000+ tokens per second on model weights, but remain severely bottlenecked by dynamic Key-Value (KV) cache expansion [00:27:08].
The Limits of Transformers & Data Subsidies
The fundamental design flaw ("original sin") of the Transformer architecture is placing the memory-bound attention layer directly alongside the compute-bound Multi-Layer Perceptron (MLP) layer [00:31:38].
The KV cache stores historical token state for attention calculations during long context interactions, frequently ballooning to exceed the memory footprint of the actual model weights [00:29:20].
Future hardware serving architectures will likely be heterogeneous: pairing SRAM/Groq-style accelerators to host the compute-heavy MLP model weights with standard GPUs to manage the large HBM-bound KV cache [00:32:03].
The public internet provided a "one-time data subsidy" of ~30 trillion high-quality text tokens (or ~300 trillion broad tokens), which modern frontier models have already completely consumed and iterated over [00:36:19].
Random human feedback (RLHF) from average users has lost its value because frontier models now surpass average human intelligence; model progress now requires expert-level human feedback or automated synthetic environments [00:37:06].
Model self-improvement is shifting toward isolated Reinforcement Learning (RL) "gyms," where models generate synthetic data and iterate continuously within verifiable, self-grading environments [00:37:25].
Scavenger Infrastructure: Arbitrage, Energy, & Distributed Power
Software kernel engineering (such as FlashAttention) is no longer a permanent defensible moat, as frontier coding models now assist or automate custom GPU kernel creation based on high-level whiteboard specifications [00:40:56].
Alternative chips (AMD Instinct, Google TPUs, AWS Trainium, Etched) suffer from market perception issues and under-developed stock software libraries, creating immense arbitrage opportunities for teams willing to write custom kernels [00:46:15].
Traditional AI data centers are built for high-density training workloads, making 100MW+ facilities nearly impossible to site or power in the US without multi-year utility delays [00:53:35].
Sal Research executes a "scavenger strategy," buying fragmented 1MW power pools across the US—a size scale plentiful in aggregate but ignored by massive training deployments [00:54:19].
By removing expensive data center redundancies (such as secondary/tertiary fiber lines, diesel generators, and uninterruptible power supply systems), infrastructure costs drop precipitously [00:56:53].
The platform intentionally tolerates low data center uptime (95% or even 80%) powered by intermittent solar/wind, using a dynamic control plane to re-route active agentic workloads whenever a localized cluster goes offline [00:57:31].
Semiconductor performance-per-watt gains across recent nodes (TSMC 5nm, 4nm, 3nm, 2nm) have slowed dramatically, proving that Western foundries like Intel are at worst 2x behind TSMC, meaning fab geopolitical shocks would be manageable [01:16:55].
The Reference Vault
4. Data & Figures
Data Point
Value
Context
Timestamp
Early Tensor Core Die Area
5% – 10%
Silicon area allocated on 2016 NVIDIA GPUs for matrix multiplication.
The Bus vs. Private Car Latency Framework [00:20:14]
Synthesis: System optimization requires choosing between absolute per-user speed or batch throughput efficiency. A private car provides a direct, un-interrupted point-to-point transit path designed for low latency, mirroring interactive chatbot serving where GPUs run un-batched to return tokens instantly. A city bus collects multiple passengers onto a shared route; while individual travel time increases due to stops, overall passenger transport volume per unit of energy surges. Real-time interfaces force GPUs into inefficient "private car" regimes, whereas long-horizon agent platforms operate as high-density "buses," maximizing hardware throughput at lower cost.
NVIDIA’s "Speed of Light" Benchmark Culture [00:16:21]
Synthesis: Rather than measuring performance against market competitors, engineering limits are derived directly from theoretical physics and hardware specifications. Speed of Light demands calculating the absolute maximum operations a piece of silicon can execute given its clock frequency, power limits, and transistor count. Every software bottleneck, memory latency lag, or thermal throttling constraint is treated as a flaw to be systematically eliminated until the hardware achieves 100% of its physical capacity.
Synthesis: Cybersecurity defense is evolving from manual static code auditing to continuous, capital-intensive algorithmic exploitation. Because modern LLMs possess multi-vector reasoning capabilities, software security becomes a direct function of compute spend. Defending a codebase requires deploying fleets of autonomous agents to attack system boundaries across thousands of parallel execution paths. Security status is verified by the dollar amount spent on inference APIs attempting—and failing—to breach the system.
The Scavenger Infrastructure Arbitrage Strategy [01:00:14]
Synthesis: Hyperscalers fight over massive 100MW+ data center sites, prime NVIDIA GPUs, and 99.999% grid reliability, pushing market prices to extreme premiums. A scavenger strategy targets ignored, non-legible supply: sub-optimal non-NVIDIA silicon, stranded 1MW power pockets, and intermittent solar/wind without expensive diesel backups. By building a software orchestration layer that dynamically moves workloads when localized power or hardware drops, non-real-time agent execution unlocks massively superior economics.
The Verifiable vs. Non-Verifiable Problem Spectrum [00:09:59]
Synthesis: AI capabilities scale rapidly where outcomes can be objectively checked by programmatic environments (software compilation, mathematical proofs, circuit designs). These domains allow recursive self-improvement through synthetic feedback loops. Conversely, non-verifiable tasks rely on subjective human preference, cultural nuances, and artistic taste. Technical compute scaling naturally concentrates on verifiable domains, turning complex formal research into a purely dollar-denominated task.
Synthesis: The Transformer architecture combines two fundamentally incompatible computational profiles into a single pipeline. Multi-Layer Perceptron (MLP) layers are compute-bound, requiring dense matrix multiplications that thrive on high FLOP hardware. Attention layers are memory-bound, relying on continuous reads/writes to the dynamic KV cache. Forcing a single chip architecture (like a unified GPU) to process both layers creates systemic hardware inefficiency, pushing the industry toward heterogeneous hardware pairings.
6. Anecdotes
The NVIDIA "Milk Club" and Corporate Frugality [00:17:32]
Context: Neil Movva highlights how NVIDIA maintained extreme operational discipline despite its rapid rise. Following the 2008 financial crash, NVIDIA eliminated office perks, including free milk for coffee. Employees who wanted milk were required to pay $1 per month into an employee-managed "Milk Club" that bought bulk Kirkland milk from Costco. The anecdote illustrates how a culture of extreme resource conservation supported NVIDIA's ability to fund long-term silicon bets.
Jensen Huang's 2016 Bet on Tensor Cores [00:14:40]
Context: Neil Movva shares his experience as a college intern at NVIDIA in 2016, reading lab notebooks written by managers attending early academic ML conferences like ICML and NeurIPS. Managers noticed graduate students hacking consumer gaming GPUs to train early deep learning models. Recognizing the trend, CEO Jensen Huang reallocated 5% to 10% of total GPU die area to dedicated matrix multiplication units (Tensor Cores), re-orienting a graphics company into an AI hardware powerhouse.
The Cisco / Intel 2000 Dot-Com Bubble Parallel [00:50:07]
Context: Neil Movva recalls growing up in Silicon Valley around the year 2000, when his mother worked at Intel while Cisco was the most valuable company in the world. Investors draw parallels between dot-com telecom overbuilds and current AI capex. However, Movva argues today's environment is fundamentally different: 2000s fiber investment was speculative, whereas current inference token demand reflects immediate utility, with companies actively capping usage due to supply constraints.
The Whiteboard-to-Kernel Model Workflow [00:40:56]
Context: Describing modern GPU kernel engineering at Sal Research, Neil Movva references Tri Dao (creator of FlashAttention). Instead of manually writing complex low-level CUDA code, engineers map hardware dispatch schemes on a whiteboard, summarize the strategy in natural language, and feed it to LLMs to auto-generate functional GPU kernels.
The Impatient Student and the Whole-Stack Vision [01:21:41]
Context: Asking for the kindest gesture in his life, Neil Movva reflects on his sophomore year of college. Impatient to jump straight into designing AI chips, he complained to his academic advisor about taking broad foundational classes like networking and operating systems. The advisor mapped out the entire computing stack, convincing him that mastering every layer—from transistor-level silicon to global web infrastructure—was a rare, noble lifetime pursuit.
7. References & Recommendations
Companies & Platforms
Sal Research - Low-cost inference provider and agent sandbox host [00:00:53].
NVIDIA - Dominant GPU designer and producer of Hopper, Blackwell, and Rubin chips [00:13:25].
Cursor - AI code editor driving early demand for low-latency inference [00:03:51].
Base10 / Fireworks AI / Together AI - Low-latency model inference providers [00:03:45].
Parallel Web Systems - Web indexing engine using background AI agents [00:06:54].
Anthropic - Creators of Claude, Haiku, Opus, and early 1M context windows [00:05:25].
OpenAI - Creators of ChatGPT, Codex, and frontier LLM models [00:06:14].
Cerebras Systems - Wafer-scale SRAM engine manufacturer [00:23:44].
EP. 20 | 10,000 Pitches, 55,000 Deals, and AI: How Pranav Pai Finds Winning Startups | 2 Sept 2026 | Clearing The BLUR
"My only regret since '91 is we have not become antisocialist." Pranav Pai 00:04 http://www.youtube.com/watch?v=lPkjuRztHcg&t=0m4s "The assumptions the government makes—every Indian businessman is a crook—is a worst assumption you can make…
~500 MB
Total Static RAM on-die memory capacity on Blackwell die.