New Server Hopes to Break Through AI’s “Memory Wall”

Memory is arguably the most serious constraint on modern AI large language models (LLMs). According to one influential paper, LLM token generation is an inherently memory-bound task, meaning the rate at which models output text is limited by how quickly data can be read in from memory. The severity of this bottleneck grows with model size. This creates a “memory wall” that holds back LLM inference performance. AI hardware startup Majestic Labs is taking a direct - and comprehensive - approach to solving this problem.