45 Billion Tokens Daily! How a US Telecom Giant Managed Soaring AI Costs
ζ―ζ₯εζεηΎδΊεεθ©ε οΌηΎει»δΏ‘ε·¨ι ε¦δ½ι ζΊζ §εζ΅ζ³ζεδΉζζζ¬οΌ | CIO Taiwan
Summary
A major US telecom provider faced a crisis as its daily token consumption for AI usage skyrocketed from 8 billion to 45 billion. In response, the company implemented an 'AI Gateway' to automatically route tasks to the most cost-effective model based on task nature. Furthermore, by developing a proprietary open model using specialized data, they successfully achieved both cost reduction and data sovereignty.
Details
The company faced the challenge of rapidly escalating AI inference costs due to the increasing token requirements for even single tasks as AI assistants became widespread. To address this, they implemented a system called an 'AI Gateway.' This gateway acts like a 'smart traffic cop,' instantly calculating response speed, cost, and quality requirements when a user submits a query, and automatically routing the task to the most cost-effective model. Specifically, by using open-source models for low-complexity tasks (like reading code) and reserving high-performance closed models only for complex tasks (like writing new code), they significantly reduced costs for code-related work. Moreover, they developed a proprietary, specialized model based on the open model Gemma, trained on over one trillion tokens of telecom professional data. By running this model on their internal servers, they eliminated the risk of data leakage and achieved independence from specific vendors. As a result, costs for some applications dropped by 80-90%, boosting the overall Return on Investment (ROI) by five times.
Original content copyright by respective publishers