HealthRecordCommunity
FHIR🌏 USAEnriched

45 Billion Tokens Daily! How a US Telecom Giant Managed Soaring AI Costs

每ζ—₯εƒζŽ‰ε››η™ΎδΊ”εε„„θ©žε…ƒοΌηΎŽεœ‹ι›»δΏ‘ε·¨ι ­ε¦‚δ½•ι ζ™Ίζ…§εˆ†ζ΅ζ³•ζ•‘ε›žδΉζˆζˆζœ¬οΌŸ | CIO Taiwan

October 12, 2026

Summary

A major US telecom provider faced a crisis as its daily token consumption for AI usage skyrocketed from 8 billion to 45 billion. In response, the company implemented an 'AI Gateway' to automatically route tasks to the most cost-effective model based on task nature. Furthermore, by developing a proprietary open model using specialized data, they successfully achieved both cost reduction and data sovereignty.

Details

The company faced the challenge of rapidly escalating AI inference costs due to the increasing token requirements for even single tasks as AI assistants became widespread. To address this, they implemented a system called an 'AI Gateway.' This gateway acts like a 'smart traffic cop,' instantly calculating response speed, cost, and quality requirements when a user submits a query, and automatically routing the task to the most cost-effective model. Specifically, by using open-source models for low-complexity tasks (like reading code) and reserving high-performance closed models only for complex tasks (like writing new code), they significantly reduced costs for code-related work. Moreover, they developed a proprietary, specialized model based on the open model Gemma, trained on over one trillion tokens of telecom professional data. By running this model on their internal servers, they eliminated the risk of data leakage and achieved independence from specific vendors. As a result, costs for some applications dropped by 80-90%, boosting the overall Return on Investment (ROI) by five times.

πŸ“°
Read Original Article
cio.com.tw
β†’

Original content copyright by respective publishers