Fastly, Inc., a leader in global edge cloud platforms, has announced the launch of Fastly AI Accelerator, the company’s first AI solution designed to create a better experience for developers by helping improve performance and reduce costs across the use of similar prompts for large language models (LLM) apps.
“Fastly AI Accelerator gives developers exactly what they want, by making the experience of their favorite LLMs a lot faster and more efficient, so they can focus on what makes their app or site unique, and what keeps their users happy.”
“AI technologies generally and large language models specifically are aggressively reshaping the technology industry, and the way millions worldwide – developers included – work every day,” said Stephen O’Grady, Principal Analyst with RedMonk. “There’s a lot of focus on the largest models. However, developers and enterprises alike are turning in large numbers to medium and smaller models. Whether it’s to lower costs, to shorten training cycles or to run on more limited hardware profiles, they’re an increasingly important option.”
Also Read: Introducing Databricks AI/BI: Intelligent Analytics for Real-World Data
Fastly AI Accelerator is designed to reduce API calls and costs with intelligent, semantic caching. Built on Fastly’s Edge Cloud Platform and leveraging industry leading caching technology, AI Accelerator uses a specialized API gateway to dramatically improve performance for apps using popular LLMs, beginning with ChatGPT and expanding support to include additional models.
Popular AI applications can process hundreds of thousands of API calls or questions daily. Many of the questions users ask are likely very similar and may have been asked before. Without semantic caching, each call to the LLM requires going back to the provider for the information, potentially increasing costs and latency. However, Fastly AI Accelerator’s semantic caching provides a cached response for repeated queries directly from Fastly’s high performance edge platform, instead of going back to the AI provider, helping to deliver a better experience by improving performance while reducing costs.
“At Fastly, we’re always listening to developers, to understand both what they’re excited about and what their biggest pain points are,” said Anil Dash, Vice President of Developer Experience with Fastly. “Fastly AI Accelerator gives developers exactly what they want, by making the experience of their favorite LLMs a lot faster and more efficient, so they can focus on what makes their app or site unique, and what keeps their users happy.”
Also Read: New HP Study Highlights AI’s Role in Advancing Social and Economic Goals
When using Fastly AI Accelerator, developers only need to update their app to use a new API endpoint, which typically only requires changing a single line of code. Fastly AI Accelerator will then transparently implement semantic caching for OpenAI compatible APIs. This approach goes beyond traditional caching as Fastly AI Accelerator is able to understand the context of the requests and queries, and will send a similar response if two or more requests are alike.
To help developers build faster, more secure and more engaging experiences, Fastly is also now making it even easier for developers to try Fastly with an expanded free account tier that helps coders set up a new site, create a new app, or launch a new service in just a few minutes. Free tier accounts also include access to Fastly’s Content Delivery Network (CDN), generous memory and storage allotments, uncapped redirects, page rules and regular expressions. Plus, the free Fastly tier includes security features such as TLS and always-on DDoS mitigation, observability tools, and much more.
[To share your insights with us as part of editorial or sponsored content, please write to psen@itechseries.com]
