FriendliAI, a leading generative AI engine company, is proud to announce the public beta release of PeriFlow Cloud. This powerful platform empowers users to run PeriFlow, an engine for generative AI serving, within a managed cloud environment.
With its innovative approach specifically tailored to large language models (LLMs), the PeriFlow engine achieves remarkable improvements in throughput while maintaining low latency. This cutting-edge engine is built upon FriendliAI’s groundbreaking batching and scheduling techniques, which are protected by patents in the United States and Korea, including U.S. Patent No. 11,514,370, U.S. Patent No. 11,442,775, Korean Patent No. 10-2498595, and Korean Patent No. 10-2479264.
CIO INFLUENCE: CIO Influence Interview with Russ Ernst, Chief Technology Officer at Blancco
PeriFlow is fast and versatile, attracting a growing number of companies that develop their own LLMs through pretraining or fine-tuning open-source LLMs. Supporting a broad range of LLMs, including GPT, GPT-J, GPT-NeoX, MPT, LLaMA, Dolly, OPT, BLOOM, T5, FLAN, UL2, and more, PeriFlow offers diverse decoding options such as greedy, top-k, top-p, beam search, and stochastic beam search. Furthermore, it supports multiple data types, including fp32, fp16, bf16, and int8. With PeriFlow, users can optimize the balance between precision and speed.
FriendliAI also offers PeriFlow as a container solution, named PeriFlow Container, which has gained considerable traction among companies for LLM serving. For instance, Scatter Lab, a prominent social chatbot company in Korea, optimizes their high user traffic by leveraging PeriFlow Container to run multiple LLMs, including the popular Luda 2.0. As a result, Scatter Lab has achieved a remarkable 50% reduction in infrastructure costs associated with serving.
The Benefits of PeriFlow Cloud
PeriFlow Cloud simplifies the adoption of PeriFlow for organizations of any scale. With PeriFlow Cloud, users can enjoy exceptional speed at low costs (70~90% GPU savings) for LLM serving without the hassle of cloud resource setup and management.
CIO INFLUENCE: CIO Influence Interview with Bill Lobig, VP of Product Management at IBM Automation
Through PeriFlow Cloud, users can centrally manage every deployed LLM from anywhere. Users are able to effortlessly upload model checkpoints, deploy models, and instantly send inference requests. Comprehensive monitoring tools empower users to track events, errors, and performance metrics while interactively testing deployed LLMs in the playground. It dynamically handles performance and fault issues while auto-scaling based on traffic patterns, freeing users to focus on creating LLMs and driving innovation.
Byung-Gon Chun, Founder & CEO of FriendliAI, emphasizes the significance of efficient LLM serving, stating “Generative AI is revolutionizing our lives, enabling more creative, intelligent, and productive services. Many organizations are now training their own models, but they have yet to fully realize how costly and painful it is to serve these models at scale for a large user base.”
“We’re due for a significant transformation in the way we serve LLMs to empower organizations to fully harness the potential of their LLMs,” Chun adds. “PeriFlow Cloud is an instant and cost-effective solution. We are incredibly excited to see the innovative services users will develop with their generative AI models, powered by PeriFlow Cloud.”
CIO INFLUENCE: CIO Influence Interview with Antoine Jebara, Co-Founder and GM, MSP Products at JumpCloud
[To share your insights with us, please write to sghosh@martechseries.com]
