Enhanced Google Cloud networking for generative AI
In providing large language models (LLMs), enterprises have different networking issues than those associated with running typical web apps. This is due to the fact that generative AI apps behave very differently from the majority of other online apps.
Examples of predictable traffic patterns in web applications include milliseconds, which are used to measure the time it takes to process requests and responses. On the other hand, because gen AI inference applications are multimodal, they show different request/response timings, which can pose some special difficulties. In addition, an LLM query frequently uses up all of a GPU’s or TPU’s computation time as opposed to the more common request processing that occurs in parallel. The inference latencies vary from seconds to minutes because to the computational expense.
Thus, conventional utilization-based or round-robin traffic management strategies are often inappropriate for general-purpose artificial intelligence systems. Google recently announced many new networking features that optimise traffic for AI applications, with the goal of achieving the greatest end-user experience for next-generation AI apps and making optimal use of expensive and scarce GPU and TPU resources.
Vertex AI has many of these technologies built in. You can use them with whatever LLM platform you choose now that they are available in Google Cloud Networking.
Let’s examine more closely.
Using the Google Cross-Cloud Network, AI training and inference are expedited
Workloads including generative AI and AI/ML are cited by 66% of businesses as one of the main uses for multicloud networking. This is a result of the fact that the data needed for retrieval-augmented generation (RAG), grounding, and model training/tuning is spread across numerous different contexts. For LLM models to have access to this data, it must be copied or retrieved remotely.
Google cloud released Cross-Cloud Network last year, which makes it simpler to develop and assemble distributed applications across clouds by offering service-centric, any-to-any connection based on Google’s worldwide network.
Products in the Google Cross-Cloud Network category offer dependable, secure, and SLA-backed cross-cloud connectivity for fast data transfer between clouds, which is helpful in moving the enormous amounts of data needed for training generation AI models. Google Cross-Cloud Network is one of the solution’s products; it provides a managed interconnect with 10 Gbps or 100 Gbps capacity, end-to-end encryption, and a 99.99% SLA.
Customers can operate AI model inferencing applications across hybrid environments with Google Cross-Cloud Network in addition to safe and dependable data transfer for AI training. For instance, you can use application services operating in a different cloud environment to access models hosted on Google Cloud.
The Service-Based Model Endpoint: a specially designed programme for AI applications
The Model as a Service Endpoint offers an answer to the particular needs of applications involving AI inference. Because generative AI is so specialised, model makers often offer their models as a service that application development teams can use. It is the goal of the Model as a Service Endpoint to facilitate this use case.
Three main Cloud components make up the architectural best practice known as the Model as a Service Endpoint:
App Hub now became live for everyone to use. App Hub serves as a hub for managing workloads, services, and apps for all of your cloud-based projects. It keeps track of all of your services, including your AI models and apps, so that they can be found and reused.
To securely connect to AI models, use Private Service Connect (PSC). In order to use Gen AI models for inference, this enables model producers to define a PSC service attachment that model consumers can connect to. Who can access the Gen AI models is determined by policies set by the model producer. Additionally, PSC makes it easier for users who don’t live on Google Cloud to access producer models and consumer applications across networks.
A new AI-aware Cloud Load Balancing feature that optimises traffic distribution to your models is one of the many advancements included in Cloud Load Balancing to effectively route traffic to LLMs. The ensuing blog parts discuss these features, which are applicable to both AI application developers and model producers.
Custom AI-aware load balancing reduces inference delay
Prior to processing user prompts, many LLM apps take them via platform-specific queues of their own. LLM applications require the shortest queue depths for pending prompts in order to maintain consistent end-user response times. Requests should be assigned to LLM models according to the queue depth in order to do this.
Cloud Load Balancing now has the ability to distribute traffic based on custom metrics, allowing traffic allocation to backend models depending on LLM-specific data, such as queue depth. With this feature, Cloud Load Balancing can receive application-level custom metrics in response headers that adhere to the Open Request Cost Aggregation (ORCA) standard. Backend scaling and traffic routing are subsequently affected by these metrics. In order to maintain the shallowest feasible queue depths for gen AI applications, traffic is automatically dispersed equally and the queue depth can be configured as a custom metric.
As a result, inference serving experiences reduced peak and average latency. As this sample demonstration shows, applying the LLM queue depth as a critical metric to traffic distribution can actually improve latency for AI applications by 5–10 times. Later this year, Google cloud will integrate custom metrics-based traffic distribution with Cloud Load Balancing.
Optimal traffic allocation for applications involving AI inference
Numerous built-in features of Google Cloud Networking can improve the dependability, effectiveness, and efficiency of Gen AI applications. Let us examine each of these individually.
Enhancing the dependability of inference
Sometimes problems in the serving stack cause models to become unavailable, which degrades the user experience. Traffic must to be routed to models that are operational and in good health in order to consistently fulfil users’ LLM cues. There are several ways that cloud networking can help with this:
Internal Application Load Balancer with Cloud Health Checks: The high availability of the model service endpoint is crucial for model makers. To accomplish this, create an internal application load balancer that can access individual model instances and has cloud health checks enabled. Only healthy models receive requests since the health of the models is automatically checked.
Global load balancing with health checks: For the best latency, model consumers should be able to reach model service endpoints that are operational and responding quickly to client queries. Numerous LLM stacks are operated by distinct Google Cloud regions. You can use global load balancing with health checks to access individual model service endpoints and make sure that requests are going to the healthy regions. This directs traffic to the model service endpoint that is operating in the nearest and healthiest region. This method can also be expanded to support clients or endpoints that are not hosted on Google Cloud in order to facilitate multi-cloud or on-premises installations.
Google Cloud Load Balancing weighted traffic splitting: This feature allows for the diversion of certain traffic to alternative models or versions of the model in order to increase the efficacy of the model. By employing this method, you can ensure that new model versions are functioning properly as they are gradually rolled out through blue/green deployments, or you can use A/B testing to test the efficacy of various models.
Load balancing for Streaming: The execution time of Gen AI requests varies greatly, sometimes taking minutes or even seconds. This is particularly valid for requests containing pictures. We advise allocating traffic according to the number of requests a backend can process in order to provide the optimal user experience and the most effective use of backend resources for lengthy queries (> 10s). With a focus on optimising traffic for prolonged requests, the new Load Balancing for Streaming distributes traffic according to the number of streams that each backend can handle. Later this year, Cloud Load Balancing will offer Load Balancing for Streaming.
Use Service Extensions to Improve Gen AI Servicing
Lastly, Google is happy to announce that Cloud Service Mesh will offer Service Extensions callouts for Google Cloud Application Load Balancers later this year. These callouts are currently generally available. Through the use of service extensions, SaaS solutions can be integrated or programmable data path modifications, like header conversions or custom logging, can be carried out.
Service Extensions can enhance the user experience in modern AI applications in a number of ways. For instance, you may use Service Extensions to provide prompt blocking, which stops undesired prompts from getting to the backend models and using up valuable GPU and TPU processing time. Additionally, you can route requests to particular backend models using Service Extensions, depending on which model is most appropriate to reply to the prompts. In order to accomplish this, Service Extensions evaluates the request header data and selects the most appropriate model to fulfil the request.
Because Service Extension callouts are configurable, you may tailor them to your gen AI applications’ specific requirements.
Make the most of Gen AI by utilising Google Cloud Networking
These developments demonstrate Google’s dedication to providing cutting-edge solutions that enable companies to fully utilise artificial intelligence. With the support of Google Cloud’s sophisticated networking suite, Google can assist you in resolving the particular difficulties artificial intelligence applications present.
