Google Cloud touts sharded fix for noisy neighbours

Google Cloud has outlined a sharded hub-and-spoke architecture for data processing environments affected by "noisy neighbour" failures. The design targets shared infrastructure used by multiple tenants or business domains.

The approach replaces a single large processing stream with a routing layer, durable buffers, and smaller isolated pipelines. It is intended to limit disruption when one tenant generates a burst of activity or when a database instance fails.

In the legacy setup Google Cloud describes, all tenants and workloads pass through one unified stream. That creates a single bottleneck, so a performance problem in one database tenant can create back pressure across the entire platform and slow processing for everyone else.

This kind of monolithic architecture can create a full blast radius when a database fails, force operators to scale infrastructure for the most demanding tenant, and make it difficult to maintain service-level agreements across critical data pipelines.

How it works

In the sharded model, a lightweight hub pipeline acts as a router. It reads from unified source topics, identifies a tenant ID or business domain, and sends data into separate buffers for downstream processing.

Those buffers sit between the hub and the processing spokes. Pub/Sub topics can serve as durable isolation layers that absorb shocks in the system and prevent a slow downstream sink from backing up the original source.

The spokes then run in separate Dataflow jobs rather than one central pipeline. This lets operators divide work by priority, tenant size, or business logic, with dedicated pipelines for high-priority tenants, grouped pipelines for smaller tenants, and separate pipelines for more complex domains.

This arrangement narrows the effect of failures to a specific spoke instead of bringing down the entire system. It also allows infrastructure teams to scale resources independently based on the load of each tenant or domain.

Operational impact

For companies running multi-tenant software services, internal data platforms, or mixed-workload processing environments, the architecture is presented as a way to improve resilience without overbuilding the whole platform. Instead of sizing a single pipeline for worst-case demand, teams can allocate more resources only where they are needed.

It also changes how maintenance is handled. In a monolithic environment, an update to one part of the system can affect every tenant because all processing is tied together. In the sharded structure, operators can change one domain or one spoke without touching the rest of the estate.

Google Cloud compared the two models across fault tolerance, blast radius, scaling, and maintenance. In its description, a monolithic system exposes all tenants to a single point of failure, while the hub-and-spoke design isolates faults and reduces operational spillover.

Google Cloud said the blast radius in the sharded arrangement can fall to less than 5%, compared with total disruption in the legacy model. It also said the design improves service stability by preventing one high-volume tenant from delaying the rest of the platform.

Implementation details

Google Cloud also outlined several operational measures for teams adopting the design. One is the use of dead letter queues so a single SQL exception does not halt processing for an entire pipeline.

Failed records can instead be routed to storage systems such as BigQuery or Google Cloud Storage for later investigation. That allows processing to continue while problematic records are separated for review.

Another recommendation is strict connection pooling at the spoke level. Because databases have connection limits, operators should use a thread-safe singleton pattern and keep the maximum pool size low for each worker to avoid exhausting database connections during autoscaling.

Google Cloud also pointed to asynchronous input and output techniques, including batching writes. By buffering writes before they reach a database, teams can reduce connection overhead and limit the latency that can trigger broader slowdowns.

The architecture reflects a wider push among cloud customers to break up large shared systems into smaller isolated units. In data processing environments where many tenants depend on the same infrastructure, the central issue is often not raw throughput but how effectively a platform can contain failures when traffic patterns or downstream systems become unstable.

Google Cloud said the aim of the sharded model is to ensure that a "noisy neighbor" affects only its own spoke rather than the wider neighbourhood.