Since then, Google has reworked both its models and the infrastructure behind them, transitioning from Bard to Google Gemini, while also introducing a new generation of its custom AI chips in an effort to change the landscape. The question now is whether this is still a story of catching up or is Google now has an edge over others in the AI race?

Understanding the basics: CPU, GPU and TPU

At the heart of this shift is hardware. Most computing today runs on CPUs, or central processing units, which are designed to handle a wide range of tasks. GPUs, or graphics processing units, are better suited for parallel workloads, which is why they became central to AI development. Companies like NVIDIA have built their dominance on this strength.

Google, however, took a different route. It built its own chips — called Tensor Processing Units (TPUs) — designed for the kind of mathematical operations AI models rely on. In simple terms, while CPUs are generalists and GPUs are strong parallel workers, TPUs are built as specialists for AI workloads.

Why Google started building its own chips

The origins of TPUs go back to around 2015, when Google realised that its growing AI ambitions were outpacing the capabilities of existing hardware.

The first TPU, introduced that year, was focused on inference — essentially helping AI models respond faster in real-world applications like search and voice recognition. But this quickly exposed a new bottleneck: training.

Training an AI model — the process of teaching it using vast datasets — required far more computing power than anticipated. That led to a major shift.

From running AI to building it at scale

In 2018, Google introduced TPU v2, marking a significant turning point. Instead of just improving individual chips, the company connected hundreds of TPUs into clusters known as pods, effectively creating a training supercomputer.

This changed the equation. Google was no longer just running AI models efficiently; it was building infrastructure capable of training them at massive scale.

Over time, this approach expanded further, bringing concepts like cooling. TPUs evolved across generations, while the systems around them scaled from single chips to pods, then to even larger clusters — referred to as superpods — and eventually to data centre-wide deployments.

The important point here is that Google’s strategy was not limited to better chips. It was building an entire AI infrastructure stack.

A deliberate choice: Keeping TPUs flexible

From the second generation onwards, Google made a conscious decision not to over-specialise its chips or the complete stack too early. Instead of building separate hardware for every stage of AI, it designed TPUs that could handle both training and inference.

This approach came from a practical concern. AI models were evolving rapidly, and it was difficult to predict what architectures would dominate in the future. Locking hardware too tightly to one type of workload risked making it obsolete quickly. By keeping TPUs relatively flexible, Google ensured that the same infrastructure could support a wide range of models — from early machine learning systems to more complex neural networks.

This philosophy is visible across multiple TPU generations. Improvements were focused on scaling performance, increasing efficiency, and improving how chips communicate with each other, rather than creating narrow-use hardware. As a result, TPUs became general-purpose AI accelerators within Google’s ecosystem — capable of adapting as models grew larger and more complex.

However, this flexibility also meant that a single system had to balance competing requirements. Training workloads demand raw compute power, while inference requires speed and responsiveness. Managing both within the same architecture inevitably involves trade-offs, which is what the latest generation addresses.

From Ironwood to TPU 8: What has actually changed

Google’s seventh-generation TPU, called Ironwood, was designed specifically for inference — that is, running AI models in real-world applications. Google described it as being built for the “age of inference”, where AI systems move beyond simply responding to queries and begin generating insights more proactively, often as part of larger workflows.

To support this, Ironwood focused heavily on:

• 
  Higher memory capacity (to handle larger models)
• 
  Faster data access and movement
• 
  Strong inter-chip communication for large-scale deployments

It could scale up to more than 9,000 chips in a single system and was optimised for running complex models such as large language models and mixture-of-expert systems efficiently at scale.

With its eighth-generation TPUs, Google introduced a more targeted approach to chip design. It introduced two purpose-built variants:

• 
  One optimised for training large models – TPU 8t
• 
  One optimised for inference, where speed and responsiveness are critical – TPU 8i

This was not about physically altering or splitting an existing chip, but about designing separate architectures from the ground up, each tailored to a specific role.

The advantage of this approach became efficiency. Training workloads benefit from higher compute throughput and the ability to scale across thousands of chips, while inference workloads require faster memory access and lower latency. By addressing these needs separately, Google can reduce the compromises that come with a one-size-fits-all design.

While Ironwood acknowledged that inference was becoming critical, TPU 8 goes further by recognising that training and inference now have fundamentally different infrastructure needs — and are both equally central to modern AI systems. There are also broader system-level improvements. Google says the new generation delivers significantly higher compute performance per pod, faster interconnect speeds, and better overall utilisation — meaning more of the system’s total compute power is actually used productively.

Why is this shift happening now

AI systems today are no longer limited to answering single queries. They are increasingly expected to reason through problems, execute tasks, and operate in multi-step workflows. This has given rise to what is often described as “agentic” AI — systems that can act, not just respond.

Such systems place new demands on infrastructure. They require faster response times, better memory handling, and more efficient coordination between different compute units. Google’s move to specialised chips reflects this change. It suggests that the company is designing not just for current models, but for how AI systems are expected to evolve.

Training vs inference

The relationship between training and inference has evolved over time. In the early phase, inference was the immediate priority — to deliver results quickly within products. But as AI models became more sophisticated, training emerged as a major bottleneck, requiring significant computational resources and time.

Today, both have become equally important, but for different reasons. Training determines how capable a model can become, while inference determines how effectively that capability can be delivered to users.

The latest TPU generation reflects this balance. By designing separate architectures for each, Google is acknowledging that training and inference are no longer just two stages of the same pipeline, but distinct challenges that require different optimisation strategies.

Not just for Google

TPUs were initially built for internal use. However, they are now available to external developers and businesses through Google Cloud. This means companies can train and run their own AI models on the same infrastructure that powers Google’s systems.

In other words, TPUs are not just a competitive advantage for Google’s products; they are also part of its broader cloud offering. Some of the companies that use Google’s Cloud TPUs are Anthropic, Midjourney, Salesforce, Citadel Securities, and more.

Gemini: A broader reset

When Google Bard was introduced, expectations were high. Google had been highlighting its AI research for years. On paper, it had the expertise and infrastructure to lead. Yet Bard’s early reception was not as great as expected. Initial responses were sometimes inconsistent, and the product lacked the polish and feature depth seen in competing systems. At the same time, rivals were moving quickly, rolling out new capabilities that captured both user attention and developer interest.

This created a perception gap. Even if Google’s underlying capabilities remained strong, its visible product did not reflect that strength immediately. In fast-moving technology cycles, perception can shape momentum, and Bard’s early limitations contributed to the idea that Google was trailing the competition. 

The transition from Bard to Google Gemini represents a broader recalibration rather than a simple rebranding exercise. Gemini is designed as a more comprehensive AI system. It is built to handle multiple types of inputs — text, images, and potentially more — and to perform more complex reasoning tasks. This reflects a shift in how AI models are being developed, moving beyond single-purpose chatbots toward systems that can assist across a wider range of tasks.

Another key difference lies in integration. Gemini is closely tied to Google’s existing ecosystem, including its cloud infrastructure and productivity tools. This allows it to operate not just as a standalone interface, but as part of a broader set of services.

Underneath both Bard and Gemini is the same TPU infrastructure that Google has been refining over the years. The refining has to be continued, though because improvements in chips and systems do posses the capability to significantly influence what the model can do and how efficiently it can operate.

Bringing it all together

When viewed as a sequence of isolated developments, Bard’s early struggles and the later introduction of Gemini might seem like a course correction. But when placed alongside Google’s long-term investments in TPUs and infrastructure, a more consistent pattern emerges.

Over the past decade, Google has steadily built out its AI stack — from custom chips to large-scale systems to advanced models. Each layer reinforces the other. Improvements in hardware enable more capable models, while evolving models drive the need for better infrastructure. The introduction of specialised TPUs fits into this trajectory. It is not a sudden pivot, but a continuation of a strategy that has focused on controlling and optimising the full AI pipeline. 

So, is Google still catching up?

The answer is not straightforward. Google may have lost early momentum in the current AI race, particularly in how quickly its products captured public attention. However the combination of all of the below factors suggest a different position:

• 
  Deep infrastructure investment (TPUs)
• 
  Full-stack control (hardware, software, cloud)
• 
  More mature model strategy (Gemini)

Rather than simply reacting, Google appears to be aligning its hardware and AI systems for the next phase of the race. Now, that may mean it is no longer just playing catch-up, but preparing to write the next chapter of growth and success.