AI inference workloads are becoming more compute-intensive, with higher demands on throughput, memory access, and power efficiency. 

AI inference chips are placing new demands on physical design. Compute is becoming denser, SRAM is moving closer to compute, data paths are becoming wider, and local power density is increasing. 

As a result, floorplanning, power delivery, clocking, routing, and thermal management all must work together. The physical design decisions made early in the flow can directly affect performance, power, and scalability.

How should floorplanning change when AI inference chips contain large compute clusters with tightly coupled SRAM and high-bandwidth data paths?

The chip floorplan methods that have worked for decades, even for ‘complex’ CPU-based designs, have seen a tectonic shift with the advent of AI inference chips. 

Golden rules while freezing the floorplan for traditional designs: 

  • Digital logic to occupy central area with least obstructions 
  • Memory macros around the edges, preferably without intruding into the central core area 
  • Relatively narrow buses and less constrained data path latency 

But AI inference chips have different needs: high data throughput, low latency, hundreds of compute units processing data in parallel, and they are power-hungry as well. 

While doing floorplanning for AI inference chips: 

  • Ensure compute logic and memory macros are co-located in dataflow-driven floorplanning 
  • Repetitive tiles (compute + memory) rather than a few big chunks of logic and macros 
  • Wide data bus can save power and reduce latency, optimizing the physical connection length and reserving routing tracks, ensuring smooth scalability, just by adding more tiles 
  • Use a more complex power-delivery network—including TSVs, microbumps, and backside power delivery—and identify and distribute power hotspots across the chip. 
  • Clock distribution network must balance within each tile and from the root to each tile, and support gating 

How do you manage IR drop and local power density when thousands of MACs or AI compute units switch simultaneously?

Managing power in AI inference chips is one of the hardest sign-off steps. Thousands of chips processing data together act like power guzzlers, pulling current from the power grid at once and causing voltage drops. Usually, high current consumption issues were tackled either by compromising on frequency of operation or by reducing operational voltage. 

AI chips are too power-hungry for these approaches to be practical. 

The solution is: 

  • Compute tiles not triggered at the same time but with staggered start 
  • Clock phase shifting – one set of tiles gets the clock edge a little bit later than another set. Hence, processing logic is active at different timestamps 
  • Lower the voltage/frequency of the idle region using DVFS (Dynamic Voltage and Frequency Scaling) 
  • Vertical Power Delivery (VPD) – power comes from directly underneath via TSVs 
  • Backside Power Delivery – moves power routing to the backside of the silicon wafer 
  • Plan for sufficient decoupling capacitors

How should clock tree design account for large AI compute arrays, high fanout control signals, and multiple performance-critical domains?

Clock tree design for AI compute arrays must find a sweet spot considering low skew across thousands of MACs, power efficiency since the clock network contributes heavily to dynamic current consumption, and multi-domain integrity (in case of DVFS) 

For regular or irregular tile arrays, no single topology is sufficient. The implementation may use a mix of H-tree, multi-point CTS, and clock mesh. 

  • Fine-grain clock gating reduces clock power in inactive compute clusters 
  • Medium-sized buffers handle clock propagation instead of large ones 
  • Correctly sized clock cells and high-Vt cells, where timing permits, can reduce leakage 
  • Multiple power and voltage domains require isolation and clock integrity. Each domain can use a dedicated clock tree, while minimizing clock crossings