Fitting Four 32 GB Resizable Bars Behind a PCIe Switch on Linux

A Linux system with four AMD Radeon AI PRO R9700s and two RTX 3090s behind a Broadcom PCIe switch needs a 32 GB Resizable BAR on each R9700 to enable ROCm peer-to-peer transfers. The motherboard’s BIOS could not allocate all six cards’ BARs, so Linux had to resize and reassign them before GPU drivers loaded. Linux 7.0.6 failed after assigning only two R9700s, because its bridge-window calculation added each 32 GB BAR and its 2 MB doorbell but omitted nearly 32 GB of alignment padding between cards. An unmerged kernel patch series corrects this sizing bug; testing it allowed all four large BARs to fit.

The system still needed an early-boot workaround: an out-of-tree module sets the R9700 BAR sizes, removes and rescans the switch’s root port, then raises the alignment of smaller BARs so each card’s window occupies exactly 64 GB. After reboot, Linux mapped all four R9700s’ full 32 GB, and their pairwise transfers reached 14.0 GB/s; simultaneous ring transfers reached 54.7 GB/s. The practical guidance is to check Linux PCI mailing-list reports for allocation errors, perform resizing and rescanning from the initramfs before drivers load, and inspect small BARs adjacent to large ones. Removing or rescanning a GPU after its driver is active can lock up the system.