Comparing the new download logic (only on Windows by default) with the old one (used on Linux and macOS by default), I can see that the old one used to disable Nagle’s algorithm while the new one doesn’t. Might be a reason for delays.
EDIT: As found in this other post, Windows Defender is responsible for a large chunk of the slowness. Normally you’d disable it with PowerShell commands like:
Set-MpPreference -DisableArchiveScanning $true
Set-MpPreference -DisableRealtimeMonitoring $true
Set-MpPreference -DisableBehaviorMonitoring $true
… but in this case you have to execute these commands at startup, before the cache restore starts. I accomplished this by backticking the commands into environment variables:
Set-MpPreference -DisableArchiveScanning \\\$true`
Set-MpPreference -DisableRealtimeMonitoring \\\$true`
Set-MpPreference -DisableBehaviorMonitoring \\\$true`
Now the whole build (not just the cache restore) runs much faster.