It really depends on workload. For ImageNet and resnet type architecture its not unusual to get 3X speed up. It also depends on if you do full fp16, leave BNs alone etc. This is a lot because instead of training for 6 days, you now training for 2 days.
> ImageNet and resnet type architecture its not unusual to get 3X speed up
Source on this? I've done a good bit of CV benchmarking work and I don't recall anything like a 3x boost. 30-40% improvement is much more in line with what I remember.
It really depends on workload. For ImageNet and resnet type architecture its not unusual to get 3X speed up. It also depends on if you do full fp16, leave BNs alone etc. This is a lot because instead of training for 6 days, you now training for 2 days.