Your perception of a problem is based on your perspective. A Network Administrator might think a momentarily slow server is a small problem, because it is running normally 99% of the time, but a Database Administrator (DBA) might think a server is performing poorly because the just 1% of the time you need a query to run quickly, it doesn’t run at all. Perspective is important when uncovering issues, determining the cause, and troubleshooting an effective resolution.
In this article by Kyle Brandt, we see this take on those different perspectives.
I think the biggest lesson is that the full view of many problems is missed unless each person in the team has at least some understanding of what is going on from other team members views. Also, the communication between different specialists is needed to solve many issues. In this case what looked initially to me like a networking problem was actually a symptom. Trying to attack the problem solely as a network administrator was treating the symptom, not the disease.
From a technical standpoint the difference between having the web tier and SQL servers on the same box compared to having a network connection between them is important. Things may work well on a single server, but when the data needs to be moved over the network shifting the load to web servers might not always work.
We still have a good amount of zero window packets going back and forth, so although the worse offenders have been mitigated I believe there is still might be work to do.