I have this problem with my Internet Service Provider. The cause of it is mostly because my ISP is terrible and blocking packets when a certain limit inbound or outbound is hit. Usually, my issue is that my outbound ceiling of 300 kbps is hit and that slams the door shut on inbound traffic as well. Sometimes the only way to resolve this is to cut down on your traffic by applying network usage reduction techniques such as:
You can verify if your ISP is indeed the problem by running a smoke ping service against your public IP address through
http://www.dslreports.com/smokeping so long as your router is not blocking ping requests. I think this may be free one or a few times, but costs a little money to have it run as a service throughout the day, but its worth it in order to get an idea of how long you're getting outages throughout the day.
If it's not your ISP then you likely have something misconfigured on your network or even a bad network cable involved. It really looks to me though like your network is being dropped as communication is underway or waiting for a response.