Caching is to help speed things up rather than slow things down. The problem if it was caching would be more down to stale data, e.g not reflecting an updated image.
Browsers will generally cache data unless the client has told it not to. Also web servers can tell the client or browser how long to cache the data for (check the response headers).
Use browser's developer tools to help track where the issues lies, looking at latency and if there's a common pattern between what you are seeing and the trace details.
