Don't cycle the modem at all often ~ not more than once a day. That might be interpreted as errors and cause a slow-down in your speed. Cycling (or disconnecting/reconnecting) the router is no problem.
Try http://mcslhr.visualware.com/myspeed/myspeed_line_capspeed.html You will probably find that the upload is quite a bit closer to the 20Mb maximum.
Still doesn't explain your slow download. A different port may help as bullitt says. Also, on the 'speed' tab of the tester above, let us know both the 'achieved download speed' (which is an average) and the 'of about' speed (which is the fastest burst. That might give an better indication of the cause of the problem (but I'm afraid probably won't).
