Check the Sender priority settings. If Sender is restricted from connecting to the destination site during the day or only certain priority send requests are allowed to go, then other send requests will be queued up to go during a time when Sender is not restricted from connecting to the destination site.
You can check these properties in the Sender Address Properties dialog box on the Schedule and Rate Limits tabs.