Thanks for all the comments guys. Just a few updates:
Yes, I've checked the battery connections, I understand how the controller works, and all that Jazz. The train works perfect for about the first 10 minutes, then it starts to lose power slowly like the batteries are dying for the next 10, then it becomes unresponsive again until the batteries in the train are changed. Like I said, it works great, but only for about 10 minutes.
As an update for trouble shooting, I've double checked all the wheels are properly on the track, I've taken most of the train apart and rechecked all connections, and I've checked all the wheels to make sure none of them are too tight and causing excessive drag. None of these seemed to be the issue, and I still killed the next set of batteries rather quickly.
My opinion so far seems to be that there might be an issue with the Motor/Wheel part. If I turn the speed up slowly, starting with the first click to speed 1 on the controller, the motor emits a high frequency noise, but the train doesn't move. Turning up the speed slightly to the next notch, the train will start to move if the batteries are fresh, but the train kinda lurches, emiting an slightly louder high pitch noise. Its not a consistent power to the wheels, and the train kind of pulsates forward instead of a constant drive. Turning up to speed three and the train starts moving smoothly.
Again, after about 10 minutes though, it all starts slowing down and getting sluggish, until it stops doing anything at about the 20 minute mark.