A bit of abstract thinking out loud, since there's been a good amount of buzz on time travel floating around lately with Stephen Hawking's most recent comments.
The thing about time travel is that because we humans are naturally blinded to see things only from our own individual perspectives, I suspect our natural assumptions about distorting time are a bit backwards.
That is to say: we think about "time travel" as a phenomenon primarily concerned about how we would be seeing the world through time.
This phenomenon might better be described as the way the rest of the world sees us.
This isn't too profound on some level - special relativity and general relativity is precisely about the perception of others.
A way to clarify this is to think about the event horizon, which is the point in space at which light can no longer escape the gravity of a black hole. As a primer:
Special relativity illustrates that because 'time' is intrinsically interwoven with perception, that perception (particularly what we 'see' on a watch) will change at very high speeds and within high gravitational fields - as we approach the speed of light (either through our moving fast enough or it's moving slow enough thanks to gravity), the perception of what it 'shows' us changes completely. This is why time slows down at high speeds and in high gravitational fields, measured by the comparison of two watches that have moved through different conditions.
One key point about relativity is that you don't notice this, others do. In fact, that's precisely why the term 'relativity' is used.
Back to the event horizon:
If you were able to observe someone moving towards a black hole, what you would be observing is the light reflecting off them. At some point - just past the event horizon - this light would never be able to escape the gravity of the black hole. At this point in the gravitational field, their watch would slow down to the point that it has actually stopped. The light that you perceive would be frozen. They of course would continue moving towards the black hole, and would perceive all the grisly experiences that come along with things like being ripped apart by gravity, but for you, they have been frozen in time.
I'm wondering if this is what we will discover "time travel" to be: not our own individual selves moving through space-time, but the relative perception of the world moving around us.
In other words, when we "travel through time," we won't notice - others will.