Einstein was deeply troubled by the problem of time. In the Newtonian World with universal time and space, the present moment was unique and unambiguous. Only the present really existed, the past was done and the future yet to come. With the advent of Special Relativity this all changed, the present moment was defined differently for each frame of reference. If I pace up and down while I am thinking about this, I am changing my position and velocity and hence my frame of reference. In a galaxy far far away, my estimate of what day today is for the citizens of that far off place will be wobbling around by more than a 100 years as I change the direction I walk in. The bigger the distances involved and the bigger the velocity change the more marked these effects become. It only makes sense if we acknowledge that the whole of space-time all coexists, but then how are we to understand the special quality of the moment we call now? Our common sense tells us that now is the only time that really exists but that it is not so. We cannot even define it unambiguously.
Einstein passionately believed in causality, the strict temporal ordering of events. The structure of special relativity appears to enforce causality. The mathematical formalism prevents matter and energy from accelerating to speeds faster than light, a necessary corollary, since anything going faster than light can also travel backwards in time. What he missed was that there might be a way to transmit pure information, unconnected with mass or energy.
The situation with General Relativity (GR) is far more complex because of the richness of the theorem. Einstein was disturbed to find that it implied many things that he had not anticipated. Shortly after publishing GR, the Russian physicist Friedmann showed that it implies that the universe is unstable, either expanding or contracting, and therefore that time must either have a beginning or an end. The Big Bang and the expanding universe were unknown at the time. In the former way of thinking it was presumed that the universe and time were eternal. This discovery was a terrible shock to Einstein who tried to get this result suppressed. Afterwards he apologised to Friedmann but it was to be the beginning of a rocky road for Einstein who never fully accepted many of the discoveries other scientists made. He was the most revolutionary of them all and yet lived in great doubt as to where scientific knowledge was going.
And here is something else to think about next time you are watching the stars: for starlight, time does not exist at all. From the frame of reference of the starlight, it leaves its distant star and arrives here all in the same instant. Einstein always claimed that it was thinking about what it would be like to ride along with a beam of light that was his inspiration for special relativity.