We argue that if everything there is in the world is physical, then time has an objective direction. If the fundamental equations of motion are time reversal invariant we show that the direction of time cannot be explained by anything else in physics (e.g. the direction of processes in time) and therefore must be added to physics. We further argue that the direction of time gives rise to both the thermodynamic and the psychological arrows of time (whenever they exist), and that it is necessary in order to construct a meaningful Past Hypothesis in statistical mechanics.