We report on long term timing observations of the 3.05 ms pulsar PSR B1821−24, made with the Proportional Counter Array (PCA) on NASA's Rossi X-ray Timing Explorer (RXTE), which have an absolute pulse time of arrival accuracy of < 3.5 µs. We compare these observations to careful radio timing observations taken at Nançay Observatory and the Green Bank Telescope, which allow us to track the variable dispersion measure so that we can compensate for radio propagation effects. These data are the first high precision long term timing measurements of a millisecond pulsar in the X-ray band and demonstrate that timing of X-ray emitting MSPs in the X-ray band can be an important complement to radio timing programs.