GRB 080503 was classified as a short gamma-ray burst (GRB) with extended emission. The origin of such extended emission (found in about a quarter of Swift short GRBs) is still unclear and may provide some clues to the identity of the elusive progenitors of short GRBs. The extended emission from GRB 080503 is followed by a rapid decay phase (RDP) that is detected over an unusually large dynamical range (one decade in time and ∼ 3.5 decades in flux), making it ideal for studying the nature of the extended emission from short GRBs. We model the broad envelope of extended emission and the subsequent RDP using a physical model for the prompt GRB emission and its high latitude emission tail, in which the prompt emission (and its tail) is the sum of its individual pulses (and their tails). For GRB 080503, a single pulse fit is found to be unacceptable, even when ignoring short time-scale variability. The RDP displays very strong spectral evolution and shows some evidence for the presence of two spectral components with different temporal behaviour, likely arising from distinct physical regions. A two pulse fit (a first pulse accounting for the gamma-ray extended emission and decay phase, and the second pulse accounting mostly for the X-ray decay phase) provides a much better (though not perfect) fit to the data. The shallow gamma-ray and steep hard X-ray decays are hard to account for simultaneously, and require the second pulse to deviate from the simplest version of the model we use. Therefore, while high latitude emission is a viable explanation for the RDP in GRB 080503, it does not pass our tests with flying colours, and it is quite plausible that another mechanism is at work here. Finally, we note that the properties of the RDP following the extended emission of short GRBs (keeping in mind the very small number of well-studied cases so far) appear to have different properties than that following the prompt emission of long GRBs. However, a larger sample of short GRBs with extended emission is required before any strong conclusion can be drawn.