We use signal enhancement techniques and a matched filter analysis to search for the K-band spectroscopic absorption signature of the close orbiting extrasolar giant planet, HD 189733b. With time-series observations taken with the Near Infrared Spectrometer (NIRSPEC) at Keck II, we investigate the relative abundances of H2O and carbon bearing molecules, which have now been identified in the dayside spectrum of HD 189733b. We detect a candidate planet signature with a low level of significance, close to the ∼153 km s−1 velocity amplitude of HD 189733b. However, some systematic variations, mainly due to imperfect telluric line removal, remain in the residual spectral time series in which we search for the planetary signal. Using principal components analysis, the effects of this pattern noise may be reduced. Since a balance between the optimum systematic noise removal and minimum planetary signal attenuation must be struck, we find that residuals, which are able to give rise to candidate planet signatures, remain. The robustness of our candidate signature is therefore assessed, enabling us to conclude that it is not possible to confirm the presence of any planetary signal which appears at Fp/F* contrasts deeper than the 95.4 per cent confidence level. Our search does not enable us to detect the planet at a contrast ratio of Fp/F*= 1/1920 with 99.9 per cent confidence. Finally, we investigate the effect of model uncertainties on our ability to reliably recover a planetary signal. The use of incorrect temperature, model opacity wavelengths and model temperature-pressure profiles have important consequences for the least squares deconvolution procedure that we use to boost the signal-to-noise ratio (S/N) in our spectral time-series observations. We find that mismatches between the empirical and model planetary spectrum may weaken the significance of a detection by ∼30–60 per cent, thereby potentially impairing our ability to recover a planetary signal with high confidence.