While it is well established that supermassive black holes (SMBHs) co-evolve with their host galaxy, it is currently less clear how lower mass black holes, so-called intermediate mass black holes (IMBHs), evolve within their dwarf galaxy hosts. In this paper, we present results on the evolution of a large sample of IMBHs from the NewHorizon simulation. We show that occupation fractions of IMBHs in dwarf galaxies are at least 50 percent for galaxies with stellar masses down to 1E6 Msun, but BH growth is very limited in dwarf galaxies. In NewHorizon, IMBH growth is somewhat more efficient at high redshift z = 3 but in general IMBH do not grow significantly until their host galaxy leaves the dwarf regime. As a result, NewHorizon under-predicts observed AGN luminosity function and AGN fractions. We show that the difficulties of IMBH to remain attached to the centres of their host galaxies plays an important role in limiting their mass growth, and that this dynamic evolution away from galactic centres becomes stronger at lower redshift.