Graphene materials have been extensively explored and successfully used to improve performances of cement composites. These formulations were mainly optimized based on different dosages of graphene additives, but with lack of understanding of how other parameters such as surface chemistry, size, charge, and defects of graphene structures could impact the physiochemical and mechanical properties of the final material. This paper presents the first experimental study to evaluate the influence of oxygen functional groups of graphene and defectiveness of graphene structures on the axial tension and compression properties of graphene-cement mortar composites. A series of reduced graphene oxide (rGO) samples with different levels of oxygen groups (high, mild, and low) were prepared by the reduction of graphene oxide (GO) using different concentrations of hydrazine (wt %, 0.1, 0.15, 0.2, 0.3, and 0.4%) and different reduction times (5, 10, 15, 30, and 60 min) and were added to cement mortar composites at an optimal dosage of 0.1%. A series of characterization methods including scanning electron microscopy, energy-dispersive X-ray spectroscopy, X-ray diffraction, thermogravimetric analysis, and Fourier transform infrared spectroscopy were performed to determine the distribution and mixing of the prepared rGO in the cement matrix and were correlated with the observed mechanical properties of rGO-cement mortar composites. The measurement of the axial tension and compression properties revealed that the oxygen level of rGO additives has a significant influence on the mechanical properties of cement composites. An addition of 0.1% rGO prepared by 15 min reduction and 0.2% (wt %) hydrazine with mild level of oxygen groups resulted in a maximum enhancement of 45.0 and 83.7%, respectively, in the 28-day tensile and compressive strengths in comparison with the plain cement mortar and were higher compared to the composite prepared with GO (37.5 and 77.7%, respectively). These results indicate that there is a strong influence of the level of oxygen groups and crystallinity of graphene structures on the physiochemical and mechanical properties. The influence of these two parameters are interconnected and their careful balancing is required to provide an optimum level of oxygen groups on rGO sheets to ensure that there is sufficient bonding between the calcium silicate hydrate (C-S-H) components in the cement matrix and minimum level of defects and higher crystallinity of graphene structures, which will improve the mechanical properties of the composite. Finding the optimized balance between these two parameters is required to formulate graphene cement composites with the highest performance.