Previous research found that quantitative information labelled on target objects of grasping movement modulates grip apertures. While the interaction between numerical cognition and sensorimotor control may reflect a general representation of magnitude underpinned by the parietal cortex, the nature of this embodied cognitive processing remains unclear. In the present study, we examined whether the numerical effects on grip aperture can be flexibly modulated by the relative magnitude between numbers under a context, which suggests a trial-by-trial comparison mechanism to underlie this effect. The participants performed visual open-loop grasping towards one of two adjacent objects that were of the same physical size but labelled with different Arabic digits. Analysis of participants' grip apertures revealed a numerical size-contrast effect, in which the same numerical label (i.e., 5) led to larger grip apertures when it was accompanied by a smaller number (i.e., 2) than by a larger number (i.e., 8). The corrected grip aperture over the time course of movement showed that the numerical size-contrast effect remained significant throughout the grasping movement, despite a trend of gradual dissipation. Our findings demonstrated that interactions between number and action critically depend on the size-contrast of magnitude information in the context. Such a size-contrast effect might result from a general system, which is sensitive to relative magnitude, for different quantity domains. Alternatively, the magnitude representations of numbers and action might be processed separately and interact at a later stage of motor programming.