This paper proposes a fuzzy reasoning based directional median filter to remove the impulse noise in an image efficiently. The differences between the current pixel and the neighbors on the four edge directions are adopted in the fuzzy reasoning to determine that the current pixel is an impulse noise pixel, detailed pixel, or the noise-free pixel. A simple approach, which can determine the edge direction in the current sliding window, is proposed simultaneously. By the acquirement of the edge direction, the proposed filter will remove the noise if the current pixel is detected as an impulse noise. Simulation results show that the proposed filter outperforms several existing filter schemes for impulse noise removal.