This paper proposes that the corporate governance structure and financial constraints determine the willingness and ability of firms to implement labor hoarding, and thus influence the speed of employment adjustment. Using Taiwanese manufacturing firms in the decade after the 2008 financial crisis as the sample and conducting a mixed-effects analysis, our results confirm the presence of labor hoarding. The willingness of hoarding labor is higher for firms having a governance structure focusing less on shareholders’ interests, which leads to lower adjustment speeds. The ability of hoarding labor is stronger, so the adjustment is slower, for firms facing less pressure from financial constraints. Moreover, financial constraints have more impact than the governance structure, and impel firms to accelerate the adjustment throughout the decade. There is also evidence of asymmetric adjustment. The firms with job destruction have higher adjustment speeds than those with job creation, and the connections of adjustment speeds to corporate governance and financial constraints are more applicable to job destruction. We suggest that encouraging firms to adopt a governance structure of less shareholder concern, or more importantly helping them to obtain sufficient funds for retaining talent may be an effective policy to deal with brain drain in Taiwan.