The easiest part of the labor market’s recovery is likely behind us, a Fed governor says.