There should be a way to take inactive followers from inactive masters. With gold like a bribe or something.