James Duff, 2. Earl Fife