Due to Charles Pelham's death at a relatively young age, the title of Earl of Yarborough later passed directly from his father to his younger brother, Sackville George Pelham, who also assumed the Charles's courtesy title of Lord Worsley.