Rayburn not going back