Everybody else was sure he was next in line for the throne, but he and his sister had their doubts