Harry and Meghan ‘declined’ to spend time with Queen Elizabeth II before her passing