The Duke and Duchess of Sussex apparently turned down the invite because their four-month-old son Archie is "too young to travel".
It comes after he flew on a private jet to Ibiza and also visited Sir Elton John's home in France.
A source told The Sun: "Harry and Meghan had no plans to go to Balmoral this summer as they felt Archie was too young. It’s a bit odd, as his age didn’t stop them taking him to Ibiza for a week and then to see Elton John in France."
The Queen was said to be "disappointed" that Harry, Meghan and Archie turned down her invitation.
The monarch asks all of her family to enjoy a stay at her estate in the Scottish Highlands, where she and Prince Philip spend two months over the summer.
Harry's brother Prince William, his wife Kate and their kids Prince George, Princess Charlotte and Prince Louis flew up on a budget airline to visit the monarch.
If they want privacy or protection – which was Harry’s excuse for taking private jets – there is no more protected or private holiday destination than the Queen’s Scottish home.
The news comes as Harry spoke out about the family's use of private jets, saying it was sometimes necessary to use them in order to protect his "family's safety".
The source added: "If they want privacy or protection – which was Harry’s excuse for taking private jets – there is no more protected or private holiday destination than the Queen’s Scottish home."
However, there are suggestions that part-time vegan and animal activist Meghan didn't want to go to Balmoral because of the activities that take place on the estate.
The source revealed: "It’s all very hunting, shooting, fishing up at Balmoral, so maybe that put Meghan off. She has never been to Balmoral and the Queen was disappointed they’re not going.”
Let us know what you think of our story on our Facebook page @EntertainmentDailyFix.