Despite only living around 20 miles away, Prince George, Princess Charlotte and Prince Louis have only seen their baby cousin a couple of times reportedly due to the rumours Harry and brother Prince William are drifting apart.
Prince Harry was reportedly told he should speak to William before he and Meghan decide to step back from their roles as senior royals.
However, the dad-of-one "made clear he hadn't any intention of doing so".
The Daily Mail claims: "Neither had it occurred to him that there would also be ramifications for the young Cambridge children, George, six, Charlotte, four, and Louis, just one.
"The siblings have only seen their cousin Archie, eight months, once or twice since he was born."
It comes after the brothers released a joint statement to deny claims Harry was feeling forced out of the royal family because of his brother's "bullying attitude" towards he and Meghan.
A spokesperson for the pair said: "Despite clear denials, a false story ran in a UK newspaper today speculating about the relationship between The Duke of Sussex and The Duke of Cambridge.
"For brothers who care so deeply about the issues surrounding mental health, the use of inflammatory language in this way is offensive and potentially harmful."
On Monday, the pair joined Prince Charles and the Queen to discuss the Sussexes' future after they announced their decision to step back from royal life.
The siblings have only seen their cousin Archie once or twice since he was born.
Meghan and Harry said they will work to become "financially independent" while still "fully supporting" the Queen.
It was claimed that Harry arrived two hours early to the summit at Sandringham so he could secure a one-on-one chat with his grandmother to put across his side of the story.
The Queen's statement said that Meghan and Harry have "made clear that they do not want to be reliant on public funds in their new lives".
It added: "It has therefore been agreed that there will be a period of transition in which the Sussexes will spend time in Canada and the UK.
"These are complex matters for my family to resolve, and there is some more work to be done, but I have asked for final decisions to be reached in the coming days."
Leave us a comment on our Facebook page @EntertainmentDailyFix and let us know what you think of this story.