The couple were originally due to head to the States with son Archie for a six-week break from the limelight.
It was thought they would spend Archie's first Thanksgiving in his mother's homeland, before heading back to his father's for his first Christmas.
However, instead it's claimed that the couple will stay in Los Angeles and enjoy Christmas with Meghan's mother Doria Ragland.
Harry has always spent Christmas at Sandringham, except from 2012 when he was in Afghanistan.
A source told The Sun the pair have declined the Queen's invite to her Norfolk estate.
"They need some time away to recharge and decide their plans for next year," they said.
Royal biographer Ingrid Seward added that Christmas at Sandringham can be "quite stressful" and the couple may not want to go with Archie at such a young age.
She said: "I think it’s sad they don’t want to be part of the family gathering, particularly now the Queen and the Duke of Edinburgh are in their dotage."
Ingrid also revealed that Harry's grandmother may be upset that he hasn't decided to spend the festive season with his family, but her "stiff upper lip" will stop her from showing it.
They need some time away to recharge and decide their plans for next year.
She added: "The Queen might be a little hurt but she’s far too gracious to ever let it show or be known."
Earlier this week it was reported that Harry and Meghan were considering spending the festive season in Los Angeles.
A source told Hello!: "It's been a hectic year and they are ready to have some downtime."
Head to our Facebook page @EntertainmentDailyFix and let us know what you think of our story.