The new building will apparently be by their favourite private members club, Soho Farmhouse.
The Beckhams are in the process of building a similar pad by the luxurious Oxfordshire retreat.
Once the Duke and Duchess of Sussex got wind of their plans, they decided to follow suit apparently.
The private members club, which welcomes non-members as guests, is loved by celebrities including Holly Willoughby, Charley Webb, Rochelle Humes, Myleene Klass and Emily Atack.
It's also a firm favourite of Harry and Meghan's.
Meghan even hosted her three-day hen do at Soho Farmhouse.
The venue has a luxurious spa, gym, boating lake, cinema room and gourmet restaurants, before marrying Harry in 2018.
According to a source, the couple are planning a three-bedroom property with its own drive on the opposite side of the Farmhouse to where Victoria and David's house will be.
They are funding the project themselves as well as paying back £2.4 million of taxpayer's money spend on renovating Frogmore Cottage.
The source told The Sun: "This house will be their bolthole when they are in the UK.
"It is costing around £3 million and will be on the opposite side of the Farmhouse to the small house being built for the Beckhams."
Harry and Meghan are also reportedly house-hunting in Meghan's hometown of LA.
They are currently living in a £10 million mansion in Vancouver Island, Canada.
The couple are forging their own way after quitting as working members of the royal family earlier this year.
Meghan and Harry will work to become "financially independent".
As well as concentrating on their charity commitments, Meghan reportedly wants to resume her acting career and has already been offered a voiceover role on The Simpsons.
It's thought that this year Harry and Meghan may also concentrate on expanding their family, with royal watchers believing Meghan will announce a pregnancy by the end of 2020.
Speaking to Yahoo's Royal Box, Royal Expert Omid Scobie revealed what he thinks the couple will focus on in 2020:
"I don’t think we’ll see many new patronages or anything like that.
"I think it will really focus on what they have on the table now - possibly another baby, who knows."
He added: "I think the big thing next year is the launch of the Sussex Royal Foundation, that is their focus now."
Visit our Facebook Page @EntertainmentDailyFix and let us know what you think of this story.