The author, who sparked controversy in 1992 when she published an official biography on Princess Diana, appeared on Tuesday's show to discuss it.
However, the interview turned sour as she branded Phil "ignorant" after he accused her of being a "gossip".
Phil asked: "We've got the book here, all sorts of different claims.
"But for someone who allegedly moves in aristocratic circles, isn't this all a bit tawdry and undignified?"
Lady C replied: "There's nothing tawdry and undignified on the truth and our history in the making.
"Harry and Meghan's story is history in the making."
Phil said: "After your Diana book, I would be surprised if any members of the royal family or aristocracy would want to willingly talk and get involved with more gossip."
Lady C calls Phil 'ignorant'
Lady C hit back: "Well that shows how ignorant you are because the fact of the matter is that my Diana biography, as People magazine said, was the most believable of all the biographies.
"Had it not been for my biography, Diana would have thrown Charles to the wolves."
She continued: "My relationships with all members of the royal family are private.
"If you knew anything about my private life, which you don't because I don't leak like a sieve, I'm not a Meghan or a Harry..."
Phil interrupted: "But you are a gossip. You're a gossip."
Lady C hit back: "What? Everybody is a gossip. Gossip is history in the making or history that has already happened."
Well that shows how ignorant you are.
Phil said: "But only when it suits you."
Lady C said: "No, not when it suits me, when it's the truth.
"I think if you had any respect for the truth you would be able to appreciate that people, who actually stand up and go against the tide, understand the truth."
Phil hit back: "I think you'll find I've done that myself," to which Lady C said: "Only recently my dear!"
Lady C then came for Holly Willoughby as the presenter asked for her to clarify a comment she made.
Speaking about mental health, Lady C said: "You can't have mental health if you can't be truthful and honest."
Holly asked: "I don't know what that means."
Lady C said: "You should know what it means, everybody who has integrity knows what it means."
Holly said: "Sorry I was only asking you to clarify, sorry would you mind clarifying it for me?"
At the end of the interview, Phil told Lady C: "It was fascinating, a little patronising but thank you for joining us today."
Viewers watching were left feeling awkward over the explosive chat.
What did viewers say?
One person said: "OMG I can't take my eyes off but I feel so awkward!"
Another wrote: "@LadyColinCampb as awkward as this is to watch she has a point!! Sorry Phil!! But she’s right!!"
A third added: "Holly’s just sat listening to lady C and Phil arguing."
What did you think? Leave us a comment on our Facebook page @EntertainmentDailyFix and let us know.