He said: "I guess being a sportsman you don't show any weakness, you know what you're good at and you stick to that, whereas doing stuff you're bad at, it's a totally new experience."
He added: "I'm getting over that bar of letting stuff go and really learning. We're all afraid of putting ourselves out there."
Opening up about how his 2010 accident, which left him with a brain injury, is affecting his Strictly journey, James revealed that it's made him feel "scared of being judged".
He said: "People can't see an injury in the head, unlike a broken leg or arm that's in a cast. In my eyes the world is normal, but people are treating you very differently, and then you feel they're judging you."
He added: "You're doing something you've never done before and then being judged by people – it's a very scary thing to put yourself in to."
After James quipped that he'd have to find a song with "no rhythm" to dance to, host Holly told the star he was being "very hard" on himself.
I've got a list of things to remember – in my dreams I hear elbows, shoulders, posture, rhythm.
James laughed and said: "It's like when you go to an exam and your mate goes: 'I haven't done any revision,' and you know they have. There's no false modesty here."
Asked how his family and friends feel about him signing up, James said they can't wait to watch him.
"In a more they want to watch me be humiliated kind of way," he said. He added: "They're supportive but have said: 'We're really going to enjoy that.'"
With Luba telling James she thought training was going well, he revealed that he's even been dreaming about his dance moves.
He said: "It's weird the way you keep going and going. You've done it for so many hours but your head can't absorb any more information. I've got a list of things to remember – in my dreams I hear elbows, shoulders, posture, rhythm."
The couple are set to dance the tango to Spandau Ballet's Gold on Saturday night (September 21).
Phillip revealed that James's pal Martin Kemp has warned him "not to mess up the song".
"The song is amazing, whether we can do the routine that we've planned…" James teased.
James's girlfriend, American finance student Jordan Connell, is currently in New York, so he'll have to stay in for a few weeks if she's to watch her man from the Strictly audience.
The wannabe dancer laughed: "I don't think Luba is a limiting factor to me making it through each week – if I go out, it's all your fault!"
Earlier this summer, the rower revealed the reason behind his split from wife Beverley Turner after almost 17 years together. He said after suffering a brain injury in 2010 he was a "very very different" person, causing a strain on their marriage.
Head to our Facebook page @EntertainmentDailyFix and let us know what you think of our story.