Speaking on the flagship programme, Lord Patten blasted the former Foreign Secretary and inferred he was a liar.
He said: “I think he has always been extremely careless with the truth.
“I think he was when he was a journalist. Max Hastings, his editor, has pointed out, and I think he has been as a politician.
“Amber Rudd said, ‘he might be the life and soul of the party but you wouldn’t want him to drive you home’, and that’s my view as well.
“I doubt there is a single member of the foreign service who didn’t say ‘hurrah’ when he resigned in that public way, looking Churchillian at the camera.
“This is no Churchill.”
The ferocious interview comes after Mr Johnson resigned following Theresa May’s Brexit proposals at Chequers.
He also released his apology publicly which he received criticism for.
Chris Patten went on to say the Conservative Party is currently “toxic” because of the turmoil surrounding Brexit.
He added: “I’ve never seen the atmosphere so toxic as it is today, I’ve never seen a Conservative Party so marginalised on the big issues and I’ve never seen so much animosity in the Conservative Party.
“And I’ve never seen the nationalist right so rampant as they are today.
“So, on all those grounds I don’t go on holiday shortly in a greatly enthusiastic spirit.”
Boris Johnson made a personal statement 10 days after he quit his ministerial post, claiming the UK was headed “for the status of a colony” under Mrs May’s soft Brexit plans.
He claimed a “fog of self-doubt has descended” upon Brexit negotiations since the June 2016 referendum.
The Brexiteer praised the Prime Minister’s “courage and resilience” but ripped into her “miserable plans” for closer ties with the European Union.
Mr Johnson added: “Chequers now makes us rule takers.”