Broadcasting watchdog Ofcom investigated after the joke, made by Clarkson about a Ferrari owned by co-host James May, provoked complaints.
He said the F430 Speciale “looked like a simpleton” and should have been called “Speciale Needs” on the BBC2 show.
Comparing it to a newer model, he said the car “was a bit wrong – that smiling front end – it looked like a simpleton – should have been called the 430 Speciale Needs”.
Charities criticised the remark, with the National Autistic Society saying it perpetuated “the prejudice and bullying which people with disabilities have to cope with”.
The BBC removed the joke from the repeat and iPlayer versions of the show and apologised for any offence caused.
It said the remark was not intended to make light of those with special educational needs or to make fun at their expense.
Ofcom said it recognised that “discriminatory language of this nature has the potential to be very offensive to some viewers as it could be seen to single out certain sections of society in a derogatory way because of their disability.”
It added: “The comments made by Jeremy Clarkson in this instance were capable of causing offence.”
It said: “While obviously intended as a joke and not aimed directly at an individual with learning difficulties, the comment could easily be understood as ridiculing people in society with a particular physical disability or learning difficulty.”
It concluded that because the BBC had apologised, decided not to repeat the comment, and said it was not intended to make fun of those with special needs, the case was resolved.
Clarkson, 50, who is reported to have been paid £2 million a year by the BBC, is used to provoking outrage with his remarks.
He sparked anger when he asked Richard Hammond if he was “mental” when he returned to the show following a near-fatal car crash and in 2008 his joke about lorry drivers murdering prostitutes led to complaints.
A year later he called then Prime Minister Gordon Brown a “one-eyed Scottish idiot” during a press conference in Australia and, in comments made to Top Gear magazine, he accused TV bosses of being fixated with having “black Muslim lesbians” on TV to balance out the amount of white, heterosexual males.