Toby Young says he regrets “politically incorrect” remarks he made in the past and should be judged on his actions in promoting social mobility in education.
The right wing journalist has faced a backlash on social media after it was announced he had been appointed to the board of a new university regulator.
Left wing critics have attacked his suitability for the role.
Boris Johnson has defended him but Tory MP Margot James said he was mistaken to appear to “belittle sexist comments”.
Following the furore over his appointment, Mr Young said he had deleted 40,000 tweets sent before 2017 because a “handful of things” dating back to 2009 were being “wrongly misconstrued” as evidence he was opposed to gender equality and LGBT rights.
However, Labour said this was not sufficient action and his appointment should be revoked.
“If Theresa May doesn’t rethink this appointment, the message she is sending to students is that under the Tories, misogyny and homophobia will not just be tolerated, but rewarded,” said Dawn Butler, shadow minister for women and equalities.
Students and academics have, meanwhile, taken to Twitter to also call for his removal, highlighting disparaging and crude remarks he has made about women’s breasts in social media posts and a 1998 article for Gear magazine on “being a lesbian for the night”.
An online petition calling for Mr Young to be sacked from his post because of past remarks has attracted more than 28,000 signatures.
A lecturers’ union has complained that Mr Young – who helped to found the West London Free School – has no relevant experience and that he had only been chosen because he is a “Tory cheerleader” and passionate supporter of the party’s education policies.
But in a staunch defence on his blog his suitability to join the board of the Office for Students, Mr Young said he had championed meritocracy and rigour in education for decades and while he accepted that some of his past remarks had been in bad taste he rejected accusations of misogyny.