MPs defended Mr Hands as they booed the speaker. The Speaker of the House of Commons said: “I don't require any help from the Rt Hon Gentleman. He wouldn't have the foggiest idea where to start. He was once a whip, he wasn't a very good whip."
MPs responded by shouting at him to withdraw his comment.
Mr Bercow added: “Members can shout as much as they like it won’t make any difference.”
However, he later apologised when he was challenged over the remark.
Sir Patrick McLoughlin said: “A few weeks ago you sent out a message to all members of Parliament asking them to treat each other with respect.
“Do you think your comments to the Chelsea MP reflected that?”
Mr Bercow said: “If I have caused offence, I will very happily apologise, very happy to do so.
“I have no difficulty with that.
“I didn’t think he was the person of tender sensibilities but if I have caused offence I am very happy to apologise.
“He is normally a most genial character.”
Mr Bercow has been under fire since he ruled against a third vote on the Prime Minister's Brexit deal unless meaningful changes are made.
Theresa May had been hoping for a third vote on her deal this week.
The response to his rule was met with anger by MPs who were going to back Mrs May's deal.
Meanwhile, some Brexiteers seemed pleased as it is likely to force the European Union to grant the UK more concessions.
Mr Hands had said the speaker was “one of the few public servants who was genuinely accountable to nobody”.