Mr Lavery, the party chair and a supporter of rival leadership candidate Rebecca Long-Bailey, said: "We need a female leader of the Labour Party. Stand aside Keir."
There are currently four candidates standing for the Labour leadership, three of them women. The other candidates are Emily Thornberry and Lisa Nandy.
Labour is likely to come in for criticism if the only man in the contest ends up winning - particularly as the party has never had a female leader, while the Conservatives have had two.
But there is also a political element to the debate as Ms Long-Bailey, one of the favourites to win, is seen as coming from the same wing of the party as Jeremy Corbyn while Sir Keir, probably the candidate with the best chance of beating her, is seen as coming from a more "moderate" or centre-ground position.
Speaking at one of Ms Long Bailey's campaign events, Mr Lavery said: "If we stand with Rebecca Long-Bailey, Baileyism, we'll have a leader who can take the fight to the Tories not in 2024 but in 2020.
"We've got a woman who is as strong as anyone within the party."
And defending his stance on Twitter later, he added: "Is it so absurd to suggest that after a century of existence the Labour Party should have its first women party leader."
The leadership contest is continuing following the decision of backbench MP Jess Phillips to pull out. Ms Phillips has now announced she is backing Lisa Nandy.
And Ms Nandy Lisa Nandy has become the second candidate to make it onto the final ballot in the Labour leadership contest. She picked up a nomination from Chinese For Labour, on top of the endorsements she had already received from the NUM and GMB unions.
Candidates require endorsements from organisations affiliated to Labour, or from constituency parties, to continue on to the final stage, a vote of party members and registered supporters.
Sir Keir already has the required nominations and Ms Long-Bailey is expected to get the support she needs to stay in the race.
Ms Nandy won praise from her supporters when she appeared on Good Morning Britain and defended the Duchess of Sussex against Piers Morgan’s claims that she has not suffered racism, asking the TV presenter “how on earth” he was in a position to judge whether there had been racism or not.
Ms Long Bailey attempted to head off claims that she offered a continuation of Mr Corbyn's leadership, saying she would take Labour in "completely different directions".
The shadow business secretary dismissed claims the outgoing leader and his allies would still run the party if she won, and said it was "disrespectful" to say she is another version of Mr Corbyn.
In an interview with the Daily Mirror, Ms Long-Bailey, a frontrunner in the contest, said: "Insinuations have been made: 'Oh these men have been pulling strings in the background'.
"I've been proud to stand on the policy platform that we've had.
"That's not to say I'm not a completely different person from Jeremy because I am, and I'll be taking the party in completely different directions."
She previously gave Mr Corbyn a score of 10/10 on his leadership, but told the paper Labour had to recognise what it got wrong and "can't them get them wrong again".
"I would do things very differently. He didn't have an easy time," she added.