The deal, which includes closer cooperation with Australia on defense, border security, policing and aviation safety, was a major factor behind the ouster of former leader Ishmael Kalsakau, who had tried to expand Vanuatu's international ties after winning a general election in November.
Kilman, who came to power on Monday in a secret ballot after Kalsakau lost a no-confidence vote, said it was unlikely the deal would be ratified by parliament as it stands.
"I think for us, at this point in time, I am not sure whether it is in the best interests of Vanuatu or not, only because we have not been consulted," Kilman said in an interview with Australian state broadcaster ABC in Vanuatu's capital, Port Vila.
"My view would be to revisit the agreement with both sides, the Australians, and the Vanuatu government, and see if there's any sticking points and then address that," he said, according to a transcript of the interview posted on the social media platform X, formerly known as Twitter.
If "there needs to be some changes, then we speak with Australia, we talk to Australia, to see what we can do together to make it something workable," he added.
The remarks by the prime minister come as China vies with the United States, a key ally of Australia, for influence and power in the Pacific region.
Kilman, who has been prime minister four times before this, has pledged closer cooperation with China in previous stints as leader.
His remarks are likely to cause some unease in Canberra and Washington, especially after Beijing signed a secretive security pact with the neighboring Solomon Islands last year.
China is currently Vanuatu's biggest external creditor.
Many nations in the Pacific region are looking urgently for outside investment, particularly in view of the increasingly negative economic impact of climate change, with a number of island countries in danger of partial or complete inundation as a result of rising sea levels.
