Police said the athletes were not robbed and their story was not true.
Brazilian police have recommended American swimmers Ryan Lochte and Jimmy Feigen be indicted on charges of falsely reporting a crime, according to ABC News.
The swimmers said they were robbed by a gun-wielding assailant while returning to the Olympic village early Sunday morning, but Brazilian police said in a press conference Thursday that no crime occurred.
Teammates Lochte, Feigen, Jack Conger and Gunnar Bentz stopped at a gas station and vandalized a bathroom, the authorities said.
The gas station manager, through a customer who translated, asked the athletes to pay for the damages, officias said.
Surveillance video taken from inside the gas station shows the swimmers sitting on a curb together at one point.
Lochte has adamantly insisted he didn't lie, and that the team members had been robbed at gunpoint.
“We wouldn’t make this story up," he told "Today" show host Matt Lauer earlier Thursday in a telephone interview.
But Brazilian authorities have been citing inconsistencies in reports given by Lochte, the 32-year-old gold medal winner, and Feigen, 26.
Judge Keyla Blank on Wednesday ordered their passports confiscated. By then, Lochte had already returned to the U.S.
That night, Bentz, 20, and Conger, 21, were pulled off a flight in Rio that was bound for Atlanta.
Jack Conger being removed from jetliner Wednesday night.
The two told Brazilian investigators that the robbery story was false, The AP reported.
Conger, Bentz and Feigen remain in Brazil. They are not under arrest and IOC officials have said they are trying to arrange meetings with local authorities to conclude the matter.
At Thursday's press conferrence, Brazilian police officials said security personnel had pulled their weapons during the incident because the athletes were uncooperative and appeared highly intoxicated.
"No robbery was committed against these athletes. They were not victims of the crimes they claimed," Civil Police chief Fernando Veloso said, ESPN reported.
Judge Blank said surveillance video of their return to the Olympic village Sunday morning shows four men in good spirits.
"The victims arrived with their physical and psychological integrity unshaken, even joking with one another,” she said.
Lochte told Lauer, "We were not joking. We were shaken up.... We're victims in this and we're happy that we're safe."
Watch: Photo Finish! Runner Dives Head-First Across Finish Line to Take Olympic Gold