The High Court judges voted 4-3 to reject a claim for damages for false imprisonment by one of the ethnic Tamils held aboard an Australian customs vessel in the Indian Ocean.
Lawyers for the Sri Lankans had hoped that a win would have also challenged the legality of the Australian policy of turning back asylum seeker boats to Indonesia.
The 157 Tamil men, women and children had left the southeast Indian port of Pondicherry in an Indian-flagged ship in late June and were intercepted by the customs vessel.
They spent weeks aboard the vessel before they were transferred to the remote Curtin Detention Center in Western Australia state on July 27, after India agreed to consider taking them back.
However, the Sri Lankans refused to meet with Indian officials and were flown to an immigration detention camp that Australia runs on the Pacific island nation on Nauru.
According to the U.N., more than 100,000 Tamil civilians fled to India to escape fighting between Sri Lanka's government forces and the now defeated Tamil Tiger rebels, who had been fighting for an independent homeland since 1983.
Australia enforces a tough policy of refusing to resettle refugees who attempt to reach the country by boat.