Police banned the fourth annual "Seksualiti Merdeka" (Sexuality Freedom) festival featuring gay-oriented films, concerts and forums just days before it was to open in November.
Police said at the time the move was taken to avoid angering Muslims.
The court in the capital Kuala Lumpur sided with government prosecutors who said police acted within their powers in a bid to prevent unrest, and that those powers could not be challenged in court, festival organiser Pang Khee Teik told AFP.
Pang said he was "stunned" by the court’s endorsement of that argument and planned to appeal.
"We must keep fighting for equal rights," Pang told AFP.
The ruling underscores a pattern of persistent persecution of homosexuals in Malaysia, said Phil Robertson, Asia deputy director at Human Rights Watch.
"We are concerned over the discrimination and the intimidation of the lesbian, gay, bisexual, and transgender community. The rights of this entire community are under threat," he told AFP.
"(Malaysia) is using national security as a catch-all phrase to justify any human rights abuse, and that is unacceptable."
Homosexuality remains a taboo subject in Malaysia and sodomy is punishable by up to 20 years in prison.
Opposition leader Anwar Ibrahim was acquitted in January of having sex with a male former aide — charges he said were fabricated in an attempt to cripple the opposition.
Rights groups have urged the government to repeal the law criminalising sodomy, calling it repressive.
The ban on the gay festival followed Prime Minister Najib Razak’s pledge in September that he would grant greater civil liberties and break with the country’s authoritarian past.