Dalam Botol, or In A Bottle, made just over one million ringgit (£205,000) in five days, its producer told AP.
It cost 970,000 ringgit (£199,850) to produce and market.
There were fears that the film’s gay theme would put off conservative Malaysian audiences.
The film is about a man who undergoes gender reassignment to the detriment of his relationship with his male partner.
It ends in tragedy and filmmaker Raja Azmi said the story was based on people she knew.
She said the box office figures showed that Malaysia was “open and not so conservative anymore”.
Gay rights groups have complained at the negative portrayal of LGBT people.
Until last year, Malaysia banned gay characters in films, meaning that Sacha Baron Cohen’s film Bruno could not be shown in the country.
Last March, the Malaysian Film Producers’ Association announced that gay characters could be depicted as long as filmmakers show that homosexuality is wrong and has negative consequences.
Last week, the country censored the lyrics to Lady Gaga’s pro-gay song Born This Way for contravening strict public morality laws.