Women and children were among those killed when the vessel was hit near the Bab al-Mandeb strait, the IOM said, according to the BBC.
Images from the scene showed a boat strewn with bodies.
Among the dead were Somali refugees carrying ID documents issued by the UNHCR, the UN’s refugee agency said.
About 80 survivors were rescued and taken to hospital, according to the IOM.
It remains unclear who carried out the attack.
The port is controlled by Houthi rebels who are fighting a Saudi-led coalition in Yemen’s two-year war. The coalition, in general, controls Yemen’s airspace.
The rebel-controlled Saba news agency said the refugees had been attacked from the air by the coalition, and coastguard Mohamed al-Alay told Reuters the boat had been hit by helicopter gunships.
But a port official told the AFP news agency that the craft had docked and the dead and injured had been hit with light weapons fire.
The coalition has not commented on the incident.
It is also not known where the boat was travelling to and from.
Despite the humanitarian crisis which has brought Yemen to the brink of famine, some migrants from Africa continue to travel to the war-torn country.
UNHCR officials said the people in the boat may have been heading towards Sudan or other countries to the north to escape deteriorating conditions in Yemen.
Somalia is also currently at risk of famine, according to the UN, with 2.9m facing food insecurity – following decades of civil war and a previous famine in 2011.