Professor John McGrath, a University of Queensland psychiatrist, said several large international studies had linked both low and high levels of vitamin D with increased risk of schizophrenia and mortality.
"This is a very gloomy prospect," Prof McGrath told a vitamin D in foods symposium in Melbourne.
"This will put the brakes on how excited we get about telling everyone, 'You must take vitamin D'" Prof McGrath said.
But several experts said the proposed amount to be added to foods would not be toxic.
McGrath said a large Copenhagen study of almost 250,000 people attending GP clinics found low levels of vitamin D increased the risk of death two-fold.
But high vitamin D levels had the same effect.
University of Melbourne health researcher Professor Peter Ebeling said that people surveyed for the study had been referred by doctors and were likely already unwell, which could bias the findings.
He said a large current US study of vitamin D and mortality might provide more answers.
The conference, which discussed whether more foods in Australia and New Zealand should be fortified with vitamin D, was told that Canadian products including milk had been enriched with the vitamin since the 1940s.
Fortification of milk, soy products and margarine is mandated, University of Saskatchewan professor, Susan Whiting, told the conference.
Deakin University researcher Professor Caryl Nowson said about 30 per cent of Australians had low vitamin D levels.
She said it was difficult for people in the southern states and New Zealand to get enough vitamin D from sunlight in the cooler months and it should be added to more foods.
Currently, Australia only mandates fortification of margarine.
Deakin University Professor Rob Daly said vitamin D fortification could reduce severe deficiencies.
Monash University Associate Professor Andre Renzaho said he had reservations about fortifying foods unless it could be proven to help migrants, who are at high risk group of vitamin D deficiency.
Facts about Vitamin D:
*Most vitamin D is created by the body naturally after exposure to sun, but it is available to a lesser extent in some foods including fish and eggs.
*Vitamin D distributes calcium from food throughout the body and deficiency can lead to bone disease including rickets in children.
*Vitamin D has increasingly been linked to a host of diseases, including type 2 diabetes, multiple sclerosis and cancer.
*People at risk of deficiency include migrant communities who are dark skinned or fully clothed, those in residential care and people who spend long periods of time indoors.