Getty Images

By Lynne Peeples

WEDNESDAY, May 18, 2011 (Health.com) — Sleep problems and depressed mood often go hand in hand. Insomnia, early-morning waking, and nighttime disturbances can be symptoms as well as warning signs of depression—as can too much sleep and daytime fatigue.

For the past several decades, the treatment of depression has centered on medications that affect levels of brain chemicals involved in mood, such as serotonin. But the growing awareness of the link between sleep problems and mental health has led researchers to take aim at a new target to combat depression: the body's internal clock.

Specifically, doctors and drug companies are investigating the role of melatonin, a hormone released by the brain in response to darkness. Melatonin levels vary naturally throughout the day in what's known as circadian rhythm. When properly tuned, this rhythm helps us sleep and wake at the proper times. But when it gets out of whack, it can disrupt energy, alertness, and mood.

Doctors have long prescribed over-the-counter melatonin for insomnia and other sleep disorders, and even as a remedy for jet lag. Although there are skeptics, some experts now believe that drugs that mimic the effects of melatonin might simultaneously stabilize the internal clock and help relieve depression.

"We've been working with the same theory of depression since 1960, focused on moderating a group of chemicals that include serotonin, dopamine, and norepinephrine," says psychiatrist Ian Hickie, MD, of the Brain and Mind Research Institute at the University of Sydney, in Australia. "Now, we're considering a completely different notion of what the problem is."

In an article published this week in the Lancet, Dr. Hickie suggests that melatonin-based drugs could prove to be safer and more effective treatments for depression than the antidepressants currently in use. In particular, he and his coauthor highlight the potential of a new drug called agomelatine, a synthetic version of melatonin that also increases concentrations of dopamine and norepinephrine in the brain.

The drug, known as Valdoxan, is currently approved for the treatment of depression in Europe and Australia, and may be reviewed by the Food and Drug Administration (FDA) as early as next year. Both Dr. Hickie and his coauthor have received research funding and other financial support from Servier, the French pharmaceutical company that developed the drug.

In clinical trials, agomelatine has improved the symptoms of depression more effectively than placebo, and it has appeared to be as effective as widely prescribed antidepressants such as Zoloft and Prozac. Many patients taking the drug also reported improved sleep.

Melatonin is a relative of serotonin, and while agomelatine does bind to serotonin receptors, it does not actually increase serotonin levels. Dr. Hickie says this may boost mood without the side effects of many serotonin-based drugs, which can include nausea, headaches, weight gain, and a loss of sexual desire. "That's the real beauty of it," he says. "You don't get the serotonin side effects."

Next page: Some psychiatrists are skeptical

Not all psychiatrists are as excited about melatonin-based drugs, however. Marc Serfaty, MD, of Priory Hospital North London, in the U.K., says it's "debatable" whether the relatively small improvements in depression symptoms seen in the agomelatine studies are "clinically relevant."

Moreover, Dr. Serfaty's own research—including a recent clinical trial that compared melatonin and placebo in people with depressed mood and sleep problems—suggests that the over-the-counter melatonin pills found at the local drugstore may be as effective as agomelatine.

Researchers have explored the use of melatonin in people with mood disorders in a series of small studies dating back to the late 1990s. Melatonin consistently improved the sleep quality of the study participants, but it appeared to have nonexistent, slight, or even negative effects on mood.

The research on melatonin in depression is thin and inconclusive partly because funding is difficult to come by, says Robert Sack, MD, a psychiatrist at Oregon Health and Science University, in Portland.

"I don't think melatonin itself has been sufficiently tested as an antidepressant," says Dr. Sack, who has consulted for Takeda Pharmaceuticals, the maker of a melatonin-based insomnia drug, ramelteon (Rozerem). "The problem is that there is no commercial incentive for drug companies. [Melatonin] is readily available and not subject to copyright."

However, Dr. Sack adds, depression comes in many varieties, and agomelatine and similar drugs may prove especially effective in certain subgroups of depressed patients in whom circadian factors play a major role. (In patients with bipolar disorder, for instance, sleep disruption can even trigger episodes of mania.)

Dr. Hickie, too, is quick to acknowledge that agomelatine will not replace other antidepressants. "This is not the answer for all people with depression," he says. "I still think Prozac-like drugs are marvelous for anxiety and obsessionality, while this new drug might help a different group of people."

The Novartis Corporation, which has partnered with Servier to market agomelatine in the U.S., has already completed several clinical trials of the drug for depression. Julie Mascow, a spokesperson for Novartis in New York City, says the company will continue testing the drug and plans to file for FDA approval in 2012.