As you rightly pointed out, the regulatory landscape of dietary supplements is quite different from that of drugs or food, and this often leads to a host of issues, including misleading labels and unproven health claims. Our goal is not to feed into these problems but instead to use technology to parse through the vast amount of data and draw out meaningful information. Dietary supplements are not a cure-all or replacement for professional medical advice.

They can, however, be beneficial in some instances if used correctly and under the guidance of healthcare professionals. Melatonin, a hormone that our bodies naturally produce in response to darkness, plays an essential role in regulating our sleep-wake cycle. Research has shown that supplemental melatonin can benefit those with disrupted sleep patterns. For instance, a systematic review and meta-analysis published in "Sleep Medicine Reviews" found that melatonin reduces sleep onset latency (the time it takes to fall asleep), increases total sleep time, and enhances overall sleep quality [1]. Another study in the "Journal of Clinical Sleep Medicine" reported that melatonin could benefit those with Delayed Sleep-Wake Phase Disorder, a condition characterized by a significant delay in sleep onset and waking times [2]. These studies suggest that, under specific circumstances, melatonin can be an effective sleep aid, emphasizing the importance of consulting with a healthcare provider to assess individual needs and potential benefits.

[1] https://pubmed.ncbi.nlm.nih.gov/23691095/ [2] https://pubmed.ncbi.nlm.nih.gov/26414986/

How is any of it relevant to a system that uses LLMs trained on user-written reviews of health supplements to recommend products based on end user's descriptions of health conditions?

In the end, how do you want to make self-treatment options accessible? We have reflected exactly on your valid critique before doing and publishing this project. User or patient written feedback around medication or self-treatment options is always risky as a source. But, it doesn't help if big pharma is the funnel that determines the information situation, there the interest is purely monetary-driven. Yes, this project is explicitly about supplement reviews, but the overall big picture and the next iteration it also include the analysis of Reddit comments in support groups also for rare diseases.

There are dozens of anecdotes for treatment options that have not reached the mainstream only because they use substances that are not or not longer patentable and patients have to resort to pharmacotherapy that has significantly more adverse side effects, to complete the picture, i have experienced it myself as I have a rare autoimmune disease and only learned about a new experimental treatment options thanks to self build NLP pipeline that analyzed Reddit comments. In the end I was able to stop my classic medications that had unpleasant side effects. I don't want to make this too personal because I want to address your legitimate criticism. All I want to say is that we do not want to be uncritical about the adverse effects of supplements, as they're evident depending on the substance we're discussing.

The narrative "it is just a supplement, so there are no dangers" is definitely something to avoid. You should not take them without critical reflection, as supplements are not without adverse drug effects. Sometimes the same substance depending on your location, is a medication or an uncontrolled substance like Berberine for example. I took it too long before researching to find out it could adversely affect your microbiome etc.

This project is not about being uncritical of supplements. Btw the NLP pipeline also detects adverse drug effects because, as already said, anything that has an effect can also have side effects.

So long post is long, now help us improve here. What would you do besides the big fat red disclaimer we have in the project to address your concerns? Happy to adjust the project to improve here, we're going to have a blog post about this project, so I'm already thinking of having a passage around what I have written in this post.

I probably wouldn't use an LLM for this problem domain. Or, if I did use an LLM, it'd be as a way to map the user's requests into an expert system. The ES would generate a recommendation as well as a set of diagnostic assumptions extracted from the prompt. The user should be presented with a checklist of extracted diagnostic assumptions along with the recommendation. The recommendation should include any specific warnings about the active ingredient(s) together with a general warning about the wild west nature of health supplements - the active ingredient may not be present and other harmful ingredients may be present.

To build out the expert system, I would find a team member who is a medical professional with expertise in health supplements. An MD or researcher with relevant SME would be the obvious choice, but I've also talked with some truly excellent registered dieticians, nurses, and PharmDs.

Finally, I would only recommend a limited white-list of health supplements that have some form of third party verified quality control in place.

Honestly, if you're interested in innovating in this space, third-party vetting a la the GAO reports from my original post seems like a MUCH more valuable product than anything using AI hotness. I don't think people need an LLM here; what they need is ground truth, and an LLM can't help with that. If I wanted to innovate in the health supplement space, I'd put the NLP away and figure out how to automate ingredient testing.