I'm reading this book, "Why We Sleep" by Matthew Walker, and it says the half-life of caffeine is like up to 8 hours ... so having a 3pm coffee means going to bed at midnight with half of it still active.
I've cut coffee out of my diet after noon lunchtime, and I think it's helping ... tough to be sure but as long as I believe it, it's pretty good
I don't think that book is generally considered scientific. It contains a bunch of opinions, but it is not peer reviewed article and not everything in it is backed by good studies.
I have noticed the same effect as well. There is a noticeable difference to the quality of my sleep on the days I have consumed caffeine and the days that I have skipped it. The quality of your sleep likely has effects on your mood, energy levels and over all stress levels.
However, I was referring to the direct effects of caffeine on your stress and anxiety levels. I would go so far as to say that if you are taking steps to reduce your stress and anxiety levels, quitting caffiene should be on the list of some of the first things you try. (Alongside the usual advice of good sleep, exercise, meditation and diet)
"After adjusting for additional dietary, demographic, and lifestyle covariates, positive associations between total weekly caffeine intake and anxiety and depression remained significant"
"Caffeine also increases anxiety in PD patients as well as among healthy adults at these doses although the exact relationship between caffeine-induced anxiety and panic attacks remains uncertain. The results suggest that caffeine targets important mechanisms related to the pathophysiology of PD."