Your ability to metabolise caffeine is dependent on your genetics. In some people it is active for days, while others break it down faster.
The most scientific approach is to record 2 weeks of data where you drink coffee before bed and then another 2 weeks where you remove coffee from your routine. Compare the results to see if there is a significant difference. Only, then, will you know, with certainty, if it affects you or not.
If you do the experiment, please let me know how it goes. I would be interested to hear about your results.