I‘d say it depends whether drinking coffee triggers insulin release for you, or other growth factors like mTOR. According to a quick search, coffee actually seems to decrease mTOR - that would be ideal for fasting. Milk OTOH is ideal for promoting mTOR (babies need to grow fast), so I‘d avoid milk during a fasting window.