I do use the materials when describing my spell casting, which I only describe if it feels pivotal.

Like insect plague needs fat, sugar and grains, so my cleric hocks a big loogie of powdered whale blubber sashimi he’s been chewing at the center of the radius of the spell.

You only need to track the stuff with a GP cost; everything else your can use a focus for or assume it’s in a component pouch of you don’t use a focus. The GP cost items, yeah, I definitely track. Also the costs in time and ink (ie gold) for copying spells into your spellbook. My wizard of constantly broke, and the rest of the party have thousands of gold each.