Donald Knuth had two rules for Optimization:

1. Don't do it

2 (For experts only) Don't do it yet.

I remember a case a few decades ago when a vital part of a product in development broke because of a change made by another team. The day before a demo. Myself and a small team completely rewrote our piece overnight mostly to avoid the broken component. We ended up with something that was much cleaner, ran faster, and used less memory.

Related to that, many years ago (can't dig out the reference, sorry), the same problem was given to different teams of programmers but with different required outcomes. One was to maximize performance, one to minimize memory usage, one to maximize understandability, etc. The team that ended up with the best performing and lowest memory usage was the one that aimed at improving understandability.