Performance is an important quality attribute of software systems. Numerous prior work focused on performance issues that can be fixed by simple code fixes. This paper categorizes and investigates 225 real-life performance issues in two separate groups: 1) localized issues (60%) that can be fixed by simple, localized code fixes and 2) architectural issues (40%) that require changes to a group of architecturally related files.

We summarized six typical root causes to performance problems, namely inefficient data structure, inefficient loop, general wasted work, slow fail, special input, and multi-thread issues, that apply to both localized and architectural issues. In particular, 19% of architectural issues are associated with multiple causes. Meanwhile, we observed four general types of architectural solutions to performance issues: 1) introducing new design elements (e.g. design patterns); 2) adding new architectural connections to leverage the benefits of an individual optimization; 3) making localized optimization that propagates changes to architecturally connected files; 4) duplicating code optimization to fix inefficient code cloned in multiple locations. This paper improves the understanding of causes and solutions to performance issues, specially for architectural performance issues.