A) When you vibe-code, you don't have the mental model of the codebase. If you don't have a mental model of the codebase, you have zero way of assessing the complexity of any change.
B) There is a famous software engineering rule - First write the code to accommodate the request you have now, then refactor it so it can be easily extended with the tomorrow requirement. Your agent doesn't know what the next requirement is going to be. Therefore, even if it was capable of meaningful autonomous refactors, it would not be able to do the correct refactors. And these boil down to recognizing the patterns for application of correct design patterns.
C) Even when you can nail the points A and B, there often come requirements that are simple by themselves, but go against how the project was architected so far and therefore require drastic changes.