Hi Heryck.
As Nikhil pointed, a lot of things can be done to improve the performance.
Mainly having a good architecture following the 4LC (4 Layer Canvas) architecture, removing unused references and making the right deployment plan will help a lot!
If you are experiencing impact in Business when doing the deploy, as a fast solution you can apply a 2-stage deploy. See here more details about it.
It will let you compile and generate the code and scripts before deploying it.
With this configuration you can gain some time while you check the main points Nikhil suggested.
Hope it helps :)
Cheers and Regards,
RR