One click clone production database to development environment

By Braxton on 29 Oct 2015
I would like to see a "Clone Database" button in LifeTime, which copies the Production database backwards into the QA and Dev environments. 


Each time a problem occurs on our production environment that can't be recreated on the dev environment for lack of similar data I get more and more frustrated with this platform.

When developing on a dynamic web platform you often need real world, live data to test a behavior, even more so when troubleshooting an unusual behavior that has been reported by a user. This necessitates being able to quickly and easily clone the production database into the development environment.

Unfortunately, OutSystems has an incredibly complex key and token system which turns a simple database clone into such a Herculean task it requires a 20 page technote! http://www.outsystems.com/home/document-download/1484/8/0/0
This is completely unacceptable.



I would like to see a "Clone Database" button in LifeTime, which copies the Production database backwards into the QA and Dev environments. This would make diagnosing complex and nuanced issues that happen in the Production environment much easier and greatly improve the RAD timeframe and lifestyle.


Sincerely,
Braxton Bragg
Kilian Hekhuis2 Nov 2015
Yeah, that would be nice. However, there's a few issues to take into account:
1) Law. In many countries, you are not allowed to have unmodified production data on your testing environments, if it contains customer data.
2) Unless you actually copy physical database files (which the platform couldn't or shouldn't do), and depending on the size of the database, transferring the data could take ages and effectively shut down the production and test servers and/or the network. We have a database that's 100GB. We are a small company. Imagine a large company, which could easily have 100TB. You really, really, really don't want that transferred over your netword record by record...
3) It is technically insanely complex if you want to restore the data record by record because of foreign key dependencies. There may be tricks to circumvent this, but it ain't easy...
4) *Everything* is stored in the database, including users and eSpaces. I assume your development environment has different users than production. I'm pretty sure your development environment has eSpaces with different content than production. This has all to be taken into account as well, adding to the complexity.

That said, I totally recognize the use case. I just don't think that it is feasible to ever implement something like this.
Braxton4 Nov 2015
You could in theory restrict the restore job to individual applications/espaces like you do deploys via Lifetime.
This could check to make sure any applications that reference the database also are restored as well etc.

This changes your restore from 100TB to something reasonable since you're no longer restoring all old espaces, outsystems logs, etc.
Kilian Hekhuis5 Nov 2015
Braxton,

You could, but it'd be a monumental task. CoolProfs has developed such a tool, which took them ages and it's still not perfect.