Bootstrap to upload initial bulk data through excel(data from legacy systems)

Hello,

We are relatively new to outsystems and would really appreciate any guidance regarding our queries as we prepare to architect and design our apps.

We are planning to create a mobile app for our sales department. As an initial step, we want to create a web app for our business to search and view data and then use the same data as the input for the mobile app.

App 1- Web App for viewing the data

App 2- Mobile App for updating / viewing the same (with some additional) data

The data for the App 1 (Web App / List / for viewing) is currently available in our existing legacy systems. We can extract this data and prepare spread sheets matching our entity models. Say for example customer, address, contact etc.

We started by creating a new data structure model and building an entity relationship diagram in outsystems.

Now, our initial approach is to bootstrap the entities and data from excel as we have 1-1 match between spread sheet and entities. The customer entity has a primary auto generated id which is being referenced in the Address entity (Foreign key). Similarly, the address has an auto generated primary key which is being referenced (Foreign key) in the contact entity.

This being the scenario, I have a few questions that needs some validation from experts in the community.

1. Since we have the entities available in excel, is it advisable to use the boot strap option to create and load the initial data? We see that it is working in development environment, can we follow the same procedure in production? What is the best practice in production for initial data load / existing data migration during go live for a new outsystems app (view only- list app as in our case)?

2.Can we make changes in the entity name/relationships after bootstrapping the initial data from excel?

Ex : Table1 Customer – Primary auto generated Id

Table2 Address –   Foreign key from Table1(CustomerId) 

Is advisable to make that change after creating the entities using boot strap option?

As per our understanding it not possible to specify foreign key during bootstrap entity creation.

3.We want to use the web app to view the data and the mobile app to update the data. What would be the dependencies that we would need to take care of to achieve this?

Any best practice for such a scenario?

Appreciate all the help!

Thanks

Sav

Hi Savitha,

That's a lot of questions :). I'll try to weigh in on some of them.

  1. Bootstrapping data. If you have a lot of data, and a lot of tables in the legacy system, using the standard Excel bootstrap option may not be the best way to go. Alternatives could be to actually connect with the legacy database (if it's a supported database) and either keep using it or use the connection for bootstrapping, or use the Excel route to get the data into the OutSystems database, but in temporary Entities including all foreign key relations as plain Integers (instead of actual Ids) and bootstrap from there, so you can map the legacy system's Ids with the OutSystems created Ids.
  2. Making changes to data. It's possible to do this, but it would be cumbersome and leave your database with data you won't be using afterwards. It's imho better, as I wrote above, to import the data into temporarty Entities that you can afterwards drop in the database, than to have temporart Attributes that are more difficult to remove.
  3. Web app for viewing data and mobile for updating. I'm not sure what you mean by "dependencies"? How to achieve this depends a lot on how you're going to set up the Mobile App, especially with regards to any off-line capabilities and data synchronisation.

Hello Savitha,

I have recently been working on a project  where we had to bootstrap some of the inital data from excel files as well from a legacy source. While I am no expert in OutSystems and am not 100% sure that this would be considered best practice, I can share some of my experience on the issues you describe / the questions you have.

1. Yes, that is pretty much the whole point of the bootstrap option. We did modify some of the bootstrapping actions generated from scaffolding though, to make sure our data from the excel files matched the entities properly. I'll get into that in my answer for question 2. a bit more.

Bootstrapping timers are usually set to run "when published". Therefore, we have a practice of turning off the bootsrapping timer(s) via ServiceCenter once a module is deployed to test/production for the first time. This serves to make sure the timer only runs once during the inital deployment to an environment and NOT again during follow-up deployments or updates, which might clutter up your database.

2. You can even make those changes before bootstrapping, but also afterwards (to the same extent that you can do so for any other data that wasn't bootstrapped). Just make sure you keep in mind that your bootstrapping actions might not work again if you change things afterwards and that you will need to bootstrap the data for each environment separately. No data will migrate between environments with the code when deploying, except static entites which can't be bootstrapped as far as I know.

As for how to include foreign keys in your bootstrapping: You just need to make sure that you bootstrap the files for different entities in sequence.

Then, you can include aggregates or SQL queries in your bootstrapping action and assign the foreign kyes before creating a new entry for the currently bootstrapped data. Here an example from my project:

In this case, we actually need to include 2 foreign keys per created "GruppeUser". The assign looks as follows, the two foreign keys are marked in red:

In your case, you just need to make sure to bootstrap in sequence customer > address > contact with an aggregat or SQL in the last two that are able to provide you the foreign key you need.

For those aggregates / queries, you need to make sure they can only ever give you one result. In order to achieve this, we had to include an additional column in some of our excel files that do not get saved into the entities but are just there for the purpose of retrieving the proper foreign key from the earlier entites at the time of bootstrapping. This might seem like wasted effort but it allowed us to set up the bootstrapping such that we did not have to modify our entities after they were set up, even though they did not match the excel files completely and even included foreign keys.

3. For the case you describe, having all the entities in your web app, setting them to public with write access and using this as a dependency for your mobile app should be fine. Better practice would probably be to put all your entities and bootstrapping actions into a service module (with no screens) and have a separate module for your web app and mobile app that both reference the service app as dependency. It's easier to make sure you follow the principles of the 4 layer canvas that way.

I hope this helps. If you have any further questions, just ask. For anyone with more OutSystems experience, I'd also be interested what the best practice approach would be for this.

Best regards,
Michael

Hi SAVITHA MANDARATH,

Adding to what was said before here is my opinion:

(...)

1. Since we have the entities available in excel, is it advisable to use the boot strap option to create and load the initial data? (...) can we follow the same procedure in production? 

2.Can we make changes in the entity name/relationships after bootstrapping the initial data from excel?

3.We want to use the web app to view the data and the mobile app to update the data. What would be the dependencies that we would need (...)

1. No. I'd advice to use integration or migration tools (avoid having production data in excel files).

2. Sure. But maybe not needed from last question, however consider new nomenclature when uploading again same excel file structure.

3. A core service module. Centralize info for other two modules (for mobile and web). You need to see the mobile course also: https://www.outsystems.com/learn/courses/12/developing-outsystems-mobile-apps/

Regards,

@Kilian @Michael  @Marco 

Thank you so much for all your valuable suggestions.

We are yet to finalize an approach , will come back and update the thread once we have a solution in place. Once again appreciate all your help and have a great weekend !.