Closing the gap between UI and Data
Certified

Closing the gap between UI and Data
Certified

  

 


Over the past few months my team at OutSystems has been focused on giving the best possible experience to newcomer developers, with minimal amount of training required. To do this, we keep a constant stream of feedback coming our way: from real world observation and usability tests, to the opinions of OutSystems community members, such as yourself.

For the upcoming release of OutSystems Platform (9.0.1.16) we took a few more steps in this direction.

Modelling data visually


We’ve made it simpler to leverage the natural and standard way of creating databases by allowing you to visually model your data. This means you can now add, edit, and connect entities without ever leaving the entity diagram. All of this while having instant feedback on the database relationships as you change them.



Geek alert!

Our previous entity diagram was built using Windows Presentation Foundation (or WPF), a 9-year-old desktop technology that made it challenging to deliver polished visuals and animations swiftly. We have now rebuilt the whole thing from scratch using standard technologies like HTML and angular.js, which will enable us to keep innovating and bringing new functionality with much less effort. We also took the opportunity to revamp the look and feel of entities, relationships and comments.


Bringing Data and UI closer together


We also looked at the path users take from entering the development environment until publishing their first page. We searched for hidden patterns and difficult interactions, especially for first time users. With those findings, we streamlined some of the action that weren’t that obvious or efficient.


In this new release, you will be able to generate List and Edit Screens directly from the Entity Diagram. This brings forth some of our hidden drag&drop scaffolding patterns into the context where data is modeled. It also allows a more natural leap from data modeling to interface building.



We also built it the other way around. Now you can add a list or detail screen to your Web Flow and bind it to the desired entity. This also means that you will be able to generate an Edit Screen without having to build a List Screen first.




For more information check out Creating Your First CRUD App.

So as an expert OutSystems developer, what will you get from this?


You will be able to model your data in less time, with fewer clicks and while having instant visual feedback on the relationships between your entities. You will also change context fewer times, as the platform is better fit to deliver the results you expect where you need them.

Finally, making the platform easier for new users means less time to make a new member of your team fully productive, as well as a bigger community, which in turn results in more open source projects and richer discussions.


As always, we are here to grow with you, so keep that feedback coming our way.


Cheers,


Sara Cruz

Software Engineer at OutSystems R&D

 

Sara

Awesome feature! 

Outsystems is definitely heading in the right direction here!

Also have a look at SAP/Sybase "Power Designer" this is one of the best data modeller on the market! 
It works just like the new outsystems feature, you are able to add attributes within one page, and you can assign foreign key to any table with a simple drag and drop, but it has alot more features than that! 

I'm glad outsystems platform data modeller works quite similar to power designer! :)
Everything's great but I have to say it loud that the quality of Service Studio 9 is terrible. Edit a variable type has a 20% change of raising an error. Afterwards, I'm not able to edit the variable type...I click and nothing...

Comparing to previous versions, it's 100x instable!! Please, fix all of this for god's sake!

Thanks
Hi Carlos,
 
I'm sorry to hear about your disappointment. When releasing a new version, our goal is to always take a step forward and improve the experience of our clients. Your feedback is key in this process. Therefore, we will contact you through private message in order to better understand what specific conditions lead to this instability and provide a suitable solution. Thanks for your feedback!
 
Cheers,
 
Gonçalo Tavares
 
Hi Robert,

Thank you for your feedback. Power designer is indeed a very powerful database modeling tool. Also a more complex one, as something with that number of features must be. 
By the way, if you think there is any specific feature from power designer you would like to see in the OutSystems entity diagram, please share it with us. :-)
 
Cheers,
 
Sara Cruz

Sara Cruz wrote:
Hi Robert,

Thank you for your feedback. Power designer is indeed a very powerful database modeling tool. Also a more complex one, as something with that number of features must be. 
By the way, if you think there is any specific feature from power designer you would like to see in the OutSystems entity diagram, please share it with us. :-)
 
Cheers,
 
Sara Cruz

Other useful features
1) Show all attribute and attribute properties all on a single screen, so we can sort, change/update the data type, attribute properies as required - sometimes you want to make sure all your attributes that you have defined are unified with eachother throughtout your database table.

2) Show all indexes on a single screen, so we can change/update any index as required.

3) Being able to group entities in a package just like PowerDesigner and MySQL Workbench


4) Able to change entity table colour for easy grouping.

Those 4 suggestions made above, are features taken directly from powerful data modeler tools: SAP/Sybase PowerDesigner! and MySQL Workbench.
Thank you for your reply Robert and for all the detailed information.
Your suggestions make a lot of sense, specially when dealing with large data models.

We'll keep these tools and features under our radar for future releases.

Thanks again,

Sara Cruz
Although this feature is, at first glance, quite awesome, as others have put it, "closing the gap between UI and data", is, as far as eSpaces go, quite undesirable. OutSystems itself has tought us, rightly so, to separate UI and data in different eSpaces, as to separate the data, business logic and UI layers of a complex application. With this new feature, it seems OutSystems is telling us to throw everything in one big happy eSpace. I don't think that's at all the way to go. Now, if we could easily reference an entity diagram in another eSpace, and do the same trick for creating the screens, that would indeed be awesome.
Hi Kilian,

As I'm sure you know, taking the first steps into any unknown technology can be quite daunting and sometimes even frustrating. What we tried to do in this project was decrease the negative impact of these first experiences and make Service Studio easier to navigate for a beginner, without damaging the productivity of advanced users.
You're right, this doesn't teach users they should develop Data and UI in different eSpaces. However, users will learn this further down the road when they are equiped to build more complex applications, as they do today.

Your suggestion about referencing foreign entity diagrams and build UI from foreign tables makes perfect sense for more experienced users and seems like a natural next step for this project. Thank you for sharing it with us.

Cheers,

Sara Cruz
Hi Sara,

It would be great to have that. As for beginning vs. experienced users: in my experience, bad habits are difficult to unlearn, so anything that experienced users should not do, should not be available to beginners. Just my .2$.
I agree with Kilian. Here's what is *really* needed:

* The default pattern out of the box is to have 2 - 4 tiers of eSpaces like OutSystems recommends.
* The IntelliWarp system recognizes an Action to *replace* the entities' CRUD Actions, so instead of calling "CreateOrUpdateXYZ" in the "Save" screen from the Entity, it calls the appropriate Action from the business logic layer.
* Have the built-in menu patterns not be a mess, and work the way OutSystems recommends (URLs, not page references) in a theme.

I understand that a big goal has been to continue to improve the experience for beginners... but the improved experience MUST TEACH BEST PRACTICES instead of worst practices! It also means that for the expert users, we don't have to abandon so much of what makes the platform great, easy to learn, fun to demo, etc. because it does things wrong.

J.Ja
I would have to agree with Kilian and Justin James.  I appreciate all the hard work you guys have done, though!  :)

Side question here:

Because elements of Service Studio now use standard web technologies such as HTML5 and AngularJS, could this be a hint of a forthcoming Outsystems Service Studio release that will easily run on multiple OSes, or even a release of Service Studio running in the cloud?
BTW: The issue posted above are not related to the new feature described in the first post at all! its a great feature! this thread is now offtopic, it now ask outsystems the question "Why isn't outsystems teaching best practices?"

It might help to understand "Who is Outsystems targeted users; Who is Service Studio and Integration Studio built for?" 

1) Business Analysts
2) Developers
3) ??
4) For everyone

Its not going to be easy to teach best practices to someone without basic knowledge of computer programming.
(not saying outsystems shouldnt start teaching best practices, outsystems should teach best practices!). 
Joshua Austin wrote:
even a release of Service Studio running in the cloud?
 
 Something like force.com? ;)
Robert Chanphakeo wrote:
BTW: The issue posted above are not related to the new feature described in the first post at all! its a great feature! this thread is now offtopic, it now ask outsystems the question "Why isn't outsystems teaching best practices?"
I completely disagree Robert, the feature is *not* great, because it doesn't adhere to best practices, and teaches newbies the wrong patterns! The mere title of this thread "Closing the gap between UI and data" is misinformed in that way.
What outsystems added is an "improvement" of an old feature.

  • Before the only way you could create an foreign key is to create an entity and assign the foreign key yourself, now you can drag and drop your mouse cursors on a fork and drag it from one entity to another and the IDE will create a foreign key for you.
  • Before you could only create one entity table at a time via multiple mouse clicks, now you can create many entities all via a single screen (with less clicks).
  • etc...

The way outsystems studio architecture your application has been kept the same as the previous version, no changes was made here.

In order for you to correctly architecture your application to follow outsystems best practices, this will depend on how you use the outsystems tool.

Note: I do agree with you and justin that intelliwrap does not follow outsystems best practices! but that architecture design (of not following best practices) is something outsystems created before 9.0.1.16 and that feature was not changed/updated in 9.0.1.16! same architecture but improved UI/UX.

"The IntelliWarp system recognizes an Action to *replace* the entities' CRUD Actions, so instead of calling "CreateOrUpdateXYZ" in the "Save" screen from the Entity, it calls the appropriate Action from the business logic layer." 

Strongly Agree!

The thing is... I spend a small fraction of any project on the kind of "let's model a LOT of data at once" that this change improves. Being super happy about reducing the number of clicks needed to access indexes (which isn't a very common task) seems a lot like a Ruby or Python developer telling about how many keystrokes their language saves them... it's a small convenience that doesn't improve the actual code.

I spend a large fraction of any project dealing with setting up N-tier architecture.

This change is good, and I'm not saying otherwise... but the time needs to be spent making the output of all of the "easy buttons" inline with best practices, not making it easier to follow worst practices.

J.Ja
Justin James wrote:


This change is good, and I'm not saying otherwise... but the time needs to be spent making the output of all of the "easy buttons" inline with best practices, not making it easier to follow worst practices.

J.Ja
This really depends on how you use the tool ; after all outsystems IDE is a tool.

For me, making it EASIER to data model, helps me quickly create entities, creating entities faster, this does not translate to ..now all developers will start to build apps with bad best practice!

For example, when I build an app, I create at least 3 eSpaces, one for data, one for logic, and another for the interface.

Example
  • Account - this is the interface tier (thin client)
  • AccountServices - this is the business logic, (REST API, core logic)
  • AccountModel - this is the data tier (Entity data model)

Apart from this, I also create extensions in c#, then I would wrap those extensions in an eSpace project!

I use the same outsystems IDE tool that every other developer use, do I design bad apps? :)

Its how you use the tool, just like you can write good code or bad code in visual studio!  

BTW: Not all project require n-tier architecture , if you just want to build a simple module component and release it via outsystems forge this does not require n tier architecture!

It would be good to see outsystems,,. at least create a n tier architecture project template and start to teach best practices to new developers, educate them and show them how to design a a basic and an advance n-tier architecture the right way!

I find advance features useful, if you were microsoft and you removed the format command, because its too dangerious and most of your users dont need this command, you will find that many advance users and techcian will be very annoyed with this type of decision.

Outsystems can not keep all users happy, and it shouldnt! It should focus on one type of user only, the developers. ...Because trying to make everyone happy will make no one happy! Outsystems just wouldnt be good at anything at all, it would just be a toy for new developers with many limitations - who wants that?
in regards to intelliwrap/scuffling feature, I believe this feature is suppose to help developers get quickly started... once the pages and logic have been auto generated, for you you would then customise it to suit your own individual needs. (this is how I use this feature), there should be more advance templates? how would you want it to work ?
Following "Best Practices", does anyone actually create entities in OS anymore?

I find it easier to use modelling tools (Enterprise Architect in my case) to create and manage the database and then use Integration Studio to connect the DB to the Data Layer eSpace.

So from my perspective, while it is nice to have improved features, it is redundant. 

An integration to the popular modeling tools may be a better investment of time, although I am sure people would complain that the one they use was not included.
Yes, we create all our entities in OS. We don't use any modelling tools.
Yes, we use OS as well. Using external DBs gives up so much power of the platform. There have only been a small fraction of tables where I wish I had more control, and it was usually in the matter of indexes, and I could ALWAYS add those indexes separately.

Keith, I am really curious what benefits you are getting from that process that make it worth the effort to do things like that.

J.Ja
That's interesting to hear from seemingly two of the most experienced OS users.
 
Maybe it's because my background is more from the analysis, data analysis and database side of development and the fact that I have always used advanced modelling tools, that I find it easier to do it this way, and possibly the fact that I do a lot of new development for clients with integration to existing databases, and also who are not used to OS, and maybe have existing practices which demand naming conventions and dba practices where doing the database work outside of OS seems more appropriate.
 
What do I like about doing the DB work outside of OS / what benefits am I getting:
1. I use Enterprise Architect (EA) for all aspects of modeling and analysis so the entities are kind of there anyway. Very often a Client has similar tools of their own.
2. The use of more advanced modeling features for visualization, change, publishing, sharing etc., is beneficial (although I suppose I could then type them into OS) it would be nice to have an import facilty maybe.
3. The use of the tool to actually control the database DDL etc., I feel give me some administration benefits
4. I like the additional control of naming, placement, using multiple databases, tablespaces ... etc. that doing the work outside of OS gives me
5. I think it gives me better handling of Nulls (always an issue with OS) and some other minor options than simply using OS (one could say maybe I am not using OS properly thats always possible)
6. Maybe it also gives another level of control in a team where only the "database aware" members are allowed to change the database (although I am sure that could be controlled in the data layer anyway)
7. I find when I use OS for entity creation it is often difficult to correct mistakes and I have to go back into the management studio to drop columns etc. and end up with entity_name2 etc.,
8. I also don't like the table names that OS uses although I admit they should not be used that often. But I do use handwritten SQL to avoid unnecessary loops. This could be fixed with more generated Entity methods for various easy functions.
 
I have to admit I haven't really witnessed the "gives up of so much power of the platform". If I create the database outside of OS and use Integration Studio to create the data layer, then from then on I believe I have as much power as I would if I had created the data layer using OS entity management. 
 
I would be interested to know what I am losing, maybe I have never realized that I was losing something? 
 
Are we talking from a 'rapid' perspective (don't want to say Agile) e.g. sitting with a business user and visually creating the application with data etc.  if so I think that would be equally hard to do with a multi-tiered architecture anyway, I do do that sometimes in a seperate project as a sort of prototype, but then translate it back afterwards. 
 
Is that a waste of time or is the cleanup aspect worth it?
 
Personally I find the database aspects of OS a little weak given it's focus is there anyway and there is much discussion on the site about that especially handling Nulls etc., and I am sure (as it is often stated) that some large companies may not even consider OS because of that, that's often been said also.
 
Always willing to learn but given my experiences to date I am happier doing it the way I do it, may just be a personal thing, probably has to do with my background vs for example a web developer background.
Keith Matthews wrote:
I find it easier to use modelling tools (Enterprise Architect in my case) to create and manage the database and then use Integration Studio to connect the DB to the Data Layer eSpace. 
Have you used visual paradigm (VP)? If so does EA compare to visual paradigm?

I use both Sybase Power Designer and Visual Paradigm, I tested EA however VP seems to work best for me (I started on VP first, this might have something to do with it!)
No. Never used VP. Have used Power Designer a long time ago. Can't really compare them.
VP and EA are more of a complete UML tool with a built in data modeller; Whereas PowerDesigner is a real data modeling tool with advance life cycle data modeling management features.


Using external DBs gives up so much power of the platformAny more detail regarding the statement:

"Using external DBs gives up so much power of the platform"
There are  pro's and con's of using an external data modeller.

Pro's
  • You can build your database the way you want it! ( follow best practices, company naming conventions etc)
  • You tables will NOT contain strange table names, indexes, foreign keys such as "osusr_a3c_account", OSFRK_OSUSR_UKO_USERDEVICEAPPLICATION_OSUSR_SFF_APPLICATION41358, OSIDX_OSUSR_UKO_USERDEVICEAPPLICATION_12USERDEVICEID etc
  • You can connect your database to an external tools and it will load your database tables with nicely presented table names, columns etc you will actually know where you stored your data! and be able to write very clean SQL query code.
  • If you are using mySQL you can use multiple databases to store your data (i.e similar to outsystems database catalogs) 
  • External data modeler tools is specialised in data modelling, it gives you more control and it is easier for you to design your database and continue to maintain your data externally (You can even perform forward and reverse engineering, database management, import/export data in and out of your database easily etc)
  • You can write queries against your database and execute the command instantly (no publishing required)
  • Easier to copy data from one database to the next database  
  • plus more 
Con's
  • It can cost you money, it can start from $0 to $10,000+ for a good data modeller
  • You have to know exactly what you are doing or you can really mess things up!
  • You always have to refresh your database using integration studio, and integration studio will not detect deleted tables (its a bug).
  • If your database changes externally and you forget to refresh your database extension, errors can occur! 
External data modeller is not for everyone! It can actually be harder to use if you never used one before!  outsystems data modeller should be sufficient for most users.
Looks like the pro's outweigh the cons to me, certainly do for me, especially when working in larger teams. Enterprise Architect can cost as little as $135, amazing tool for that price.
"Enterprise Architect" is cheap $135 is a steal!

More...pro's
  • When you connect to your database from an external app, you do not need to first create a view in order to display your table names correctly, you just connect directly to the database table and everything will always be displayed correctly without you doing anything.  
  • When your database changes, you do not need to spend time maintaining your view tables either (very painful process!)
  • Depending on your data modelling tool, you can write scripts and run the script, to do anything you want to do! (key advantage!)
  • You can always write beautiful SQL code that will always work with your database from one server to the next (key advantage)
  • You can write a bootstrap script to bootstrap your database tables this will save you many minutes to many hours! (it takes longer to create bootstrap logic via service studio!)
  • When you decide to migrate your database type from say SQL Server to Oracle, you can use standard migration tool, you wouldn't have to write logic in service studio to copy data from one table to the next for each and every table (this can save you a lot of time and money)
Robert,

"You can always write beautiful SQL code that will always work with your database" - This is a bit like saying you only use extensions because you can write beautiful C# code... Also, if you really must, you can write beautiful Advanced Queries.
"
You can write a bootstrap script to bootstrap your database tables" - You can do that perfectly fine against the OS-managed tables. We do it regularly, no problem there.
"
When you decide to migrate your database type from say SQL Server to Oracle, you can use standard migration tool" - Again, there's no problem with OS-managed tables. Also, that example is a bit far-fetched...
 
 Kilian Hekhuis wrote:
 ""You can always write beautiful SQL code that will always work with your database" - This is a bit like saying you only use extensions because you can write beautiful C# code... Also, if you really must, you can write beautiful Advanced Queries."

 


If you built your own database by hand, you can write beautiful code in both service studio and also via external tool. Built it in outsystems data modeller, you can "only" write beautiful code via service studio, and you can not write beautiful code via external tool (unless you create a view table, and then you have to maintain the view tables you create)

"You can write a bootstrap script to bootstrap your database tables" - You can do that perfectly fine against the OS-managed tables. We do it regularly, no problem there.
"
When you decide to migrate your database type from say SQL Server to Oracle, you can use standard migration tool" - Again, there's no problem with OS-managed tables. Also, that example is a bit far-fetched...
 

If you built your database using outsystems data modeller, the safest method to migrate your database is to write logic to copy your data from one table to the next.
 
I've been meaning to respond to this for a while now.

IMPORTANT NOTE: Development tool selection has a large amount of personal preference involved. For example, no matter how many technical merits it may have, I have never, and will never feel comfortable using emacs. If someone cannot enjoy using a particular tool, regardless of how good it may be at a technical level, this does not mean there is something wrong with them, it means that they are a human being with a personal preference.

Do not take any of my statements here as a criticism of anyone as a person or as a developer. These are facts and opinions as I understand them and my understanding will never be perfect. I am not trying to say that "if you don't like the OutSystems Way there is something wrong with you"! I am trying to say that the OutSystems Way is not for everyone, and if it is not for you, why force yourself to fit it, and why force it to fit you?


TL;DR: OutSystems isn't a development tool, it is a development process. Clinging to the way you have always done things is holding you back regardless of their technical merits.

I think that the technical merits of both the OutSystems approach and the traditional DB modeler approach are well understood by everyone.  Robert's summarized them above. Trying to understand this topic through the lens of technical merits is a mistake.

The issue is one of *culture* and *business*.

Here's the difference between someone using OutSystems as a "platform for the development process" (my approach) and someone using it as a "platform for developing code" (your approach). The first group says, "OK, the tools works like XYZ. Let's change our development approach do also match XYZ and tweak where needed". The latter group says "OK, the tools work like XYZ, and we do ABC. How do we tweak XYZ to work like ABC?"

Some of the typical symptoms I see of a shop using OutSystems in the latter approach:

  • Putting copies of OML/OSP/OAP files in Dropbox, Box, Git, Subversion, TFS etc. so they have "proper version control".
  • Using TOAD, Erwin, etc. for their data modeling then importing the data model via Integration Studio.
  • Writing a lot of Extensions in Java/.NET to do work.
  • Creating a substantial number of Advance SQL Queries where Aggregates would suffice.
  • Deploying via manual OML/OSP/OAP file transfers instead of using Lifetime.
  • Searching for ways to easily write queries against the OutSystems DB and being frustrated by it.
  • Big piles of handwritten JavaScript and jQuery code to do things that the system could do; heavy use of "WidgetClick" and "RunJavaScript" Actions within Screens.
Do you see what I'm getting at? They are using OutSystems because they find something about it compelling, but they haven't bought into the full process. They liked the drag/drop of the screens, or perhaps the "no code" approach to jQuery widgets. It looked like a good way to get the BAs or QA staff more involved in the dev process without actually teaching them how to program. Etc. The system has a ton of "hey, that's neat!" to attract people without convincing them that the full development process makes sense.

Many times at these shops without full process buy-in, a decision maker in their company said "we're using OutSystems, we've signed the deal, now figure it out and get to work". And the developers are basically rebelling, they are saying "we don't want to use this tool, why are you making use using it?" Knowing that the Big Decision Maker at their company isn't actually going to be reviewing their work on a day-to-day basis, they "fight back" by using the actual OutSystems Platform itself for little more than a screen generator, based on a pile of code (DB models, Java/.NET code in Extensions, JavaScript) that they have built elsewhere.

Here's a fact: traditional development techniques haven't been very successful. If they were so great, or just needed a bit of tweaking, or were waiting for the final feature to show up in Eclipse or Visual Studio, why is there a market for a tool like OutSystems or its competitors? Why are all of the "cool kids" building apps on platforms like Ruby and Python now? Why is it that I can't seem to find a Web Service written in the last 5 years that is SOAP-only, and only a coupleof them support SOAP at all, and many Web Services that used to support SOAP are now going full-blown REST? Why are people rapidly adopting things like NoSQL DBs? Why is it that it seems like every greenfield project I hear of is using NoSQL as the default except for the cases where a compelling argument can be made for a traditional RDBMS? Why are so many companies, large and small, desperately looking for alternatives to the Oracle (+ DB modelers) + Java (+ Eclipse) or the SQL Server (+ DB modelers) + .NET (+ Visual Studio) stacks?

There are a lot of things in the OutSystems Platform that are similar in purpose, but nowhere near the workflow of the traditional development model. A GREAT example is the versioning. When I first encountered OutSystems (around 2009) I scoffed at the version control system built into it. I was in a TFS environment at the time, and the OutSystems versioning didn't support ANYTHING I expected from version control. "How do I branch? How do I do XYZ? What about all of these features I used in TFS?"

You know what? I *did* use many of those TFS features, but they were not helping me reach my goal. In some ways they held me back. The OutSystems team re-thought version control and delivered excactly what I needed to do my job and meet my goals while leaving aside the features that did not accomplish that or actually hindered it.

What I didn't initially understand when I moved to OutSystems, was that having everyone deploy to a central server instead of building/testing on local machines would completely revolutionize the way I worked. The "fail fast, fail hard" approach to merging and code changes forced me to actually communicate and co-ordinate with my peers (shocking, right?) and understand and discuss the impact of my changes. Before, I would branch, go do my own thing, and then a few hours before we wanted to deploy, we would all merge our code, discover we all made a bunch of horribly conflicting changes, then slip deadline by days while we resolved the problems.

To summarize: giving up a ton of features for a version control system I initially thought was "shoddy" or a piece of garbage significantly improved my development process.

So I've learned to be open minded about the decisions made by the OutSystems platform, and I've bought into the PLATFORM AS A PLATFORM FOR THE DEVELOPMENT PROCESS instead of buying into as a PLATFORM FOR DEVELOPING CODE.

With regards to the data model, I've learned/realized a few things:

  • Most people use maybe 10% of the power of a tool like TOAD or SQL Server Management Studio or Erwin or whatever. Maybe 10%. How many people use functionality in them that does not exist in OutSystems? Virtually no one. Depending on these tools after moving to OutSystems is an emotional item for most developers not a technical item. They may raise valid technical reasons for it (if you were to compare the tools in a vacuum) but if you look at the actual use in practice, the differences in capabilities used by developers is pretty minimal on most projects.
  • These tools are expensive and difficult to use. If I bring them into my OutSystems projects, it means that instead of making my development process easier, I've made it harder. Instead of letting the developers use the simple, "can be learned in 10 minutes" tools within Service Studio, they now have a process with a million round trips involved, using tools that are complicated, difficult to use, and give up all of the "oops, I made a mistake, let's fix it" that OutSystems provides. Instead of buying an expensive OutSystems license and saving money on the development side of things, I'm making the development side of things more expensive too.
  • I've worked on/in some massive OutSystems projects. The job I had previous to my current job has over 300,000 users with over 2,500 tenants (at the time that I left, they are much bigger now). That's a BIG PROJECT. There was one table that required DB work outside of OutSystems (a complex index), and that was needed because we made poor data model decisions. It was 100% my fault, and I learned a lot of valuable, painful lessons on this one Entity. If I had not made poor decisions, that table would be perfectly fine. I would have made the exact same mistake using TOAD/Erwin/etc. It was a tool neutral poor decision.
  • Analogy: I can't write love poetry to save my life, and it doesn't matter if I use pen, chalk and a blackboard, or a computer to write bad love poetry. The tools are not the problem with most of the technical problems on a project, poor decisions are.
  • There's only so much time in a day. I'd rather have my team developing code than struggling with tools they barely know. The OutSystems data modeling system is fast, easy to use, robust, and easily handles situations like multi-tenancy. Using another tool loses that functionality and slows my team down.
To summarize (again):

Using external tools this much is a major red flag. It says one of two things:

You are either not buying into the full workflow and reaping the rewards of the OutSystems Platform. You are fighting the platform for one reason or another instead of working with it. Try going without your external tools for a period of time, do things the OutSystems Way, and have an honest conversation with yourself and your team if your development process and results truly improved, or if they were worse. If you find yourself miserable without those external tools, regardless of the outcome, it's a sign that maybe OutSystems isn't a good fit for you personally.

Or you have technical considerations which do not make your projects a good fit for the problems that the OutSystems Platform is meant to solve. OutSystems is meant to help developers solve a few specific development challenges. The farther your projects stray from those specific project types, the less the OutSystems Platform will benefit you. The good news is, OutSystems has been pushing the Platform harder and harder and harder to handle more problems better, and it shows. But if you are constantly finding that it does not meet your needs, and need third party tools to meet those needs... guess what? It doesn't meet your needs. There's no shame or harm in admitting that it may not do what you need it to do!

But again... you need to be honest with yourself. You need to know if this constant urge to use external tools is because you have an emotional attachment that you won't move past (even if you can find some technical reasons to justify or reinforce it) or if your projects are truly a poor match for the OutSystems Platform.

J.Ja
Hear hear! +1 (I'd +1000 it if possible :)).
Excellent post Justin!!  And really applies to any tool stack in my opinion.  You have to completely 'buy in' to the way the tool stack is intended to be used to get the most out of it, whether that stack is Outsystems or anything else.  Take a portion of your application, do it completely in Outsystems and see what happens.  If it works for you, great - if not, find a better tool stack for your application.

Curt
 I have an engineering background- so my skill set is very technical,  I am used to using technical tools and I mastered them before discovering outsystems in 2009.

I could have chosen to build our app without outsystems - 100% it would have failed! No doubt about it! it is much faster to make changes to the business logic code in outsystems than any other traditional programming language.

When using outsystems platform, you have to take the good with the bad; outsystems is a very highly productive tool. However it is not the best at everything, outsystems know this themselves, they had to build a tool for a wider audience and make commonly used features very easy for everyone to use, so they had to make some sacrifice too! 

I have my database using external reporting and analytics tools, and I look at my database a lot, and I write sql by hand, a lot for reporting, so I like to work with clean and beautiful code, but not every project and every company need to do this, as for my app, I use the simple SQL widget where I could.

If I wanted to get really technically, there are a lot of things I could say! but I learnt to understand how outsystems makes business decisions and make do with their decision.

I have been reviewing each and every single feature since 2009, everytime a new feature is about to get released, I will get a call from outsystems to tell me what they have been working on and what they will do next, and ask me what I think about it. I get to play and review many features before it gets released to you guys, some features that I ask for, outsystems will build it send the feature to me but never release it, some features are removed because it is just too advanced for most people and outsystems don't want to make their product ugly and cluttered that most people don't get or will find useful. Outsystems makes decision based on  90% of their user needs, but if you want to do something advance, then you are the 10% and you have to use external tools for advanced features to fulfill your requirement.

BTW: I use both outsystems data modeller and external data modeller, different project different use case different requirements, you have to choose what works for you and for your project, and what works for you don't always mean it will work for everyone else, and with every project! :)  

One guy likes to use a swiss army knife for everything, while the other guy carries a swiss army knife in his pocket and also a big box of tools for every single other purpose, don't limit yourself - use the tool that works for you!

I certainly agree with Justin's philosophy, in my career I have used many advanced analysis and development tools and my first approach is always to use them in their entirity, and in their recommended way, and using their best practices. Following that philosophy my team and I have made many of them work where others who 'fight the tool' fail. 

Regarding OutSystems the only thing I do not do using OutSystems is the database design and implementation, I use all of the other aspects of the tool as natively as possible and avoid writing hand built SQL wherever possible. And I have tried it both ways. I just find the database aspects of OutSystems weak and prefer the other way for reasons previously mentioned. I think the database support could be a lot better, or even left to be done outside of the tool using common framewoiks for access. I think there is a great opportunity to make it stronger even just by generating common methods for common database actions such as "DeleteAllForParent", (no I don't ever use delete cascade), and support for more functions.

I find it interesting that we can profess support for a 3 or 4 tier architecture, creating a data layer and wrapping the generated methods with our own methods for access by a business layer supporting a presentation layer, and yet still profess taht we use the tool as intended. I don't think that that was how it was intended at all, and several posts from OutSystems engineers have suggested that we do NOT do that. 

Keith




My first attempt was to use outsystems entirely, everything worked out well for projects that did not require me to connect with external tools.

My core project, required to work with external tools,  I could have solved this problem by creating view tables and then spend a lot of time forever maintaining them but why add technical debt?  

Just use something you already know! and something already solves the problem! In my case it made more sense to use 3rd party data modelling tool because it is currently more powerful than outsystems built in data modeller, I was more productive using a 3rd party data modeller for this project (I am saying this project, not every project is considered equal) - I could do what I wanted to do and get the results I needed, that is all it's about at the end.
 

How I think about solving problems? When attempting to solve a problem, I start with the customer first, and I use technology and tools to build products that solve the customers problems and I consider all factors and shakeholders involved.

 

Example of how I choose a technology

I love c#, but If I was to build an embedded system, I might consider alternative options since c# requires extra overhead to run the .NET framework, and extra overhead requires more powerful hardware, in turn this increases the cost of production, increasing cost of hardware production might not become a viable business venture any more, in engineering all factors, and shakeholders needs to be considered into project scope.

 

If cost and performance is a key factor for a viable and sustainable business, I would choose C programming language over c# in this case. I have came across micro controllers hardware that had only 128k of memory, where every character including spacing in your code would take up memory! you can't flush your code into ROM sometimes because your code was not optmised. There has been cases where I had to write assembly language for hardware optimisation as well, sometimes you just have to do it! because you are restricted by the hardware itself.
 

If I had to build a cross platform desktop application that runs on windows, and mac, I could use c# with xamarin/mono, and not make the project dependant on a extra framework. I love c# however a better choice would be to use Java for this cross-platform desktop app. 
 

I treat all technologies as a tool, tools that I can use to solve everyday problems.
 

I wouldn’t use a sledgehammer to put a straight nail through a plank of wood, when a claw hammer is more suitable for the job!

 I came from an engineering background my views and the way I think about solving a problem might be very different from how others think about solving problems! I will always consider all key factors and shakeholders involved, for me its never about the technology, its about using the right tool to get the job done (efficiently and in the most cost effective way as possible!). 
 

Justin James wrote:
I've been meaning to respond to this for a while now.

IMPORTANT NOTE: Development tool selection has a large amount of personal preference involved. For example, no matter how many technical merits it may have, I have never, and will never feel comfortable using emacs. If someone cannot enjoy using a particular tool, regardless of how good it may be at a technical level, this does not mean there is something wrong with them, it means that they are a human being with a personal preference.

Do not take any of my statements here as a criticism of anyone as a person or as a developer. These are facts and opinions as I understand them and my understanding will never be perfect. I am not trying to say that "if you don't like the OutSystems Way there is something wrong with you"! I am trying to say that the OutSystems Way is not for everyone, and if it is not for you, why force yourself to fit it, and why force it to fit you?


....
J.Ja

The good thing about using outsystems platform is it doesn’t leave you stuck nor does it force you to work in one specific way!

You always have the freedom to choose how you want to use the outsystems platform, either use it entirely to build your whole application or use both outsystems platform + external tools to help you become more productive.

You can even choose your own NET or Java stack  there are lots of open options! this is why outsystems is a clear winner for me!

I made my decision based on this specific project requirements, I am sure if you were me, you probably make similar or the same decision,  only because I know you and have also worked with you including the project you've mentioned above :)

There has not been one time where you've disagreed with any of my decision , nor have I disagreed with yours - good decisions are made by fulfilling both the business and technical requirements for a given project.

Once the full scope of a project requirement are fully understood, everything starts to become clear :)
 
p.s I don't disagree with you, I use the outsystems platform entirely too, just not for this project with good reasons.  
 
Looks like I amongst friends here! 

Platform should try to enforce best practice as I have spent a whole career cleaning up messes made in the first week of green field sites. Kept me in money but boy was it depressing to see the tool in question used so badly and the excuses that followed like "That is how we were taught on the course" If only green field sites paid the vendor for hand holding for the first projects, BUT developers always seem to think they know best and they like to learn by playing......this is absolutely the wrong way...no one is bright enough to be able to invent best practice from scratch.

Developers fighting the platform and using what they are comfortable with, in the end hurts the site and morale. If you hand code you better be sure you code is as uniform as a generator and you better improve the code on your own free time as when you use the platform the bright people in Portugal will be improving the generated code while you are working on other things and sleeping!

BUT the database as a tier maybe a little different as if the database already exists you will not want to recreate it in OS and what about new tables, I would tend to keep new tables with the existing database. But that decision would not be taken lightly. Maybe this is my view as I see OS as a great tool to reface existing systems, and for integration projects. A brand new site with no real database I would not even consider using anything but OS as this is the platform we have chosen and its everyones job to get comfortable with it and not fight it.

What attracts me to this platform is the business savvy approach and not the technology that is kept at arms length. You have to embrace low code and resist old habits and comforts from your 3GL days.
@George

If you are a hacker and want a tool with extensive flexibility to let you solve your everyday problems! Outsystems is not for you! With outsystems you have to go all in or you will just keep on fighting and fighting and fighting! (fight so much you either give up or just quit and stop using outsystems altogether).

If you want to use outsystems and be happy, forget about technology, how outsystems does anything under the hood, if it's efficient or not, or if it follows best practice or not, you got to think much like a business man and just get the job done and make money! then you will be happy.

I wish outsystems was built by a hacker with a business hat on! (then everyone would be happy!) but it was not built by a hacker and you just have to live with that. 
@Robert

I agree with everthing you say.

But for "or if it follows best practice or not" you are talking about the generated code best practice which imo is required for 3GL buyin BUT I care more about the best practice of using the RAD tool. I have made a career of cleaning up the mistakes taken by the RAD tool developers and just wished they had reached out and got best practice first instead of thinking they knew best, or they didn't have time to do it, etc.

Don't get me wrong I owe some of my career to fixing these problems but would much rather maintained and developed on a solid application and been proud of the solutions nd not had to defend them at every corner to solution architects, for RAD tools of today not to go the way of UML or 4GL of the 90's we must not give any excuses to the 3GL people as in my experience they will grab any excuse to ridicule RAD development. So use the RAD tools best practices from the get go!

Cheers, George

@George

I don’t want to comment on what is wrong and why hackers hate outsystems! but I could tell you something positive “Outsystems allows you to make changes really really fast!” hands down outsystems wins here!




 

 

I’ve been trying to tell all my friends to use outsystems but they want to! If outsystems wanted any of the world’s top developer/hacker at google, or Facebook to use outsystems, outsystems would get rejected no one wants to touch outsystems with good reasons. 

If you are not a hacker yourself and never sat down to talk to a hacker and all you do is just ignore them, you will never understand them and they ill never love what you do.

It took 15 years to sign up 600 customers, something is very wrong here! 

Outsystems can fix all of this and make the world love outsystems, it will sell like hot cakes but if only outsystems listen to a hacker.

Every idea I proposed to outsystems was heavily rejected, but many years later outsystems did exactly what I asked them to do and it worked out! from Outsystems enterprise cloud and free personal cloud these are just two of many proposed ideas at one meeting. I was never credited for this! but I don’t really care about that, its not about me anyways! it's about you the community the ecosystem the guys that actually use the outsystems product! And I just want to make outsystems product superior and more mainstream - I care about outsystems product, its future and what happens to this company!.

But do not listen to me, I’m just a hacker! I am no one - I know nothing anyways! (I'm only the primary user that use outsystems product)