Hi Sienna,

Indeed, not everything is covered in the training. But on each certification page it is clearly stated about how many questions will be asked per topic. It also states all online resources (documentation) that you must read to understand the exam questions.

I honestly think one should read the documentation, as it gives you a lot more background information. In the end its not about getting certified quickly, it is about understanding the topics.

Regarding the way questions and possible answers are formulated I do agree, and feel it's sometimes testing more once capability to read than the capabilities of once OutSystems knowledge. Luckily, most certification details nowadays come included with some sample exam questions, so you get a feel of the way the questions are asked.

Regards,

Daniel

Hi Sienna,

Can you provide an example of such scenario? So we can better understand your feedback.

Thanks

I don't have any examples I'm afraid as I don't remember the questions (was years ago now) and of course I didn't copy any down. But I know a number of colleagues have said the same thing, that it feels like there are a lot of trick questions.

Also, in my basic certification exam I marked all the questions I wasn't sure about to go over at the end and when I went over them, I ended up changing all of them, although I literally could not work out what the correct answer was because of the poorly worded questions. Annoyingly it turned out that I changed all of them from the correct answer to the wrong one. I believe they were the questions where I was unsure exactly what the question was asking. So if you have access to my exam questions, you'll be able to see what questions I got wrong and therefore the ones I was unsure what they were asking. Thankfully I still passed but it would have been really annoying to have failed because of ambiguous questions.

@Daniel I never saw the prompts to read the documentation as you describe. I even did a search in one of the PDFs from my training for online resources and documentation just now and got no results. I did my exam for v10 so maybe that's something new in v11?

Hi Sienna,

https://www.outsystems.com/training/certifications/

Each certification on this page has a link with all the exam details, including a complete list of documention you need to read and understand to pass the exam.This info was also already available for all V10 version of certifications.

Regards,

Daniel

You are correct that this advice is provided in the current version, however I have checked my v10 training PDFs and was able to find no such advice.

But glad that it exists now at least. Though I still think it would be better to incorporate all the information into training material rather than having to read through dozens (maybe even hundreds) of pages of text online.

It exists already for years. I truly cannot see what difference it makes to move the information one must read to training the training material.

You still need to understand and therefore read it. If it is all converted in videos, you will just have hours of extra videos, it does not shorten or limit the knowledge you need to gain to past an exam.

Like the days at school you dive in the books to learn to past the exam. It takes some effort, and they makes the certification means something. 

What I was thinking was in terms of exercises and opportunities to put the information into practise, as opposed to simply reading or watching videos.

I concur with Sienna. 

Most people in the world don't have English as their first language. The questions should be put in a straightforward and clear English. Most of us speak it well enough, but certain subtleties are beyond me at least.

I agree, and will illustrate an example as I recently took the professional traditional web exam. Even for those who are proficient in English, the exam seems to be set up with trick questions. 

One such question was regarding the 'working in teams' topic and included a screenshot of a merge and publish screen, one of the options was written in such poor English that I was not able to understand what it suggested. To make matters worse, this specific question asked which option was incorrect. Obviously I chose this option, apparently that option was correct and I got no marks...

In addition, there are at least two questions that contradict directly what is mentioned in documentation.

1. A question about Screen Content Security. Button1 that is set to visible: True encapsulated within a container that is set to display: False. Button2 is set to visible: False. When using javascript to find the buttons in HTML and click them, which button can be clicked? I chose Button1 only, as display: False does not remove it from HTML whereas visible: False does. See https://jmjames.medium.com/visible-and-display-do-different-things-212457546561
2. A CSS question that so far, no one from our team has had correct even though, collectively we have tried all four options. It looks like a simple CSS hierarchy question that somehow does not follow the hierarchy explained in documentation as that answer was incorrect. We have tried as a team to figure out the trick to this question but we can't figure it out. See https://www.outsystems.com/training/lesson/1756/themes-and-styling at around 8:18

This is frustrating as the Score report email shows you only which topics you made mistakes on, however, when reviewing these topics we can't figure out what we did wrong. It provides no way to learn from your mistakes and has resulted in our team failing at their second exams as well.

Yes, scorecards from exams are completely useless; I think there is an commercial reason for this: fail, pay to do the exam again.

Sorry Joost, I don't agree, did you ever take Microsoft certifications? Those are written to fail as you barely have time to read and answer all questions.

My English isn't perfect, and I had no problem with passing the exams. The way these questions are written are indeed sometimes more a reading challenge, but that is nothing specifically to OutSytstems. You see that in certifications of other technology too.

My experience with Microsoft certifications is that there are always questions not covered by the training material. That is something I never experienced with OutSystems exams. If you read all the suggested documentation and understand what you read (you need English for that too), you should be able to pass the exams.

Also, the money on exams that OutSystems makes is probably not to be compared by far to what they make on the licenses they sell.

Hi Daniël, 

that is fine, but I don’t see the point in referring to how bad Microsoft certification is; We are OutSystems (OS)
And I agree, the price of the exams are peanuts compared to the license fees. I just really started OS last October and have passed all three exams I took in one pass.
However, there were several/multiple questions that suffered from symptoms mentioned above; several were formulated in double negative wording. Or were really long, which seemed to make it more an exam in ‘comprehensive reading’ ( Dutch: ‘begrijpend lezen’ )   This was not just my take on the exams  

And then there is the sample exam, which it really isn’t a good reflection of the real thing. Those were questions that were clear and concise. I suspect that everyone who takes these , will get a false sense of ‘I am ready’.  

Just my 0.02.

Joost

I agree, let's not compare ourselves to Microsoft - let's be better than them! And they get many things wrong, so not the best baseline to measure ourselves against.

And I don't see any benefit in making questions with tricky comprehension, or worse still, ambiguous, as I found some questions to be (v10). It should be purely a test of Outsystems knowledge and ability, not English grammar coupled with some luck.

Hi Joost/Sienna,

I wasn't trying to set the bar for OutSystems certification to what Microsoft does, the contrary! 

OutSystems is doing it very well compared to other companies for which I had to do certifications.

I was trying to explain to Joost that I don't agree to his statement:

"I think there is an commercial reason for this: fail, pay to do the exam again."

I think OutSystems does it all lot better then, lets not call names anymore, other companies that provide certification trainings.

For one thing that OutSystems differs for example in the fact that since V10 certifications no longer expire. Furthermore OutSystems does not require me to buy multiple expensive books to read, all training material is for free. They give a way free certifications, on events like ODC, and nextstep, or user group meetings. OutSystems partners get yearly free certifications and 1 free bootcamp voucher. OutSystems champions and MVP get certification vouchers.

I agree with you both about the way questions and answers are formulated,

Regards,

Daniel

I agree, there is so much that Outsystems already does that is far better than Microsoft.:)

But adjusting the training/exam material would be one way Outsystem can further improve, so here's hoping they take this onboard.

And for the record, I do hope all my comments came off friendly - as that is the way they were intended (as they always are). The ambiguity of the written word without tone or a smile means you can never sure, and sometimes reading them back I wonder if they might sound harsh, but they're never meant that way and always said with a friendly smile. :-P

Going back to the issue of the training material vs documentation. I think it would be fine to use the documentation if there was some kind of guided path through it with the ability to track what you've done.

So for example, there could be some of kind of personal marker you could set against every article in the documentation which you could use to show whether you had read that article or not. Ideally there would be 3 options, the default being hadn't read it, another being have read it, and a third you could use to flag/highlight it, which you might use for scenarios such as you had read it but wanted to find it easily to go over it again prior to an exam.

In addition to that, there could be a colour indicator which showed which exams any given article contains relevant information for. Some articles might have multiple coloured indicators because they contain information in more than one exam.

And it would be great if you could filter the documentation by a specific exam and it would display all articles containing information relevant to that exam, and they would be ordered according to the suggested learning path.

@MiguelDomingues

This is an example of a question that is incorrect. It’s a quiz question in the modelling data training in the reactive web guided path (don’t know if it’s an actual exam question or not but affects your Outsystems score)....

Business concepts that need to be stored and accessed in our applications should be modelled as...

1. entities

2. entity diagrams

3. entity relationships

4. database tables

The correct answer according to Outsystems is 1 but that doesn't make sense. A concept is an idea and can consist of actions, interactions, decision making, complex relationships and more, however entities store only data can’t capture an idea, only the datasets to be used. [https://www.outsystems.com/training/lesson-quiz/1917/quiz?LearningPathId=18]

And the Outsystems documentation itself contradicts that answer, see here where it states: "Once you identify your concepts, you must translate them into application modules." [https://success.outsystems.com/Support/Enterprise_Customers/Maintenance_and_Operations/D[…]02_Translating_business_concepts_into_application_modules]

This is typical of many questions. One of my colleagues commented that he felt a huge part of the Reactive exam questions were set up to trick you and that is a common feeling amongst many people I’ve spoken to. It may just be that they're ambiguous, incorrect or simply written confusingly rather than set to trap you, but either way, this doesn't test your knowledge of Outsystems but comes down to luck as to whether you manage to interpret the question correctly and surely this is not what you want from a technical exam?

I understand that English may not be the authors' first language but honestly, I think this kind of thing really needs to be fixed. It wastes people's and businesses’ time and money if they're failing exams because the questions are grammatically incorrect. And this is a common problem across all your exams (not just Reactive) with lots of questions being confusing, not because they're difficult technically, but because the English language used was poor or had an incorrect usage or poor choice of words.

I want to go into an exam feeling confident that I know and understand the expected content, not worried that I'm going to fail because I was unlucky enough to incorrectly interpret questions with wrong/poor/confusing English – but unfortunately that is how I currently feel about Outsystems exams.

Sorry not meaning to be rude, but I do find it frustrating.

@MiguelDomingues Here is another example of poor English.

D doesn't make sense and is confusing.

@Sienna why is D poor English, and yes you are right, it doesn't make sense, like A also doesn't make sense. Because of that it are the wrong answers.

@Daniel I presume it's use of poor English that has lead to them writing a question that doesn't make sense.

So many questions are worded badly like this plus lots of use of things like double negatives and phrases that are easy to trip up on, such asking 'choose what is not correct' and the answer to that question being 'not implementing X causes Y' which is clumsy and makes it more difficult to get your head around what they are actually asking.

And the use of words like 'concept' exampled above, and many others, are used in slightly the wrong way, which makes you question what they are actually asking and causes a lot of confusion and no doubt wrong answers, even when you have technical understanding of the topic.

Well an entity is a concept, and a module is also a concept. Maybe my English is equally bad not seeing the problem here. 

I've done multiple choice exams (non OutSystems) in my native language Dutch, and got the same "doesn't make sense" sentences. I think it's inherent to multiple choice questions.

Done many English (non OutSystems) multiple choice exams with the same type of questions, with double negative etc. 

So most of the time you have 2 answers from the 4 that are clearly wrong. Then from the two remaining one is better than the other. That's it. It's just a matter of elimination.

Never stopped me from passing any exam (and I was bad at school with Dutch and English grammar).

A concept is an idea - I don't see how a database can encapsulate an idea, only parts of it.

I have done many multiple choice questions in English (my native language) and I did really well in English at school and I've never had problems except with Outsystems.

But anyway, even if others do things the same way, it doesn't mean it's the right way and doesn't mean it isn't a problem, even though it doesn't stop people passing exams either.

Take a driving test exam question: "What is the clutch used for?" It's not asked: "What is the clutch not used for?" Answer A: Not accelerating, etc.

Well, I think the main reason they conjure up these questions in combination some awkward formulation of the answers is the underlying inherent problem of multiple choice (mc). 

The main criticism of mc is that it mainly tests the recognition of the (correct) answer. Which just open question actually tests the knowledge. However, it makes scoring much more difficult to do with computers and of course makes the answer open to interpretation. 

Multiple choice on the other hand is really great for scoring large amounts of questions for large groups of people without much less room for interpretation.

So, I think they do this to the questions and answer to try to tap more into the understanding and less into the recognition part.

Nonetheless, items ( as we like to call questions ) should pass a QA test and certainly, quite a few items, do not. And some answers are just plain wrong, to make matters worse.

Personally I don't believe it's a multiple choice issue as I've done many and never seen this problem in any others, even highly regarded ones.

Here's another example of a mistake. And as Joost said, these should be QA'd to ensure there aren't errors like this (although in this case it's pretty obvious what it should say, all too often it's ambiguous):

The Professional Traditional Web Exam was a lot better in both questions and answers.