Just a quick question guys how do you apply the robots.txt file in your application? Currently I uploaded one in my application resource robots.txt and deploy to target directory where target directory was left empty.

Now let's say I have https://test123.com/app1 when I type in https://test123.com/app1/robots.txt I do see the robots.txt file but I tried using a free web penetration scanner and it says my robots.txt file was not found. Am I supposed to set an SEO Site rule for this case?

SEO SiteRule:

Base URL: https://test123.com Root: Robots

also tried

Base URL: https://test123.com/app1 Root: Robots

I would like this to apply only for "app1" and not to other apps I have at my environment.


My Robots.txt is

User-agent: *
Disallow: /

I did try following this but the only effect was the scanner says the website is inaccessible.

https://www.outsystems.com/forge/component-overview/1179/robots

This the scanner I tried but it isn't free after 2 scans.

https://pentest-tools.com/website-vulnerability-scanning/website-scanner


I think this is the nearest to what I want to achieve

https://www.outsystems.com/forums/discussion/11369/robots-txt-needs-to-be-moved-without-301/

Hi Clintjie,

The robots.txt file needs to reside in the root directory of your website. So if the domain URL is https://test123.com, you need to have a https://test123.com/robots.txt. I'm not too familiar with SEO rules, so I'm not sure that's possible.

Kilian Hekhuis wrote:

Hi Clintjie,

The robots.txt file needs to reside in the root directory of your website. So if the domain URL is https://test123.com, you need to have a https://test123.com/robots.txt. I'm not too familiar with SEO rules, so I'm not sure that's possible.

Hi Kilian thanks for the prompt response,

I did look into the SEO site rule and the base URL to be changed always points to an Application Root so I'm not sure if it's possible to point it at the environment root.


Is there a way for us to upload the robots.txt to the root directory of the environment? Because you are correct about where the robots txt should be most web checkers look for the file there but I do not know how we can put the file in that location specified.


I don't think there is. It's not possible to do so via resource deployment: if you specify a path below the application, you get this error:

Kilian Hekhuis wrote:

I don't think there is. It's not possible to do so via resource deployment: if you specify a path below the application, you get this error:

Currently the robot metatags works similarly to a robots.txt file. Our client just wants the robots.txt file to be located where it should be found. But if it's not possible we could settle with the meta tags.


Thank you Kilian, you have a great day ahead cheers.