Can I make batch that repeatedly checks whether or not a file exists?

Hi, I am new to OutSystems.
Can I make batch that repeatedly checks whether or not a file exists and executes when the file exists?
And if I can make it, how can I do?

* The file shall be placed in the "Resources" folder, for example.

Hello Keiya,

Can I ask you what is your use case? Because there are ways to do what you want, but I don't think it is the best way to do anything I can came up...

Cheers

Hi keiya,

In Outsystems we have Timers to do such background jobs , but it will be better if you can tell more about your requirement like Eduardo mentioned . It will help us to help you finding most feasible way to do that.

See this link to know more about timers.

Regards

-PJ-

Thanks for reply Eduardo, PRAMOD.


I consider receiving a data file from another server and importing the file into the database.

When file reception is completed, an empty file indicating completion is placed by another product.

I want my batch to detect the empty file (and import data to DB).


See this link to know more about timers.

I understood that the timer can be executed repeatedly, but I do not know how to check if the empty file exists.

Can I write it in the "If" condition in the timer's Server Action?

Hi Keiya,

You can write anything in the Timer like you write in any other Action , also if the file resides in the server than there is an extension (Filesystem) which you can use to check file is empty or not.


Regards

-PJ-

Thanks PRAMOD.

Is it impossible to check the existence of a file only with outsystem's standard functions?

I would like to implement this with only standard functions as much as possible.

Hi Keiya,

  How are you receiving the file? Is it via an Outsystems action of some sort such as a Form post or REST API call, or is it from a completely external process such as an FTP file upload? And where is the file being saved?

If it is via something like a REST API call then you can call a process from within that REST API action itself or you could insert an entry into a data entity and then use a BPT (process) flow to put a trigger on a new record being created in that entity. I recommend having a look at the BTP Masterclass in the training.

One thing I would be very careful of though when dealing with files, if you are storing them on the actually file system of the web server then you need to take into account any future scale-ability issues. Outsystems is designed to be scale-able  and have multiple front end servers in a load balanced configuration. This means that if you upload a file to one server but the processing request is handled by the second server it won't be able to find the file. So you need to ensure that any files are either saved on a shared network file system or are stored somewhere else such as inside the database or on an external cache.

Thanks John,

The file is from completely external process like FTP upload and I do not have multiple front end servers.

I thought the implementation to check the existance of the file in one folder("Resource" folder) could be done with a simple If statement in Timer's Server Action. But as you mentioned, because Outsystems is designed to be scale-able, it is not possible to implement it by simple statement, right?

I checked the BPT Masterclass, but I could not tell whether I can check the existence of the file placed in the "Resource" folder only by OutSystems standard function..


Solution

I see 2 possible options. 

1) As Pramod suggested you can use the FileSystem extension from the forge in a timer to check if the file exists. You will need to ensure that the file is saved on a network share that is accessible by all front end servers. This is probably the easiest but you will need to take into account what happens if the file is partialy uploaded when you check as the file may be there but not accessible. One option is to try and rename the file, if the file is still being uploaded it should fail, if it works then the process the renamed file. Not elegant but should be servicable.

2) Use a separate "watcher" application such as a .net app using the file systems watch events. This application could then post to the Outsystems application via a rest API to say the file is ready, then access the file using the FileSystem extension (via a network share again). Not as easy but has the benefit of not needing Outsystems timers to continually check the file.

How big is the file anyway? If it is less than a couple of MB can you change the process to transfer the file via a REST post instead? Or if you can use option 2 above then have the watcher transfer as a REST Post.


Solution

Other thing to check is what FTP server you are using and if it has the option to trigger a process or action when a file has been uploaded. If it does you could get the FTP server itself to do a post to Outsystems to say a file is finished. For example something like this guy is trying to do

https://superuser.com/questions/1296329/monitor-ftp-on-iis-8-5-and-run-script-upon-upload

https://docs.microsoft.com/en-us/windows-server/storage/fsrm/fsrm-overview



Sorry, just reread your reply, initially read it as you DO have multiple front end servers :).

If you only have one and only ever intend having one then you can store the file locally but that would require setting up an FTP server on the Outsystems front end server itself. So you will probably have an FTP service running on a separate machine, that service will save the file on a network share and then you use the FileSystem extension to do the checks so my replies above still apply even though you have only 1 front end server. Either way the process is still the same from the Outsystems side, use the FileSytem extension to do your checks in a timer or have a separate background application watching the file and posting.

Thanks for reply, John

I will check the FileSystem extension first.