Agile Platform as a "Batch/Back-Office Processing Solution"

Agile Platform as a "Batch/Back-Office Processing Solution"

  
HI,
I am on a steep learning curve with regard to the Agile Platform, but a very passionate learner  :-) and I need to get really clear on the degree of suitability of Agile Platform for a particular use case, and I need to understand WHY or WHY NOT....I KNOW that it is really clear to all that a great "sweet spot" for application development for Agile Plaform is web based apps, portals, integration - (please do not remind me of this).

My question is to the suitability for a different use case and why or why it is not ideally suited..

I will set out below a broad description of the software application that needs to be developed by a retail investment company.

Imagine an investement company that has, say 100,000 members.
The company is looking to run transaction processing and database updating on a continual basis as files are electronically received -(eg FTP) . The investment company is looking to run this processing on a 64 bit server in batch/background mode in real-time.
.
By way of background to the business, the company invests money on behaf of its 100,000 member investors into various investments, in varying amounts. The investments are contributed by the members by way of deduction of investment contributions from their pay.
The 100,000 members  work for a disparate range of employers, and the contributions are deducted at the time of each employer's payroll processing, so there will be weekly, fortnightly and monthly pays being done at different times, so conceivably, there could and will be a constant stream of batch files coming from literally thousands of employers over the course of a month.

The company needs to do daily "batch" type processing of thousands of input (transaction) records per day that provide the key information on contrbutions.
Nearly all contributions from members will be by way of amounts deducted from their pay, that are remitted by employers, via deposit into the investment companys bank account.

The investment company will therefore receive files from all employers who make such deductions and process these transactions. Naturally, this is not a web/user/protal process - it is a back office, batch type process - high volumes of records, need for validation and then multiple table updates, logging, audit trail and reconciliation. Just to provide a "flavour" of the type of processing required, and without being exhaustive or prescriptive, but just by way of a "sketch", imagine the updates that one contrbution transaction can cause to the following tables;

  • Employer table - total of receipts per file for reconciliation,
  • Bank Table        - Jurnals to record the contribution receipts that have been made
  • Investor/Member - Amount of contribution this period
  • Taxation Table  - Amounts contributed, tax concessions by member/investor
  • General Ledger - All the requisite accounting transactions -
  • Investment Ledger - Increment sums invested under designated categories for each member
I would be very pleased with any concrete information that could help me to explain to a prospective client if Agile Platform is the right tool for this application, and to be able to explain "why" or "why not".

The upshot of this story is ...Lots of records, validation, background batch processing, heavy duty database updating..real traditional back office processing
Never done anything like that but here are my observations.

a) This looks like a situation where you should buy a software.

b) In case you want to build it, I would suggest something that can process a big amount of files. But Biztalk is my other area of expertize, so I would never think of the Agile Platform to process files on a daily basis.

c) That degree of complexity is possible. I've heard of companies that even use the Agile Platform to process EDI documents and those are the trickiest files I ever had to read.
Get file from FTP/disk. Change to data structure. Trigger action to write it to database. Repeat a thousand times. Easy :)


Concerns
1. Someone from OutSystems must assure you if it can handle the number of files you have.
Assuming that you get a daily file from each member and they are approximately in the same time zone, you'll have something like 10000 files per hour at least. Processing 3 per second in a single server with an unspecialized software can be hard.

2. Be careful with the ordered processing of the files.

Don't worry too much with the rest.
Under the hood, Agile Platform is Java or .NET; if they can do it, Agile Platform can.

We have been using Agile Platform for some similar purposes. Not at that same volume, but still a high volume, with no issues. Server resources, as expected, make a world of difference.

I love using Agile Platform for this kind of work, because the maintenance is a joy. I used to process feeds from a data vendor in a combination of .NET. Perl, SQL, FoxPro, and Excel, all to do what Agile Platform can do within itself rather nicely.

J.Ja
Nuno Reis wrote:
Never done anything like that but here are my observations.

a) This looks like a situation where you should buy a software.

b) In case you want to build it, I would suggest something that can process a big amount of files. But Biztalk is my other area of expertize, so I would never think of the Agile Platform to process files on a daily basis.

c) That degree of complexity is possible. I've heard of companies that even use the Agile Platform to process EDI documents and those are the trickiest files I ever had to read.
Get file from FTP/disk. Change to data structure. Trigger action to write it to database. Repeat a thousand times. Easy :)


Concerns
1. Someone from OutSystems must assure you if it can handle the number of files you have.
Assuming that you get a daily file from each member and they are approximately in the same time zone, you'll have something like 10000 files per hour at least. Processing 3 per second in a single server with an unspecialized software can be hard.

2. Be careful with the ordered processing of the files.

Don't worry too much with the rest.
 Thanks Nuno

I will read and digest - much appreciate the feedback
 
Justin James wrote:
Under the hood, Agile Platform is Java or .NET; if they can do it, Agile Platform can.

We have been using Agile Platform for some similar purposes. Not at that same volume, but still a high volume, with no issues. Server resources, as expected, make a world of difference.

I love using Agile Platform for this kind of work, because the maintenance is a joy. I used to process feeds from a data vendor in a combination of .NET. Perl, SQL, FoxPro, and Excel, all to do what Agile Platform can do within itself rather nicely.

J.Ja
 
 
Further to my question above, in relation to Nuno's comment on the number of files..there could be, as an illustration, 100,000 members, but they will not all be subject of an individual file ...there will be one file per employer and an employer will likely have multiple employees who are contributors... So whilst the numbers of files are large, it is likely hat there will be a consolidation process of input files prior to triggering the back officce processing..so say all he files received in hour will be consolidated and then processed as a single onsolidated transaction file.

I also realise that , in relation to the gartner Pace Layer model, that this  application would be viewed as  the COMMODITY LAYER for an industry, and this would usually be provided to users as a package
And,  and that is he case here...the requirement being qualified is one where the package developer is interested in better ways of developing and maintaining their core package.
Hi,

In my opinion the quantity of information to process is not that big.

I had a personal project that I had to process 200Mb-400Mb files, parsing, modifying and inserting most the data in the database.
Since I'm geeky and I was worried about performance (it was going to run on my personal machine, not even a real server) did most of it as custom code on an extension.

Took about 6 hours of work (and lets say I complicated it alot more than it really was necessary) ..and at the end it was doing all the work in about 5seconds.


My point is more about what other values can you take from using the platform.
Doing batch processing is very frequent in OutSystems, even it not being the best target of it. Not because its hard or bad, but because it gets a bit obfuscated by the value of the other features.

Regards,
João Rosado

Nicely said, João!