Daniel,

Thank you very much for your response.

>>you published a lot of changes. Each time you press a version of the module is saved in the database. If you have a large module, maybe because you Bootstrap big excels or they include large images, then you quickly run out of space. <<

Okay, now we are getting somewhere! My development cycle currently has me "publishing" several times per hour.  One of the apps did indeed have large excel bootstraps, and the publish process seemed very slow.  

Since I now have the data I need, and it is not likely to be updated from the spreadsheets again (it's static state/county/city/zip code information), I presume that it is okay to delete the bootstrap timers and all related constructs once I have verified that I have the Excel data in the database.  Seems like there needs to be an option to only run the bootstrap only once, instead of "whenever I publish."

>>You can use dbcleaner on steroids safely to removed deleted apps, modules and entities as well as old versions of modules. <<

Very good to hear that.  Thanks for the verification.  And I will be patient to see the results. :)

Best,

=Joseph

Clearly whatever I am doing is not working and I need additional support. 

Once again, this morning I found my personal environment "put to sleep" because of excess database storage issues. In fact, today is actually worse than yesterday (now at 148%), not better:

This result is after using DB Cleaner on Steroids like this and then waiting until today for any cleanup processes to run on the server side:

My question is: what am I doing wrong and how do I fix this? Doing the automated clean up does not seem to have helped, and I want to get under the provided limit.

Thanks in advance.

=Joseph

 Hi,

Go to the table space usage tab, there you can see the top tables that use space.

That will guide you in understanding what is happening. Could be several causes:

- many 1CP on large modules
- detailed integration logging 
- lots of processes running generating lots of data
- do you store large binary objects (documents) in the database?

I think the last 2 bullets are not relevant seeing 148% is system data, so I guess either lots of logging or lots of module versions. Try to clean up as much as old versions  as possible change the keep days to 1 or 2 weeks.

Same for logging set the 30 days to keep back to 7 days for example.

Regards,

Daniel

Looking at Table Space Usage yielded the following:

There are a couple of clear candidates with tens of thousands of rows (all of the rest, including those not visible in the screen cap above, are either single-digit thousands or mere hundreds), but there is still no clear and obvious (to me) path to solving the issue.  I've got a button to "Clear Table" on all items named oslog_*, but no mechanism seems to present itself for working with those larger offending items.

Regarding your points, in order:

- many 1CP on large modules
I don't believe that my modules are particularly large
- detailed integration logging
I am not using this and don't need it - these are merely POC's -  Can I turn it off or throttle it in some way?
- lots of processes running generating lots of data
I don't have a lot of processes running
- do you store large binary objects (documents) in the database?
No, no BLOB's 

And just as further background:
There are only four entities, and only two of them are of significant size (note: all of my other apps have data size that is trivial and not likely to be of issue here):

• 51 rows in State
• 1811 rows in County
  • Counties are unique by State
• 24482 rows in City
  • Cities are unique by County
• 29839 rows in Zip
  • Cities can have multiple zips
  • Zip can be shared by multiple cities

I have a sense that it is the City and Zip tables (and the logs associated with them), and perhaps my frequent publishing (to save changes while developing, about 4-6 times per hour) that are causing the issue.

There are no longer any processes/timers/bootstraps in the project:

And the entities from the diagram above are the only ones in the project:

I am at a loss for what to do next. 

What are the specific, detailed steps that I need to take to solve this issue?  Sorry to be so clueless about this but as I said I've only been doing this for about 10 weeks and have really been focused on learning the development side of things so that I could pass my first certification.  

I am sure the more administrative side of things will become familiar over time, but right now this is a significant challenge for me.  I will go back and re-use the cleaner with your suggested values and see if that makes a difference.

Thanks again for all your time, attention and assistance.

=Joseph

>>You need to check what resources and images you have in your module and separate those into separate modules. So that you only do 1CP on logic that changes frequently. <<

There are NO resources/images in my module.  Only 2 simple screens and the four data entities described up-thread.

The only data-related changes I am making are to the Aggregates (a.k.a. read-only queries), I'm not adding, updating or deleting any data in the tables. Is the publishing process not smart enough to avoid reloading those 4 entities if none of the data has changed?  Am I really uploading 10's of thousands of rows of data with each 1CP click??  If so, I am astonished.

>>Each 1CP click is a version...it's not rocket science. <<

Never said it was, just that I don't know how, and not knowing how it might as well be.  There is no obvious mechanism in the UI of the DB Cleaner to clean up that huge item (ossys_Espace_Version). No button on the row you display above, and nothing here:

Note: not sure what that info-banner is telling me.

>>save the modules you work frequent on to your computer disk, then you can see which one is big <<

Have now done that, and yes, the one with the entity structure I sent previously is by far the largest.  But as I said, the only thing big in there is the data, and now that it's on the server, I only want to upload my code and query changes with each 1CP.

Is there a way to tell it to skip uploading the data every time?  To save only the last 5-10 versions instead of having to manually go clean up once-per-week?

Or should I just modify my POC to only use the counties, cities and zip codes of 2 or three states (instead of all 50+D.C.) so that I am using a much-reduced and much less dataspace-gobbling data set?

And for future planning: Should I always plan to bootstrap my data in one module that only gets used once at the beginning of the development process, and then put all of my app logic and queries to that data in a different module?  

These are the sort of questions that always seem to come up when learning any new platform; "tribal" knowledge that is not in the official documentation but is "learned wisdom" acquired from communicating with experienced developers like yourself. :)

Thanks again for all of your help and insight.

=Joseph

Dorine,

Thank you for your input.  Much appreciated.  I will try what you suggest and work on it again tomorrow.

=Joseph

A-ha! THAT's the ticket!

Changing the filter to 0 weeks was the key.  Thanks so much for your help.

................................................................................................................................................

...and 5 hours later:

Success!  Thanks to you both for all of your insight and assistance!

=Joseph

You'r welcome happy you got it resolved 

>>happy you got it resolved <<

Much appreciated.

Some questions still outstanding from my posts above:

1. When I have a module that contains bootstraps for significant amounts of Excel data, does every 1CP make another copy of the data, even if the data hasn't changed?
2. Does it do so even if I have deleted the bootstrapping timers/procedures?
3. If this is the case, is the recommended solution design to do the bootstrapping of the data in one module and then create second module with the relevant aggregates and code/logic so that I am not loading the bootstrapped data with every 1CP? Or is there some other architecture that is considered "best practice"?

Thanks,

Joseph

Hi again,

Some questions still outstanding from my posts above:

1. When I have a module that contains bootstraps for significant amounts of Excel data, does every 1CP make another copy of the data, even if the data hasn't changed?

Yes, your module is a file (.oml). Each version is a complete file, not the differential with the previous version. The main reason you had storage capacity issues.

So moving assets like images as public assets in their own module, saves a lot of space, as the pace of changes to the images is low.

Moving bootstrap logic and files to another module, has the same effect, these files and logic don't change often over the course of your application life cycle, so its worth to refactor those in seperate modules too,

2. Does it do so even if I have deleted the bootstrapping timers/procedures?

If you delete objects from your module, the module size will decrease, so future versions will be smaller, it doesn't effect of course the size of older versions store in the system table.3

3. If this is the case, is the recommended solution design to do the bootstrapping of the data in one module and then create second module with the relevant aggregates and code/logic so that I am not loading the bootstrapped data with every 1CP? Or is there some other architecture that is considered "best practice"?

Yes, I would suggest moving bootstrapping logic in another app and module, most likely it also requires other people to have access to it.

Hope these answers are useful for you.

Regards,

Daniel