Waiting for the number of running backups to drop below the system limit of 3 ..

I finally got serious about starting to configure back-ups for all users, and after setting up only a few with largely default options and "simple schedules" I've run into a problem that I just knew intuitively was going to be an issue with using just "simple schedules" for all domains. Back-ups have started failing with this error:

Waiting for the number of running backups to drop below the system limit of 3 ..
.. gave up waiting after 120 seconds

What is the acceptable, logical and scalable way forward here? It never did make sense to me that all domains on the server would run back-ups at midnight (along with all of the other cron jobs that run at midnight), but manually keeping track of hundreds of domains running back-ups a few minutes apart is going to be a nightmare of monumental proportions, exactly the kind of manual busy work that control panels like Virtualmin are supposed to eliminate.

I'd appreciate your thoughts on this.

Craig

Status: 
Active

Comments

Howdy -- hmm, can you describe your backup setup in more detail there?

I think there's a piece or two that I may be missing.

I'll offer that most commonly, an admin will setup one backup schedule at midnight that's configured to backup all Virtual Servers.

It sounds like you're describing a system containing many more backup schedules than that though?

You can also increase the number of allowed concurrent backups at System Settings -> Virtualmin Configuration -> Backup and Restore.

Eric, to answer your question, I'm setting up a daily (incremental), weekly (full) and monthly (full) back-up on each virtual server for all of the sub-server under that virtual server. I am not doing a server-wide back-up of all virtual servers, as I have a different system for doing server-wide back-ups outside of Virtualmin.

Jamie, I assumed there was a place to configure this, and I strongly suspect I left this setting at it's default when I set up the server originally. I'm leery of setting anything to "unlimited", but if I can set it to something high (like in the dozens or even hundreds) and scheduled back-ups will wait patiently in the queue without all trying to run at exactly the same time and crippling the server, then that would work for me. Is that what you suggest?

Craig

What you're describing would certainly work.

I'll offer that what you're describing sounds like it could be fairly tedious to setup... and the problem with having multiple backups running at a time is it can end up impacting performance of your server.

My first thought is that even though you're doing backups elsewhere, that it might be easier to also do a full and incremental backup of all domains from within Virtuamin using a single backup schedule.

If that's not desirable for some reason, I might then look to making a script to perform a backup the way I needed it.

For example, you could create a script that looped over all top level domains (using the command "virtualmin list-domains --toplevel").

Then, you could have an inner loop that queries all sub-servers of that top-level domains using the command "virtualmin list-domains --parent DOMAIN.TLD".

From there, you could run a backup on each resulting domain with "virtualmin backup-domain ...".

With that, you could backup all Sub-Servers and not the top-level domains -- and then you'd be preventing multiple backups from running at the same time.

Those are just ideas though, it all comes down to what your needs there are.

Eric, what you describe seems way more tedious to set up. Admittedly, it's going to be tedious to set this up for existing virtual servers, but if I'm not mistaken I seem to recall that I can include a back-up schedule in a hosting plan template. Feel free to correct me if I'm wrong, as I'm going by memory and haven't checked it for myself yet.

If I understand you correctly, you're suggesting that root set up a single server-wide back-up schedule. Sounds fine (but is what I already have happening outside of Virtualmin), except that individual users will not be able to manage their own back-ups, and I will have to be asked to step in to restore back-ups instead of individual users doing it themselves through Virtualmin. This seems to be totally against the whole point of a self-service control panel. Or am I misunderstanding you?

What I want -- to set up a basic back-up schedule when creating virtual servers and then allowing users to manage it all themselves through Virtualmin -- doesn't seem to be an edge case to me; rather it seems to most logical and basic set-up. Are you suggesting that doing it this way is going to cause performance problems on a server that is otherwise capable of handling nominal traffic to existing websites? Is the performance problem going to be because, as I said in my last post, scheduled back-ups won't "wait patiently in the queue without all trying to run at exactly the same time and crippling the server"?

Okay, I see what you're after now... I'm talking to Joe about how best to accomplish that.

So it sounds like what is wanted is a way for root to setup a single backup for all domains, but to allow domain owners to do their own self-service restores from that backup destination? If so, this is something we could implement ..

Jamie, that wasn't what I said, but maybe it's a good idea. Except that what I had in mind (and I'm surprised this seems so unusual, apparently) is setting up new virtual servers with a default back-up schedule (daily, weekly and monthly) as if the user himself had done so. He can then delete or change or add to the default schedule as he wishes, as well as restore individual back-ups on his own.

The issue seems to be not in the act of setting up scheduled back-ups, but in how they are actually being run when the schedule comes up. The subject of this ticket is the error I saw recently with a couple of scheduled back-ups failing because (to put it simply) too many were scheduled at the same time, and they were scheduled at the same time because I chose a "simple schedule" rather than a complex one. I could choose a complex schedule, but then I either have to manually keep track of spacing back-ups or pick random times for each new set of back-ups.

I guess what I'd like to see is Virtualmin running all scheduled back-ups in some sort of intelligent rotation, instead of starting 100 back-up jobs at midnight and having 97 of them fail because the 3 that succeeded took longer than 3 minutes. I realise I can change those defaults at System Settings -> Virtualmin Configuration -> Backup and Restore, but my questions about the knock-on effect of doing so went unanswered.

Which led me to my other earlier question: If Virtualmin or cron won't somehow balance things out or let a job sit in the queue for however long is necessary to let it run in turn so as not to cripple the server, what numbers should I use in the "Maximum concurrent backups" and "Time to wait if maximum is exceeded" fields at System Settings -> Virtualmin Configuration -> Backup and Restore?

Hmm, I was actually thinking Jamie had come up with an awesome solution for what you were after with his idea above.

Perhaps we're not understanding though.

It seemed like you were looking into a way of setting up all these individual backups, so that the Virtual Server owner would be able to restore the backups.

The problem though is that setting up all these individual backups is very tedious, and there's no real way to organize when they'll run.

Jamie's thought is that rather than making you setup each backup individually, he's going to add a feature so that a Virtual Server owner ran restore a backup made by the root user.

Then, you can just setup one backup schedule for your entire system, and then the Virtual Server owners can restore on their own.

I know you weren't initially looking to use Virtualmin to backup the top-level domains, but you could always just remove those from the schedule if you really don't want them.

It's pretty rare that we see a system with more than a handful of backup schedules, and we'd highly recommend something like the above rather than attempting to setup numerous individual backups.

Hi Eric,

Thanks for your reply. I don't want to appear ungrateful, but I think that the whole point of this problem has been missed, despite the longer and longer messages trying to describe it.

The issue is that back-ups are failing because of how Virtualmin is scheduling them by default. Full stop.

The issue is NOT the actual setting up of scheduled back-ups, although now that I look at Jamie's idea again I do agree that it is indeed an awesome solution (and better than the one I had planned to implement), as long as I can set it up once and each new virtual server I add in the future will automatically be included. Thank-you Jamie.

In the meantime, to prevent the errors and failures I'm seeing -- and I am currently only backing up 7 top-level virtual servers with 37 domains, not all 150 or so domains on the server -- I have increased the "Time to wait if maximum is exceeded" field at System Settings -> Virtualmin Configuration -> Backup and Restore to 600 seconds. I will increase it again if I get further failures of the same kind.

But again, Jamie's idea does sound like a good one if:

  • it solves the time-out problem, which presumably it will if it's just one cron job,
  • I can set it up once and future virtual servers will automatically be added or included, and
  • user's can restore their own back-ups as if they had set up the cron job themselves.

On that first point, the Webalizer cron job seems to be running in the background all day, so I'd imagine Jamie's idea would be too, at a low priority.

Thanks.

Craig

Jamie, do you have any thoughts on Craig's issue of auto-scheduling backups?

Craig, I do see your point completely. If we're going to offer a way to automatically generate a backup job, it should be able to run without error.

We'll certainly look into that.

While I do think that Jamie's idea is perhaps a better way of going about all this for the long-term -- so long as we're offering the ability to auto-schedule individual backups we'll probably need to look into a way of getting that to run smoothly.

Thanks Eric. You make perfect sense.

Here's an idea: Years ago when I started using Slicehost (which later became Rackspace Cloud) they had some sort of automated way to suggest that if you wanted to run updatedb (part of mlocate) on your VPS that you do it at a specific number of minutes past the hour, so that not all VPSes on the physical server were running it at the same time. In other words, they gave your VPS a slot during which only you (or perhaps only a portion of the VPSes on the physical host) would be running a disk-intensive operation.

If Virtualmin were to have similar logic, it too could suggest to users -- both root and ordinary users -- when they are setting up a scheduled back-up that it be run at a particular time. In other words, there would be three options instead of two: simple schedule, complex schedule, and a suggested schedule, the latter being the default. So for Bob setting up his daily back-up the system might suggest he run it at 02:42, instead of running it with everyone else at 00:00.

But that idea is under the current system, and probably more complicated than Jamie's idea. However it addresses your last paragraph.

The underlying problem here is that I didn't consider the case where there are many separate Virtualmin backup schedules that might end up running at the same time - instead, I assumed that the admin would create a single backup that would go through all the domains one by one in series. I actually recommend this approach vs. individual user-owned backups, as it removes the risk that domain owners can accidentally turn off their own backups.

That said, it should be possible to schedule multiple backups at approximately midnight and have them work. This is typically done by adding a random factor to the starting minute, which some versions of cron support. However, as a short-term work-around, increasing the config option for the number of minutes to wait for the number of concurrent backups to drop to a very large number would work.

Craig, I wasn't actually aware of the option you're describing... the option to automatically create a backup job when making a Virtual Server. Where is that option located?

Or are you saying that issue occurs just by having multiple users go in and setup the backup jobs on their own?

After reading Jamie's response above, I just wanted to ensure we're all on the same page about how that issue is occurring.

Jamie: OK, I'm surprised at your assumption, since Virtualmin does make possible what you assumed would not happen, and some idiot like me is going to come along and want as much control in the hands of the users as possible and proceed as I did.

But I see your point, and I think what you've suggested -- root backing up all domains, but ordinary users being able to restore their own domains -- is a good compromise

Eric:

Craig, I wasn't actually aware of the option you're describing... the option to automatically create a backup job when making a Virtual Server. Where is that option located?

I'm assuming you're referring to my first paragraph in message #5. As I said at the time, I was going by memory and hadn't double-checked yet. I expected to find it under System Settings -> Server Templates, but I don't see it so I was wrong. This makes me even more in favour of Jamie's planned new feature, as without it I was planning to set up a back-up schedule manually each time I added a new virtual server.

But under Jamie's new feature I'm expecting that if I have a server hosting abc.com and xyz.com, and a back-up job that includes "All virtual servers", that when I later add example.com it will automatically be included in the back-up of "All virtual servers" from the next time the back-up runs. I'm repeating myself, but I also want to make sure we're all on the same page.

Or are you saying that issue occurs just by having multiple users go in and setup the backup jobs on their own?

Mostly I have been setting up scheduled back-up jobs under individual users' accounts, when logged in as them, but individual users have also set up their own scheduled back-up jobs. Either way there are multiple scheduled back-up jobs instead of the one big job that Jamie apparently expected server admins to set up.

There's no feature to automatically create a backup for each new domain, but this could in theory be automated with a post-creation script (although I don't recommend it).

Regarding your question about what "all domains" means - yes, this will include all domains created in the future as well.

Thanks for the tip, and the advice against it.

And thanks for clearing up the "all domains" / "all virtual servers" thing.

Sorry to reactivate this ticket, but I'm just wondering when you anticipate adding this feature. Thanks.

It's on my TODO list, but hard to say what the timeline is.