Submitted by stevepisg on Fri, 09/29/2017 - 04:39 Pro Licensee
Dropbox backups are failing with output message "v1_retired". It looks like Dropbox has retired V1 of their API. See:
https://www.dropbox.com/developers/reference/migration-guide
As a result, VirtalminPro cloud backups to Dropbox are failing. Thanks, stevep
Status:
Closed (fixed)
Comments
I'm getting this too ... mind you there appears to a lot of work to do
Submitted by andreychek on Fri, 09/29/2017 - 10:10 Comment #2
Thanks for the heads up guys!
Yeah this does indeed appear to be a bug (or perhaps a missing feature). I just ran into this last night on my own personal server.
Submitted by JamieCameron on Fri, 09/29/2017 - 19:18 Comment #3
Damn, I guess dropbox changed their API without telling us :-( We'll work on a fix ..
Dropbox did notify that v1 api was retiring some while a go looking at the changes you may have some work to do let's hope you can do it quite quickly
Submitted by JEMEDIACORP on Fri, 09/29/2017 - 23:20 Pro Licensee Comment #5
We started getting this too as of midnight Friday )09/29) in regards to our daily backups. Dropbox is our primary backup source as space is cheap so I hope Virtualmin is quickly updated to use the new Dropbox API.
Submitted by JamieCameron on Sun, 10/01/2017 - 18:42 Comment #6
Good news - support for the new Dropbox API has been implemented, and will be included in the upcoming 6.1 Virtualmin release.
Submitted by JamieCameron on Sun, 10/01/2017 - 19:40 Comment #7
Submitted by stevepisg on Mon, 10/02/2017 - 00:18 Pro Licensee Comment #8
Thanks for getting this done quickly. Is there any ETA on when we can get our hands on it?
Submitted by JamieCameron on Mon, 10/02/2017 - 16:26 Comment #9
Couple of days at most
When using the new code I get the following error :- .. upload failed! HTTP/1.1 409 path/malformed_path/
when the backup module tries to 'upload' the data to dropbox is there a change required in the backup job settings ?
Submitted by JamieCameron on Wed, 10/04/2017 - 12:22 Comment #11
@jimr - did you upgrade to 6.01 already? If so, what destination path are you backing up to?
Webmin handled the upgrade to 6.01 so yes I am running 6.01 with the original version the dropbox path required was something like webmin//%y.%m.%d with v2 the path required is now webmin/%y.%m.%d . I made the change to the schedule to reflect the new path however domains greater than 1GB backup fail with the following :- upload failed! HTTP/1.1 409 incorrect_offset/... (performed 3 attempts) all the smaller domains backup correctly however I can see the archives in the correct dropbox folder for the larger domains so perhaps there is an issue with the V2 api that is returning and error ?
I notice that the 1gb archive is about 2kb on drop box so the previous point is valid
Submitted by stevepisg on Thu, 10/05/2017 - 16:14 Pro Licensee Comment #14
I think there are still a few issues around the new Dropbox API calls. My backups get copied up to Dropbox after updating to 6.01, however, the cleanup of old backups does not function:
"servers backed up successfully, 0 had errors. Backup is complete. Final size was 322.29 MB.
Deleting backups from web2_%d-%m-%Y in Dropbox folder vs_abvmp older than 10 days .. .. failed to list Dropbox files : Error in call to API function "files/list_folder": request body: path: 'vs_abvmp' did not match pattern '(/(.|[\r\n]))?|id:.|(ns:[0-9]+(/.*)?)'"
I looked on the dropbox website and found :- NOTES /files_put has a maximum file size limit of 150 MB and does not support uploads with chunked encoding. To upload larger files, use /chunked_upload instead.
as every domain backup that is smaller than 150mb is uploaded correctly & those bigger than 150mb fail this may shed some light on my problem
trying again I got :- Uploading archive to Dropbox .. .. upload failed! JSON decoding failed : malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "(end of string)") at /usr/share/webmin/virtual-server/pro/dropbox-lib.pl line 246. (performed 3 attempts)
Backup failed! See the progress output above for the reason why.
Deleting backups from %d.%m.%y in Dropbox folder noideersoftware older than 3 days .. .. failed to list Dropbox files : Error in call to API function "files/list_folder": request body: path: 'noideersoftware' did not match pattern '(/(.|[\r\n]))?|id:.|(ns:[0-9]+(/.*)?)'
Submitted by JamieCameron on Fri, 10/06/2017 - 23:40 Comment #17
That's odd, because Virtualmin does use the chunked upload feature for files above 100 MB. Let me see if I can re-produce this, and we will do a patch release with a fix for it and the purge issue.
I could give you access to one of the virtualmin servers if you like so you can see for yourself if that helps
Submitted by steve@itgroup.net.au on Sat, 10/07/2017 - 15:42 Pro Licensee Comment #19
this looks like what is happening to me too Jamie (see my other help ticket) - some upload, some do not.??
As of version 6.01-2 dropbox backups now appear to work. I forced a back up and all files were uploaded to dropbox however it was a good hour or so to transfer 3.5 GB of files I guess when the unattended back up will work .. I'll have to wait till 3:00 am to be sure .. I'll report back
just a thought .. does restore work ?
Submitted by JamieCameron on Wed, 10/11/2017 - 23:45 Comment #22
It does in our automated tests. Restores are less complex to support in the Dropbox API, as there's no special code needed for large downloads.
OK I can report that both servers backed up fine over night using the scheduled backup task to dropbox. I have tried to do a restore .. restoring from dropbox failed with the larger domains .. so I tried from a local file (upload to server) this also failed with a chrome error of 'connection reset' I'll try a restore from google see if that works
Hi Just tried to restore from dropbox .. this failed with the message 'Restore failed : The specified source is not a Virtualmin backup : Download incomplete' this applies if you either use the restore function or try to restore from a back up log .. I guess the need work title is still valid ;)
Submitted by JamieCameron on Wed, 10/25/2017 - 22:35 Comment #25
@jimr - how large was this backup you were attempting to restore?
Submitted by JEMEDIACORP on Wed, 10/25/2017 - 22:37 Pro Licensee Comment #26
I have not filed an item in the bug tracker regarding this issue, but have continued to test and no matter what Dropbox backup I attempt to restore, I always see the "Download incomplete" message. The backups range anywhere from 10 to 16 GB in size.
After finding out the restore doesn't work I have tried to restore backups of all sizes from 2mb up to 4.3gb
Submitted by JamieCameron on Fri, 10/27/2017 - 00:52 Comment #28
When restoring, are you entering the full path to a tar.gz file (like folder/domain.com.tar.gz), or just a directory on Dropbox?
I have tried both entering the location and the exact file neither work also trying to restore from the backup logs also fails. To date I have not manually uploaded the tar to the server from Dropbox and used the local file option to restore .. But I guess this option will work
Submitted by JamieCameron on Fri, 10/27/2017 - 17:43 Comment #30
Does the restore fail immediately, or does it take some time after the process starts for that "Download failed" message to appear?
I would guess about 3~5 seconds, using authentic the little red line across the top gets to about 50% of the screen width after you hit the "show what will be restored" button
Submitted by JamieCameron on Fri, 10/27/2017 - 17:57 Comment #32
Ok, so definitely not long enough to download a significant amount of a 10 GB backup.
If you SSH in as
root
and use the commandvirtualmin download-dropbox-file
to fetch the same tar.gz from Dropbox to your system, does that work OK?Submitted by JEMEDIACORP on Fri, 10/27/2017 - 17:59 Pro Licensee Comment #33
The GUI restore tool runs for about 15 seconds for me then fails with the "Download incomplete" message; running the download-dropbox-file command fails after even less time, only about 5 seconds.
command line fails with :- Unsupported file or mode >xx.tar.gz at WebminCore::../web-lib-funcs.pl line 2413
Submitted by JamieCameron on Sat, 10/28/2017 - 00:44 Comment #35
Make sure you enter a full path for the destination, like
/tmp/domain.com.tar.gz
I ran :- virtualmin download-dropbox-file --file noideersoftware/27-10-17/ickleh.co.uk.tar.gz --dest /home/ickleh.co.uk.tar.gz and the response was ERROR: Download incomplete within about 5 seconds
Submitted by JamieCameron on Wed, 11/01/2017 - 18:27 Comment #37
How large is that noideersoftware/27-10-17/ickleh.co.uk.tar.gz file?
Submitted by DavidKuykendall on Thu, 11/02/2017 - 00:00 Pro Licensee Comment #38
I have tried every permutation of every parameter on Backup Virtual Servers and Scheduled Backups page and I can not get Backup to work with Dropbox. I also used forward and backward slashes.
Ickleh. Archive size is about 68mb... I did try with larger archives but hit the same issue
Submitted by JamieCameron on Thu, 11/02/2017 - 23:30 Comment #40
As a test, I tried a few Dropbox uploads and downloads with a 522 MB file, but it worked fine.
Any chance we could login to your system to see what's going wrong here?
I have sent remote log in privs for you
Submitted by JamieCameron on Fri, 11/03/2017 - 23:38 Comment #42
Thanks, but I wasn't able to SSH in as root to the IP you granted access to. Perhaps root logins are blocked?
You can also send me login details at jcameron@virtualmin.com
Submitted by JamieCameron on Sat, 11/04/2017 - 17:59 Comment #43
Ok, thanks to another user (Logan) I found the cause of this - there is a mismatch between the HTTP requests Webmin sends and what Dropbox expects. If you want an immediate fix, the patch is here : https://github.com/webmin/webmin/commit/40fa0bcf399d8c5f7ac4440c24686844...
Or you can wait for the next Webmin release.
Submitted by JEMEDIACORP on Sat, 11/04/2017 - 22:17 Pro Licensee Comment #44
Thanks for the mention, Jamie! How would one go about applying this patch immediately to the current installed version of Webmin? Or, since you already have my login details, would you mind applying it for me? Thanks!
I added the patch and it Works !!! so the last thing to do is to get the older archives removed e.g stuff older than X days then we will be back to a working system
Submitted by JamieCameron on Sun, 11/05/2017 - 14:49 Comment #46
That should happen automatically after the next backup runs.
I ran an automated backup @ 3:00 after I applied the patch ( I guess no bearing on file deletion from dropbox) and I got in the log 'Deleting backups from %d-%m-%y in Dropbox folder noideersoftware older than 3 days .. .. found 3 backups, but none were old enough' ... even though 1 backup was 4 days old ... is this a problem using %d-%m-%y rather than any other time format ?
I now have quite a few backups not deleted ... with the option backup->Schedule->Schedule and reporting->Command to run after backup I could code something that deletes whats needed until such time Virtualmin gets back on track within this module ?
Submitted by JamieCameron on Sat, 11/11/2017 - 00:20 Comment #49
The problem is likely that Virtualmin isn't parsing the backup date correctly from Dropbox.
If you SSH into your system as
root
and run :virtualmin list-dropbox-files --path noideersoftware --multiline
does it show the correct timestamps?
The result from the above command is :-
ERROR: Error in call to API function "files/list_folder"
: request body: path: 'noideersoftware' did not match pattern '(/(.|[\r\n])*)?|id:.*|(ns:[0-9]+(/.*)?)'
but using this command :- virtualmin list-dropbox-files --path /noideersoftware --multiline works as expected ... note the / with the slash the following is returned:-
09-11-17
Type: Directory
Full path: /noideersoftware/09-11-17
10-11-17
Type: Directory
Full path: /noideersoftware/10-11-17
11-11-17
Type: Directory
Full path: /noideersoftware/11-11-17
Submitted by JamieCameron on Sat, 11/11/2017 - 23:23 Comment #51
Ok, it looks like there is a bug in the dropbox purging code for date-based directories like this. It will be fixed in the next release though ..
how far off is the next release ?
Submitted by JamieCameron on Tue, 11/14/2017 - 01:49 Comment #53
A few weeks, but if you need a fix sooner we can send you a beta.
well I would like to have it up and running as soon as ... if you point me to the patch I'll edit the local file(s) ... if it is a bit more far reaching just give me access to the full beta
Submitted by JamieCameron on Tue, 11/14/2017 - 23:39 Comment #55
I can send you a beta - just let me know which Linux distribution you're running there.
Hi Jamie I am using ubuntu 16.04 on one server and ubuntu 14.04 on the other
using webmin 6.02 I get
Deleting backups from %d-%m-%y in Dropbox folder noideersoftware older than 3 days ..
Deleting file 03-12-17, which is 3 days old ..
.. deletion failed : HTTP/1.1 409 Conflict
.. found 4 backups, but none were old enough
so still not working
Submitted by JamieCameron on Sat, 12/09/2017 - 01:18 Comment #58
What files are in that 03-12-17 folder?
Hi Jamie it looks like fixed but I am seeing
Deleting backups from %d-%m-%y in Dropbox folder noideersoftware older than 3 days ..
Deleting file 13-12-17, which is 6 days old ..
.. deleted 0 bytes.
Deleting file 14-12-17, which is 5 days old ..
.. deleted 0 bytes.
Deleting file 15-12-17, which is 4 days old ..
.. deleted 0 bytes.
Deleting file 16-12-17, which is 3 days old ..
.. deleted 0 bytes.
.. deleted 4 old backups, and skipped 3 that were not old enough
which may be down to the script I wrote to fix the issue ... I'll report back when my other server updates to the new version
Submitted by IssueBot on Thu, 10/11/2018 - 20:07 Comment #60
Automatically closed - issue fixed for 2 weeks with no activity.