Virtualmin.com migration updates

31 posts / 0 new
Last post
#1 Sun, 06/07/2009 - 17:38
Joe
Joe's picture

Virtualmin.com migration updates

Howdy all,

So, after a very, very, very long night, I've wrapped up our migration to Drupal. It turned out to be a little bit more than I could chew in a single sitting.

There are a few quirks in the new site, and I hope you'll bear with us while I get them worked out.

A few known issues:

Forums and issues have weird "last updated" times, leading to the order of the forums and tickets being absolutely chaotic. The actual dates on the posts are correct (that took some work!), but Drupal keeps a separate last updated time table, and I haven't figured out how to correct it yet. This will be fixed...in the meantime, search works better than on the old site, and new posts should naturally fall where they're supposed to. So, if you can't find your old topics, don't fret, just start a new one. Once I've had a nap, I'll dig back in.

The shop is possibly not quite working. We've taken this opportunity to switch to a new card processor that is more cost-effective, and also a lot nicer to work with than the old one. So, it works in test mode, but I don't know if it works with real credit cards or PayPal orders. Don't panic if something goes funny while you're making a purchase. We'll make sure everything works out. We're also still waiting for the software repositories to finish syncing to the new system...so purchased products (assuming the shopping cart works) won't show up immediately in the repo password database...you'll have to wait a little while to install. These will be fixed by tonight (they're at the top of my list, after a nap).

Documentation is still in a state of flux. We're moving to a whole new organization, based on tasks rather than random blobs of docs spread across a bunch of files. It will be dramatically easier to find relevant documentation once we're finished, but it'll be another few days before everything has been merged and moved. In the meantime, there is a link to the old doc index, so you can still find everything that existed on the old site.

Lots of refinement is still needed, and I haven't enabled a handful of additional modules we plan to turn on, but I felt like it was finally a better experience in almost every way than the old site, so it was worth a few bits of pain here and there while we work them out. Forums are finally sane (edit your posts!), and the ticket tracker doesn't do crazy things with links (unless they were imported from the old one, in which case all bets are off). And, we can finally start adding new features to the site.

Anyway, I'm off to flip the switch, so you all can actually see this! Holler if anything is seriously broken and needs immediate attention.

Regards, Joe

Sun, 06/07/2009 - 19:03
Joe
Joe's picture

OK, so I forgot to mention about notifications. If you had notifications setup on the old site, they're gone now (the notifications model just isn't really transferable between Joomla and Drupal...dramatically different way of thinking of the problem). I haven't yet turned on all of the notifications modules we'll be enabling...so the options are a bit sparse, right now. You can follow threads by checking the subscriptions checkbox box, and you can make new notifications of nodes in the My Account page if you know the node. But, it's pretty limited until I've installed and configured some additional modules.

Just wanted to reassure you that better notifications are coming very soon.

--

Check out the forum guidelines!

Mon, 06/08/2009 - 06:08
Joe
Joe's picture

Howdy again,

So, I mostly got notifications working nicely now. You can setup notifications for content types (so, if you wanted to follow every forum, you could choose to get notification of all new forum posts), tags (so, if you wanted to only get forum posts from the News forum you'd go that route), or by a half dozen other less useful selection mechanisms (this will probably be simplified a bit soon, as it's a bit intimidating at first glance).

I don't actually know if notifications are actually working...we're still in limbo on the mail server migration. It'll happen tomorrow some time, but I think we have everything spinning for sending out mail, even though we don't receive it on the new server, so it should work. You just can't respond to it yet and expect it to go any place good (yes, I'm working on making emailed replies to forum posts work...it conflicts with the project issue tracker module, unfortunately, so those bugs need to be worked out, so it may or may not happen soon).

I've also published a new version of the download page. It was long due for an overhaul. I think it'll make folks experiencing Virtualmin installation for the first time a little more calm about the process...and also a bit more clear about what they need to do to make a system Just Work with Virtualmin.

Lots of other little tweaks and fixes, including turning off the dastardly required preview "feature". That was awful. I very nearly forget to actually save the comment every single time when the preview is enforced. But, it's fixed now. Yay!

OK, back the trenches.

--

Check out the forum guidelines!

Mon, 06/08/2009 - 12:23
nmournian@yahoo.com

logged in and tried to submit support request - the form wants a 'component' but there are none (other than 'none') in the list which causes an error that sends the user back to the form to select from the non-existent list...can't imagine how long your weekend was doing this change-over.

Mon, 06/08/2009 - 15:26 (Reply to #4)
Joe
Joe's picture

Hey Nick,

The "Component" thing bugs me too. We sponsored development of a "default" component, so it automatically selects one, but it still requires me to actually make sure there are components to choose for each project. So what was project were you filing a ticket for? I'll make sure it gets at least one component.

--

Check out the forum guidelines!

Mon, 06/08/2009 - 18:01 (Reply to #5)
Joe
Joe's picture

OK, all projects in the issue tracker should now have components, and all should have a default component. So, this problem should be gone. Holler if you find a project that doesn't have any components. (All new projects will get default components...I somehow assumed that the default components would apply to existing projects, but they don't. They're like Server Templates in Virtualmin!)

--

Check out the forum guidelines!

Mon, 06/08/2009 - 17:13
Joe
Joe's picture

Premature optimization is the root of all evil. So I haven't prematurely optimized the site...and now that we're seeing normal Monday load, we're finding that Drupal is somewhat slower than Joomla in some cases (forum indexes are dramatically faster, though...they sometimes just didn't work on the old site, timing out after 120 seconds).

So, I've been tweaking the MySQL database configuration a little bit, and learning how to look at a MySQL database to know what it's doing. We'll probably be seeing some downtime today for a minute or two at a time, while I restart things for the experiments.

Since problems always come in groups, one of our Xen instances (the Virtualmin demo) has begun to become needy and very demanding. It's eating tons of resources for some reason (I suspect foul play...chickens, or some geese, maybe). Jamie is on the case, and will have it sorted soon, I'm sure. Looks like we'll get to experiment with resource limits in Xen soon (Cloudmin has some capabilities in this direction, but they're pretty new, so we'll have a good reason to play with them).

Back to the salt mines!

--

Check out the forum guidelines!

Fri, 06/12/2009 - 10:32 (Reply to #7)
fuerst

Do you use aggressive caching for anonymous users in Drupal (/admin/settings/performance)? You may also try to use the cacherouter module together with memcached to get the cache out of the database. I see below you are using APC yet, that's good.

Mon, 06/08/2009 - 17:23
Joe
Joe's picture

OK, so I mixed up the IPs when migrating software.virtualmin.com. That probably contributed a little bit to our performance problems in Drupal, as Apache was serving virtualmin.com out to pretty much everybody for every request. This would have given 404s for install.sh among other things. Sorry!

That's been fixed now (I think). And so the load on the CMS is a bit lower, and the download links should work again.

--

Check out the forum guidelines!

Mon, 06/08/2009 - 17:23
Rogi

Joe,

No matter how many teething problems and temporary hitches there may be with this new version of the site, rest assured of one thing;

It's just so much better than the old one already!

:-)

R.

Mon, 06/08/2009 - 18:00 (Reply to #10)
Joe
Joe's picture

Thanks for the reassurance, Rogi. It has been a bit more painful than I'd planned, and with a few more bumps in the road than I thought we'd hit. But, well...it is a major architectural change (Joomla and Drupal could not be more different; given that both are "CMS written in PHP", the differences in underlying design are striking).

I'm fixing problems as fast as I find them, and I think in another day or two all the pains will be forgotten and all that will remain is a better site.

--

Check out the forum guidelines!

Tue, 06/09/2009 - 02:32
Joe
Joe's picture

OK, so the permission denied error has, after much gnashing of teeth, but knocked out.

You can now access your licenses again.

So I'm kinda dumb for taking so long to figure it out (note to self: Drupal modules cache hooks...so you have to disable and re-enable a module after making changes to hooks; not true of other code...just hooks).

--

Check out the forum guidelines!

Tue, 06/09/2009 - 02:42
said

I love it :)

Very good work Joe

Tue, 06/09/2009 - 15:08 (Reply to #13)
Joe
Joe's picture

Awesome. Glad you like it. It'll only get better from here. ;-)

--

Check out the forum guidelines!

Tue, 06/09/2009 - 15:14
Joe
Joe's picture

So, I've added more docs (forgotten bits from the old site), and fixed a couple of broken links in the old documentation.

All CGIs should now be fixed (things like the script ratings, license checkers, etc.). The scripts.virtualmin.com website has been migrated (this is the fall-back download server, in the event a script, like Joomla, Drupal, Wordpress, etc. cannot be downloaded from its canonical source; it keeps copies of the latest version we know about on our own server).

The shopping cart seems to have an issue where it doesn't create licenses. I don't know why this is (it was working before I switched out of test mode), but I'm creating them manually in the meantime. So purchases can take a few hours to processes until I get that figured out.

--

Check out the forum guidelines!

Tue, 06/09/2009 - 15:15
Joe
Joe's picture

Oh, and the performance problems have all been worked out. Jamie figured out why our Xen instances were being so hateful (disk I/O contention, strangely enough), and corrected it. So they are no longer trying to chew up the whole server.

I tuned MySQL a little bit, and enabled the APC PHP cache, and the site seems to be nice and snappy today.

--

Check out the forum guidelines!

Wed, 06/10/2009 - 16:44
Joe
Joe's picture

Howdy all,

Some updates:

I figured out why orders in the shopping cart weren't issuing licenses, and I believe I've fixed it (I can't test further without doing a full order and sending money, and the terms of service at our payment processor only let us do that in test mode, which was already working). Life is hard sometimes. ;-)

We fixed a number of bugs in the reseller system (most folks don't see that, but it was pretty broken, and a lot of folks get their licenses through that system, so we had to drop everything and get it straightened out).

Cloudmin download links were broken, and are now mostly fixed (at least for Linux, working on Solaris shortly). There also might be a problem with installs of Cloudmin from the new server, but I haven't been able to track that down yet--the auth details seem to be getting put into the right places, but I still got an error report. So, that's on my todo list.

I'm in the midst of fixing the missing API docs and bug submission from the Virtualmin support module; strangely enough, the same problem must be solved for both of them (adding a Drupal node from a Perl program), so once one is solved, the other will also naturally be mostly solved as well.

Back to the coal mines.

--

Check out the forum guidelines!

Thu, 06/11/2009 - 01:07
sgrayban

Imagine that.. I have no licenses for any product under my account and yet I do.

Thu, 06/11/2009 - 22:31 (Reply to #18)
Joe
Joe's picture

Imagine that.. I have no licenses for any product under my account and yet I do.

Are you still seeing no licenses?

--

Check out the forum guidelines!

Thu, 06/11/2009 - 23:00 (Reply to #19)
sgrayban
Fri, 06/12/2009 - 00:00 (Reply to #20)
Joe
Joe's picture

Oh, that link shouldn't even exist. Vestigial stuff from earlier experiments.

Licenses are where they always have been: http://www.virtualmin.com/serial

--

Check out the forum guidelines!

Fri, 06/12/2009 - 00:05 (Reply to #21)
Joe
Joe's picture

I'll also add a link to the My account page for the serial numbers page. I'd written a block for that early in development, but it was too crowded with just a few licenses (it actually listed the licenses). I'll revisit that one now, though.

--

Check out the forum guidelines!

Fri, 06/12/2009 - 00:18 (Reply to #22)
sgrayban

Ok I was just going to suggest to add that link to the account page

Thu, 06/11/2009 - 01:18
sgrayban

Hmm what happened to the resellers VM login ? Seems that is broken as well.

Thu, 06/11/2009 - 22:32 (Reply to #24)
Joe
Joe's picture

Hmm what happened to the resellers VM login ? Seems that is broken as well.

It should currently be fully functional (to the best of my knowledge...there were many bugs remaining on Tuesday but we straightened them all out by last night when you posted). What problem are you seeing specifically?

--

Check out the forum guidelines!

Thu, 06/11/2009 - 22:58 (Reply to #25)
sgrayban

going to https://www.virtualmin.com:10000/

It says Login failed. Please try again. I had access to it before.

Fri, 06/12/2009 - 00:08 (Reply to #26)
Joe
Joe's picture

Not sure what to make of that. Your account looks fine from here. No lock due to failed logins or anything like that.

I can reset the password if you'd like. Email me if you want a reset, and I'll send back a new password. Webmin will soon be able to sync with the website password. We're working on some integrations of Cloudmin and Drupal that requires it, so it's on Jamie's todo list. Probably a week or two out, though, as we obviously have a number of more pressing issues to address.

--

Check out the forum guidelines!

Fri, 06/12/2009 - 00:20 (Reply to #27)
sgrayban

Ya reset the password for me - send to sgrayban @ gmail.com

Fri, 06/12/2009 - 10:38 (Reply to #28)
cjdavis

I would be very, very interested in just what integrations you are making, and whether they will be made available to customers or publicly..

I host many Drupal sites on a Virtualmin managed machine, and more integration between the two would be welcome news. If it were to tie into such Drupal projects as Aegir.. be still my beating heart!

Fri, 06/12/2009 - 12:07 (Reply to #29)
Joe
Joe's picture

I would be very, very interested in just what integrations you are making, and whether they will be made available to customers or publicly..

We plan to Open Source most of the Drupal components, with the possible exception of the ordering system (which is clearly and entirely for making money with; our current policy on GPL vs. Professional is "does it primarily help someone make money?" if yes, it goes into Professional, if not, it goes into both). I'm not actually sure yet, as I don't want to become a full-time Drupal developer, so it might go Open Source just so I don't feel compelled to add new stuff to it all the time.

It looks like this one will be an Ubercart plugin, but it doesn't quite fit with the "buy one time" model that Ubercart is tuned for, so it may end up being standalone. I'm not sure. In short, we're building a tool to allow folks to buy Virtualmin and Cloudmin services via a Drupal shop. But, this isn't as simple as saying, "OK here's my moneys, give me my stuffs". Cloudmin supports accounting at much finer granularity than the traditional monthly. You can bill by the hour with Cloudmin...so we'd like to have some sort of ability to do that. But automatic payments are really hard. Monthly subscriptions are easier, but don't allow a real "cloud computing" experience. I'm still researching this, but expect to begin work on the actual code in a couple of weeks...I will probably also contract some of this out to the Ubercart developers or the guy who helped us out with the ticket tracker development to make it go a bit faster. I'm not sure the big problem (fine-grained recurrent billing) has ever been solved in Drupal, though. If you have any ideas on how to solve that one (even a payments provider that offers that kind of thing; simple subscriptions don't work for this, which is all I've been able to find).

There will also be a ticket submission plugin, which will come out very soon (like in a week or two), which will allow you to use the same Virtualmin Support module that we use to allow clients to file tickets and grant temporary root access to the system to your administrators from within Virtualmin or Cloudmin itself.

We don't currently have plans to have Virtualmin dig into Drupal any deeper than it currently does. The plan is to allow folks running Drupal sites to sell Virtualmin-related products and services (and possibly perform simple management via the UI), and support them better and more easily. We'll also probably distribute a Drupal module that provides logged in users access to full Virtualmin documentation under your brand. This one is further out, as we have to finish the docs first.

I'd never seen Aegir before, it looks pretty cool. We'll see whether there is any useful overlap there, and get in touch with the folks working on it, if so.

--

Check out the forum guidelines!

Thu, 06/11/2009 - 22:44
Joe
Joe's picture

Howdy all,

So, I fixed the shop, again. Maybe it actually works this time. (Yesterday it was issuing two licenses for every purchased license, because the payment processing module was calling the completed order function twice. Believe it or not, this one wasn't a bug in my code! Hooray!)

The support module will be updated in a few hours to support the new issue tracker. I've ported the connector script to PHP, since we can't safely hit the database anymore, so converting the Perl code was going to be difficult. So, it'll not be part of the site so it can directly use the real Drupal API.

I'm also working on the API docs convertor.

Jamie let me know at lunch today that the new help search is complete, and the new Webmin release will be out in a couple of days. This is really cool news, and Jamie has put in a huge amount of work into the help system. If you've ever used Search in Webmin, you know that the results it returns for large modules can be...frustrating. It tells you what module the help file appears in, but not what page within the module. The new help search tells you exactly where to find the option or page that the help file applies to. It's awesome. I was surprised by how much code it required to support this seemingly simple thing, until I actually thought about it for a while and realized there is no way to go "backward" from the help files in current Webmin. Making that backward trip possible is pretty complicated. So, it's really nice that it's now done. It will also enable a new Webmin reference guide compiled from all help files. Once I have an API docs convertor/importer, it should be trivial to make the same tools work for the Webmin docs. More docs is always good, right?

Cloudmin is flying off the shelves faster than we'd planned at this early date (the Cloudmin site hasn't even launched yet!), and we're a bit overwhelmed helping folks out with it. Mostly new customers, so we're hearing from them via email so far (I suspect we're also seeing slow uptake on the new tracker and such because it's new and complex, so we'll keep tweaking the site a bit so folks can find their way to the right places).

I haven't heard any opinions on the new doc layout or the download page, but clicks and downloads are up, while forum posts and ticket tracker submissions are down, so I think we done good. Thanks to Eric for a lot of that work.

If it sounds like I'm winding down this thread...well, that's because I am. Most of the really urgent issues with the site have been resolved. I'm still working on a lot of stuff (obviously), but when people need help now, they can get it. Likewise, when people want to buy software, they can buy it, and it'll probably show up in their licenses page right away (except for Scott, it seems, but we'll get that taken care of).

So, I'll start posting new topics henceforth.

--

Check out the forum guidelines!

Topic locked