API Update / Domain Path / Top-level Server Filter

Joe/Jamie,

I figured I'd kill a few bird with one stone this time around.

API UPDATE
I think it'd be cool if you could tell API commands to return a limited amount of data. For instance, "list-domains" API when using the "multiline" option returns almost everything about a domain. It'd be nice to have the option to tell the command to list say a few options instead of everything. This could possibly reduce the overhead on these scripts, and in particular would reduce the overhead of parsing the output given data returned could be minimal.

Ex.

remote.cgi?program=list-domains&multiline=&json=1&features=quota_limit,quota_used,bandwidth_limit,bandwidth_usage

*** Where the above would return the "domain name" along with "bandwidth limit", "bandwidth usage", "quota limit", and "quota used" data instead of every detail. ***

DOMAIN PATH
I've been pondering a new directory structure, and wonder how realistic it would be to do something like the following:

    Top-level Server
    /home/username/domains/maindomain.com
    /cgi-bin
    /fcgi-bin
    /Maildir
    /public_html
    etc

    Sub-Server
    /home/username/domains/sub.maindomain.com
    /cgi-bin
    /fcgi-bin
    /MailDir
    /public_html
    etc

*** The purpose of this directory structure is to keep all domains under an account in a consistent structure rather than having "/public_html" and "/domain/mydomain.com/public_html" for Top and Sub servers respectively. Clean and simple is what I'm after. ***

TOP-LEVEL SERVER FILTER
My final idea relates to a way to restrict what domains a user can add, to ensure they are not adding domains which other people own. Currently I've disabled the ability to add domain "not" under the current domain, therefore only allowing "sub-domains" to be added. However, I'd like to take this restriction off, but still have some kind of control over domains being added. Basically, I figure if another user has say "mydomain.com" on their account, another user should by default be "restricted" from adding any domain under that domain like "sub.mydomain.com". However, if the domain "myotherdomain.com" is NOT added to an existing account, the member should be able to add it to their account and therefore take ownership over this domain.

Here's what I'm proposing as a way to make this work.

Alter the "Add Domain" form so that there are two or three separate fields. Field one and/or two would then get filtered at form submission so that no "." appears in it (field one could technically be allowed to support the "." character therefore allowing third-level domains to be added. Ex: third.second.top.com). Field three would be a drop down of TLDs that are allowed. This could and should be configurable by the administrator saving you guys from having to maintain the master list of TLDs which will change over time. (this also allows administrators to choose which TLDs to support)

. .com .net .org

*** If you need any further information on any of my questions, or ideas please let me know. ***

-Peter

Status: 
Closed (fixed)

Comments

tpnsolutions's picture
Submitted by tpnsolutions on Tue, 02/16/2010 - 16:01

BTW: Not sure if you noticed, but I updated the "I've Been Blocked" thread with a request.

  1. API limits - there is an option already to select only certain domains to output information for, with the domain or user parameters to list-domains. Does that help?

  2. There's no way to do this currently .. but you could fake it by creating for each customer a 'fake' toplevel domain which is just their username, and then creating all the real domains as sub-servers.

  3. Good idea, I will add this in the next release.

tpnsolutions's picture
Submitted by tpnsolutions on Wed, 02/17/2010 - 00:21

Jamie,

I think you misunderstood what I meant regarding limiting data from an API response.

Based on what you wrote, I take it you saw my request as wanting to limit the amount of "domains" being shown, but this is NOT what I was proposing.

I'm aware of how to do this.

What I'm proposing is limiting the amount of data "per domain" shown rather than the amount of domains shown.

So for example.

Say I have 1000 domains, and want to produce a simple and clean report showing the "quota used and quota limit" for each of the domain, but do NOT care about how much bandwidth, which features are enabled, etc, etc....

Right now the API would output to my script EVERY DETAIL about each domain that is available using the following command.

remote.cgi?program=list-domains&multiline=&json=1

domain1.com
quota limit
quota used
bandwidth limit
bandwidth usage
editable features
html directory
cgi directory
features enabled
etc
etc...
domain2.com
quota limit
quota used
bandwidth limit
bandwidth usage
editable features
html directory
cgi directory
features enabled
etc
etc...
domain3.com
quota limit
quota used
bandwidth limit
bandwidth usage
editable features
html directory
cgi directory
features enabled
etc
etc...

etc... (the other 997 domains)

*** i've displayed this info in human readable context for example purposes and have listed only a few of what the script would actually output, because frankly the list goes on for about 20 - 30 details per domain. ***

What I'd like to have instead of the above, is a way to tell the script that I could CARE LESS about 20 - 30 features about a domain for a given call to the API, and really only CARE about a few details. So given the following command (parameter "show-details" is fictitious at this time)

remote.cgi?program=list-domains&multiline=&json=1&show-details=quota_limit,quota_used

domain1.com
quota limit
quota used
domain2.com
quota limit
quota used
domain3.com
quota limit
quota used

etc.... (the other 997 domains)

Simple, sweet, to the point!

While a call to the API at a later time using....

remote.cgi?program=list-domains&multiline=&json=1&show-details=html_directory,cgi_directory

I'd get....

domain1.com
html directory
cgi directory
domain2.com
html directory
cgi directory
domain3.com
html directory
cgi directory

etc... (the other 997 domains)

In theory, this could speed up the API response given it's only showing a fraction of the data at any time, while I could still make a call to the API requesting ALL the details. At my choice however.

In reality, this would speed up the parsing script, since each record set would be much smaller, therefore making parsing that much faster. (sure with a few domains, you probably wouldn't notice much of a difference, but once you start getting into the hundreds, and thousands of domains, you do start to see speed increases.)

While I've illustrated how this could be used with "list-domains", need we stop there....

There are other API requests which could also benefit from this kind of output limitation.

Finally, I'd like to stress once again... I am NOT referring to a way to "filter" the "records" (aka rows) but rather the "details" within a "record" (aka columns) which are sent back from the script.

If this were an SQL statement, it'd be like doing something along the lines of...

SELECT column1,column2 FROM table
(how I'd like the script to run)

instead of:

SELECT * FROM table
(how the script currently runs)

*** the above SQL statements are simply to illustrate how I'd like to see the script function, and are not literal statements, since I realize that data is NOT stored in a database. ***

I hope this clarifies my idea a bit...

-Peter

Ok, I see what you mean. So are you most concerned about the size of the current output, or the time it takes to be generated?

tpnsolutions's picture
Submitted by tpnsolutions on Wed, 02/17/2010 - 00:41

Jamie,

I'd say both...

Outputting useless data when a given call to the API command doesn't require it means more overhead at least on my scripts end, which could be better utilized on other tasks (aka, another async API request)

*** having said that, I'm not proposing writing a separate API command for each type of output, but rather an update to the existing one, expanding the options to be a bit more flexible. ***

The time it takes to generate a response on a low memory system (i run virtualmin on systems with around 1-2GB generally, and attempt to squeeze every ounce out of the system prior to adding more memory) is also somewhat a concern, which I'm not sure can necessarily be helped, but it'd be nice if it could be.

*** I realize that there's a lot going on under the hood, ex: remote.cgi talks to list-domains.pl which opens a config file, which parses potentially another file, etc, etc. ***

I guess anything that could be done to make API requests more faster to process from the backend, to the frontend would be ideal, now and into the future.

-Peter

One thing that could be done to speed it up is add a flag to skip output of all the info that has to be looked up from the apache config, quota system and so on - this is what makes the multiline mode significantly slower.

I did some tests, and this made the display 4 times faster.

tpnsolutions's picture
Submitted by tpnsolutions on Wed, 02/17/2010 - 03:15

Jamie,

For data that slows down the execution, perhaps some level of "caching" could be done.

Ex.

--use-cache=1

A timestamp could be included in the cached data, so that applications could decide weather to use the cached data, or pass a call to the API which tells it to retrieve a "fresh" copy.

Ex.

--update-cache=1

A cron job could be easily scheduled in order to update the cached data at a set frequency as desired by the admin or directly via the API call.

*** naturally caching would only be applicable for "get" requests ***

The default behavior would then remain the same, as in get data in "real-time".

Personally, for statistical data, I could live with a cached copy of the data which between 5 - 30 min old, however using cron a person could decide how frequent they want their cached data updated.

-Peter

*** FYI: this concept could also be applied to any GET request, therefore saving on system resources and speeding up the API significantly. ***

I suppose that could be done, but it feels like a bit of a hack. Wouldn't it be better for the caller to do its own caching? You could get the full information perhaps daily, and the faster limited information on demand..

tpnsolutions's picture
Submitted by tpnsolutions on Wed, 02/17/2010 - 15:47

Jamie,

Sure, that'd also work. I was simply putting one possible idea on the table.

-Peter

tpnsolutions's picture
Submitted by tpnsolutions on Thu, 02/18/2010 - 04:03

Jamie,

Based on your suggestion, I added an application caching layer which saves different API responses for a certain about of seconds.

Ex.

http://dev.tpnh.com/cp

  • stats are cached for 45 min.
  • domain aliases / subdomains are cached for 5 min.

*** after an "add", "edit" or "delete" command, some cached data would get cleared ***

Additionally, I updated the API call used for retrieving the "domain alias" and "subdomain" lists to use "name-only" which also speeds up the script a bit.

ex.

http://dev.tpnh.com/cp/?cmd=domainaliases
http://dev.tpnh.com/cp/?cmd=subdomains

"I love Virtualmin because, the people who develop it are easy to work with, and are happy to collaborate on fixing bugs, and adding/updating features to meet my needs."

*** A sample of the kind of comment I would post on "ilovevirtualmin.com" based on my own experiences ***

-Peter

Cool, that's a good plan.

Are there any other issues still outstanding from this bug?

tpnsolutions's picture
Submitted by tpnsolutions on Thu, 02/18/2010 - 14:56

Jamie,

I think we've hit them all on the head.

As always, you were prompt, offered great suggestions, were willing to see my point of view, and are looking into a few of the suggestions which may further enhance/improve Virtualmin.

This is definitely better than a simple canned response, and no action taken by other vendors.

Always a pleasure to contribute, and be heard :-)

*** BTW: I was wondering about potentially becoming a moderator on the forums, if for no other reason to help prune those nasty SPAM threads. That way you and Joe could focus your attention on more important matters. ***

-Peter

Great .. I will close this bug.

We are working on automated spam filtering, so hopefully those spam threads will disappear in future. I will ask Joe about setting up moderator access though, assuming Drupal supports it.

Automatically closed -- issue fixed for 2 weeks with no activity.