Recently I turned my interest at content delivery networks (CDN) and found a plethora of blog posts both praising and bashing Google App Engine as a CDN. This is the first post in a series of posts I will publish discussing Google App Engine as a CDN.

Google App Engine wasn’t created to be a CDN provider, but it does have CDN capabilities. Some posts argue that App Engine is not a real CDN, others describe how to utilize it as a CDN. Just Ping is a service that will ping/resolve domain names from 41 different locations. Pinging my App Engine results in 12 different IP’s over the world!

Misinformation published about App Engine

Google App Engine was first released as a beta version in 2008. There are a number of posts detailing problems and restrictions in App Engine. Most of them were published around the launch and I started looking into App Engine a couple of weeks ago. Here are some fact corrections:

Static file size limit?

Many blogs say there’s a 1Mb file size limit. While that might have been true once, it is no longer true! The limit as of right now (2010-04-05) is 10.000.000 bytes, which equals ~9.5Mb. Engine Launcher will throw you an error message if you try to upload a file larger than this and my tests show that the service API will not accept a file larger than that.

Cache control, Etags etc?

Cache control is important when using a CDN. Some blogs posts indicate that Etags are missing and that there is no way to set your own cache expiration. This is also not true as of right now. You can configure cache expiration for individual folders as well as globally. Etags are also supported. Verify yourself here: http://www.seoconsultants.com/tools/headers, test against http://bjorn-sallarp.appspot.com and you will also notice that the cache expiration is set to one year.

Static file storage limitations?

The total storage limit has confused me a bit. The quota details says there’s a 1Gb limit for total stored data. Engine Launcher throws an error message if you try to upload files that total over 150.000.000 bytes (~143Mb). However, using my own code to upload files I have now successfully put roughly 400Mb of static files on my account. Still, the quota summary says I’m using 0Mb of my 1Gb storage! I’m not keen on getting my account closed down permanently as it’s registered in my real name but someone will probably push this to the limit and if you do, please comment and let us know where you hit the limit.

Google, where’s the service API?

Google is usually very good at not only delivering a good service, they also provide a robust simple to use API to go with it. However, I haven’t been able to find any documentation on how to upload files to App Engine. All deployment and uploading “should” be done through “Google App Engine Launcher”. The problem with Engine Launcher is that it’s available in a Java or Python version, and although the application communicate with the App Engine servers through a REST API there’s no documentation. If I want to automate uploading files to my App Engine account, i.e. using it as a CDN, I don’t want to run the Engine Launcher to do so. Perhaps this is Googles way to prevent abuse of their shiny cloud service?

Google, here’s your API!

Ok, I haven’t taken the time to investigate the entire App Engine API, I’ve just been curious about deploying an application version, i.e. uploading files.

The base url to the service layer is not your own application url, it is http://appengine.google.com. Authentication against the API is done through your standard Google account. If you Google for ClientLogin you’ll find code sample for many languages. In my C# sample code I’ve included an authentication method. The ClientLogin procedure gives you a cookie that you pass along your API requests.

This is the standard message flow for uploading files:

1. Create application. With this request you pass along your application configuration. My app engine is set up to use Python so I pass the contents of my app.yaml
2. Clone blobs. With this request you pass along a list of files that your application consist of. The response will be files you need to upload, i.e. are either now or updated
3. Upload your files. Pass the file binary data.
4. Deploy.
5. Wait for deploy to finish. You need to poll to find out.
6. Start serving. This brings your “new” application to life

It’s mandatory to pass two arguments with every request: app_id and version. App_id is name of your application, in my case: bjorn-sallarp. Version should contain the version number you want to manipulate. If you want to use App Engine as your CDN you only need one version so this will always be 1.

Create application – /api/appversion/create

URL: http://appengine.google.com/api/appversion/create?app_id=bjorn-sallarp&version=1
Method: POST
Content-type: application/octet-stream

The payload for this request is your application configuration, in my case I pass my python app.yaml file:

This configuration sets up a default expiration time on files to 365 days. I have three directories to store files in: stylesheet, images and files.

Server response

HTTP Status code: 200. If all goes well an empty string is returned.

Clone Blobs – /api/appversion/cloneblobs

URL: http://appengine.google.com/api/appversion/cloneblobs?app_id=bjorn-sallarp&version=1
Method: POST
Content-type: application/octet-stream

It’s important to note that if you don’t pass all your files at clone blobs they will no longer be available for download from your application. By calling this method the files that you specify that already exist in the application will remain there. However, I noticed a funky thing while working on this project. If I don’t pass a file through CloneBlobs, it becomes unavailable fro download, then later add it again in CloneBlobs, even with a different name, I won’t have to upload the file. It will just be there again automagically. This is somewhat confusing but could also be super smart, if the same file exist already somewhere in App Engine (you know by the SHA-1 hash) why should Google store it again? If that is not the case then it’s probably a huge delay until the file is actually deleted.

The payload for this request is a list of files that your application consist of, or the files you want to serve if you like. The list is just plain text with file details on a separate line, file information is separated with a pipe sign (|). Below is a sample file list:

index2.html|bb1029ca_5642322a_769836da_db0bdd0c_411a9fcf|text/html
stylesheets/example.css|da39a3ee_5e6b4b0d_3255bfef_95601890_afd80709|text/css
files/Archive0.zip|6049564c_128fbe63_3ea88961_17ca269e_d2b7f1c4|application/zip

First is the relative file path, then a SHA-1 hash of your file, last the mime type. Line breaks should be done using “\n”. Here’s a C# code sample for how to calculate a proper hash:

public string CalculateHash(byte[] fileData)
{
    byte[] result;
 
    SHA1 sha = new SHA1CryptoServiceProvider();
 
    // This is one implementation of the abstract class SHA1.
    result = sha.ComputeHash(fileData);
 
    StringBuilder hashValue = new StringBuilder(40);
    int i = 0;
 
    foreach (byte b in result)
    {
        if ((i > 0) && (i % 4 == 0))
        {
            hashValue.Append('_');
        }
 
        hashValue.Append(HEX[(b >> 4 & 0xF)]);
        hashValue.Append(HEX[(b & 0xF)]);
        ++i;
    }
 
    return hashValue.ToString();
}

Server response

HTTP Status code: 200
The response is a list of files that are either new or changed. If nothing is returned you don’t need to upload anything. Here’s an example response:

stylesheets/example.css
files/Archive0.zip

Line feed character is “\n”.

Upload file – /api/appversion/addblob

URL: http://appengine.google.com/api/appversion/addblob?app_id=bjorn-sallarp&version=1&path=stylesheets/example.css
Method: POST
Content-type: file mime type

Post this request for each file you want/need to upload. You have to pass the relative file path as the parameter “path”. The content-type should be set to the mime type of the specific file you’re uploading.

The payload for this request is the binary file data.

Server response

HTTP Status code: 200. If all goes well an empty string is returned.

Deploy – /api/appversion/deploy

URL: http://appengine.google.com/api/appversion/deploy?app_id=bjorn-sallarp&version=1
Method: POST
Content-type: application/octet-stream

The payload for this message is an empty string

Server response

HTTP Status code: 200. If all goes well an empty string is returned

IsReady – /api/appversion/isready

URL: http://appengine.google.com/api/appversion/isready?app_id=bjorn-sallarp&version=1
Method: POST
Content-type: application/octet-stream

This is a method to find out if the deploy has completed. This should be implemented with a timer back off, meaning you sleep longer for each request that doesn’t indicate that the application is ready. Usually it takes between less than a minute for a deploy to complete.

The payload for this method is an empty string.

Server response

HTTP Status code: 200. If the server returns “1″ it means the deploy has finished.

Startserving – /api/appversion/startserving

URL: http://appengine.google.com/api/appversion/deploy?app_id=bjorn-sallarp&version=1
Method: POST
Content-type: application/octet-stream

When the deploy has completed this request tell the application to start serving. Payload is an empty string.

Server response

HTTP Status code: 200. If all goes well an empty string is returned

Rollback – /api/appversion/rollback

URL: http://appengine.google.com/api/appversion/deploy?app_id=bjorn-sallarp&version=1
Method: POST
Content-type: application/octet-stream

If something goes wrong after you called create version, which is usually that you get back a http status code other than 200, you need to roll back the deploy.

Payload is an empty string

Server response

Http Status code: 200. If all goes well and empty string is returned

C# API Proof of concept

No post is complete without a proper sample. Below is my API sample project containing everything you need to deploy/push your own files to Google App Engine. The sample project is VS2008 .NET 3.5.

Download Google App Engine API

Related posts

• Google App Engine as your CDN?