Blog.Project13.pl

I’ve been coding an simple RESTful “ip to country” conversion service. I’ve decided to put it up to appengine – so that everybody may use it freely even if I’d change my server etc… And if looked quite nice the first day – buw when I got to do some “real stuff” app engine started to get in my way, here’s a quick rundown of the problems I’ve found with appengine:

1. I need to download and update the ip<->country database every few days. I’d use an GZIPInputStream and BufferedReader to get the file I was interested in and update the database. Did appengine allow me to use such an simple aparoach?
   1. pro: appengine provides a very nice cron-like mechanism. So I just had to create an cron.xml in WEB-INF and this part was ready to go! This was in fact easier and more fun than in classic Spring + Open Symphony Quartz.
   2. con:I checked if this tactic was OK with the JRE whilelist and checked the Quotas – http://code.google.com/appengine/docs/quotas.html#UrlFetch, “seems ok” – I thought – “nothing about per connection limits, only daily quotas.”. After writing the code, I discovered that even though on the main quota page there was no word about per connection limits, in fact there are such quotas, but a little more hidden: http://code.google.com/appengine/docs/java/urlfetch/overview.html Max request/response sizes are capped at exactly 1MB.
      As my gzipped file is around 1.1MB, appengine killed my simple idea… I’d have to split the file into separate ones – on another server, and then fetch the separated files onto appengine.
   3. con:The mentione CRON mechanism is quite funny. You don’t call methods but URLs – and they are normally called as if one would launch them from the browser – thus, traffic and “max time” quotas do count there as well. So rather than calling an method, as you’d do with OpenSymphony – you create an Servlet that does all the work. This may me both good, and bad… You cound do all the CRON stuff by hand if it got out of sync etc… I didn’t really like it, and as mentioned… When doing my “big batch database update” the servlet would simply timeout…
2. Let’s talk about the dataStore. As you all probably know… AppEngine does not provide and “database” per se. It’s not relational and has quite some limitations. BUT! That’s quite ok, as it’s very quick and very very scalable! And most of the time… Do you really need all those fancy relations? ;-) It was an ideal place to hold my super simple data: “ip region = country” mapping in a persistent way. “Another nice thing on appengine for this application I’m going to write.” – I thought. Was I right?
   1. pro/con: Not really… I used JPA but JDO (which is prefered by appengine from what I’ve seen) also works nice on GAE. The setup did run quite ok while I was running tests on my local machine. Deployment is also an breeze and I didn’t have to use any complicated dependencies to get it running – “yay, no maven!”. You just have to enchance the classes you want to make persistent (just as hibernate does).
   2. con:The problems started when I wanted to clear my datastore. Nothing easier than that, right…? “delete from BlaBla” and we’re done. Yeah, but not on GAE. As even the “max rows a query touches” are limited – to 500. So there I am, with my 100.000 rows, and I have to delete them in 500 rows per query… Of course I can’t call this in an loop – as the timeout quota would get in my way and kill the app.From what I’ve seen, people do solve this using a CRON task that calls this “clearDatabase” servlet until it’s  done – ugh, not a nice solution but I can’t think of any other solution :\
   3. con:The only query I need to do in this app is basically:

      SELECT range FROM RemoteIpData as range
                   WHERE range.ipFrom <= ?1
                     AND range.ipTo >= ?2
      #and this would always return 1 entry!
      

      And guess what… AppEngine does not support multiple “less/more than” operands in one query! If you think hard about what BigTable is, it does make some sense. More information about “GQL” can be found here: http://code.google.com/appengine/docs/python/datastore/gqlreference.html All the restrictions about the Queries you can do are documented here – some are really counter intuitive when you come from an RDBMS enviroment… Ah well ;-) Oh, and yet another great link about GAE limitations.

      Having this limitation, really sucks for my normally “super simple query”, and I’d have to change the data structure somehow or do some awful 2 queries and then combine them in Java (omg teh terrorr… :<). So, while developing on appengine, keep in ming – simple things might turn out quite complicated due to the nature of BigTable. If you know all the limitations when designing the system and not while finishing it, you’ll be a happier man… ;-)

   4. neutral: Primary keys can’t be Integers etc, as AppEngine uses it’s own “Key” type. :-)
3. Having that all said. Appengine’s administration panel is quite ok. And the not-so-newly-added log search etc are really fine tools. Something an plain old tomcat can not offer. On the other hand, the limitations can really be a deal breaker! My app was really fairly simple, and yet appengines quotas managed to really get in my way. Keep this in mind while thinking about using it. You may also try Amazon’s cloud or CloudForce from SalesForce etc… They all do offer a quite less restrictive enviroment.
4. If you’re interested, deployment does take about 5-7 minutes before the new version is visible on the web – so don’t panic if you’re still seeing the old version after deploying the new one.
5. My opinion about GAE…? (semi serious ;-))

“Screw you clouds, I’m going /home!”
(intended southpark pun)

I’ll finish this project using my ol’ pal, Tomcat6 which I’ve already got running for netbeans.edu.pl (but that was an grails app).

I’ve also decided to use Spring, which I didn’t really need on appengine – as the only thing I was doing was so small that all the logic was around 10 lines in the servlets… But if using hibernate and all the other “real” JEE stuff, I felt I’d need to “do this right” so I’ve decided for Spring 3 and Maven2… I’ll try to build this project from gradle soon too!

ALSO! If interested in an more experienced programmers view on appengine (I’m still a novice), go and read this blog post about GAE on Sławek Sobótkas blog. All in all we seem to agree that the limitations can be an pain in the a… ;-)

The source for the appengine version is on my github account. I’m porting it to an plain old tomcat environment and will post this version there too when it’s ready to run (tomcat deployment is somehow hell with such apps for me… Any tips are really welcome :-))