I’ve been coding an simple RESTful “ip to country” conversion service. I’ve decided to put it up to appengine – so that everybody may use it freely even if I’d change my server etc… And if looked quite nice the first day – buw when I got to do some “real stuff” app engine started to get in my way, here’s a quick rundown of the problems I’ve found with appengine:

1. I need to download and update the ip<->country database every few days. I’d use an GZIPInputStream and BufferedReader to get the file I was interested in and update the database. Did appengine allow me to use such an simple aparoach?
   1. pro: appengine provides a very nice cron-like mechanism. So I just had to create an cron.xml in WEB-INF and this part was ready to go! This was in fact easier and more fun than in classic Spring + Open Symphony Quartz.
   2. con:I checked if this tactic was OK with the JRE whilelist and checked the Quotas – http://code.google.com/appengine/docs/quotas.html#UrlFetch, “seems ok” – I thought – “nothing about per connection limits, only daily quotas.”. After writing the code, I discovered that even though on the main quota page there was no word about per connection limits, in fact there are such quotas, but a little more hidden: http://code.google.com/appengine/docs/java/urlfetch/overview.html Max request/response sizes are capped at exactly 1MB.
      As my gzipped file is around 1.1MB, appengine killed my simple idea… I’d have to split the file into separate ones – on another server, and then fetch the separated files onto appengine.
   3. con:The mentione CRON mechanism is quite funny. You don’t call methods but URLs – and they are normally called as if one would launch them from the browser – thus, traffic and “max time” quotas do count there as well. So rather than calling an method, as you’d do with OpenSymphony – you create an Servlet that does all the work. This may me both good, and bad… You cound do all the CRON stuff by hand if it got out of sync etc… I didn’t really like it, and as mentioned… When doing my “big batch database update” the servlet would simply timeout…
2. Let’s talk about the dataStore. As you all probably know… AppEngine does not provide and “database” per se. It’s not relational and has quite some limitations. BUT! That’s quite ok, as it’s very quick and very very scalable! And most of the time… Do you really need all those fancy relations? ;-) It was an ideal place to hold my super simple data: “ip region = country” mapping in a persistent way. “Another nice thing on appengine for this application I’m going to write.” – I thought. Was I right?
   1. pro/con: Not really… I used JPA but JDO (which is prefered by appengine from what I’ve seen) also works nice on GAE. The setup did run quite ok while I was running tests on my local machine. Deployment is also an breeze and I didn’t have to use any complicated dependencies to get it running – “yay, no maven!”. You just have to enchance the classes you want to make persistent (just as hibernate does).
   2. con:The problems started when I wanted to clear my datastore. Nothing easier than that, right…? “delete from BlaBla” and we’re done. Yeah, but not on GAE. As even the “max rows a query touches” are limited – to 500. So there I am, with my 100.000 rows, and I have to delete them in 500 rows per query… Of course I can’t call this in an loop – as the timeout quota would get in my way and kill the app.From what I’ve seen, people do solve this using a CRON task that calls this “clearDatabase” servlet until it’s  done – ugh, not a nice solution but I can’t think of any other solution :\
   3. con:The only query I need to do in this app is basically:

      SELECT range FROM RemoteIpData as range
                   WHERE range.ipFrom <= ?1
                     AND range.ipTo >= ?2
      #and this would always return 1 entry!
      

      And guess what… AppEngine does not support multiple “less/more than” operands in one query! If you think hard about what BigTable is, it does make some sense. More information about “GQL” can be found here: http://code.google.com/appengine/docs/python/datastore/gqlreference.html All the restrictions about the Queries you can do are documented here – some are really counter intuitive when you come from an RDBMS enviroment… Ah well ;-) Oh, and yet another great link about GAE limitations.

      Having this limitation, really sucks for my normally “super simple query”, and I’d have to change the data structure somehow or do some awful 2 queries and then combine them in Java (omg teh terrorr… :<). So, while developing on appengine, keep in ming – simple things might turn out quite complicated due to the nature of BigTable. If you know all the limitations when designing the system and not while finishing it, you’ll be a happier man… ;-)

   4. neutral: Primary keys can’t be Integers etc, as AppEngine uses it’s own “Key” type. :-)
3. Having that all said. Appengine’s administration panel is quite ok. And the not-so-newly-added log search etc are really fine tools. Something an plain old tomcat can not offer. On the other hand, the limitations can really be a deal breaker! My app was really fairly simple, and yet appengines quotas managed to really get in my way. Keep this in mind while thinking about using it. You may also try Amazon’s cloud or CloudForce from SalesForce etc… They all do offer a quite less restrictive enviroment.
4. If you’re interested, deployment does take about 5-7 minutes before the new version is visible on the web – so don’t panic if you’re still seeing the old version after deploying the new one.
5. My opinion about GAE…? (semi serious ;-))

“Screw you clouds, I’m going /home!”
(intended southpark pun)

I’ll finish this project using my ol’ pal, Tomcat6 which I’ve already got running for netbeans.edu.pl (but that was an grails app).

I’ve also decided to use Spring, which I didn’t really need on appengine – as the only thing I was doing was so small that all the logic was around 10 lines in the servlets… But if using hibernate and all the other “real” JEE stuff, I felt I’d need to “do this right” so I’ve decided for Spring 3 and Maven2… I’ll try to build this project from gradle soon too!

ALSO! If interested in an more experienced programmers view on appengine (I’m still a novice), go and read this blog post about GAE on Sławek Sobótkas blog. All in all we seem to agree that the limitations can be an pain in the a… ;-)

The source for the appengine version is on my github account. I’m porting it to an plain old tomcat environment and will post this version there too when it’s ready to run (tomcat deployment is somehow hell with such apps for me… Any tips are really welcome :-))

I used 2 libs to create this tag:

• twitter4j – a very good twitter client library (java, even works on android)
• twitter-text-java – the official library for parsing plaintext to produce valid @ and # links, more information about it can be found on github or http://engineering.twitter.com/2010/02/introducing-open-source-twitter-text.html

Anyways, first the source of the actuall taglib. We simply call the methods, get the response fill out the content of the tag with the variables we got:

Super easy, ain’t it? More stuff about taglibs in grails can be found here. Now let’s go over to our view and see how to use this taglib – it’s (once again) amazingly simple:

All we need to do here, is set an variable thanks to some nice GSP tag, and then use it to query and echo our data the taglib is supplying us with. Note that the tweets:fromSearch acts like an foreach – no need for any extra loops here :-) That’s just a very simple tag but it’s quite nice to use, hope you enjoied it and will be more than happy to see the sources of netbeans.edu.pl published on github soon… :-)

Helo again! Today’s post will be an super quick usage scenario for NetBeans Platform Lookups – the probably most essential part of the Platform. An super simple explanation of them is “an Map<Class, Object>”. Sounds simple right? Well it is, and it’s quite powerful at the same time. It enables you to loosely couple parts of your app, create extension points etc. Another use case is passing some date around in the app, but you don’t know where the modules interested in this data are (because you’re loosely coupled, right?). In an app I’ve been writing for my Uni since some days, I’ve used the Lookup to do exactly this – pass data from an algorithm to an LineChartDrawer that then will update it’s chart each time the data is being updated.

This is going to be quite similar to an  Observer Pattern implementation, but using the lookup as notification mechanism. Let’s start out with our Algorithm class (and module – note that this module is not dependent on the chart drawing module!)

The code is well commented for this post so take a moment and read through it. It’s just some pieces of the actual class we’re using – so no actual computing stuff is shown here, let’s focus on the lookups. Note that I’m getting the TopComponent that I’ll add as an listener “by name”, it’s super simple but would you need to “find some implementation of some drawer interface” just use an lookup and get the instance, or even all the instances of such interface implementations. That said, this solution is not the most universal here (getting the instance) but it’s good enough for our app since it’s a very small application. No let’s look at the “Observer”, better named as “LookupListener”:

That’s about it. We simply have an resultChanged() method that will be called if the contents of the resultset are changed. We might also want to arr more lookups to be observed, so it’s better to check this instanceof I think than not, since there might be some unexpected stuff inside of it…

All in all, you write an Lookup.Provider and LookupListener, implement both’s methods and then at some place in your app hook them up. It doesn’t really matter “who looks for who” as long as it’s consistent and logical. In this case, the algorithm is looking for people interested in his data. Note that at any moment in time, we can add another listener, no problems here. Note that the Lookup.Provider and LookupListener are quite similar to the Observer and Observables mentioned before.And the good thing is, the algorithm does not know about any concrete drawer, and the drawer has no idea about some algorithm. They just have one shared API module – the toolbox, in which there is the DataTransferObject we’re using to pass the data from one to another – yay, an loosely coupled system. (yeah, I know our code is tightly coupled at some other places, but this part is quite ok :-))

If you have anything you’d like to comment on this code, feel free to do so. It’s just parts taken from http://github.com/ktoso/TravelingSalesman-NBP our implementation and visualization of the Traveling Salesman Problem being solved by Genetic Algorithms.

For more information about lookups goto: a nice post by Anton Epple on dzone.com or to Geertjan’s post with some good links or http://www.antonioshome.net/blog/2008/20081023-1.php

What I don’t “like” there is that an server is bound to one special domain that they call the server’s “main domain“. Fair enough, he ought to have some “main” thing right? Basically, the domain is identified 1 to 1 with the server, and this one, has some special stuff it can do. It’s like “I’ll call this server example.com” and then you work with example.com, kinda not caring wtah server it is. And that’s not harmful at all but:

The problem comes up if you host multiple domains – each of which would like to be “mainish” – be able to setup email accounts easily etc.

The part with multiple websites is simple: you set it up using what they call “multidomains” (weird name…) – it’s basically like httpd’s VirtualHost stuff. “Ok, nice.”  – you think and then want to setup mail adresses for those domains you’ve hooked up to this server. Should be easy – right? You could just enter some [login]@[domain] and be done with it. But it’s not that simple since all you can do is add emails inside the servers “main domain” (if you’d try to youd get into relay domain denial stuff). You’re forced to buy some sucky (25MB POP accounts) extra “MX pack”… I’ve encountered this while using an domain not registered in OVH, so it’s possible that this is easier with domains bought “from them”, but I doubt it (nothing suggesting this in sight on their website).

To sum it up: You can only easily set up email accounts for an domain that is the servers “main domain” – it’s 2 clicks. Setting the same stuff up for an “not main domain” – will cost you a little extra.

Another weirdity is that if you want to change an servers main domain – it’ll cost you a little (not much).

That said – ovh is an reiable hosting company and does not try to “rob” the user from every penny he’s got like nazwa.pl does on their “WOWOWO FREEEE* domains” – you can guess what for the * there is… ;-) I’ve seen lot’s of hosting operators and ovh is certainly one of the best when it comes to “techy stuff”. I also often recommend (and have used) linuxpl.com it’s also very nice for virtual hosting – I’ve seen an VPS there and it was horrible – many “magical” things were happening there (it was openvz based if I recall propperly) but Virtual Hosting is top class there (svn, python, rails, ssh, synfony, all preinstalled :-))

Oh man, that would take sooo much time...

Thanks goodness I’m not really keen on doing stuff the slow “not_smart” way, and the new server is a dedicated CentOS machine! :-) So, what did I do?

Yeah, no time wasted here!

There’s a lot of ways to accomplish this if you have shell access. You could:

1. wget -r --user=login --password='pass' ftp://admin:example.com/folder/

2. ncftp #normal ftp should also work the same way
   open ftp://example.com
   login
   pass
   mget -r folder

3. Yes, SCP is a great tool for this kind of situation, but i couldn’t use it since the old server didn’t have ssh acces :-( Here is how it could be used in such a situation:

   scp -r login@old.example.com:/home/folder login@new.example.com:/home/folder

I used the second method and am quite satisvied with it, as the servers did all the migrating i set up the database and did some other project related stuff… :-)

Download the compressed backup
Put it onto your target server, by FTP, SFTP or SCP, doesn’t really matter
Launch SSH and use your terminal hero skillzorz:

cd where/i/put/the/file
gunzip backup.sql.gz #or unzip backup.sql.zip
mysql-u username -D database_name -pPASSWD < backup.sql
cowsay "data imported, yay!"

It’s pretty self explanatory I guess, and MUCH faster and more reliable than any other way of doing this :-)

My backup should finish downloading any second now, so… cya soon!