The main issue with Windows from my perspective is that it seems to go downhill very quickly. A clean system, once fully patched, runs fine but overtime (short amounts of time at that) the whole system begins to feel sluggish, disk access becomes near continuous and overall responsiveness and stability suffer. I don’t have these feelings in Linux at all, nor to I even feel tempted to scandisk, defrag, virus scan, remove old programs and so on.

My feeling is that all the problems come down to a few key decisions in the development cycle of Windows. For example, I don’t understand for example why the dedicated swap and hibernate files sit on the same hard drive as my programs. I certainly feel their presence when defragmenting (or being told I can’t defragment).

However, since I’m forced into Windows by this tablet it’s good to see that VIrtualbox has saved me from being without Linux. Version 2 (and from 1.6 I believe) includes seamless desktop (with Windows as host).

That’s a great feature - not only because it gives me just the one desktop but also because it allows me to put all my development programs, test programs and general clutter into another expendable virtual machine. Thus my Windows OS is light and hopefully won’t get bogged down with the remains of old version of old programs and DLLs.

Time will tell…

In the past 2 weeks through I’ve picked up two items which hare changed my life - noting that phrases like that are always overstated!

The first is a Acer C300 laptop. The C300 is a tablet pc with swivelling screen. Whilst my primary design for the purchase (second hand) was for use as an ” always on media control station” must confess to being
impressed with the laptop as a laptop and as a piece of electronic paper.

At the same time I’m sad that the handwriting, journal, push to rotate screen between portrait and landscape seems to be unavailable as out of the box functionality in Linux. That said writing is infinitely slower than typing! For reference this entire blog entry has been written and converted using MS Windows Journal. Also worth stating that be my ‘lack of love’ for Windows XP is unchanged!

Whether l remain in Windows or move to Linux with some painful configuration or loss of functionality remains to be seen. Whilst the handwriting recognition is nice it’s ultimately a gimmick. However the ability to go portrait to read books, webpages, etc is more appealing.

My second purchase was the G1 phone from T-mobile. In the past I’ve found it hard to commit to a

new phone - cost vs functionality was always balanced in the wrong direction. Similarly as a dedicated developer my memories put me off the closed iphone ad Blackberry. The G1 changes the development side but surprisingly manages to do that on budget (£30 per month for unlimited data, free phone and some combination of 1400 texts and 700 mins).

I will leave a detailed entry on the phone for another day - my arm hurts from the writing! However suffice to say that - with the obvious comments that smaller phone, longer battery life and better camera would make for perfection - the G1 is a fantastic piece of kit for any one looking for a true smart phone. For developers it’s a must-have; for any one with Gmail it’s essential; and for everyone else its a big but cheap iphone (without the Apple T&Cs).

(PS. Apologies for some dodgy hand writing recognition but I thought I’d leave them in. I’m impressed on the whole!)

My Compass (vs Hibernate Search) results from yesterday also had me thinking about why I was using a database at all. When using ORM you are often more interested in the easy storage of objects vs the properties of the database behind. Specifically are you aren’t using all the SQL muscle.

Depending on the data type (and database) you are storing database ORM might not have any advantages. For example for large amount of unstructured text or for objects where you don’t have a defined schema.

Today I had a quick play with Compass as a schema free database. Using the resource mapping you can simply create a object (a resource) and add properties (ie name, value pairs) to it. These resources are indexed by Compass and available through searching. Resource mapping allows you to configure how resources are pushed into Lucene (for example if they are stored, indexed, which analyser and how objects are converted to strings).

Since I haven’t posted any code in a long time, I’ll post the important pieces here:

META-INF/compass.cfg.xml:

<?xml version="1.0" ?>
<compass-core-config xmlns="http://www.compass-project.org/schema/core-config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.compass-project.org/schema/core-config
http://www.compass-project.org/schema/compass-core-config-2.0.xsd">
<compass name="default">
<connection>
<file path="compass_index"/>
</connection>
<mappings>
<resource location="META-INF/entity.cpm.xml" />
</mappings>
</compass>
</compass-core-config>

META-INF/entity.cpm.xml:

<?xml version="1.0" ?>
<!DOCTYPE compass-core-mapping PUBLIC
"-//Compass/Compass Core Mapping DTD 2.0//EN"
"http://www.compass-project.org/dtd/compass-core-mapping-2.0.dtd">

<compass-core-mapping>
<resource alias="entity">
<resource-id name="id" />
<resource-property name="contents" store="no" index="tokenized" />
</resource>
</compass-core-mapping>

Main.java:

    public static void main(String[] args) {
CompassConfiguration cfg = CompassConfigurationFactory.newConfiguration();
Compass compass = cfg.configure("/META-INF/compass.cfg.xml").buildCompass();
// This remove locks from previous crashed instances (useful when debugging trouble!)
compass.getSearchEngineIndexManager().releaseLocks();
CompassSession session = compass.openSession();
ResourceFactory resourceFactory = compass.getResourceFactory();
CompassTransaction tr;
int count=0;
String sourcePath = "/path/to/files";
// OPTIONAL: Turn the optimiser off here for increased speed
// compass.getSearchEngineOptimizer().stop();

// Use LUCENE transaction for faster batch processing
tr = session.beginTransaction(CompassTransaction.TransactionIsolation.LUCENE);
for (File file : new File(sourcePath).listFiles()) {
try {
Resource r = resourceFactory.createResource("entity");
r.addProperty("id", UUID.randomUUID().toString());
// Just for fun add a property we haven't mapped at compile time
r.addProperty(resourceFactory.createProperty("filename", file.getAbsolutePath(), Store.YES, Index.UN_TOKENIZED));
r.addProperty("contents", new FileReader(file));
session.save(r);
// Simple batching to improve speed
if(count++ % 1000 == 0) {
tr.commit();
tr = session.beginTransaction(CompassTransaction.TransactionIsolation.LUCENE);
}
} catch (IOException ex) {
logger.log(Level.SEVERE, null, ex);
tr.rollback();
}
}
// OPTIONAL: Turn the optimiser back on (if you turned it off and optimise)
// compass.getSearchEngineOptimizer().start();
// compass.getSearchEngineOptimizer().optimize();

// Now search
tr = session.beginTransaction();
CompassHits hits = session.find("Your search query here");

for(CompassHit hit : hits) {
Resource r = hit.getResource();
System.out.println(r.getValue("filename"));
}
tr.commit();
session.close();
compass.close();
}

I chose resources over POJO mapping because it’s ideally for allowing users to store meta-data information around an entity. New properties can be added and queried at runtime (though some control might be advisable to help find things later on).

I’ve got an specific application in mind where I want to use this, but I think it’s got wider utility as the web moves us more towards textual (meta-data) information. With a bit of work you could probably achieve a CouchDb in Lucene…

One of my motivations for trying Compass was that I found Hibernate Search to be a little slow. Indexing was fine, but bringing back the results was quite painful. I wondered if Compass was faster.

Out of the box Compass was pretty easy to get going (though I seem to have problems creating/dropping/updating the database when the Compass session listener is enabled in the persistance.xml). I moved back to TopLink for no other reason than to use the JPA reference implementation rather than Hibernate (which I know Compass works with because of Grails). The help documentation is pretty complete for Compass which means that once you read it you are good to go. My simple app didn’t require cascades, inheritance, etc but it’s all in there and available to use.

One of the differences between Compass and Hibernate Search is that when HS returns a query result they are objects from Hibernate itself. For Compass you are getting Compass’ object back which might not have all the fields of the JPA version. If you want the JPA version you have to go and fetch it yourself (using the primary key shared between Compass and JPA).

On the surface this might seem like Hibernate has an advantage and to an extent it does. However the reason HS is “slow” as it needs to pull all that text from the database and into entity beans when you get the results list (you could use Projections to do something about that). For my data set of 20000 docs that’s 75mb of text, or perhaps more relevantly 7mb to process for every 1000 documents worth of search results.

If you contrast with Compass, it’s effectively maintaining two data repositories - one JPA and one Compass/Lucene index. When you get results from a Compass search you aren’t hitting the database, so you only get the fields back you asked you annotated as @Searchable*.

So comparing on a level playing field (ie getting the JPA persisted object for each result)  Compass and Hibernate Search seemed to perform around the same speed, though Compass was maybe a little faster (10%). But if you just want to do a search through Compass and not grab the original JPA object (say because all the fields you need are stored in the Lucene index), then it takes about 400ms (vs 19 seconds)!

I was hoping to get the best of both worlds by lazy loading the large string field of the document class. However that didn’t work (with TopLink at least, others not tried). The alternative is to put the content in another dedicated class, use lazy fetching for JPA and SearchableComponent / SearchableCascade in Compass. That way you would search on the text but return the owning class (eg the metadata of a document) rather than the content. Speed would improve at the cost of complexity. I’ve tried implementing this without sucess in the last few minutes, TopLink seems to be eager fetching the content class despite my lazy request!

One of the failures of Lucene is that you can’t update a document without deleting it from the index and then adding (and hence re-indexing). Thus you have to have the original content around if have entries that change - sadly that means you either store it in the database (and take advantage of HS or Compass) or you can leave it on the file system and then write your own code to re-read as necessary.

• TopLink is bundled with Netbeans, so that’s really easy to get started with. However Hibernate has Lucene integration through Hibernate Search, which I wanted to try out.
• Hibernate is very difficult to get started with requiring you to download multiple Hiberate zip files and combine together a massive set of jars.
• Additional JARs are required to be downlaoded such as an implementation JAR for SLF4J (you should download the whole SLF4J zip and pick one applicable to you - though that’s not obvious from the SLF4j site)
• You still need commons-logging for Hibernate Search.
• My code from TopLink didn’t work with Hibernate out of the box (though that’s probably my assumptions about JPA rather than a comment on either TopLink or Hibernate). Having a Singleton EntityManagerFactory sorted this out.
• Hiberate was faster than TopLink ( 25% reduction in execution time) and also more robust. It didn’t “out of heap space” even with large transactions (which I’d had to batch up for TopLink).
• The additional (non-standard) Hibernate annotations (such as Index) reduced the execution time to a 25% of the original TopLink. Of course I could have added indexes manually (either through the database commandline or as JPA native SQL queries) but the annotation method is much cleaner.

Main points captured but not in the nicest form nor with the context which justifies them - sorry for that!

Following up on my previous post around an hour ago, Hibernate Search (that is Hibernate integrated with Lucene) does indeed work perfectly. Setting up entity beans for indexing and performing search is easy.

Almost prefect - another couple of out of heap size messages due to the single transaction which batches up 20000 documents (around 75mb on disk) and which is held in memory for Lucene to index on commit. Hibernate seems to do fine persisting this to the database though obviously  Lucene has more work to do at commit point and runs out of memory constructing the index.

I could of course increase the heap size but instead simply batch off the transaction to commit 1000 documents at time. There is another way of doing that but it’s similar in results/method. I chose to explicitly batch the transaction simply so that I keep the database and index in harmony (and don’t have to worry about rollbacks to different stages).

Might be worth getting the Hibernate Search in Action book, though I think that you’d been to be doing some complex search and entity mapping in order to require more than the basic.

As an aside, I seem to be coming across JMS more and more in ‘everyday applications’. For example, Hibernate Search uses JMS for dedicated indexing, Glen Smith talks about his use of JMS for his Groovy Blog app and LinkedIn presented their JMS based architectured at JavaOne 2008. Make me wonder if JMS is the pragmatic, bloatfree ESB?

I don’t understand the overlap with Groovy Swing/GraphicsBuilder which seems to offer a large majority of the demonstrated functionality of the JavaFX and offers more power and better integration with Java itself. (Admittedly there are additional media, etc elements to JavaFX).

All this isn’t helped by the fact I can’t get my hands on it. Verying disappointing that there’s no Linux distribution (even feature limited version). This is particularly frustrating as it has been demonstrated that it is possible to hack the current Mac JavaFX SDK to run on Linux (just not supported officially).

Strange that Grails is the first time I’ve felt this way - having used CakePHP, Django, etc in the past. I think this is because Grails is the first framework which feels integrated to me. Django for example seems very disjointed going between the template language and Python. I wonder if this ‘disjointedness’ meant that I had to ‘put my web designers’ hat on. In Grails I’m just wearing my Groovy hat (so to speak) - though the template language isn’t Groovy (but it feels closer thanks to easy to write taglibs and “${it}” everywhere).

So I need to check out some javascript/ajax html  libraries to help me speed up the front end development. I confess that I would have been tempted to go for Ext JS where it not for the licence approach. In some sense, I don’t have a problem with the new GPL nature (though LGPL was better) but I find the whole concept quite confusing, as do many others.

Still since my work it usually open source I might give it a go. Alternatively jQuery is by far the best when ’starting from scratch’ and Yahoo UI is certainly impressive.

This week I’ve developing some Grails applications. Unfortunately nothing too “bloggable” due to the nature of the work. Tomorrow I’m going to work on a streaming video application in Grails which I’ll hopefully release (as example code).

I must also check out the new Drools plugin which could be an excellent way of implementing business rules.

I’ve noticed that the Repast agent simulation toolkit has updated it’s site and also released Repast Simphony 1.1. I’m going to have a play with this over the next few days using Groovy.

As a side note, installing Repast I had the opportunity/motivation to try out Eclipse 3.4 (Ganymede) for Java and Groovy development. The Groovy plugin seems to be at about the same level of functionality as Netbeans (maybe a less mature but then noting it runs on a stable version Eclipse not a development milestone like Netbeans). Java development with Eclipse seems ‘less out of the box’ than Netbeans, which is less friendly for the newcomer.