Andy Hedges’ Tech Blog

1003 · 2^2076535-1 is prime! That’s all.

This Beta release of Java has had the typical Sun treatment of not being promoted hard enough. Maybe I’m being unfair and maybe they are saving the hype for JavaOne - we’ll see.

Anyway one of the main feature of the snappyly names Java 6 update 10 (nee Java 6 update n) is that it has a modular basis. The kernel - all you need to run most things is 200K (flash player is 1.6M, Silverlight 1M, .NET Runtime 23M). It will then dynamically fetch what it needs to run each application or applet and cache that in case anything else needs it.

This isn’t the end of the story however there is now good support for 3D acceleration of 2D and 3D graphics and some great configuration parameters for Applets (like turning the system tray icon off).

As always a Java Applet can be run as a Java application and a webstart application.

As a demo of this here is Quake 2 running as an applet (you’ll need Java 6 update 10+ for the applet version):

• applet versions/embeded
• this will run the no embeded version

Yes, multiple player works you can run a Quake server within the applets and then anyone else running the applet on your network can join. This is a great demo of low latency high response network requirements being delivered in a browser and standalone environment.

Other examples showing JavaScript interacting with a Java applet is the NASA World Wind demos

• World Wind JavaScript Demo
• Other World Wind Demos

Most of this stuff I think will run with good old Java 5 but who would know nobody from Sun seems to be promoting it hard enough.

There is uproar in many Open Source circles that Microsoft might be buying Yahoo. Yahoo reportedly runs on a predominantly FreeBSD platform and Microsoft would likely change this as they did when they bought Hotmail to run on their own proprietary operating system.

Whilst this deal is almost certainly a blow for Open Source I don’t think it is that much of a big deal. There are plenty of large successful companies running their core systems on Open Source - I know, I’ve worked on countless numbers of them. Yahoo gradually moving to Windows will take an extremely long time. Firstly because Windows and Microsoft technology isn’t built for use at this kind of scale and secondly there will be a huge amount of legacy at Yahoo that is tied to their platform. If it weren’t a PR nightmare for Microsoft running their platforms on Open Source then it would make much more economic and technical sense to leave the systems at Yahoo the way they are.

Anyway I’m happy this is happening it is about time Google had some competition in the search space. Search hasn’t moved on since 2001 when Google reached tipping point as the dominant search engine.

I keep having conversations with people on given topics. Then read their blog posts posted before our conversation and realised they have blogged the points covered in our f2f discussion prior to our conversation. I hope they don’t think things are happening in the other order.

Some may remember I predicted in a post back in April 2006 that Firefox would overtake Internet Explorer in terms of user base by December 2007 (based on the statistics from my website The Humor Archives). It looks as though I was right. The statistics for December 2007 stands as follows:

Firefox: 50.9%
MSIE: 45.1%
Safari: 2%
Opera: 0.5%

based on 261,386 unique visitors to date in December.

Figures for last month were still in MSIE’s favour

MSIE: 53.1 %
Firefox: 42.5 %
Safari: 2.1 %
Opera: 0.6 %

based on 351,316 unique visitors in November 2007

Cedric’s post regarding software headaches got me thinking about the more pleasant learning experiences I have had in the last 15 or so years I have been writing code (for real machines not Species and STs). So I have pulled together a short list of Epiphanies that most mortal developers go through sooner or later.

• Object Orientation: now on the face of it OO isn’t that complicated, you create a set of classes that represent both real world objects and behavioral entities. This is a simple enough concept however it take a while for it to click, it takes a while for you to understand why that is a good thing
• Functional Programming: who’d have thought that Lambda Calculus wouldn’t be intuitive? Seriously though, Functional Programming is a category of programming languages that is essentially based on mathematic functions and their evaluation without the need for state. This isn’t such an easy concept to grasp like OO and even when you do the benefits can be non-obvious until that epiphany moment.
• Recursion: is a very powerful thing, used well and it can achieve great things, used badly and it will crash your program.
• Do one thing well: certain companies are famous for their feature rich products. People love their features, product managers get paid for adding extra features and features are what (lazy) marketers concentrate on. However more features means unecessary complexity of code and complex code leads to bugs. More features means that the important features get less attention. The trick is to do one thing well and make sure your code, program, service can interoperate with others that also do one thing well.
• Proprietary Software is only good for the vendor: if you give a company that you do not control, monopoly provision over the support of a piece of software, that you rely on, you expose yourself to significant risk. Unless you have a very big stick to hit them with they will not be responsive to your needs (in my experience even if you do have a very big stick they still won’t be very responsive). If you need to fix a bug in a piece of software and you need to fix it immediately then you need the source code - end of story. Also you probably don’t need all the features they are offering (see Do one thing well).

I’m sure there are more… would love to hear other peoples epiphanies

PS I almost titled this 101 Epiphanies in Software Development, but thought better of it

I have been messing around with Povray again. Below is my effort

I have read numerous documents about ‘SOA’, ‘WOA’ and ‘Mashups’, some offer some insight or some intelligent ideas most however are unadulterated drivel. Most, as is the way with emerging technology, are touting some product or other that is surely going to revolutionise the way we do everything or form part of some market analysts propaganda/echo chamber/poorly written explanation for lazy technical management.

I have noticed something interesting; virtually all the product pitches and analysis that use Google Maps as their example are hiding the crapness of whatever it is they are trying to sell. The cool thing about Google Maps is that they have given free access to anyone with a internet connection maps and satellite images of the entire Western World and some way beyond - that is amazing, hats of to the Google. However this would be cool no matter if it was exposed as a CSV file over a raw socket or Corba (that would just be slightly annoying). It is cool no matter what product you put between it and the web-browser.

If you can create something with Google Maps that isn’t cool or amazing then you are quite frankly an idiot. If you can create something that is cool and amazing that is because Google Maps is cool and amazing, not the technology be it webservices, mashups or the newest TLA. If you want to show how brilliant your product or <something> oriented architecture is pull something together from apparently innocuous services to provide something of (much) greater value than the parts.

Looks like the new checksum in the iPod database to restrict non-Apple products has been reverse engineered in the open source libgpod (a library that many open source music library managers use to talk with iPods) and so iPods can now work with Linux again. Given all the problems I have had with my iPods and their interoperability with any standards I will not be buying any Apple products for a long time.

I have now moved the jmimeinfo project to a more suitable home at java.net. The project now has all the trimmings of a well run project: defect tracking, source control, forums, mailing lists and file distribution. Thanks to Jeff Dinkins for setting that up.

There have been no changes to the license it is still GPL and I will hopefully be doing a bug fix release in the near future.

It seems to me much of the two essays, Surgical Team and Mythical Man Month in Fred Brooks important work are elaborations on Adam Smith’s theories of Specialisation and Division of Labour and production line optimisation but applied specifically to software development. Which is excellent and worthwhile cause; as software engineering still lacks the rigour associated with other engineering professions.

I need to re-read both the great men’s works but I think there is millage in applying more of Adam Smith’s principles to software engineering.

See:

• An Inquiry into the Nature and Causes of the Wealth of Nations
• The Mythical Man-Month: Essays on Software Engineering

• I have fixed a bug in jmimeinfo regarding files not being closed under certain circumstances. Thanks to KewlCat and Ryan for pointing out the bug (sorry guys I don’t have your email addresses to thank you personally). The latest version is available for download.
• I am considering moving the project to somewhere more suitable possibly java.net or maybe sourceforge (need to compare the two).

I have been considering buying a Mac for sometime now - I have been put off by the seemly annoying way in which the user interface works. My Mac loving friends and colleagues ensure me that once I have been using one for a while I will never want to go back. This I can buy, I had the same experience when I moved from Mac to Windows (c 1995) and from Windows to Linux (c 1998).

The only Apple based software that I use regularly is iTunes (on my work Laptop or within VMWare on my Linux desktop at home). I assume that iTunes is representative of the standard of other Apple Mac based software and quite frankly it is rubbish. These are the things I hate about iTunes.

• I have to give up my credit card details to Apple in order to get album art work
• There is no good user story around using multiple computers with one iPod or multiple iPods with one computer. If I set my iPod to sync my music collection and I connect it to my work laptop all my music is erased. I don’t have my music collection on my work laptop and don’t want to have to. If I don’t set it up to sync then it doesn’t store my play statistics and I lose a bunch of functionality.
• My subscribed podcasts are stored on the computer and not the iPod or online and so I have to manually sync my subscribed podcasts (this is the RSS feeds in the Library podcast view as opposed to the actual audio files)
• iTunes and the iPod firmware are a closed proprietary pieces of software
• iTunes and the iPod do not support ogg vorbis or flac or any other open standard

I just need to find a laptop that is as good value for money as the Mac Book Pro or buy one and install Linux on it… or just procrastinate a bit longer and save some money.

Many of the commercial enterprisey tools/components/servers available today major on WYSIWYG, point click, drag drop style development. Which is great, I’m all up for labour saving tools and flashy sales presentations. I can open an IDE (usually Eclipse with plugins) and get going creating an Orchestration, ESB routing path, Portal portlet, Rule Flow, you name it.

However,… when the dust settles there are broadly 2 major problems with the current implementations of these Enterprisey Tools.

1. I need to put these creations into a project version control system. Why is this ‘the trouble’? Well these tools create an assortment of xml configuration file, java code, .properties, .blah and a proliferation of other unfathomable artifacts. I don’t know what is specific to my machine, specific to the IDE layout, what is test code - I short I don’t have a scooby doo (clue) what these files are - the documentation is much help either. I end up checking the lot into source control and then the fun starts when my team mates try to merge their changes with mine when they also have no idea of what these files are. For the record I don’t ever want to use the in built version control system. Why? Because, there will be usually more than 5 separate tools with there own source control systems and this quickly becomes unmanageable.
2. I need to be able to compile, package, build, deploy (remotely), run test and release to a release system working software from a continuous integration system. This system needs to run periodically and autonomously (that is without any human intervention). This system most like doesn’t have a graphical subsystem (windowing interface) it is likely a big bad *nix machine (sans X). I can not run the IDE on this machine. I also don’t want to install the IDE or indeed the server components on this machine. I don’t want to install this software on the machine because it is a huge overheadand keep the versions in line with the project (which can last up to 2 years). On a typical project this could mean installing 6 IDE and 5 servers on the continous integration servers and upgrading and patching each of these 3 to 4 times.

So here is my plea to the tools industry (you know who you are):

• Everything that can be done in the IDE should have a corresponding ant task
• Anything that can be done though your server console can be done as an ant task (or scripted in some way which can be call from ant).
• All the ant tasks should be in a jar file and this jar file should have documented dependencies (ideally as an Ivy or Maven dependency file).
• Do not impose your jar file directory hierarchy on me for the ant task (you know who you are).
• Every file that your IDE plugin creates should be documented, what it is, what it’s structure is and what every part of that structure means. If you need help understanding the level of detail look at the hibernate mapping documentation for an exemplar
• And whilst I’m asking please don’t insult me by telling me this isn’t important - it is. This is very important.

Update: whilst trying to find my way through the borkedness of one product I stumbled upon this blog entry describing some of the problems I am hitting with one product. This article was written in Nov-2004 and it seems that things have moved on very little since then. BEA, in this case, still use OS environment variables such as CLASSPATH and PATH rather than allowing them to be specified at the task level in an ant way. They still bundle a whole lot of crap in their weblogic.jar and reference still more from the manifest that confuses builds. Why is the ant API included in their server environment classpath?!

A great article on why Reddit is better than Digg. I almost completely agree with the article, Digg has become a very bland place. Interesting articles are discovered make the front page get noticed by the unwashed and modded down as spam or ‘lame’. Reddit on the other hand doesn’t put the power of censorship in the hands of the minority and so these small groups cannot get articles removed from the front page.

I have really cut down on the visits I make to Digg because most of the time the articles there are fairly low brow (how many more 50 CSS tricks articles can there be? How many ‘hilarious’ Colbert sketches must we be subjected to?). Nope Reddit is for me at least for now.

Just a short post to say that those who don’t read xkcd should. It calls itself “a webcomic of romance, sarcasm, math, and language.” and is very funny if you are technically inclined.

So Mr Jones has decided to blog-tag me, thanks. I guess this will get the Google Juice flowing if nothing else.

I have decided to decrease the number from 5 to 3 for two reasons, I’m lazy and it keeps things interesting (read: I’m lazy and not very interesting).

3 things:

1. I once found the 204^th largest prime number it has 160578 digits, sadly it is now only 702^nd largest, I still search for them but haven’t found any larger yet.
2. I sleep with my eyes open (literally rather than metaphorically)
3. I am a big fan of Single Malt Whiskies and collect them; however I seem to drink them almost as fast as I buy them

Andrew Law, Heather & Doug Cutting

Ant P. > How fast is it against MyISAM?

greg1104 > I’ve managed to get my PostgreSQL installation tuned to very high speeds simply by switching the database disk over to /dev/null. It runs fast as hell, and the data integrity is basically the same as MyISAM.

(via tabo @ aureal via Slashdot)

Another reason, if you needed any more, to use PostgreSQL over MySQL is independent benchmarks have shown Postgres to be far faster than MySQL, 50% in fact. The gap gets even bigger when multicores are taken into account; MySQL adds 37% performance going from 1 to 4 cores; postgresql adds 226%, that really is an absolute pounding.

(via spyced)

Technorati Tags: postgresql, postgres, mysql, database, performance

jmimeinfo has moved please update your links!

I’ve been looking around for a java library that will do a similar task to the unix file command. The reason I need such a library is because, shortly, I’ll be allowing the upload of files from the public to a website of mine. When uploading a file via http it is encapsulated as a multipart message which includes the mime type. However what mime type gets put in the multipart is entirely dependent upon the uploaders browser and/or operating system. So if the user uploads a flash movie file but their system is unaware of the flash movie format the mime-type will likely be set to default such as application/octet-stream which basically tells me that it is a bunch of binary stuff - not much use.

Windows also provides a number of headaches (just for a change) in this areas, as every program installed on the machine is allowed to set mime-types at the OS level there is often some confusion. So depending on what programs a user has installed the same file maybe encapsulated with a different mime-type. For example on a Windows system I recently looked at there was a file called video.mpg, the *.mpg extension was associated with the movie/quicktime mime-type and upon looking at the file in a hex editor it was actually an RIFF based avi file - as a colleague of mine would say ‘genius’.

So this is how Windows deals with things, how do other OSes do it? Well I’m glad you asked. Linux or more correctly freedesktop.org which produces standard and software for numerous OSes (Linux, BSD, Solaris, HP-UX, AIX etc) have a standard for this, it is called ‘Shared Mime Info‘. Shared Mime Info has a specification and a reference implementation (in ANSI C). Shared Mime Info uses what are colloquially know as magic numbers and globbing to determine a file’s type.

Magic numbers simplistically work by looking for sequences of byte patterns in a file. So for example a java class file starts with the hex string ‘CA FE BA BE’ or an access database file starts with a normal string ‘\x00\x01\x00\x00Standard Jet DB’ or to pick a more complex example ELF executables start with ‘\177ELF’ and then 5 bytes in have the byte 1 then a further 16 bytes in have a little-endian byte 16.

Globbing is just a simple form of regular expressions matched against a file path. So for example the glob for a word document would be *.doc and the glob of a GNU make file would be [Mm]akefile.

Shared Mime Info also goes one step further providing plain English (or anyone of 30 other languages) description of each file format as well as suggesting applications that can deal with the file.

Anyway, so Java doesn’t have an implementation of Shared Mime Info… until now. I’ve timeboxed myself 2 hours over 2 days to put this together, it is a basic implementation of Shared Mime Info for Java, currently the code is a messy and it isn’t a full implementation yet, but it does provide a full implementation of magic and glob based mime identification and it’s licensed under the GPL. It requires Java SE 5 or higher.

The source and compiled jars can be downloaded here:

jmimeinfo-0.0.1.tar.gz (08-Nov-2006)

jmimeinfo-0.0.2.tar.gz (12-Nov-2006)
Fixed concurrency issue, I (stupidly) assumed that Matchers in ORO are threadsafe.

jmimeinfo-0.0.3.tar.gz (20-Apr-2007)
Fixed problem with RandomAccessFiles not being closed under certain conditions in net.hedges.mimeinfo.magic.Rule - Thanks to Ryan and KewlCat for point that out (see comments below)

To determine the mime type of file code the following:


File file = new File("/path/to/my/file.ext");
MimeInfo mimeInfo = new MimeInfo();
System.out.println("mime-type: " + mimeInfo.getMimeType(file));


To test the library run the following from the expanded archive:


java -cp jmimeinfo-0.0.1.jar:lib/jakarta-oro-2.0.8.jar net.hedges.mimeinfo.ui.Test /path/to/file.ext


All comments and feedback welcome as well as patches and bug reports

Technorati Tags: mime, shared mime info, freedesktop