I had to post this because right now I’m in a state of gastronomic elation – I just had the best kippers I can ever remember eating. In desperation more than I had a case of frozen kippers shipped from a company on the east coast, not really expecting anything special, but they are amazing. Shipped in a polystyrene box filled with dry ice, thawed and cooked in foil on the barbecue with a little butter – amazing. Kind of expensive but worth it.
How many ways can we as tax-payers protest at the payment of any single penny in bonuses to the employees of any institution which received a single penny of bail-out money. It is irrelevant whether they paid that money back or not – the fact is that without that bailout those companies would have closed down, and they would have ceased paying salaries, let alone bonuses. There is no valid argument about staff retention or incentives – the fact is that we should treat each of those companies as if they were bankrupt and their employees as recipients of government hand-outs. I say again – there is no, and indeed there never was one, valid reason to give them a penny of our money.
I’ve said before that I’m not really a fan of Google, but it seems to me that a lot of the reaction to their Chrome OS announcement has been overly negative. Much of it centres on the supposed limitations of the OS, especially with regard to running large complex apps. I think this is missing the point. The OS seems to be aimed at the netbook segment of the market – hardly the first choice to run large complex apps anyway. Netbook users want a quick way to access data stored on other machines, across the internet, and dare I use such a cliched term, in the cloud. It seems that the day of the Network Computer (which I was involved in over a decade ago) may yet be dawning. Back then the idea of remote data and applications was a little too revolutionary but now people are used to it. And anyhow – assuming Google base Chrome OS on linux and give it a good UI (the missing piece in all Linux distros to date – cf with what Apple did to BSD with OSX) then that alone may garner it sufficient support in the technical community to make it a success. And with that kind of basis, and a decent effort at a Windows emulator like Wine, it could be at least the start of a credible alternative to the Redmond offering. I’m not denying it’s a tough market to try and break into, but I really wouldn’t write Google’s efforts off at this early stage. The world is changing and Google are as likely as anyone to find a way to take advantage of those changes.
Here’s a prediction – the last remains of Microsoft will be the Exchange server. When their OS monopoly is a dim memory and their applications are treated a legacy apps, the Exchange server will still be serving corporate mail and calendars across the world. I’d say that the clients displaying said mail and calenders will be predominantly non-MS, but the server will still be there.
As promised, I’ve written a brief overview of jQuery for old farts experienced programmers like me, who just need a brief overview to get them started.
The main function of jQuery is to select elements of the HTML page’s DOM tree. The jQuery function itself does this, and its arguments are selectors to specify which elements to return. The jQuery function also has an alias – the $ sign can be used instead of typing out jQuery.
To select elements based on id, use
jQuery(#id)
or
$(#id)
to return all elements with id id (cf with document.getElementById which returns the first element with the given id.
To select elements based on class name use
jQuery(.className)
or
$(.className)
to find all elements with class name className.
Note how the .id and #className formats borrow from css.
Finally you can select by tag name, thusly:
$(tagName)
where tag name is the name of the HTML tag you’re searching for – $('a') to find all anchors, for example. These can be combined in many ways, for example:
$('a.home')
to find all anchors with the class ‘home’, or
$('div.share')
all divs with the id ’share’. Or even
$('a[@href^="http://www.example.com"]')
which returns all links whose href attribute starts with http://www.example.com. Note that the @ sign is used to represent attributes, in common with XPath.
So far, so simple. Most of this functionality is equivalent to existing JavaScript functions, for example document.getElementById, although in a much simpler and easy to use format, and with a rich query language built in. The result of the query isn’t just the JavaScript object or DOM element though – it is a jQuery object with a lot of additional functionality built in. In fact it can be an array of jQuery objects which can be acted on together, similarly to the resultSet from a SQL select. There are lots of methods that can be used on the result, but here are a few of them:
alert($('a$test').html())
to generate an alert displaying the text of the link element with the id test.
$('a').html('Link1')
to change the text of all links on the page to ‘Link!’.
$('li').css({color: 'red', backgroundColor: 'blue'})
to change the css attributes of all
var c = jQuery('li$id').css('color')
to save the color of the selected element.
The most powerful feature of jQuery though, I think, is its support for Ajax. To dynamically change the content of an element with an Ajax call, just do this:
$('div#placeholder').load('/external/address.html')
invokes a call to /external/address.html and puts the resulting HTML into the div with id placeholder. How simple is that. And there are event handlers to allow this basic functionality to be modified – for example displaying loading indicators or changinh attributes to indicate success.
Events are what makes all the magic happen in jQuery – you can hook on to events for clicks, hovers, button presses, etc. and invoke functionality appropriate to that event. The most basic is the document ready event, which is accessed like this:
$(document).ready()
This is invoked when the HTML DOM has been loaded. To do something useful on document ready, for example, we could do this:
$(document).ready(function() {
$("a#link1").click( function() {
$("p.vanish").toggle(100);return false;
});
});
And if that weren’t enough there are bunch of additional packages built on top of jQuery that give a lot of great functionality for free – two that I have used recently are a carousel control and a history package that allows the back and forward button to interact with Ajax based pages. I’ve got notes on both of these that I intend to post here sometime.
This is a very short introductory overview, I hope to be able to expand it in the future, but in conclusion jQuery is a very powerful package, and what’s more it’s actually fun to use. Highly recommended.
[ad]
For years now I’ve tried as much as possible to stay away from writing anything approaching a UI. This goes back to a product I’d built a prototype for at Borland when a junior product manager type was invited to a demo. I remember the guy turned up (barefoot, for some reason) and proceeded to demand that the UI be re-written as a wizard – that being the trendy paradigm at the time. I told him that that model wasn’t appropriate for my project (it would have been like having clippy the paper clip replace command mode in vi) and a bit of an argument broke out. We eventually did it my way, promising to “investigate” a wizard approach later (yeah right) and I left the company about a month after release and the product eventually died. I realised at that point that because the UI was the point of contact most people had with the code all the idiots would focus on that and demand to have their input as a way of making their mark. Form that point on I tried to keep to back end processing – databases, messaging, parsing, compiling; that kind of thing. And because the back end is generally regarded as the tougher discipline than the UI, it pretty much worked out. But now that kind of split is going the way of other specializations (I know a TV news sound guy who hasn’t worked worked for 5 years because the sound guy and cameraman were combined. Now of course the reporter often does the sound and the camera as well so there are two specialisms gone). Nowadays most development is based around a web site, and the specializations of front end and back end development are also going. Now you have to be able to do both. Not design, thankfully, of the world would be full of very ugly websites. But all the code that makes the thing work is often done by one person – Javascript and CSS on the client, JSP or PHP or whatever in the middle and Java, PHP or whatever on the back end. So I’ve been brushing up on Javascript (I once wrote a Javascript engine for a web browser so I used to know it quite well) and CSS and HTML and things. And I’ve got to say it’s not so bad. The guy who has to deal with the idiots is usually the designer and everything else is hidden from view. And all of this is really just a preamble to the point of all this, which is that I’ve recently discovered jQuery and it rocks. It does such a great job of abstracting out the interface to the HTML code from Javascript that it is actually a pleasure to develop with. For those who do not know it, it is a Javascript library that uses an xpath-type query language to access HTML elements in a declarative manner (by class name, type, id or a combination of each, and optionally taking into account the parent objects in the selection too, so you can select all radiobuttons that are children of a div with a certain id, for example), and allows operations to be performed on each matching result, often with a lot of built in convenience methods. So if, for example, you want to make a div appear when a button is pressed, you can write a handler for the button click that select the div and call show(’slow’) on it to make the div appear expanding slowly in the middle of the page. Which is all very cool. If I have a problem with jQuery it is that the mapping from the usual Javascript objects to the jQuery ones (for example, using the show method instead of accessing the style directly) isn’t too well documented. I’d guess that a lot of people started out using jQuery and never had to transition. So I’m going to try and create some tutorials for old-timers like me who know what they need to do but need some hints on the new way. So watch this space.
[ad]
I’ve been looking at some of the ratings Apple has assigned to the apps on its iPhone AppStore. Yesterday I found a recipe book rated 17+, which I assumed to be a mistake. Today I found several other recipe apps rated 12+ – the common theme seems to be anything that references alcohol gets you a 12+ minimum, with occasional 17+s probably depending on the reviewer. You have to wonder how many Mormon reviewers they’ve got. Although there are guides to making cocktails that are only rated 12+, so I really want to know what kicked that recipe book to 17+. Other random notes – a harmless looking CRM client (TopFloor HighRise Client) gets a 17+ for, amongst a long list of other things, “Frequent/Intense Alcohol, Tobacco, or Drug Use or References”. It doesn’t look like it’s CRM for crack dealers, but just maybe it is. Healthcare and fitness has a lot of quit smoking apps rated 17+, which is tough if you’re under 17 and want to stop smoking. In other categories referring to alcohol seems to be the best way to get a 17+ (although the rules seem inconsistent and some are rated 12+). The largest selection of 17+s is in the lifestyle section, where the dating and bikini apps are joined by one that creates Marijuana advocacy letters (presumably because that’s really hard to do when you’re stoned) and a couple of interfaces to Craig’s List. More as I find ‘em…
[ad]
The introduction of age ratings to the iPhone appstore recently made a lot of people think that some of the problems over app blocking that have made the headlines lately might be solved. After all, if I submit my app and say that it has an 17+ rating, shouldn’t Apple just take my word for it and post it without even rating it? A little simplistic perhaps but Apple really don’t need to get involved with censoring 50000 plus apps. The model just doesn’t scale. Unfortunately it looks like there may be few teething problems with the new system. Unless the app calling itself “Easy Recipes – Food & Drink” is hiding something that neither its description nor its many positive reviews are hinting it, it seems strange that it gets a 17+ (the highest) rating. The reason is apparently “Frequent/Intense Alcohol, Tobacco, or Drug Use or References”. I haven’t bought a copy of the app (I’m tempted but not that tempted) but I really want to see what they’re cooking. Or is it possible that a few too many recipes contained alcohol and either a very uptight employee or a too thorough automated scanner set the rating?
You can see the app for yourself in iTunes.
As I found out last week, the browse method of scraping the iTunes AppStore is limited by the fact that a maximum of 2500 apps are listed per category, or per sub-category in the case of games. And given that may apps are listed under multiple categories, it shoud be apparent that that method will not expose all the apps in the store. Therefore a different approach is required. The method I chose mimics selecting a category on the main AppStore front page in iTunes and the clicking ‘Next page’ until the last page is reached. This gives you twenty apps per page, so there are a lot of pages to click. For example, there are currently 301 pages of Books apps. In practice this doesn’t seem to be too slow, although it takes a lot more pages that the browse method.
So to get to the top page per category, the URL is:
http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/viewGenre?id=id
where id is the category id (eg 6018 for books). This returns the first page with the first 20 apps. Unfortunately it also includes a lot of extra bumph – mainly the top paid apps and top free apps sidebars. The first thing we need to find is the URL of the next page – it is hidden in a bunch of horrible looking XML something like this:
<HBoxView topInset="5" bottomInset="10" leftInset="0">
<TextView topInset="0" truncation="right" leftInset="0" stretchiness="0" styleSet="normal11Align" textJust="left" maxLines="1">
<SetFontStyle normalStyle="matrixTextFontStyle">
<B>Page 1 of 301</B>
</SetFontStyle>
</TextView>
<VBoxView alt="">
<View alt="" stretchiness="1"/>
<GotoURL target="main" url="http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/viewGenre?sortMode=2&id=6018&batchNumber=1">
<PictureButtonView leftInset="3" width="12" topInset="1" picts="plain,pressed,rollover" transparentClicks="1" alt="next page" url="/images/arrowoutline/arrow_000000_r.png" height="12"/>
</GotoURL>
<View alt="" stretchiness="1"/>
</VBoxView>
</HBoxView>
Got that? The next pages are the same as the root URL with a batchNumber attribute (0 indexed apparently). So you could just increment the index until you hit an error, or you could use an xpath query to read the next page URL from each page. I though the latter approach was less messy, so I did it that way. The xpath I used is:
/*[name()='Document']/*[name()='View']/*[name()='ScrollView']/*[name()='VBoxView']/*[name()='View']/*[name()='MatrixView']/*[name()='VBoxView']/*[name()='MatrixView']/*[name()='VBoxView']/*[name()='VBoxView']/*[name()='View']/*[name()='VBoxView']/*[name()='VBoxView']/*[name()='VBoxView']/*[name()='VBoxView']/*[name()='HBoxView']/*[name()='VBoxView']/*[name()='HBoxView']/*[name()='VBoxView']/*[name()='GotoURL']
which returns the node with the URL in the attribute “url” and the text “next page” in the alt attribute. You have to check that alt because on all pages but the first there is a similar node containing the link to the previous page before the link to the next one.
To read the apps on the page, the xpath
/*[name()='Document']/*[name()='View']/*[name()='ScrollView']/*[name()='VBoxView']/*[name()='View']/*[name()='MatrixView']/*[name()='VBoxView']/*[name()='MatrixView']/*[name()='VBoxView']/*[name()='VBoxView']/*[name()='View']/*[name()='VBoxView']/*[name()='VBoxView']/*[name()='VBoxView']/*[name()='MatrixView']/*[name()='HBoxView']/*[name()='VBoxView']/*[name()='MatrixView']
returns 20 nodes which contain links to the product detail pages, which can be read as before.
And that’s how to read all the apps on the AppStore. The big drawback of this method is that you don’t get category information – if you need that you’re going to have to traverse the browse links as well and create separate links to the apps you found on the front page browse.
Enjoy!
[ad]
I had assumed that in a browse URL such as http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/browse?path=/6014/7001/1, the last number was some kind of paging option, with each page returning up to 2500 apps. iTunes only seems to display that umber or items per category, and that’s the format of the URL it uses, so it made sense. But having actually tried it I find out that the xml returned is the same regardless of the number at the end. It returns an error if you don’t put a number, but put anything from 0 to 99 and you get the same list of apps. Which is kind of a pain, because that leaves a lot of apps unreachable. I can get to around 35000 using the browse method, but according to apptism there are currently around 49000 apps. The only way round this that I can see is to abandon the browse approach and scrape from the front page link for each category and page through 20 at a time. It’s probably going to be slow but I don’t see any choice at the moment. Of course I’ll report my findings here.
MacWorld have launched their appguide site for iPhone apps. I think this raises the bar on appstore scrapers – it displays a lot of information and makes it simple to find cool apps. And the editorial content adds a lot of value too – there are just too many apps out there to navigate without some kind of guidance. And they’re giving away iTunes gift cards for reviews – of course they can’t restrict it to purchasers only as Apple can so it’s open to abuse, but historically they’ve been pretty hard on spammers and abusers so we’ll have to give them the benefit of the doubt. Check it out.
[ad]