Archive for June, 2009

jQuery

Wednesday, June 24th, 2009

For years now I’ve tried as much as possible to stay away from writing anything approaching a UI. This goes back to a product I’d built a prototype for at Borland when a junior product manager type was invited to a demo. I remember the guy turned up (barefoot, for some reason) and proceeded to demand that the UI be re-written as a wizard – that being the trendy paradigm at the time. I told him that that model wasn’t appropriate for my project (it would have been like having clippy the paper clip replace command mode in vi) and a bit of an argument broke out. We eventually did it my way, promising to “investigate” a wizard approach later (yeah right) and I left the company about a month after release and the product eventually died. I realised at that point that because the UI was the point of contact most people had with the code all the idiots would focus on that and demand to have their input as a way of making their mark. Form that point on I tried to keep to back end processing – databases, messaging, parsing, compiling; that kind of thing. And because the back end is generally regarded as the tougher discipline than the UI, it pretty much worked out. But now that kind of split is going the way of other specializations (I know a TV news sound guy who hasn’t worked worked for 5 years because the sound guy and cameraman were combined. Now of course the reporter often does the sound and the camera as well so there are two specialisms gone). Nowadays most development is based around a web site, and the specializations of front end and back end development are also going. Now you have to be able to do both. Not design, thankfully, of the world would be full of very ugly websites. But all the code that makes the thing work is often done by one person – Javascript and CSS on the client, JSP or PHP or whatever in the middle and Java, PHP or whatever on the back end. So I’ve been brushing up on Javascript (I once wrote a Javascript engine for a web browser so I used to know it quite well) and CSS and HTML and things. And I’ve got to say it’s not so bad. The guy who has to deal with the idiots is usually the designer and everything else is hidden from view. And all of this is really just a preamble to the point of all this, which is that I’ve recently discovered jQuery and it rocks. It does such a great job of abstracting out the interface to the HTML code from Javascript that it is actually a pleasure to develop with. For those who do not know it, it is a Javascript library that uses an xpath-type query language to access HTML elements in a declarative manner (by class name, type, id or a combination of each, and optionally taking into account the parent objects in the selection too, so you can select all radiobuttons that are children of a div with a certain id, for example), and allows operations to be performed on each matching result, often with a lot of built in convenience methods. So if, for example, you want to make a div appear when a button is pressed, you can write a handler for the button click that select the div and call show(’slow’) on it to make the div appear expanding slowly in the middle of the page. Which is all very cool. If I have a problem with jQuery it is that the mapping from the usual Javascript objects to the jQuery ones (for example, using the show method instead of accessing the style directly) isn’t too well documented. I’d guess that a lot of people started out using jQuery and never had to transition. So I’m going to try and create some tutorials for old-timers like me who know what they need to do but need some hints on the new way. So watch this space.
[ad]

More AppStore Age Ratings

Thursday, June 18th, 2009

I’ve been looking at some of the ratings Apple has assigned to the apps on its iPhone AppStore. Yesterday I found a recipe book rated 17+, which I assumed to be a mistake. Today I found several other recipe apps rated 12+ – the common theme seems to be anything that references alcohol gets you a 12+ minimum, with occasional 17+s probably depending on the reviewer. You have to wonder how many Mormon reviewers they’ve got. Although there are guides to making cocktails that are only rated 12+, so I really want to know what kicked that recipe book to 17+. Other random notes – a harmless looking CRM client (TopFloor HighRise Client) gets a 17+ for, amongst a long list of other things, “Frequent/Intense Alcohol, Tobacco, or Drug Use or References”. It doesn’t look like it’s CRM for crack dealers, but just maybe it is. Healthcare and fitness has a lot of quit smoking apps rated 17+, which is tough if you’re under 17 and want to stop smoking. In other categories referring to alcohol seems to be the best way to get a 17+ (although the rules seem inconsistent and some are rated 12+). The largest selection of 17+s is in the lifestyle section, where the dating and bikini apps are joined by one that creates Marijuana advocacy letters (presumably because that’s really hard to do when you’re stoned) and a couple of interfaces to Craig’s List. More as I find ‘em…
[ad]

AppStore age ratings

Wednesday, June 17th, 2009

The introduction of age ratings to the iPhone appstore recently made a lot of people think that some of the problems over app blocking that have made the headlines lately might be solved. After all, if I submit my app and say that it has an 17+ rating, shouldn’t Apple just take my word for it and post it without even rating it? A little simplistic perhaps but Apple really don’t need to get involved with censoring 50000 plus apps. The model just doesn’t scale. Unfortunately it looks like there may be few teething problems with the new system. Unless the app calling itself “Easy Recipes – Food & Drink” is hiding something that neither its description nor its many positive reviews are hinting it, it seems strange that it gets a 17+ (the highest) rating. The reason is apparently “Frequent/Intense Alcohol, Tobacco, or Drug Use or References”. I haven’t bought a copy of the app (I’m tempted but not that tempted) but I really want to see what they’re cooking. Or is it possible that a few too many recipes contained alcohol and either a very uptight employee or a too thorough automated scanner set the rating?

You can see the app for yourself in iTunes.

AppStore scraping – the front door method

Monday, June 15th, 2009

As I found out last week, the browse method of scraping the iTunes AppStore is limited by the fact that a maximum of 2500 apps are listed per category, or per sub-category in the case of games. And given that may apps are listed under multiple categories, it shoud be apparent that that method will not expose all the apps in the store. Therefore a different approach is required. The method I chose mimics selecting a category on the main AppStore front page in iTunes and the clicking ‘Next page’ until the last page is reached. This gives you twenty apps per page, so there are a lot of pages to click. For example, there are currently 301 pages of Books apps. In practice this doesn’t seem to be too slow, although it takes a lot more pages that the browse method.
So to get to the top page per category, the URL is:

http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/viewGenre?id=id

where id is the category id (eg 6018 for books). This returns the first page with the first 20 apps. Unfortunately it also includes a lot of extra bumph – mainly the top paid apps and top free apps sidebars. The first thing we need to find is the URL of the next page – it is hidden in a bunch of horrible looking XML something like this:

<HBoxView topInset="5" bottomInset="10" leftInset="0">
<TextView topInset="0" truncation="right" leftInset="0" stretchiness="0" styleSet="normal11Align" textJust="left" maxLines="1">
<SetFontStyle normalStyle="matrixTextFontStyle">
<B>Page 1 of 301</B>
</SetFontStyle>
</TextView>
<VBoxView alt="">
<View alt="" stretchiness="1"/>
<GotoURL target="main" url="http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/viewGenre?sortMode=2&id=6018&batchNumber=1">
<PictureButtonView leftInset="3" width="12" topInset="1" picts="plain,pressed,rollover" transparentClicks="1" alt="next page" url="/images/arrowoutline/arrow_000000_r.png" height="12"/>
</GotoURL>
<View alt="" stretchiness="1"/>
</VBoxView>
</HBoxView>

Got that? The next pages are the same as the root URL with a batchNumber attribute (0 indexed apparently). So you could just increment the index until you hit an error, or you could use an xpath query to read the next page URL from each page. I though the latter approach was less messy, so I did it that way. The xpath I used is:

/*[name()='Document']/*[name()='View']/*[name()='ScrollView']/*[name()='VBoxView']/*[name()='View']/*[name()='MatrixView']/*[name()='VBoxView']/*[name()='MatrixView']/*[name()='VBoxView']/*[name()='VBoxView']/*[name()='View']/*[name()='VBoxView']/*[name()='VBoxView']/*[name()='VBoxView']/*[name()='VBoxView']/*[name()='HBoxView']/*[name()='VBoxView']/*[name()='HBoxView']/*[name()='VBoxView']/*[name()='GotoURL']

which returns the node with the URL in the attribute “url” and the text “next page” in the alt attribute. You have to check that alt because on all pages but the first there is a similar node containing the link to the previous page before the link to the next one.

To read the apps on the page, the xpath

/*[name()='Document']/*[name()='View']/*[name()='ScrollView']/*[name()='VBoxView']/*[name()='View']/*[name()='MatrixView']/*[name()='VBoxView']/*[name()='MatrixView']/*[name()='VBoxView']/*[name()='VBoxView']/*[name()='View']/*[name()='VBoxView']/*[name()='VBoxView']/*[name()='VBoxView']/*[name()='MatrixView']/*[name()='HBoxView']/*[name()='VBoxView']/*[name()='MatrixView']

returns 20 nodes which contain links to the product detail pages, which can be read as before.

And that’s how to read all the apps on the AppStore. The big drawback of this method is that you don’t get category information – if you need that you’re going to have to traverse the browse links as well and create separate links to the apps you found on the front page browse.

Enjoy!

[ad]

AppStore scraping – back to the drawing board

Tuesday, June 9th, 2009

I had assumed that in a browse URL such as http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/browse?path=/6014/7001/1, the last number was some kind of paging option, with each page returning up to 2500 apps. iTunes only seems to display that umber or items per category, and that’s the format of the URL it uses, so it made sense. But having actually tried it I find out that the xml returned is the same regardless of the number at the end. It returns an error if you don’t put a number, but put anything from 0 to 99 and you get the same list of apps. Which is kind of a pain, because that leaves a lot of apps unreachable. I can get to around 35000 using the browse method, but according to apptism there are currently around 49000 apps. The only way round this that I can see is to abandon the browse approach and scrape from the front page link for each category and page through 20 at a time. It’s probably going to be slow but I don’t see any choice at the moment. Of course I’ll report my findings here.

MacWorld AppGuide

Wednesday, June 3rd, 2009

MacWorld have launched their appguide site for iPhone apps. I think this raises the bar on appstore scrapers – it displays a lot of information and makes it simple to find cool apps. And the editorial content adds a lot of value too – there are just too many apps out there to navigate without some kind of guidance. And they’re giving away iTunes gift cards for reviews – of course they can’t restrict it to purchasers only as Apple can so it’s open to abuse, but historically they’ve been pretty hard on spammers and abusers so we’ll have to give them the benefit of the doubt. Check it out.
[ad]