Archive for the ‘Development’ Category

AppStore Rejection

Monday, December 21st, 2009

I had an iPhone app in review which was intended to promote the re-release of Tony Jacklin’s interview “Jacklin Speaks“. It’s very similar for the one I made for the Bill Shankly interview, “Shankly Speaks“. The Shankly version was initially rejected because it did not close down if there was no internet connection. To fix this I added an error message and a call to

[[UIApplication sharedApplication] terminate];

if there was an error reading from the internet. I resubmitted the app and it was accepted. The new app contains exactly the same code, but it was rejected for using an undocumented api, namely terminate.

I hadn’t realised the terminate was undocumented (it is documented for its desktop version, but not for the iphone). The alternative is apparently to use the standard C call exit(0), although that bypasses application shutdown code and may result in memory leaks and other bad behaviour. However, since that call is documented, I’ve made that change and resubmitted. What they say to that will be reported here.

UPDATE 12/23 – It’s been approved. It is (or shortly will be) available here.

jQuery for Programmers

Thursday, July 2nd, 2009

As promised, I’ve written a brief overview of jQuery for old farts experienced programmers like me, who just need a brief overview to get them started.

The main function of jQuery is to select elements of the HTML page’s DOM tree. The jQuery function itself does this, and its arguments are selectors to specify which elements to return. The jQuery function also has an alias – the $ sign can be used instead of typing out jQuery.
To select elements based on id, use

jQuery(#id)
or
$(#id)

to return all elements with id id (cf with document.getElementById which returns the first element with the given id.
To select elements based on class name use

jQuery(.className)
or
$(.className)

to find all elements with class name className.

Note how the .id and #className formats borrow from css.

Finally you can select by tag name, thusly:

$(tagName)

where tag name is the name of the HTML tag you’re searching for – $('a') to find all anchors, for example. These can be combined in many ways, for example:

$('a.home')

to find all anchors with the class ‘home’, or

$('div.share')

all divs with the id ’share’. Or even

$('a[@href^="http://www.example.com"]')

which returns all links whose href attribute starts with http://www.example.com. Note that the @ sign is used to represent attributes, in common with XPath.

So far, so simple. Most of this functionality is equivalent to existing JavaScript functions, for example document.getElementById, although in a much simpler and easy to use format, and with a rich query language built in. The result of the query isn’t just the JavaScript object or DOM element though – it is a jQuery object with a lot of additional functionality built in. In fact it can be an array of jQuery objects which can be acted on together, similarly to the resultSet from a SQL select. There are lots of methods that can be used on the result, but here are a few of them:

alert($('a$test').html())

to generate an alert displaying the text of the link element with the id test.

$('a').html('Link1')

to change the text of all links on the page to ‘Link!’.

$('li').css({color: 'red', backgroundColor: 'blue'})

to change the css attributes of all

  • items.

    var c = jQuery('li$id').css('color')

    to save the color of the selected element.

    The most powerful feature of jQuery though, I think, is its support for Ajax. To dynamically change the content of an element with an Ajax call, just do this:

    $('div#placeholder').load('/external/address.html')

    invokes a call to /external/address.html and puts the resulting HTML into the div with id placeholder. How simple is that. And there are event handlers to allow this basic functionality to be modified – for example displaying loading indicators or changinh attributes to indicate success.

    Events are what makes all the magic happen in jQuery – you can hook on to events for clicks, hovers, button presses, etc. and invoke functionality appropriate to that event. The most basic is the document ready event, which is accessed like this:

    $(document).ready()

    This is invoked when the HTML DOM has been loaded. To do something useful on document ready, for example, we could do this:

    $(document).ready(function() {
    $("a#link1").click( function() {
    $("p.vanish").toggle(100);return false;
    });
    });

    And if that weren’t enough there are bunch of additional packages built on top of jQuery that give a lot of great functionality for free – two that I have used recently are a carousel control and a history package that allows the back and forward button to interact with Ajax based pages. I’ve got notes on both of these that I intend to post here sometime.

    This is a very short introductory overview, I hope to be able to expand it in the future, but in conclusion jQuery is a very powerful package, and what’s more it’s actually fun to use. Highly recommended.

    [ad]

  • jQuery

    Wednesday, June 24th, 2009

    For years now I’ve tried as much as possible to stay away from writing anything approaching a UI. This goes back to a product I’d built a prototype for at Borland when a junior product manager type was invited to a demo. I remember the guy turned up (barefoot, for some reason) and proceeded to demand that the UI be re-written as a wizard – that being the trendy paradigm at the time. I told him that that model wasn’t appropriate for my project (it would have been like having clippy the paper clip replace command mode in vi) and a bit of an argument broke out. We eventually did it my way, promising to “investigate” a wizard approach later (yeah right) and I left the company about a month after release and the product eventually died. I realised at that point that because the UI was the point of contact most people had with the code all the idiots would focus on that and demand to have their input as a way of making their mark. Form that point on I tried to keep to back end processing – databases, messaging, parsing, compiling; that kind of thing. And because the back end is generally regarded as the tougher discipline than the UI, it pretty much worked out. But now that kind of split is going the way of other specializations (I know a TV news sound guy who hasn’t worked worked for 5 years because the sound guy and cameraman were combined. Now of course the reporter often does the sound and the camera as well so there are two specialisms gone). Nowadays most development is based around a web site, and the specializations of front end and back end development are also going. Now you have to be able to do both. Not design, thankfully, of the world would be full of very ugly websites. But all the code that makes the thing work is often done by one person – Javascript and CSS on the client, JSP or PHP or whatever in the middle and Java, PHP or whatever on the back end. So I’ve been brushing up on Javascript (I once wrote a Javascript engine for a web browser so I used to know it quite well) and CSS and HTML and things. And I’ve got to say it’s not so bad. The guy who has to deal with the idiots is usually the designer and everything else is hidden from view. And all of this is really just a preamble to the point of all this, which is that I’ve recently discovered jQuery and it rocks. It does such a great job of abstracting out the interface to the HTML code from Javascript that it is actually a pleasure to develop with. For those who do not know it, it is a Javascript library that uses an xpath-type query language to access HTML elements in a declarative manner (by class name, type, id or a combination of each, and optionally taking into account the parent objects in the selection too, so you can select all radiobuttons that are children of a div with a certain id, for example), and allows operations to be performed on each matching result, often with a lot of built in convenience methods. So if, for example, you want to make a div appear when a button is pressed, you can write a handler for the button click that select the div and call show(’slow’) on it to make the div appear expanding slowly in the middle of the page. Which is all very cool. If I have a problem with jQuery it is that the mapping from the usual Javascript objects to the jQuery ones (for example, using the show method instead of accessing the style directly) isn’t too well documented. I’d guess that a lot of people started out using jQuery and never had to transition. So I’m going to try and create some tutorials for old-timers like me who know what they need to do but need some hints on the new way. So watch this space.
    [ad]

    AppStore scraping – back to the drawing board

    Tuesday, June 9th, 2009

    I had assumed that in a browse URL such as http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/browse?path=/6014/7001/1, the last number was some kind of paging option, with each page returning up to 2500 apps. iTunes only seems to display that umber or items per category, and that’s the format of the URL it uses, so it made sense. But having actually tried it I find out that the xml returned is the same regardless of the number at the end. It returns an error if you don’t put a number, but put anything from 0 to 99 and you get the same list of apps. Which is kind of a pain, because that leaves a lot of apps unreachable. I can get to around 35000 using the browse method, but according to apptism there are currently around 49000 apps. The only way round this that I can see is to abandon the browse approach and scrape from the front page link for each category and page through 20 at a time. It’s probably going to be slow but I don’t see any choice at the moment. Of course I’ll report my findings here.

    Apple Blocking Access to the AppStore?

    Wednesday, May 27th, 2009

    I got a chance to investigate the strange access problems I found last week trying to scrape the iPhone Appstore. It looks to me like something has definitely changed on the server, but it’s hard to see what. My original script used curl with its default user string, and that seemed to suffer timeout problems on every page. So I by changed the agent string to the firefox one, which seemed to result in an immediate improvement, but it too started to suffer slowdowns as it progressed through the store. Finally I changed all timeouts to 5 minutes and it looks like every call returned successfully. So all I can think of is that requests that aren’t from the iPhone or iTunes are being served, but they’re being sent to the back of the queue. I don’t see much logic in all this, but the Apple do move in mysterious ways sometimes, and it’s always possible it’s a quirk rather than a policy decision. But the good news is that access is still being permitted at some level and we can go on cutting and dicing appstore content into something useful.

    Apple blocking curl from the Appstore?

    Tuesday, May 19th, 2009

    Not quite sure what’s going on with the AppStore. I just resumed my experiments and it appears that a couple of things have changed. Firstly calls from curl seem to be blocked – although changing the user agent seems to get round that. Why they would impose such a trivially bypassed hurdle is a bit of a mystery – surely if there is a target of a block there are better ways to keep them out, like ip address blocking. It is interesting that they aren’t moving to impose a total block from non-iTunes clients though, clearly that is a tacit admission that they are allowing store scraping at some level. More seriously, some of the browse URLs I was using previously don’t appear to work any more. I’m sure I can figure out what’s going on but I’m going to need more time than I have now to investigate. I’ll post back as soon as I figure it out.

    [ad#co-1]

    iTunes AppStore scraping – decoding the browse URL

    Monday, May 11th, 2009

    Further to my recent posts covering scraping the itunes appstore – I have made some progress towards decoding the browse URL that returns the list of apps by category. There is a slight wrinkle with categories that have sub-categories (currently only games) and a potential work-around to the 3500-per-page limit.

    The browse URL breaks down to this:

    http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/browse?path=/category/subcategory/page

    The top level browse URL, ie

    http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/browse

    on its own gives a list of top level categories and their associated ids- eg TV shows is 32, Music videos is 31, Music is 34 and AppStore is 36.

    So to browse a category from the root, you append the URL with the query string path=/id. Ie the AppStore URL is

    http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/browse?path=/36

    which returns a list of AppStore categories and their ids – Weather = 6001, Travel = 6003, Games = 6014, etc.

    Then, to browse all weather apps the URL is

    http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/browse?path=/36/6001/1

    where the final 1 seems to be a paging control – so where there are > 3500 apps you can increment the last number to retrieve the next set of app details.

    Where there are subcategories, they can be accessed by replacing the top level id with that of the category – so to browse all games subcategories the URL is

    http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/browse?path=/6014

    which returns the names and ids of the games subcategories (Action = 7001, Adventure = 7002, and so on). Then to browse the action games the URL becomes

    http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/browse?path=/6014/7001/1

    It looks to me that currently if the tree is traversed from the root until the list of subcategories returns an empty list, and then the leaf node is used to retrieve the apps, there are no need for paging with a value of greater than 1. This is also the only method I can see for determining which subcategory an app is listed under – the apps themselves link to the category and a genre but not a subcategory. I also don’t know right now if this will produce multiple instances of the same app – ie if an app can appear under multiple subcategories.

    [ad#co-1]

    Scraping iTunes App Store part iv – reading application details

    Tuesday, April 28th, 2009

    So we now have the page containing all (or the first 3500) applications for each category. To read details of individual apps I used the following XPath query -


    /*[name()='Document']/*[name()='View']/*[name()='ScrollView']/*[name()='VBoxView']/*[name()='View']/*[name()='MatrixView']/*[name()='MatrixView']/*[name()='VBoxView']/*[name()='VBoxView']/*[name()='TextView']"

    Each node of this contains a long list of name/value pairs as shown in my previous post. Some of the fields are:

    • artistId – The unique id of the app developer
    • artistName – A string containing the name of the developer.
    • genre – the name of the genre to which the app is assigned
    • genreId – the numeric id of the genre
    • itemId – the unique id used to identify the app throughout iTunes
    • itemName – the name of the app
    • kind – always “software”as far as I can see
    • popularity – a ranking indicator. Not quite sure how this is calculated right now
    • price – the price in tenths of a cent
    • priceDisplay – the price as a formatted string
    • releaseDate
    • softwareIcon57×57URL – the URL of the app’s icon
    • url – the URL to view the app in iTunes

    [ad#co-1]

    Scraping iTunes part iii – reading the categories list

    Thursday, April 23rd, 2009

    My last post on scraping the iTunes store showed how to read the categories page. We had a URL something like this:
    http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/browse?path=/36/6081/1/
    (for books). Sending that in produces a large xml response – the interesting parts look like this:

    <dict>
    <key>artistId</key>
    <integer>293260414</integer>
    <key>artistName</key>
    <string>Saxorama.net</string>
    <key>buy-only</key>
    <true/>
    <key>buyParams</key>
    <string>
    productType=C&salableAdamId=294770918&pricingParameters=STDQ&price=0&ct-id=14
    </string>
    <key>genre</key>
    <string>Productivity</string>
    <key>genreId</key>
    <integer>6007</integer>
    <key>itemId</key>
    <integer>294770918</integer>
    <key>itemName</key>
    <string>EasyWriter</string>
    <key>kind</key>
    <string>software</string>
    <key>playlistName</key>
    <string>EasyWriter</string>
    <key>popularity</key>
    <string>0.13890815</string>
    <key>price</key>
    <integer>0</integer>
    <key>priceDisplay</key>
    <string>Free</string>
    <key>releaseDate</key>
    <string>2009-04-06T07:00:00Z</string>
    <key>s</key>
    <integer>143441</integer>
    <key>softwareIcon57x57URL</key>
    <string>

    http://a1.phobos.apple.com/us/r30/Purple/40/ce/15/mzl.dtewfrse.png

    </string>
    <key>softwareIconNeedsShine</key>
    <false/>
    <key>softwareSupportedDeviceIds</key>
    <array>
    <integer>1</integer>
    </array>
    <key>softwareVersionBundleId</key>
    <string>net.sax.easywriter</string>
    <key>softwareVersionExternalIdentifier</key>
    <integer>1589121</integer>
    <key>softwareVersionExternalIdentifiers</key>
    <array>
    <integer>875361</integer>
    <integer>1472572</integer>
    <integer>1486886</integer>
    <integer>1589121</integer>
    </array>
    <key>url</key>
    <string>

    http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=294770918&mt=8

    </string>
    </dict>

    This is a set of summary details for one product, and there will be up to 3500 of them on the page that we just downloaded. While the information here is useful, we need to go to another page to get the complete set of information for the produce. We do this by extracting a URL using an xpath statement, which I will post next time.
    [ad#co-1]

    Scraping the iTunes AppStore part ii – categories

    Tuesday, April 21st, 2009

    My previous post on scraping the iTunes Appstore showed how to retrieve the top level page for iPhone apps – basically a list of the available categories and their associated IDs. The next step is to get a list of the applications in each category.
    The previous call gave us a url:
    http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/viewGenre?id=
    and a list of name/value pairs like this:

    <dict>
    <key>itemName</key><string>Books</string>
    <key>itemId</key><integer>6018</integer>
    </dict>

    It looks like we need to combine the URL we just received with the ids to get a page containing links to all the apps in that category, so for all books we would use-
    http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/viewGenre?id=6081
    but unfortunately I couldn’t get this to work. I have no idea what that URL is doing there. Instead I saw that iTunes was using a variant of the original URL, and sending this:
    http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/browse?path=/36/6081/1/
    This indeed returns a list of applications belonging to the requested category, although it seems to be limited to 3500 results per category That is enough for most categories right now. As the store gets bigger more categories will exceed this – I would then expect new categories to be added, or the limit to be raised. If you really need all the apps in the store, you’ll have to use one of teh other approaches I mentioned earlier.
    Again, the response is an xml message. It’s quite a complicated one, but my next post on the subject will show you how to get details of individual applications.
    [ad#co-1]