Comments on: iTunes AppStore scraping – decoding the browse URL

By: paul

paul — Thu, 02 Jul 2009 02:47:42 +0000

Hi Cyrille – yes both your points are correct. The 3500 was a finger slip (sorry) and the paging either stopped working or never worked. It just looked so logical – why else is that ‘/1′ at the end there? There is a further posting about an alternative approach – see here. You’ll still need to use this method if you want to associate categories with an app as well as a genre though so it’s not a total loss. Good luck!.
–Paul

By: Cyrillus

Cyrillus — Wed, 01 Jul 2009 19:47:23 +0000

Hi Paul,
It seems that the page browsing does not work — here’s the output of my in-project AppStore crawler :
Got category Business (#6000)
Downloading page 1… done, got 2500 items
Downloading page 2… done, got 2500 items (2500 duplicates)
Downloading page 3… done, got 2500 items (2500 duplicates)
Downloading page 4… done, got 2500 items (2500 duplicates)
Downloading page 5… done, got 2500 items (2500 duplicates)

And it goes on and on… Could it be that Apple disabled the page browsing ? (Also, the limit is set to 2500 and not 3500 as you say in the article)

By: paul

paul — Tue, 09 Jun 2009 20:07:04 +0000

Hi Tim – Yes I have confirmed that you do get duplicates, for just the reason you say – there are several apps that appear in multiple categories. They do have the same ID though so it’s no big deal. And if it’s important to track category as well as genre it’s the only way to find it because the apps don’t link back to all sub-categories (at least as far as I can see).

By: Tim

Tim — Tue, 09 Jun 2009 18:56:18 +0000

I suppose these could produce duplicates. I haven’t gone through the XML files to check that but if you go to the browse function in iTunes (which is was we’re doing with curl I suppose), go to Books and find “ABC Book”. But the genre for this one is actually “Education”.
If you now go to the “Education” category, you’ll find it there too.
There is no reason for the ID to be different though, so it should be easy to check.

By: paul

paul — Tue, 12 May 2009 16:38:20 +0000

Hi Peter – glad they’re useful. I know there’s not much information out there so I wanted to record everything I found. I’ll continue to update as I find out more.

Cheers.

–Paul

By: Peter B

Peter B — Tue, 12 May 2009 07:02:00 +0000

Paul,
I’m finding your iTunes appstore articles incredibly useful for a project I’m currently working on. Many thanks for sharing your findings in this area.
Cheers, Peter.