Paul Hinks' Blog

Many years ago I worked for a company called GrandCentral, which was the most dysfunctional organization I have ever encountered, and believe me that’s a pretty tough list to top. Anyhow, through a series of transformations, that company is now a part of Google and its services are called Google Voice. It’s currently in beta, and you have to apply for an account, kind of like you did for gmail in the early days. I recently got myself an account and I’ve been playing with it to see what it offers.
Which is actually quite a lot. You get a (free) telephone number of your choosing, in one of several area codes (I got a 415 number with no problem). Calls to that number can be forwarded to various other numbers of your choosing, and different callers can be forwarded to different phones or sent straight to voice mail. When you get a voice mail, you get an email notifying you of the fact along with a transcript of the message. I tested leaving myself a message and the transcription seems to work pretty well – when it did mess up it was obvious enough what the meaning was that it wasn’t a problem. Overall I’m very impressed. I’m wondering what it will be like when anybody can sign up – there can’t be an unlimited supply of phone numbers, and what’s to stop people signing up for multiple accounts with different phone numbers (in different areas, for example, to maintain a virtual presence in different area codes). It would presumably be simple to prevent people chaining GC accounts (redirecting calls from one account to another) but with an intermediary voip provider those restrictions would be simple to circumvent. I’m also wondering about phone number portability – say I want to move my phone number onto GV, or off of it when it’s become my established contact number. Lots of questions, but overall it’s a good service and I predict another success for google.

I just went through this and it was a little painful due to incomplete documentation, so I’m summarizing the steps I took here – mainly for my own reference but also in case anybody else needs to do the same thing. Note – this does not cover using Apache to serve the repository remotely, this is a local install only. I’ll deal with Apache when I get round to it.

1. Installation
yum install mod_dav_svn subversion

This grabs the basic subversion packages, and resolve any other dependencies you may have.
When complete try typing:
svn -help
to make sure that all is well and that svn runs.

2. Configuration
Now you can create the actual repository from which you will check in and out your files.

mkdir /var/svn
cd /var/svn
svnadmin create repos

3. Creating projects
The recommended repository directory layout is as follows:

|– project1
| |– branches
| |– tags
| `– trunk
`– project2
|– branches
|– tags
`– trunk

To start with, we’re going to create a simple project:

mkdir proj1
cd proj1
mkdir trunk tags branches
vi trunk/main.c

Edit main.c to contain your code. Now we can add this project to svn.

3.1. Importing
To add your code to the repository do this:

cd ..
svn import proj1 file:///var/svn/repos/proj1 -m “Initial checkin for proj1″

3.2 Checking Out
To create a working copy of the project, you need to check it out. To do this, do:

mkdir work
cd work
svn co file:///var/svn/repos/proj1

3.3 Edit and Update
Modify the copy of main.c you just checked out:

vi trunk/main.c — Add or delete something and save.

And then check in your saved file:

svn commit -m “Modified main.c”

Similarly, if you add a new file to the project, this command will add them to the repository. To delete a file use the delete command:

svn delete trunk/main.c

To recover a previous version of a file, co with the revision number:

svn co -r file:///var/svn/repos/proj1

If you’re not sure which revision number, check the logs:

svn log file:///var/svn/repos

or

svn log file:///var/svn/repos

4. And more

This should be enough to get you started, but there are lots more commands to explore, and of course this is just a local install – svn becomes really powerful when it is set up as a remote server. I’m going to wade through the docs to do that myself next and I’ll record the results here. The svn bible is :Version Control with Subversion.
[ad]

As I found out last week, the browse method of scraping the iTunes AppStore is limited by the fact that a maximum of 2500 apps are listed per category, or per sub-category in the case of games. And given that may apps are listed under multiple categories, it shoud be apparent that that method will not expose all the apps in the store. Therefore a different approach is required. The method I chose mimics selecting a category on the main AppStore front page in iTunes and the clicking ‘Next page’ until the last page is reached. This gives you twenty apps per page, so there are a lot of pages to click. For example, there are currently 301 pages of Books apps. In practice this doesn’t seem to be too slow, although it takes a lot more pages that the browse method.
So to get to the top page per category, the URL is:

http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/viewGenre?id=id

where id is the category id (eg 6018 for books). This returns the first page with the first 20 apps. Unfortunately it also includes a lot of extra bumph – mainly the top paid apps and top free apps sidebars. The first thing we need to find is the URL of the next page – it is hidden in a bunch of horrible looking XML something like this:

<HBoxView topInset="5" bottomInset="10" leftInset="0">
<TextView topInset="0" truncation="right" leftInset="0" stretchiness="0" styleSet="normal11Align" textJust="left" maxLines="1">
<SetFontStyle normalStyle="matrixTextFontStyle">
<B>Page 1 of 301</B>
</SetFontStyle>
</TextView>
<VBoxView alt="">
<View alt="" stretchiness="1"/>
<GotoURL target="main" url="http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/viewGenre?sortMode=2&id=6018&batchNumber=1">
<PictureButtonView leftInset="3" width="12" topInset="1" picts="plain,pressed,rollover" transparentClicks="1" alt="next page" url="/images/arrowoutline/arrow_000000_r.png" height="12"/>
</GotoURL>
<View alt="" stretchiness="1"/>
</VBoxView>
</HBoxView>


Got that? The next pages are the same as the root URL with a batchNumber attribute (0 indexed apparently). So you could just increment the index until you hit an error, or you could use an xpath query to read the next page URL from each page. I though the latter approach was less messy, so I did it that way. The xpath I used is:

/*[name()='Document']/*[name()='View']/*[name()='ScrollView']/*[name()='VBoxView']/*[name()='View']/*[name()='MatrixView']/*[name()='VBoxView']/*[name()='MatrixView']/*[name()='VBoxView']/*[name()='VBoxView']/*[name()='View']/*[name()='VBoxView']/*[name()='VBoxView']/*[name()='VBoxView']/*[name()='VBoxView']/*[name()='HBoxView']/*[name()='VBoxView']/*[name()='HBoxView']/*[name()='VBoxView']/*[name()='GotoURL']

which returns the node with the URL in the attribute “url” and the text “next page” in the alt attribute. You have to check that alt because on all pages but the first there is a similar node containing the link to the previous page before the link to the next one.

To read the apps on the page, the xpath

/*[name()='Document']/*[name()='View']/*[name()='ScrollView']/*[name()='VBoxView']/*[name()='View']/*[name()='MatrixView']/*[name()='VBoxView']/*[name()='MatrixView']/*[name()='VBoxView']/*[name()='VBoxView']/*[name()='View']/*[name()='VBoxView']/*[name()='VBoxView']/*[name()='VBoxView']/*[name()='MatrixView']/*[name()='HBoxView']/*[name()='VBoxView']/*[name()='MatrixView']

returns 20 nodes which contain links to the product detail pages, which can be read as before.

And that’s how to read all the apps on the AppStore. The big drawback of this method is that you don’t get category information – if you need that you’re going to have to traverse the browse links as well and create separate links to the apps you found on the front page browse.

Enjoy!

[ad]

One fairly persistent issue I have is some people I know insist on sending me emails from Windows mail (apparently Outlook is the worst offender) with the ‘Use Windows mail format’ option checked. This anti-social behaviour results in non-Windows users receiving attachments packaged up into a file called winmail.dat which they can’t open. Apparently this is just a wrapper around standard attachments that could be handled ok but MS prefer to use their own ’standards’ instead. Sometimes you can ask the mail sender to send the message in a standard format, but this is not always possible. I have done a little investigation and have discovered a few ways to deal with the problem.

Firstly, the OSX Mail app has an option to try viewing messages as alternate types – the option is under View|Message|Next Alternative option. This often manages to decode an attachment into plain text. If this fails there is a free application called Enough that you can drop the winmail.dat file onto and it seems to do a pretty good job of extracting the attachments from it. And finally there is a plug-in for Mail called Letter Opener which is expensive but handles the conversion inside Mail and make the attachments appear where they should be in the message. I’m guessing it’s expensive because it does a bunch of other things too – calendar conversion for one. It looks like it would be fairly to implement the attachment processing functionality on its own – I’m adding it to my to-do list of things to do on a quiet evening sometime.

I’ve been meaning to post this for about a week now, but I’m sure the information is still accurate. GoGrid have a $50 coupon on their site that’s easy to find and works out at about a month’s free basic service. I got an email from them saying that to get a $100 credit you just need to call 877.946.4742 or +1 415.869.7444 and ask for one. I guess they think the extra details they can grab from you in person is worth the extra cash. If they ask where you got the number say you got forwarded the StartupSF email.

Had a discussion today with one of those people who gets all defensive when you mention that you use a Mac. I haven’t had one of those conversations in a while – most people don’t care, or they already know that the Mac is a better choice. I’ve been using a Mac for the best part of a decade now (since soon after the release of OSX in 2001) and the arguments started out being frequent and kind of fun quite soon became less frequent but rather tedious. I’m not going to re-hash the arguments here – the perceived higher cost (or “Apple tax” – c.f. “Microsoft tax”) and software compatibility issues, and even the use of Parallels as a case *against* a Mac (“you’ll only use it to run Windows anyway”) are incredibly boring and totally beside the point. I was a Windows developer for years. I know all about Windows. I still have to work on Windows for clients once in a while – and no I don’t try to convert them, unless there’s a pretty good reason to. I use a Mac because I like it – it does everything I need (and really well to boot) and I like the way it works. What really frustrates me is the implication that the Mac is trendy and therefore only people who can’t think for themselves would possibly use one. This kind of argument really annoys me, not least because I have met it lots of times over the years for different reasons – for example when I first started working on computers PCs were regarded as a fad (yes I’m old) and I was regarded with suspicion for advocating them over the minis and mainframes that the companies I worked in were using. I was also a strong advocate of Windows when that was released. On thing I will say is that back when they were considered subversive PCs were adopted by a small subset of leading edge opinion formers (mainly the more tech-savvy user and developer) and they gradually spread as people cottoned on to their advantages. I see a similar movement now with regard to people adopting Macs – many software developers, especially those on the leading edge of web development. use Macs as standard. I thought the argument was pretty much over, to be honest. I think the PC user now is like the Vax user I upset years ago when I said his beloved machine would be dead in five years time. Not that the PC will necessarily be dead in 5 years, but it’s certainly no longer the future of the industry. But as I said all this is boring. The next time someone questions my choice of OS I shall merely smile politely and let it pass.

Interesting news from GoGrid today – they are adding two new features which actually go some way towards addressing the cloud computing issues I have mentioned before. Firstly they are allowing user-defined images to be uploaded, from which multiple virtual servers with varying memory allotments; this means that I can clone my entire application stack and deploy multiple copies of it in minutes. They are also connecting their Colocation servers to their cloud servers, so it will be possible to use the standard hosting service and expand into the cloud as and when required, just as I said I wanted to here. This issue is also potentially addressed by Dynect’s solution, although I have yet to fully investigate that service. Anyhow, it’s good to see that somebody is thinking through the cloud offering and what it takes to make it a practical solution. As soon as I’ve fully investigated Dynect and kicked GoGrid’s tyres for deploying my new project I’ll post the details here for people to throw stones at.
[ad#co-1]

Further to my recent comments about cloud computing – I am currently helping a new site set itself up – we want to use a traditional hosting service but with the option of dynamically expanding into the cloud if bandwidth or other capacity limits are approached. Somebody had a bright idea – set up a free starter account a goggrid, configure a loadbalancer as the public address of the site and point the loadbalancer back at our server. Then when we need to we can start up a gogrid server and have the loadbalancer distribute load between the two servers. Brilliant – just the kind of scenario I wrote about, and a logical use of cloud capacity. Unfortunately gogrid don’t let you configure a loadbalancer to point to anything but their own servers. Rats. Something is going to have to give here if this concept is ever going to take flight.

Every company I have worked at for the last 5 years has tried to implement some kind of Agile development process. It seems to be the new mantra, the silver bullet of software development, similar to the way Object-Oriented Programming was ten, no wait fifteen years ago. Of course OOP was a development methodology and Agile is more of a management process, but that in many ways makes it more appealing to managers since they can understand it better. Each company I have seen seems to have taken different parts of the process and incorporated them in different ways into their existing process. All have had some kind of meeting which they called a scrum, usually an existing meeting with a new name. Most have instigated “sprints”, usually of 2 weeks, to implement suitably small pieces of functionality. Usually the enthusiasm wears off fairly quickly, often when a major re-write is required or something else that doesn’t fit the model and daily meetings are reduced to people repeating the same thing every day for weeks on end. Which is fine by me. I’m sure the whole Agile thing is perfectly appropriate for some sub-set of environments and projects, but for those that it isn’t it’s really painful to try to make it work. And I really dislike doing something because somebody thinks I ought to be doing it. Plus it makes me feel like a Toyota factory worker singing the company song in the morning. I don’t think the development process at any of these places was broken before they were “agile”, and I don’t think any of them were significantly better while it was. If a project is well managed and the lines of communication between different stakeholders are open (which they should be) then a project is generally successful. If it isn’t and they aren’t then it generally won’t be, and it will take more than a few buzzwords to fix it. Keeping a team lean and focused is important, if you have to focus on that then go ahead and implement Agile. If you’re already lean and focused, and delivering, then it ain’t broke – don’t fix it.
[ad]

We have been evaluating cs-cart as a replacement shopping cart on our store front. It has more features than Actinic, our current solution; it seems generally more flexible and configurable; and it isn’t tied to Windows. We installed a store based on a beta release of the forthcoming 2.0 version, did a lot of data and design migration, and all was going well. Until yesterday when I tried to log into the admin area and found I was locked out. A little investigation revealed the site had been hacked from Russia using an apparently well known SQL injection hack. I replayed it myself and got a nicely formatted list of user names and password hashes in the product description area. I thought the admin password was strong enough to resist a hash lookup attack, but apparently it took him just 9 minutes to figure out the password and log in.
A post to the cs-cart forum was met with the response that the sql injection attack was well known, and that was it. Studied indifference. I still like cs-cart but this attitude worries me. I am going to modify the password code on my copy to incorporate a salt into the password, and I may make some other changes to make my password generation different from the standard. I may even package up an add-on for other users. But if any other such blatant insecurities come to light we will be looking elsewhere for our shopping cart software.