Articles Comments

The ClickFail of Australian Retail.

There but for the grace of Web Gods, go I…

In Australia tonight we were supposed to witness the coming of age of Australian Online Retail. Our first “Cyber Tuesday” – a moment where the industry said to Australians who have been lured to the sales of the US and Europe “We can do this too”. And then we didn’t.

We didn’t because technologists were too arrogant to heed the lessons of “Web Scale” development that have been done in the rest of the world. We didn’t because our retail industry has spent more time crying foul over the lack of GST on international retailers than gaining proficiency in doing online commerce well. And we didn’t because we still have marketers in this country who can say and hype what they want but are not held accountable to delivering good results.

I’m not happy. Not happy at all. I’ve spent just about my entire adult life working to stop exactly the type of problems that happened during the Click Frenzy debacle tonight.

From designing web applications that could withstand the load of half the young adult population of the UK to listen to and download the latest track featured on a Smirnoff Ice advert. To ensuring the website for the Melbourne Cup could stay up when hundreds of thousands of people all decide to find out the winner at 3:15pm on the first Tuesday of November each year. Or maybe it’s just designing ecommerce applications so that when a marketing director decides on a whim to do a 40% off sale with 3 hours notice over lunch time and then email several hundred thousand customers the website will manage to survive.

For the first two hours of the 24 hour sale, the website was simply unavailable due to meltdown. Three hours in as I write this I can get a smattering of pages from the site but no deals to speak of.

Most of the retailers have responded admirably, turning to their social media channels to direct customers to their own websites where they could get the offers anyway. This, for me, was a highlight, showing yet again the web’s ability to route around failure. Indeed many smaller brands quickly capitalised on the fiasco, launching their own deals on their sites to try and lure in customers who were eager to spend – hijacking the twitter and Facebook back-channels to their own ends.

Unfortunately, due to the huge amounts of PR in the lead up to Click Frenzy, many of our biggest retailers failed to withstand the traffic also – Myer, David Jones and Harvey Norman all went down within minutes as customers started going to their sites directly. Not surprising really given these are businesses that have been dragged every inch of the way into online retail since its inception in the ’90s.

Australia was late to the ecommerce party. With an industrial boom that kept customers purchasing in store at inflated rates and high shipping costs holding off international competition they had it easy until the GFC and the resources slump. Since then they have woefully mis-invested in their ecommerce platforms. They bought into the sales hype of platforms like Magento and WebSphere Commerce from consultants that have been all too eager to get reseller fees and provide little value or experience alongside it.

Most Australian retailers have shown little more than disdain for the channel. Staffing has been a nightmare – with juniors recruited because they are young, use Facebook  and “get” the Internet. Never mind that they had no experience and as a result, no ability to judge whether what their agencies or vendors were telling them was correct or a good idea.

And then there was click frenzy – an organisation that hyped its ability to drive sales for lagging retailers yet was totally unproven. An organisation who outwardly presented such a level of arrogance in the face of what they were attempting that it can only be called hubris.

This was a wasted opportunity and tomorrow, when the press releases go out, we’ll see more arrogance from the team that created such a monumental disaster. Tomorrow it will be, “The scale of traffic was unprecedented , “We could NEVER have predicted this was going to happen”, “This has never been done before in Australia”.

Seriously STFU.

Every aspect of this endeavour could have been anticipated. A simple bit of maths by someone who understands traffic and retail customer behaviour could have told you that you’d need to do something special. The sort of special that keeps the Ticketek site running when Justin Beiber releases tickets. The sort of special that keeps the Amazon site up when they launch a new gold box. The sort of special that keeps the home page of a newspaper running in the wake of a war or global natural disaster.

Doing this maths is what keeps me awake at night leading up to big launches or events. Thankfully I’ve always had a stellar team around me – people like Steve McDonald (Tech Lead on Melbourne Cup – now at Fairfax) and Matt Black (Product Lead on JBA real time behavioural processor). People I count on to think through all the problems and come up with solutions well before they are needed. People who have been there before and who are humble enough to not believe they know everything and will learn from other people’s mistakes in order to prevent us making them as well.

These are solved problems. There was absolutely no reason why any of these sites needed to go down tonight. Indeed looking around others managed to stay alive - Target, Deals Direct, Kogan were all fine.

There was no need for Click Frenzy to be running magento and the type of architecture it was running from watching it fail for a couple of hours suggested it was never going to deal with the traffic it was going to get (Magento is notorious for database overload which takes a lot to architect around). Likewise Myer, Harvey Norman and David Jones should all have good enough systems to be able to deal with these sorts of traffic spikes – what are they doing for Christmas and Boxing Day if they don’t?

From my research on the effects of trust on customers’ purchasing behaviour in the wake of site outages, I can tell you that all those brands who went out tonight are going to see serious lags on their sales. Forget the money spent on advertising, the ongoing loss of sales from erosion of trust (where a customer decides to go elsewhere) will hurt a lot more in the long run. The chest thumping spin that will be in the media about “we were just too popular” doesn’t wash in reality with consumers – it’s insulting. Every second a site is out harms your brand and you have to work extra hard to regain that trust.

That there was such a deep failure of our ecommerce systems shows that Australian Retail is not ready for a global stage.

My hope in all of this is that next week, after the dust settles, after the fingers are no longer being pointed, that learning can begin and that collectively this industry learns from the mistakes.

My fear is that it will be business as usual in Australian Retail – where this is used as yet another reason why ecommerce is not viable and investment and learning will evaporate.

#Update: 21/11/2012 13:50 AEDT

I mentioned in my post above that Kogan survived alongside other retailers such as Target etc. According to some people who have chatted to me, apparently Kogan did experience a small period out down time however I didn’t see that as I was going back to the site every hour or so last night.

Also I just want to point out for the record, as part of this post has been used in articles on The Age and SMH, that this was not written in my capacity as CTO of JBA. JBA work with a lot of retailers around crafting better customer experiences, who fully recognise the challenges of doing ecommerce at scale, and who are working hard to try and build or refactor platforms to to deliver the type of experience that should have happened last night.

Written by

Maker and breaker of smart things that combine mobile web, cloud computing, ubicomp and large data processing. Sometime programmer and CTO @ JBA, Breaker of Stuff @ Rocket Melbourne.

Filed under: cloud computing, development, internet, media, rant, web

26 Responses to "The ClickFail of Australian Retail."

  1. Asher Wolf says:

    Thank you for writing this. Spot on.

  2. Daryl Antony says:

    Great rant!

    And needed.

    We’re going to be working really hard to make sure our clients keep surviving these kinds of sales – and I’m sure there’ll be greater demands in the future of Australian online shopping.

    We’re already knee deep in works helping kogan scale out — to meet world wide demand.

    Will keep you posted.

  3. keith says:

    Totally agree with your post. I’ve done the big M type sites as well, there is a certain scaling mindset that is required and unfortunately ClickFrenzy did not employ people with that mindset… You need detailed knowledge of CDN’s, knowledge of how to protect DB’s with caches, knowledge of when to push content to the server over DB pull, how to cluster and shard DB’s, when to denormalise, etc, etc. I think they were doomed from the start.. Putting together web sites that scale is a science as well as an art.

  4. Thanks for the shout-out!

  5. BT says:

    I think the most disappointing part of the ClickFrenzy failure is the fact that the companies and hosting providers aren’t embarrassed (and in fact posting the opposite on their Facebook). Is it really that hard to put your hand up and say, “look, we could have done a better job”? The quotes in the papers sound a bit Iraqi Information Minister.

  6. ajfisher says:

    Thanks Daryl – and it’s great to see Kogan was one of the ones that did well last night. Hopefully this shows the retail industry it can be done here, with local teams and local tech. It was also a great endorsement for Python / Django when the other “enterprise” stacks were failing last night too.

  7. ajfisher says:

    @Keith: Yes, Kate Carruthers (@kcarruthers) touched on this via twitter last night as well – doing this type of work requires experienced and well skilled dev / ops teams to do it right. It’s not sufficient to just bang in a few servers and say “there you are, we’re right to go”. These are skills we lack in Australia by not being exposed to this type of activity – but we should be buying in those skills from abroad or at least learning from what people are doing in Europe or the US or China.

  8. ajfisher says:

    @StephenMcD: Credit where it’s due – it would have been interesting to see the differences had you been involved – no Magento for a start ;P

  9. ajfisher says:

    @BT: Indeed. The “any PR is good PR” mentality will be in full effect for the next week I expect – and seeing responses on FB & Twitter suggests we’ll see a lot more of that. Admitting failure would go a long way to shoring up consumer trust. This is where something like extending the sale until midnight tonight or tomorrow night would do wonders.

  10. Ingrid Thorpe says:

    Amen. Couldn’t say it better myself.

  11. [...] The ClickFail of Australian Retail (2012-Nov-20) [Technology Treason] .. We didn’t because technologists were too arrogant to heed [...]

  12. Ellen says:

    Great feature. I cannot understand why Australian companies keeps getting online retail so, so wrong.

  13. itgrrl says:

    Spot-on analysis as usual, AJ.

    You ask: “…what are they (Myer, HN, DJs) doing for Christmas and Boxing Day if they don’t?”

    Well, as you know the answer is that they haven’t (and won’t, for a while yet) make online a core part of their retail strategy. So they have perfunctory sites with limited catalogue and zero stickiness, whose primary function is to drive sales through their retail stores.

    Same old same old.

  14. Stuart says:

    Great write-up.

    Traffic is no longer an excuse in 2012, especially when scalable site architectures are increasingly commonplace and easy to acquire.

    I think a lot of their problem was due to poor (or zero) research. The founder of CF was on the Today show this morning saying that ‘we expected one million users and got two million’, presented as if it were something to be proud of. For mine, a prediction that is *that* wrong shows only managerial negligence or incompetence.

    I agree with you in that it will forever be a good example of what not to do. I was able to show this to my dev management this morning and say ‘this is why performance is important’.

  15. ajfisher says:

    @itgrrl: Yep, well the storm is coming. Boxing day online spending is going up in Oz as people don’t want to deal with the crush in store. Same as what happened at Thanksgiving in the US. They will learn though.

  16. ajfisher says:

    @Stuart: Exactly. Hopefully there will be a lot of dev teams highlighting this as a case study for why development, testing and investment in their retail platforms is so important.

  17. Clickfail are the new dotcom / get rich online / out for a quick buck / business built on hype.
    Is what they do any different to *deal of the day* type sites? Its a business model that doesn’t work. Retailers need to accept the fact that on-line sales are as hard fought as off-line ones.

  18. Lants says:

    Great post. Hopefully a lesson was learned (although I doubt it).
    People were tagging “#CrashFrenzy” in anticipation before 7pm even rolled around.

  19. Craig Mattson says:

    Very good article describing the common pitfalls I seem to run out of breath daily trying to explain, yet when these things do happen – it’s like water off a duck’s back – in that “unprecedented”, “couldn’t have estimated”. On the bright side, the retailers *did* clearly get visitors – you can’t exactly blame ClickFrenzy for your own crummy server setup.

    The amount of times medium-large scale websites have been “OzBargained” over the years really makes you wonder.

  20. Thanks, a Great post.

    My company ToolTwist has been building websites to handle Black Fridays and Cyber Mondays in the US for years, and as you say, the loads last night weren’t that large.

    Funny that Grant Arnott says “we expected 1 million but got 2 million” – when you’re planning for acceptable response times you shoot for way over 100% headroom.

    I loved your description of the problem…

    “They bought into the sales hype of platforms like Magento and WebSphere Commerce from consultants that have been all too eager to get reseller fees and provide little value or experience alongside it. [...] no ability to judge whether what their agencies or vendors were telling them was correct or a good idea.”

    We have customers around the world, but don’t even bother trying to sell to the big Australian retailers, for exactly this reason. Perhaps we should partner up, now you’re a media celebrity?

    Cheers,
    Phil Callender
    ToolTwist Ltd.

  21. Peter Murphy says:

    If you can’t learn from history, at least learn from this month.

    One of the contributing reasons to the recent Democratic victory in the US is that they had a working voter system call “Narwhal”, with competent systems engineers making it go, doing load testing, and basically making sure when the big day came, their system didn’t die. It kept going, unlike “Orca” – the Republican equivalent – on the only day it was going to be used.

    http://www.theatlantic.com/technology/archive/2012/11/when-the-nerds-go-marching-in/265325/

    Whatever your political preferences, it should have been a wakeup call for anyone working with large websites – the Click Frenzies of the world. Certainly was an eye opener for me. Shame too many people slept through the episode.

  22. ajfisher says:

    @Peter Yes that was fantastic article highlighting how seriously they took it and how simulation (as used in many other high risk, high stress endeavours) was used to work through the problems before they mattered. Contrast that to the Republican team who didn’t. Hopefully the tech industry will use these two high profile incidents to change up the way they approach some of this stuff in the future.

  23. ajfisher says:

    @Phil: Thanks for the comment and the insight about trying to sell into the Aussie retailers – there’s definitely some challenges there but hopefully this should wake them up a bit.

  24. So glad to see this. I am not an IT professional, just a desk jockey who uses software all the time and was so frustrated to see all the free PR they got on the News, hello, News? When it had all the indicators of being just a hyped up scam. Like another on here I also wrote about it, not from a tech point of view but from a punters view who supports small business and trying to get a foothold on the net. http://yathink.com.au/article-display/sounds-too-good-to-be-true-usually-is,28 I am just hoping that the 1000′s who decided to give online shopping a go for the first time because the big boys were involved, don’t toss it out as a fad because of this fiasco. Even as a layperson just using online software I could see that this was going to fail.

    I also have an ethical issue with grabbing all that pre-registration data???

  25. I think their were many problems with this project from the start.

    1. The T & C’s and privacy policy for the website was only added in the last 2 weeks after many users complained.

    2. They obviously have not dealt with this level of traffic in the past, hence they made poor choices with CMS, hosting solutions and every thing else. I think they could have even played it smart and quickly made up an EDM (and sent it out) I mean even if it was the landing page they are currently using, after the site tanked from 7pm-11pm (4 hours of down time, this was the time when people wanted to know what was happening) I noticed some other Aussie retailers were smart and sent out huge EDM blasts with Frenzy in the title near 7pm ;)

    3. Another point I have not seen many people mention which cost click frenzy huge amount of traffic was that did not even register the trademark they acquired for “Click Frenzy” with Google, so this just opened up the flood gates for people to bid on all terms relating to Click Frenzy.

    Kogan did go down for a short period of time (see image below of downtime) but kudos for their cool error page, it is funny how some retailers did not even have error pages.

    https://pbs.twimg.com/media/A8HVJzNCAAAqIxi.png

    Thank you for the analysis and write up but.

    Kind Regards,

    James Norquay.