Visitor Question about Data Feeds

Dear Aegist, firstly congrats on publishing a truly informative arbitrage guide. well done.

My question is ,how do the arb services get their data.? Do they pay for a feed from each bookie?

cheers Baz

Hi Baz,
Before I get in to this, I should just make it clear that I am no expert on this subject, and I would love to get some comments on this from real programmers and people who know what they are talking about below. However, my dealings in this area and with various companies over the years have given me this basic level of knowledge…

My understanding about how alert services get their odds is from 3 main sources – I think the most common source of bookmaker odds for arbitrage alert services is the XML feed directly from the bookmakers who offer that service. This data is sometimes supplemented with a third party XML feed who provide odds which they have aggregated from numerous bookmakers, although I may have those two sources back to front – the third party might be primary, with supplementation from free bookmakers. Anyway, OddsMiner.com is one such example of a third company source of odds, and it is through companies like them that most of those odds comparison websites out there get their odds feeds. The third option is direct HTML ‘scraping’, which is where the alert service owner creates a bot to scan the bookmaker websites directly, much like Google does to index websites for their search engines, but instead of trying to store all of the data on the page, this bot is designed to just grab the odds for each market and store them in a useable form – and do it quickly, and often.

All three methods have the advantages and their problems.

Bookmaker Provided XML Feeds

Direct XML feeds from bookmakers are free, which is a huge advantage, and the bookmakers want you to use them. However, not all bookmakers offer them and those that do still control what sports and odds are covered in them, often providing only selected offerings – usually the most lucrative ones for them. The users have no say in what odds they can get, and as anyone who has ever used an alert service knows, sometimes the XML feed odds simply don’t match up with the odds on the website! So you have to accept that the free, bookmaker preferred system is not always the best data source for the users.

HTML Scraping of Odds

Direct HTML Scraping is much more reliable when it comes to the odds collected. Because the system takes the odds directly off the website and reports them asap, the odds are reliably what a user will find when they visit the website themselves. The creator of the html scraping program can also control what pages, and therefore, which sports and odds are scraped – so there are no restrictions to what odds can be collected. It is also free – ignoring the time/effort requirement to develop an adequate scraping program, and then maintain it with respect to the bookmaker website ie: If the bookmaker changes their website layout, the scraper needs to be updated to match the new layout.

For all of the benefits of HTML scraping though, it has the significant problem of requiring constant HTML access to the bookmaker website for re-scraping. In order to keep your scraped odds up to date, you need to scrape the whole website regularly – for arbitrage purposes, ideally you would scrape it every minute – but no website owner wants a bot hitting their website and loading every page on it every minute. That wastes their bandwidth, slows down the website for all of their users and costs them money. So any ip found repeatedly hitting a page at that frequency will usually be ip-blocked by the bookmaker. So you need to be much more clever with how you scrape the odds from bookmakers.

Third Party Odds Providers

Third party odds providers cost money. In return for that, you get a feed of odds of whatever sports, markets and bookmakers you want, without all of the other hassles. However, these data sources are still never perfect. I don’t know exactly how they get their odds, but I suspect it is a combination of bookmaker XML (probably in a closer partnership with the bookmakers, so they can get a better source and get more books than most common people), and possibly some html scraping used too. But I really don’t know. However they get their odds, the odds are still not perfect, but they are useable and mostly good enough. I think the biggest cost of using a third party odds provider, aside from the financial cost, is that you lose any unique advantage that you might be trying to claim. If everyone has the same source of odds, then every alert service is going to look very similar where it counts (what arbs it finds). Nonetheless, I think this is the most common source of odds because the simplicity of just letting someone else take care of it for you cannot be ignored.

Some more great information on this topic can be seen on OddsMiner’s FAQ page.

Tagged as: alert service, data, feed, html scraping, odds, oddsminer, xml