Logo
Published on

Webhooks - scraping is a thing of the past

469 words3 min read
Authors
  • avatar
    Name
    Phil
    Twitter
    lead engineer @ vatsim

Webhooks

Introduction

The more observant among you may have seen the "webhook" discussion pop up in the Community Tech discord over the last few months, but what is it? How does it change the game?

A brief history

Historically, to get a members status, vACCs and Divisions would have to poll the VATSIM API to scan for changes. If you're an especially large division, or subdivision, this may result in thousands or tens of thousands of API calls a day. We get a lot of calls to our web services, see this post to learn more about the scale of the services we provide.

These calls put significant pressure on the VATSIM database, and the supporting web services. Wouldn't it just be easier if something came to tell you that a member in your division, or subdivision has had properties changed?

Enter Webhooks

Firstly, what is a webhook?

Webhooks are a simple way for two applications to "speak" to each other and get notified automatically when something new happens. Most people will probably recognise the term from Discord. Discord webhooks provide a way of posting information into discord channels by sending a crafted payload matching the schema they expect.

API v2.0

We've been thinking about the problems with load on our APIs a lot for the last year.

Ryan, Mike R, Nick, Jacob, and others sat down one night to discuss how we can improve the service we provide to our talented developer community, and from that meeting, webhooks were born. They've been hard at work redeveloping the new version (v2.0) of our API to distribute webhooks ever since.

Impact

So, what does this mean for you, as a developer?

api vs webhook

As shown above by Kartik's handy diagram; when the service goes live, you'll give us a URL to post changes to, and you simply react to the changes submitted, and hey presto. No more polling, no more excessive loads, no more throttling or Cloudflare challenges.

The webhooks have a defined schema for the changes which will be published shortly. If you're interested in what other changes are being made to the API, you can check out our handy API docs here.

NOTE: We're not currently looking for additional divisions/subdivisions to help test the webhook system, as we already have a long list, but keep an eye out for when the service goes live!

Conclusion

webhookvsget

Need we say more?

Information you need, when you need it, without having to write background workers or scrapers to update member data...

As always, if you have any questions regarding these changes, or what it might mean for you, reach out to us via tech(at)vatsim.net, or via the community discord.

Until next time!

~the vatsim technology team

Sources: Kartik Mahendra's Webhook Guide