Rail Data FAQ: Difference between revisions

From Open Rail Data Wiki
m →‎What data is available from Network Rail?: Remove unnecessary underscore in Train Movements link
Q (talk | contribs)
Update & merge details from FAQ page
Line 3: Line 3:
It depends on what you want to do.
It depends on what you want to do.


* Use the '''Network Rail feeds''' if you want very low-level detail, such as train describer berth-level data, line/platform/path data in train schedules, or if you want to receive data for non-passenger TOC trains.  The data in these feeds is of a highly technical nature and you will require a fair amount of railway domain knowledge to be able to interpret them correctly.
* Use the [[About_the_Network_Rail_feeds|'''Network Rail feeds''']] if you want very low-level detail, such as train describer berth-level data, line/platform/path data in train schedules, or if you want to receive data for non-passenger TOC trains.  The data in these feeds is of a highly technical nature and you will require a fair amount of railway domain knowledge to be able to interpret them correctly.
* Use the '''National Rail Enquiries feeds''' if you want industry-wide consistent data on passenger train movements which matches that shown in railway stations and on other sites powered by [http://www.nationalrail.co.uk/56908.aspx Darwin].
* Use the '''[[About_the_NRE_Feeds|National Rail Enquiries feeds]]''' if you want industry-wide consistent data on passenger train movements which matches that shown in railway stations and on other sites powered by [http://www.nationalrail.co.uk/56908.aspx Darwin].


= What data is available from National Rail Enquiries? =
= What data is available from National Rail Enquiries? =
Line 93: Line 93:


SOAP is not the friendliest of protocols - the Open Source [https://unop.uk/dev/huxley-rail-api-proxy/ Huxley] proxy, which you can run on your own system, may be of use if you want to access the web service using JSON.
SOAP is not the friendliest of protocols - the Open Source [https://unop.uk/dev/huxley-rail-api-proxy/ Huxley] proxy, which you can run on your own system, may be of use if you want to access the web service using JSON.
= I get a 'User not subscribed' error when trying to download from the SCHEDULE feed =
The format of the correct URL to use is documented on the [[SCHEDULE]] page.  A previous version of the developer pack referenced an incorrect URL.
= I have an issue accessing the real-time data-feeds =
Make sure you are following all of the [[About_the_feeds#Good practice|good practice advice]].
= I have registered and keep getting a "bad credentials" error =
Check that you have registered at https://datafeeds.networkrail.co.uk correctly.  Your password for this wiki cannot be used to access the Network Rail Data Feeds.
= There are messages missing from the feed =
Make sure you're using a [[Durable_Subscription|durable subscription]] and acknowledging all the messages you receive.  You can have around 100 messages pending before the server will stop sending data to you.
Check the 'time' field in the messages body (not the header) is within a few seconds of the current time.  This field is populated by Network Rail's systems and should be within 10 seconds of the current time.  If it isn't, contact support.
= I see odd data/descriptions in my TD feed =
A berth may contain any valid 4 chars, not just what appears to be a train ID. Train descriptions are normally in the format NANN:
* 1B01 - Normal Class 1 passenger service
* 2T01 - Normal Class 2 passenger service
* 5V61 - Normal ECS Service
* 3H52 - Express ECS Service
* 0Z01 - Light Loco
* 6B92 - Class 6 freight service
Freight and engineering trains will be in the format NNNA, consistent across the TRUST and TD feeds:
* 431L - Class 4 obfuscated ID
* 651A - Class 6 obfuscated ID
* 011X - Class 0 obfuscated ID
Not all non-passenger services are obfuscated, and there are some known bugs where an obfuscated FOC service will cause TD data for a valid passenger service to become obfuscated.
You may also see some of the following things in berths:
* *T3* - T3 Possession
* DETS - Detonators
* STBY - Standby (Unit)
* STBL - Stabled (Unit)
* --33 - Reference to an item number in that weeks WON
* 147- - Reference to an item number in that weeks WON
* IT64 - Reference to an item number in that weeks WON
* BLOK - Blocked
* NOGO - No Go
* FRED - Name of PICOP or similar
* **** - Non described
* -REG - Regulate
* TBA- - To be advised
* SHNT - Shunt
* 4CAR - 4 Cars
* 12CR - 12 Cars
* RHTT - Rail Head Treatment Train
* TAMP - Tamper
* FAIL - Failed unit
* DUFF - Failed unit
* WEED - Weed Killing Train
= I can't connect to any feed, or the web portal =
If you suddenly find that you are unable to connect to any feed, as well as the web portal then your IP address may be blocked by the NROD firewall.  This appears to automatically block an IP address for 15 minutes if it detects conditions such as:
* Duplicate durable subscription name
* Too many authentication attempts
You can check this by trying to connect to the web portal from a different IP address, if you can connect then its likely you have been banned from your original IP. If you have been banned, disconnect all your clients and wait at least 15 min before trying to re-connect again, and hopefully once you have figured out what caused you to get banned in the first place!
= I'm connected but getting no data or any errors =
There is an issue with NROD in that you suddenly find that you are connected to a feed but no longer receiving data, or you have reconnected. (for maintenance etc.) This can usually be resolved by changing your durableSubscriptionName then reconnecting. In doing this you will loose any unconsumed data from your previous connection. This issue has been reported to support many times but is still unresolved.
{{Navtable-DataFeeds}}
[[Category:Network Rail Data Feeds]]

Revision as of 15:45, 28 March 2021

Which feed should I use?

It depends on what you want to do.

  • Use the Network Rail feeds if you want very low-level detail, such as train describer berth-level data, line/platform/path data in train schedules, or if you want to receive data for non-passenger TOC trains. The data in these feeds is of a highly technical nature and you will require a fair amount of railway domain knowledge to be able to interpret them correctly.
  • Use the National Rail Enquiries feeds if you want industry-wide consistent data on passenger train movements which matches that shown in railway stations and on other sites powered by Darwin.

What data is available from National Rail Enquiries?

The following data is available from National Rail Enquiries' Data Feeds platform:

Data Format Details
Reference data XML Reference Data
Timetable data XML Push Port Timetable
In-day historical log files of real-time data XML Accessing non-real time data
Train service updates (movements, cancellations, forecasts) XML Accessing real-time data

What data is available from Network Rail?

The following data is available from Network Rail's Data Feeds platform:

Data Format Details
Planned train movements (schedules) JSON SCHEDULE
Planned train movements (schedules) CIF SCHEDULE
Late notice changes to the train plan JSON VSTP
Train movement/cancellation-related messages JSON Train Movements
Raw train describer feeds JSON TD
Real-time PPM data every 60 seconds JSON RTPPM
Temporary Speed Restrictions JSON TSR

Why are you using message queues and Stomp?

Whilst it may seem odd to use a relatively heavy message queueing platform, it gives many advantages over using a raw TCP socket, or a stream of data over HTTP:

  • If/when your client disconnects, the server will hold on to several minutes worth of data that you'd otherwise lose if you weren't using a message queue
  • Topics, as used in the Network Rail Data Feeds platform, are a scalable and efficient way to distribute the same data to hundreds of subscribers, each of which may consume data at a different rate
  • Queues, as used in the National Rail Enquiries Data Feeds platform, are an efficient way to distribute a custom set of data to each user
  • There is no need to write a protocol handler to cope with the nuances of HTTP, particularly when using 'chunked' encoding
  • Stomp clients are available for many of the most common development platforms

Why are you using XML in the Push Port?

The Push Port uses XML natively, as it was first built for industry/enterprise use. The data is distributed in XML at this stage for a number of reasons:

  • Converting it to a different format, such as JSON, will take the focus away from getting the data released quickly and later evolving based on the community's feedback
  • A straight XML-to-JSON conversion is difficult, as the Push Port schema makes heavy use of XML namespaces, and some element names are repeated across namespaces and used for different purposes
  • The existing documentation for the Push Port refers to XML messages, and it was felt easier to make as few changes to the format, and therefore the documentation, at this early stage

Why are you using XML and SOAP in the Live Departure Board web service?

The Darwin Web Service was built using XML and SOAP as it was first built for industry/enterprise use.

SOAP is not the friendliest of protocols - the Open Source Huxley proxy, which you can run on your own system, may be of use if you want to access the web service using JSON.


I get a 'User not subscribed' error when trying to download from the SCHEDULE feed

The format of the correct URL to use is documented on the SCHEDULE page. A previous version of the developer pack referenced an incorrect URL.

I have an issue accessing the real-time data-feeds

Make sure you are following all of the good practice advice.

I have registered and keep getting a "bad credentials" error

Check that you have registered at https://datafeeds.networkrail.co.uk correctly. Your password for this wiki cannot be used to access the Network Rail Data Feeds.

There are messages missing from the feed

Make sure you're using a durable subscription and acknowledging all the messages you receive. You can have around 100 messages pending before the server will stop sending data to you.

Check the 'time' field in the messages body (not the header) is within a few seconds of the current time. This field is populated by Network Rail's systems and should be within 10 seconds of the current time. If it isn't, contact support.

I see odd data/descriptions in my TD feed

A berth may contain any valid 4 chars, not just what appears to be a train ID. Train descriptions are normally in the format NANN:

  • 1B01 - Normal Class 1 passenger service
  • 2T01 - Normal Class 2 passenger service
  • 5V61 - Normal ECS Service
  • 3H52 - Express ECS Service
  • 0Z01 - Light Loco
  • 6B92 - Class 6 freight service

Freight and engineering trains will be in the format NNNA, consistent across the TRUST and TD feeds:

  • 431L - Class 4 obfuscated ID
  • 651A - Class 6 obfuscated ID
  • 011X - Class 0 obfuscated ID

Not all non-passenger services are obfuscated, and there are some known bugs where an obfuscated FOC service will cause TD data for a valid passenger service to become obfuscated.

You may also see some of the following things in berths:

  • *T3* - T3 Possession
  • DETS - Detonators
  • STBY - Standby (Unit)
  • STBL - Stabled (Unit)
  • --33 - Reference to an item number in that weeks WON
  • 147- - Reference to an item number in that weeks WON
  • IT64 - Reference to an item number in that weeks WON
  • BLOK - Blocked
  • NOGO - No Go
  • FRED - Name of PICOP or similar
  • **** - Non described
  • -REG - Regulate
  • TBA- - To be advised
  • SHNT - Shunt
  • 4CAR - 4 Cars
  • 12CR - 12 Cars
  • RHTT - Rail Head Treatment Train
  • TAMP - Tamper
  • FAIL - Failed unit
  • DUFF - Failed unit
  • WEED - Weed Killing Train

I can't connect to any feed, or the web portal

If you suddenly find that you are unable to connect to any feed, as well as the web portal then your IP address may be blocked by the NROD firewall. This appears to automatically block an IP address for 15 minutes if it detects conditions such as:

  • Duplicate durable subscription name
  • Too many authentication attempts

You can check this by trying to connect to the web portal from a different IP address, if you can connect then its likely you have been banned from your original IP. If you have been banned, disconnect all your clients and wait at least 15 min before trying to re-connect again, and hopefully once you have figured out what caused you to get banned in the first place!

I'm connected but getting no data or any errors

There is an issue with NROD in that you suddenly find that you are connected to a feed but no longer receiving data, or you have reconnected. (for maintenance etc.) This can usually be resolved by changing your durableSubscriptionName then reconnecting. In doing this you will loose any unconsumed data from your previous connection. This issue has been reported to support many times but is still unresolved.


Network Rail Open Data Feeds
Data Feeds About the Feeds Account States Durable Subscriptions Example Code ( PHP / C# / Java / Ruby / Node.js) • Advanced UsesFAQ Release Notes
RTPPM RTPPM Feed
Train Movements Train Movements Feed Train Activation Train Cancellation Train Movement Train Reinstatement Change of Origin Change of Identity Change of Location TSPEED Field Planned Cancellations Cancellation Codes
TD TD Feed C-Class Messages S-Class Messages Train Describers TD Berths
VSTP VSTP Feed
TSR TSR Feed Route Codes
SCHEDULE SCHEDULE Feed TIPLOC Records Schedule and Location Records Association Records CIF Codes How Scheduling Works Allowances
Reference Data Reference Data Feed TOC Codes CIF Codes Delay Attribution Codes Identifying Locations (STANOX, TIPLOC, NLC and 3-Alpha Codes) STANOX Geographical Areas Train Planning data (BPLAN)