Difference between revisions of "Darwin:Push Port"

From Open Rail Data Wiki
Jump to navigation Jump to search
m (Tidy up page)
Line 1: Line 1:
The Darwin Push Port is an XML push feed that continuously streams information about the creation of, and changes to, train schedule records, together with train running predictions made by Darwin. It requires the user to build a database capable of capturing extremely high volumes of information, as well as a query engine to draw the information from your database. There is a large amount of interpretation work involved in this; however this allows substantial flexibility to apply the information to any product within the limitations of your own infrastructure.
+
The Darwin Push Port is an XML push feed that continuously streams information about the creation of,
 +
and changes to, train schedule records, together with train running predictions made by Darwin.
  
It can be filtered to a specific area of interest by TIPLOC, or to provide information for the entire country. Either way, the information delivered is complex and must be properly interpreted before presentation to end users.
+
The data is made available through http://opendata.nationalrail.co.uk.
 +
 
 +
The Push Port requires the user to build a database capable of capturing extremely high volumes of
 +
information, as well as a query engine to draw the information from your database. There is a large
 +
amount of interpretation work involved in this; however this allows substantial flexibility to apply
 +
the information to any product within the limitations of your own infrastructure.
  
 
= Data =
 
= Data =
 +
The Push Port has two components:
 +
* Timetable and Timetable Reference Data.
 +
* Real-Time Update Data.
  
The Push Port has three components:
+
All Darwin data is gzipped ''(except for the Darwin Status Topic)''.  [[Darwin:Push_Port_XML_Schemas|XSDs]] for the interface are available, along with the specification.
  
* Reference Data, available over FTP once a day
+
= Timetable and Timetable Reference Data =
* Timetable, available over FTP once a day
+
Darwin makes available Timetable and Timetable Reference Data exposed as static files that are
* Real-time updates, available over Stomp and historically, in-day, via FTP
+
generated ''usually'' on a daily basis.  The creation of new Timetable and Timetable Reference files
 +
are alerted via ''TimeTableId'' messages in the real-time Update Data.
  
[[Darwin:Push_Port_XML_Schemas|XSDs]] for the interface are available, along with the [[Media:Push_Port_Interface_Specification.gz|specification]].
+
== Timetables ==
 +
Timetable data contains a set of schedules covering at least a 48-hour
 +
period held in the Darwin database. This list of schedules provides the basis on
 +
which a Darwin snapshot can be applied.  
  
= Reference Data =
+
The schedules in the timetable do not include forecast or actual times although
 
+
they reflect the latest state that Darwin has when the timetable file was generated,
The reference data contains:
+
so any schedule changes, new schedules, false destinations, cancellations and
 +
associations will be included.
  
 +
== Reference Data ==
 +
The Timetable Reference Data contains the following data referenced in timetables:
 
* TIPLOCs, CRS codes, TOC codes and location names
 
* TIPLOCs, CRS codes, TOC codes and location names
 
* TOC codes, names and website URLs
 
* TOC codes, names and website URLs
Line 24: Line 40:
 
* [[Darwin:CIS codes and names|CIS codes and names]]
 
* [[Darwin:CIS codes and names|CIS codes and names]]
  
= Timetable =
+
= Update Data =
 
+
Darwin makes available real-time updates that alert the user to changes in the state of the
There are two components to the Darwin Timetable:
+
Darwin database, or the creation of new Timetable and Timetable Reference Data.  Darwin exposes two
 
+
message topics:
* A timetable snapshot, available once a day via FTP
+
* Darwin Live Feed Topic
* Schedule updates via the Real-time feed (see below)
+
* Darwin Status Topic
 
 
= Real-time Updates =
 
 
 
NOTE: All messages in the real-time update stream are sent in XML with the message body compressed using gzip.
 
 
 
Each message in the update feed contains [[Darwin:uR_Element|update response]] element, which is in turn nested inside a 'Pport' element:
 
  
 +
== Darwin Live Feed Topic ==
 +
The live feed topic exposes all update messages.  Update Messages contain one or more of the following elements:
 
* [[Darwin:Schedule_Element|Schedule data]]
 
* [[Darwin:Schedule_Element|Schedule data]]
 
* [[Darwin:Association_Element|Association data]]
 
* [[Darwin:Association_Element|Association data]]
Line 45: Line 57:
 
* [[Darwin:Tracking_ID_corrections|Tracking ID corrections]]
 
* [[Darwin:Tracking_ID_corrections|Tracking ID corrections]]
 
* [[Darwin:Alarm_Element|Alarms]]
 
* [[Darwin:Alarm_Element|Alarms]]
 +
* [[Darwin:Formations|Schedule formation]]
 +
* [[Darwin:Train_Loading|Loading]]
 +
 +
The Live Feed Topic also exposes ''TimeTableId'' messages that alert the creation of a new Timetable
 +
or Timetable Reference file.
 +
 +
== Status Messages ==
 +
The Status message topic contains status messages about the health and state of the Update Data.
 +
The possible messages are:
 +
; HBINIT : The upstream live feed is running but is initialising its timetable.
 +
   
 +
; HBFAIL : The upstream live feed is shutting down.
 +
   
 +
; HBPENDING : The upstream live feed is operating, but part of the system is currently in failover mode. Data may be queued for a short period. Clients may remain connected and data will be delivered when available.
 +
   
 +
; SNAPSHOT : The Darwin Live Feed has encountered a discontinuity of messages from upstream and is starting a snapshot to re-sync it's state.
 +
   
 +
; SHUTTING-DOWN : Darwin is shutting down and the message topics will soon become unavailable.
  
 
= Usage =
 
= Usage =
  
== How do I get the data? ==
+
== Subscribing to Darwin ==
 +
The Darwin Push Port is made available through http://opendata.nationalrail.co.uk.  By creating an account, you can register
 +
for a subscription to the Darwin feed.
  
To access the data feeds you must register for an account at [http://datafeeds.nationalrail.co.uk http://datafeeds.nationalrail.co.uk]. Once you have submitted the registration form, you should receive a confirmation email. Follow the instructions in the email and set your new password.
+
As a user with an active Darwin subscription, navigating to the My Feeds page will display the following details:
  
Once you have set your password and signed in, navigate to the data feeds screen by clicking on the My Feeds link at the top of the page.  You will find your queue name, username and password, as well as details of the username and password you should use to download data from the FTP service.
+
; Darwin File Information : This section provides user details for accessing the Timetable and Timetable Reference Data via an Amazon S3 Bucket.
  
Keep your queue name private, and do not let others know what it is.
+
; Darwin FTP Information : This section provides user details for accessing snapshots and 5-minute logs of the real-time Update Data via FTP.
  
== Accessing real-time data ==
+
; Darwin Topic Information : This section provides user details for accessing real-time Update Data via OpenWire and STOMP message topics.
  
The data from the Push Port is available on datafeeds.nationalrail.co.uk via:
+
'''Important''' - Please note NRDP accounts expire after extended periods of no use. The unused account expiry period is  
 +
currently set to 30 days. If you create an account and do not consume any of the feeds during this time your account
 +
will be deleted. If your account has been deleted, you will receive a notification email, and you will be able to
 +
re-register for a new account.
  
* [http://en.wikipedia.org/wiki/Streaming_Text_Oriented_Messaging_Protocol STOMP] on TCP port 61613
+
== How do I consume the data? ==
* [http://activemq.apache.org/openwire.html OpenWire] on TCP port 61616
 
  
You will need a STOMP client (available in most languages - a list is available [http://stomp.github.io/implementations.html here]) or an [http://activemq.apache.org/openwire.html OpenWire client].
+
=== Timetable and Reference Data ===
 +
Timetable and Reference data can be obtained via an Amazon S3 Bucket.  You will be required to connect and authenticate to S3 via the details given in ''Darwin File Information'' on your ''My Feeds'' page.
  
Messages bodies are compressed using gzip before being placed on per-user queues.
+
=== Keeping up to date ===
 +
Timetable and Reference Data is updated ''usually'' on a daily basis.  To indicate that a new Timetable or Timetable Reference
 +
file is available, the real-time topic will send a ''TimeTableId'' message, to identify the new Timetable or Timetable Reference Data file name.
  
Each user's queue maintains 5 minutes worth of Push Port messages.
+
A separate ''TimeTableId'' message will be sent for each individual Timetable or Reference
 +
Data file that becomes available. Thus, multiple ''TimeTableId'' messages will be generated
 +
in succession, one for each Timetable and Reference file schema version.
  
'''Important''' - Please note that once you have registered for an account, if you leave your message queue idle for an extended period, i.e. you do not connect your STOMP client to pull data down, your account and queue will be deleted. The unused account expiry period is currently set to 100 days. If your account has been deleted, you will receive a notification email, and you will be able to re-register for a new account.
+
''Note that due to existing schema limitations, the TimeTableId message has mandatory
 +
attributes for timetable file and timetable reference data file names. Since the
 +
TimeTableId notification message is only reporting the presence of a single file, only
 +
one of these attributes will be populated with a valid file name. The other attribute will
 +
consist only of white space.''
  
== Accessing non-real time data ==
+
=== Update Data via FTP ===
 +
The FTP server provides non real-time Update Data for users that missed the real-time updates.  All files are gzipped.
  
Darwin releases the timetable and reference data file once a day. These files along with a snapshot file that is taken shortly after the timetable has been downloaded are made available for download via an FTP site. All Push Port messages that are older than 5 minutes are also placed in this FTP site in log files,  with each log file containing 5 minutes of Push Port messages. It should be noted that the messages in the log files in the FTP site are not compressed.
+
Darwin regularly creates ''Snapshot'' files, containing the entire state of Darwin at a given point in time. The latest snapshot file is available over FTP for end users.
  
The FTP site's URL and access credentials are displayed in the My Feeds screen.
+
Every 5 minutes of Live Feed Data since the last snapshot will be available in log files, and available over FTP.
  
== Good Practice ==
+
=== Real-Time Update Data via OpenWire & Stomp Message Topics ===
 +
The Darwin Live Feed Topic and Darwin Status Messages Topic are exposed via ActiveMQ, and can be connected to
 +
via [http://activemq.apache.org/openwire.html OpenWire] or [http://en.wikipedia.org/wiki/Streaming_Text_Oriented_Messaging_Protocol STOMP]. The credentials for connecting can be obtained via your ''Darwin Topic Information'' section on your ''My Feeds'' page.
  
You should follow the [[Good_Practice|good practice]] guide when using this service.
+
STOMP and OpenWire allow durable and non-durable subscriptions.  If you would like Darwin to retain messages for you on disconnection,
 +
you should use a durable subscription.  Please note that message retention is limited, and is implemented to allow for short term
 +
subscriber failure, not long term message persistence.
  
== Examples ==
+
'''Important''':  The following must be true when connecting to a Darwin Topic:
 +
* Watching Advisory Topics must be turned off.
 +
* If you are using a Durable Subscriber, your Client ID '''must''' begin with your username.
  
 +
=== Detecting Real-Time Discontinuity ===
 +
Each Update message contains a ''SequenceNumber'' header.  The sequence number runs from 0 to 9,999,999.
 +
Upon reaching the end of this range the sequence number wraps around to 0.
 +
 +
Darwin guarantees messages are produced with sequential sequence numbers, therefore a missing sequence number
 +
indicates a missed message.
 +
 +
For example, if you received the following sequence numbers in order:
 +
 +
    0, 1, 2, 4, 5, 6
 +
 +
Then you have missed the message with sequence number 3.
 +
 +
=== Filtering ===
 +
If you wish to, you may filter the Darwin Live Feed by message type using JMS Selectors on the MessageType header. Available message types and their respective
 +
codes include:
 +
{| class="wikitable"
 +
!Description
 +
!Code
 +
|-
 +
|Schedule updates (consisting of Schedule, DeactivatedSchedule)
 +
|SC
 +
|-
 +
|Association updates
 +
|AS
 +
|-
 +
|Schedule formations
 +
|SF
 +
|-
 +
|Train order
 +
|TO
 +
|-
 +
|Actual and Forecast Information
 +
|TS
 +
|-
 +
|Loading
 +
|LO
 +
|-
 +
|Station messages
 +
|OW
 +
|-
 +
|Notifications (consisting of TrainAlert, TrackingID, RTTIAlarm)
 +
|NO
 +
|}
 +
Please note that if you choose to filter messages, you will not be able to detect discontinuities in the Darwin feed.
 +
 +
= Good Practice =
 +
You should follow the good practice guide when using this service.
 +
 +
= Examples =
 
Code examples for STOMP clients are available in [https://github.com/openraildata Github].
 
Code examples for STOMP clients are available in [https://github.com/openraildata Github].
  
The [[Advanced Uses|advanced usage]] page contains examples of some advanced applications for the data feeds, including bridging the ActiveMQ feeds to your own messaging server.
+
The advanced usage page contains examples of some advanced applications for the data feeds, including bridging the ActiveMQ  
 +
feeds to your own messaging server.
  
== Support ==
+
= Version 12 Support =
 +
Version 12 is still available for those who are already subscribed. You will need to transition to Version 16 by mid
 +
May 2019 as Version 12 will be decommissioned from this date. You will not be able to change your Version 12 filter
 +
settings during the transition period.
  
 +
= Support =
 
If you are having problems with the feeds:
 
If you are having problems with the feeds:
 
 
* First, read this wiki - there's a lot of material here that will help you
 
* First, read this wiki - there's a lot of material here that will help you
 
* Check [https://twitter.com/open_rail_feeds twitter] to see if an issue has been reported
 
* Check [https://twitter.com/open_rail_feeds twitter] to see if an issue has been reported

Revision as of 10:23, 21 February 2019

The Darwin Push Port is an XML push feed that continuously streams information about the creation of, and changes to, train schedule records, together with train running predictions made by Darwin.

The data is made available through http://opendata.nationalrail.co.uk.

The Push Port requires the user to build a database capable of capturing extremely high volumes of information, as well as a query engine to draw the information from your database. There is a large amount of interpretation work involved in this; however this allows substantial flexibility to apply the information to any product within the limitations of your own infrastructure.

Data

The Push Port has two components:

  • Timetable and Timetable Reference Data.
  • Real-Time Update Data.

All Darwin data is gzipped (except for the Darwin Status Topic). XSDs for the interface are available, along with the specification.

Timetable and Timetable Reference Data

Darwin makes available Timetable and Timetable Reference Data exposed as static files that are generated usually on a daily basis. The creation of new Timetable and Timetable Reference files are alerted via TimeTableId messages in the real-time Update Data.

Timetables

Timetable data contains a set of schedules covering at least a 48-hour period held in the Darwin database. This list of schedules provides the basis on which a Darwin snapshot can be applied.

The schedules in the timetable do not include forecast or actual times although they reflect the latest state that Darwin has when the timetable file was generated, so any schedule changes, new schedules, false destinations, cancellations and associations will be included.

Reference Data

The Timetable Reference Data contains the following data referenced in timetables:

Update Data

Darwin makes available real-time updates that alert the user to changes in the state of the Darwin database, or the creation of new Timetable and Timetable Reference Data. Darwin exposes two message topics:

  • Darwin Live Feed Topic
  • Darwin Status Topic

Darwin Live Feed Topic

The live feed topic exposes all update messages. Update Messages contain one or more of the following elements:

The Live Feed Topic also exposes TimeTableId messages that alert the creation of a new Timetable or Timetable Reference file.

Status Messages

The Status message topic contains status messages about the health and state of the Update Data. The possible messages are:

HBINIT 
The upstream live feed is running but is initialising its timetable.
HBFAIL 
The upstream live feed is shutting down.
HBPENDING 
The upstream live feed is operating, but part of the system is currently in failover mode. Data may be queued for a short period. Clients may remain connected and data will be delivered when available.
SNAPSHOT 
The Darwin Live Feed has encountered a discontinuity of messages from upstream and is starting a snapshot to re-sync it's state.
SHUTTING-DOWN 
Darwin is shutting down and the message topics will soon become unavailable.

Usage

Subscribing to Darwin

The Darwin Push Port is made available through http://opendata.nationalrail.co.uk. By creating an account, you can register for a subscription to the Darwin feed.

As a user with an active Darwin subscription, navigating to the My Feeds page will display the following details:

Darwin File Information 
This section provides user details for accessing the Timetable and Timetable Reference Data via an Amazon S3 Bucket.
Darwin FTP Information 
This section provides user details for accessing snapshots and 5-minute logs of the real-time Update Data via FTP.
Darwin Topic Information 
This section provides user details for accessing real-time Update Data via OpenWire and STOMP message topics.

Important - Please note NRDP accounts expire after extended periods of no use. The unused account expiry period is currently set to 30 days. If you create an account and do not consume any of the feeds during this time your account will be deleted. If your account has been deleted, you will receive a notification email, and you will be able to re-register for a new account.

How do I consume the data?

Timetable and Reference Data

Timetable and Reference data can be obtained via an Amazon S3 Bucket. You will be required to connect and authenticate to S3 via the details given in Darwin File Information on your My Feeds page.

Keeping up to date

Timetable and Reference Data is updated usually on a daily basis. To indicate that a new Timetable or Timetable Reference file is available, the real-time topic will send a TimeTableId message, to identify the new Timetable or Timetable Reference Data file name.

A separate TimeTableId message will be sent for each individual Timetable or Reference Data file that becomes available. Thus, multiple TimeTableId messages will be generated in succession, one for each Timetable and Reference file schema version.

Note that due to existing schema limitations, the TimeTableId message has mandatory attributes for timetable file and timetable reference data file names. Since the TimeTableId notification message is only reporting the presence of a single file, only one of these attributes will be populated with a valid file name. The other attribute will consist only of white space.

Update Data via FTP

The FTP server provides non real-time Update Data for users that missed the real-time updates. All files are gzipped.

Darwin regularly creates Snapshot files, containing the entire state of Darwin at a given point in time. The latest snapshot file is available over FTP for end users.

Every 5 minutes of Live Feed Data since the last snapshot will be available in log files, and available over FTP.

Real-Time Update Data via OpenWire & Stomp Message Topics

The Darwin Live Feed Topic and Darwin Status Messages Topic are exposed via ActiveMQ, and can be connected to via OpenWire or STOMP. The credentials for connecting can be obtained via your Darwin Topic Information section on your My Feeds page.

STOMP and OpenWire allow durable and non-durable subscriptions. If you would like Darwin to retain messages for you on disconnection, you should use a durable subscription. Please note that message retention is limited, and is implemented to allow for short term subscriber failure, not long term message persistence.

Important: The following must be true when connecting to a Darwin Topic:

  • Watching Advisory Topics must be turned off.
  • If you are using a Durable Subscriber, your Client ID must begin with your username.

Detecting Real-Time Discontinuity

Each Update message contains a SequenceNumber header. The sequence number runs from 0 to 9,999,999. Upon reaching the end of this range the sequence number wraps around to 0.

Darwin guarantees messages are produced with sequential sequence numbers, therefore a missing sequence number indicates a missed message.

For example, if you received the following sequence numbers in order:

   0, 1, 2, 4, 5, 6

Then you have missed the message with sequence number 3.

Filtering

If you wish to, you may filter the Darwin Live Feed by message type using JMS Selectors on the MessageType header. Available message types and their respective codes include:

Description Code
Schedule updates (consisting of Schedule, DeactivatedSchedule) SC
Association updates AS
Schedule formations SF
Train order TO
Actual and Forecast Information TS
Loading LO
Station messages OW
Notifications (consisting of TrainAlert, TrackingID, RTTIAlarm) NO

Please note that if you choose to filter messages, you will not be able to detect discontinuities in the Darwin feed.

Good Practice

You should follow the good practice guide when using this service.

Examples

Code examples for STOMP clients are available in Github.

The advanced usage page contains examples of some advanced applications for the data feeds, including bridging the ActiveMQ feeds to your own messaging server.

Version 12 Support

Version 12 is still available for those who are already subscribed. You will need to transition to Version 16 by mid May 2019 as Version 12 will be decommissioned from this date. You will not be able to change your Version 12 filter settings during the transition period.

Support

If you are having problems with the feeds:

  • First, read this wiki - there's a lot of material here that will help you
  • Check twitter to see if an issue has been reported
  • If you want to discuss your problem with other people working with the service, the openraildata-talk group on Google Groups will be useful
  • Finally, if you're still having a problem, email dsg_nrdp.support@caci.co.uk


National Rail Enquiries Data Feeds
Data Feeds About the Feeds Darwin Webservice (Public) Darwin Webservice (Staff) Historical Service Performance Push Port KnowledgeBaseDTDLocations (PoC)Real Time Journey Planner
LDB API About
LDB-SV API About
HSP About
DTD About Fares Timetable
Push Port About XML Schemas Schedules Associations Train Status Station Messages Alarms Train Order Train Alerts Formations Formation loading