Difference between revisions of "SCHEDULE"

From Open Rail Data Wiki
Jump to: navigation, search
(Downloading)
m (Include CIF data and links)
Line 1: Line 1:
The SCHEDULE feed is an extract of train schedules from Network Rail's ITPS (Integrated Train Planning System), converted in to JSON format.  Network Rail are not planning to make raw CIF files available.
+
The SCHEDULE feed is an extract of train schedules from Network Rail's ITPS (Integrated Train Planning System), available in JSON and CIF format.
  
Schedules can either be downloaded for all TOCs, or for individual TOCsTwo types of schedule file are available - a 'full' file which contains a snapshot of all schedules at that point in time, and an 'update' file which can be applied to an older set of files to bring them up-to-date.
+
It is recommend that, if you are just starting out with the service, you use the JSON filesThe CIF data is more suited to advanced users of the service and requires additional parsing compared to the JSON data.
  
The [http://www.atoc.org/about-atoc/rail-settlement-plan/data-feeds/types-of-data CIF User Specification] is available from ATOC's website, which details the format of the CIF file and the fields within, and should be read in conjunction with these wiki pages.
+
= Types of file =
 +
 
 +
== JSON format ==
 +
 
 +
The schedule files in JSON format are available for all operators, or for individual operators.
 +
 
 +
Two types of file are available:
 +
 
 +
* A full extract is produced each day, which can be used to initialise your schedule database
 +
* An update extract is produced each day, which can be applied to the previous update extract to bring your database up-to-date
 +
 
 +
These files are generated
 +
 
 +
== CIF format ==
 +
 
 +
The schedule files in CIF format are only available for all operators.  A 'full' extract is available each Friday, and an 'update' extract is available daily.
 +
 
 +
Two types of file are available:
 +
 
 +
* A full extract is produced each Friday, which can be used to initialise your schedule database
 +
* An update extract is produced each day, including Friday, which can be applied to the previous update extract to bring your database up-to-date
 +
The [http://www.atoc.org/about-atoc/rail-settlement-plan/data-feeds/types-of-data CIF User Specification] is available from ATOC's website, which details the format of the CIF file and the fields within.
  
 
= Downloading =
 
= Downloading =
  
The schedule data is compressed using gzip and is downloaded from Amazon S3 via a private URL which is valid for a few minutes after generation.  Authentication will be required either by logging in through the datafeeds website, or by sending your username and password as part of the HTTP request.  On successful authentication, you will receive a 403 redirect to the location of the schedule files.  Requests are of the format:
+
The schedule data is compressed using gzip and is downloaded from Amazon S3 via a private URL which is valid for a few minutes after generation.  Authentication will be required either by logging in through the Data Feeds website, or by sending your username and password as part of the HTTP request.  On successful authentication, you will receive a 403 redirect to the location of the schedule files.  Requests are of the format:
  
 
   https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=type&day=day
 
   https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=type&day=day
Line 23: Line 44:
 
!''day''
 
!''day''
 
!Description
 
!Description
!Example URL
+
!Example URL (JSON format)
 +
!Example URL (CIF format)
 
|-
 
|-
 
|CIF_ALL_FULL_DAILY
 
|CIF_ALL_FULL_DAILY
Line 29: Line 51:
 
|Daily all-TOC snapshot
 
|Daily all-TOC snapshot
 
|[https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_FULL_DAILY&day=toc-full Daily]
 
|[https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_FULL_DAILY&day=toc-full Daily]
 +
|[https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_FULL_DAILY&day=toc-full.CIF.gz Weekly]
 
|-
 
|-
 
|CIF_ALL_UPDATE_DAILY
 
|CIF_ALL_UPDATE_DAILY
 
|toc-update-''DAY''
 
|toc-update-''DAY''
 
|Daily all-TOC update
 
|Daily all-TOC update
|[https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-mon Mon], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-Tue Tue], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-wed Wed], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-thu Thu], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-fri Fri], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-sat Sat], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-sun Sun],  
+
|[https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-mon Mon], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-Tue Tue], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-wed Wed], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-thu Thu], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-fri Fri], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-sat Sat], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-sun Sun]
 +
|[https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-mon.CIF.gz Mon], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-Tue.CIF.gz Tue], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-wed.CIF.gz Wed], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-thu.CIF.gz Thu], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-fri.CIF.gz Fri], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-sat.CIF.gz Sat], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-sun.CIF.gz Sun]
 
|-
 
|-
 
|CIF_''XX''_TOC_FULL_DAILY
 
|CIF_''XX''_TOC_FULL_DAILY
Line 39: Line 63:
 
|Daily snapshot for TOC ''XX''
 
|Daily snapshot for TOC ''XX''
 
|[https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_FULL_DAILY&day=toc-full Daily]
 
|[https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_FULL_DAILY&day=toc-full Daily]
 +
|Not available
 
|-
 
|-
 
|CIF_''XX''_TOC_UPDATE_DAILY
 
|CIF_''XX''_TOC_UPDATE_DAILY
Line 44: Line 69:
 
|Daily update for TOC ''XX''
 
|Daily update for TOC ''XX''
 
|[https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-mon Mon], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-Tue Tue], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-wed Wed], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-thu Thu], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-fri Fri], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-sat Sat], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-sun Sun]
 
|[https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-mon Mon], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-Tue Tue], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-wed Wed], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-thu Thu], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-fri Fri], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-sat Sat], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-sun Sun]
 +
|Not available
 
|-
 
|-
 
|CIF_FREIGHT_FULL_DAILY
 
|CIF_FREIGHT_FULL_DAILY
Line 49: Line 75:
 
|Daily FREIGHT snapshot
 
|Daily FREIGHT snapshot
 
|[https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_FREIGHT_FULL_DAILY&day=toc-full Daily]
 
|[https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_FREIGHT_FULL_DAILY&day=toc-full Daily]
 +
|Not available
 
|-
 
|-
 
|CIF_FREIGHT_UPDATE_DAILY
 
|CIF_FREIGHT_UPDATE_DAILY
Line 54: Line 81:
 
|Daily update for FREIGHT
 
|Daily update for FREIGHT
 
|[https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_FREIGHT_UPDATE_DAILY&day=toc-update-mon Mon], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_FREIGHT_UPDATE_DAILY&day=toc-update-tue Tue], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_FREIGHT_UPDATE_DAILY&day=toc-update-wed Wed], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_FREIGHT_UPDATE_DAILY&day=toc-update-thu Thu], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_FREIGHT_UPDATE_DAILY&day=toc-update-fri Fri], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_FREIGHT_UPDATE_DAILY&day=toc-update-sat Sat], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_FREIGHT_UPDATE_DAILY&day=toc-update-sun Sun]
 
|[https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_FREIGHT_UPDATE_DAILY&day=toc-update-mon Mon], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_FREIGHT_UPDATE_DAILY&day=toc-update-tue Tue], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_FREIGHT_UPDATE_DAILY&day=toc-update-wed Wed], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_FREIGHT_UPDATE_DAILY&day=toc-update-thu Thu], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_FREIGHT_UPDATE_DAILY&day=toc-update-fri Fri], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_FREIGHT_UPDATE_DAILY&day=toc-update-sat Sat], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_FREIGHT_UPDATE_DAILY&day=toc-update-sun Sun]
 +
|Not available
 
|-
 
|-
 
|}
 
|}
Line 61: Line 89:
 
A list of [[TOC Codes]] is available.  Note that the TOC code used in the URL is not the ATOC code as seen in schedule files, but the business code as used in Network Rail systems.
 
A list of [[TOC Codes]] is available.  Note that the TOC code used in the URL is not the ATOC code as seen in schedule files, but the business code as used in Network Rail systems.
  
The data files are generated between 02:45AM and 03:00AM GMT.
+
The JSON data files are generated overnight and will be available from around 0600.  The CIF files are go through a different process and will be available from around 0100.
  
 
== Downloading via curl ==
 
== Downloading via curl ==
Line 71: Line 99:
 
(Thanks to [[User:Jules Self|Jules Self]] for figuring this syntax out.)
 
(Thanks to [[User:Jules Self|Jules Self]] for figuring this syntax out.)
  
= File Format =
+
= JSON File Format =
  
Schedule data is contained in the schedule file in JSON format, with a single JSON record per line.  The file can be parsed line-by-line to extract the data required.  Each schedule file contains:
+
The schedule data in JSON format has a single JSON record per line.  The file can be parsed line-by-line to extract the data required.  Each schedule file contains:
  
 
* A header row
 
* A header row
 +
* Zero or more TIPLOC records
 
* Zero or more association records
 
* Zero or more association records
 
* Zero or more schedule records
 
* Zero or more schedule records

Revision as of 10:56, 15 January 2014

The SCHEDULE feed is an extract of train schedules from Network Rail's ITPS (Integrated Train Planning System), available in JSON and CIF format.

It is recommend that, if you are just starting out with the service, you use the JSON files. The CIF data is more suited to advanced users of the service and requires additional parsing compared to the JSON data.

Types of file

JSON format

The schedule files in JSON format are available for all operators, or for individual operators.

Two types of file are available:

  • A full extract is produced each day, which can be used to initialise your schedule database
  • An update extract is produced each day, which can be applied to the previous update extract to bring your database up-to-date

These files are generated

CIF format

The schedule files in CIF format are only available for all operators. A 'full' extract is available each Friday, and an 'update' extract is available daily.

Two types of file are available:

  • A full extract is produced each Friday, which can be used to initialise your schedule database
  • An update extract is produced each day, including Friday, which can be applied to the previous update extract to bring your database up-to-date

The CIF User Specification is available from ATOC's website, which details the format of the CIF file and the fields within.

Downloading

The schedule data is compressed using gzip and is downloaded from Amazon S3 via a private URL which is valid for a few minutes after generation. Authentication will be required either by logging in through the Data Feeds website, or by sending your username and password as part of the HTTP request. On successful authentication, you will receive a 403 redirect to the location of the schedule files. Requests are of the format:

 https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=type&day=day

where type determines the TOCs contained in the schedule extract and whether the file is a full schedule or an update file, and day determines which day's update file is downloaded. For example:

 https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_FULL_DAILY&day=toc-full

would download a full schedule file for all TOCs.

Values for type and day can be one of the following:

type day Description Example URL (JSON format) Example URL (CIF format)
CIF_ALL_FULL_DAILY toc-full Daily all-TOC snapshot Daily Weekly
CIF_ALL_UPDATE_DAILY toc-update-DAY Daily all-TOC update Mon, Tue, Wed, Thu, Fri, Sat, Sun Mon, Tue, Wed, Thu, Fri, Sat, Sun
CIF_XX_TOC_FULL_DAILY toc-full Daily snapshot for TOC XX Daily Not available
CIF_XX_TOC_UPDATE_DAILY toc-update-DAY Daily update for TOC XX Mon, Tue, Wed, Thu, Fri, Sat, Sun Not available
CIF_FREIGHT_FULL_DAILY toc-full Daily FREIGHT snapshot Daily Not available
CIF_FREIGHT_UPDATE_DAILY toc-update-DAY Daily update for FREIGHT Mon, Tue, Wed, Thu, Fri, Sat, Sun Not available

DAY should be replaced with the shortened version of the name of the previous day. For example, on Monday, Sunday's ('sun') update should be requested.

A list of TOC Codes is available. Note that the TOC code used in the URL is not the ATOC code as seen in schedule files, but the business code as used in Network Rail systems.

The JSON data files are generated overnight and will be available from around 0600. The CIF files are go through a different process and will be available from around 0100.

Downloading via curl

The schedule file can be downloaded from the command line using the curl utility (included with linux, versions available for other operating systems) with the following syntax:

 curl -L -u username:password -o file.gz 'https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_FULL_DAILY&day=toc-full'

(Thanks to Jules Self for figuring this syntax out.)

JSON File Format

The schedule data in JSON format has a single JSON record per line. The file can be parsed line-by-line to extract the data required. Each schedule file contains:

  • A header row
  • Zero or more TIPLOC records
  • Zero or more association records
  • Zero or more schedule records
  • and an end-of-file (EOF) record.

Each association and schedule record has an action - either 'create' or 'delete'. These are used in update files to add or remove records from the master schedule. In full files, there will be no 'delete' records. Update files must be applied sequentially to a full file.

Record Types

Schedule data contains two types of records:

Validity

Associations and schedule validities are between a start date and an end date, and on particular days of the week. They each have a Short Term Planning (STP) indicator field as follows:

  • C - Planned cancellation: the schedule does not apply on this date, and the train will not run. Typically seen on public holidays when an alternate schedule applies, or on Christmas Day.
  • N - STP schedule: similar to a permanent schedule, but planned through the Short Term Planning process
  • O - Overlay schedule: an alteration to a permanent schedule
  • P - Permanent schedule: a schedule planned through the Long Term Planning process

For any date a permanent schedule ('P') is valid, it may be overridden by an overlay ('O') or a planned cancellation ('C'). However, if any two schedules appear to be valid for a particular day, the schedule with the lowest alphabetical STP indicator wins.

Scheduling Process

A description of the scheduling process is available on the how scheduling works page. This describes how multiple schedules can be used to describe a specific train service.


Network Rail Open Data Feeds
Data Feeds About the Feeds Account States Durable Subscriptions Example Code ( PHP / C# / Java / Ruby / Node.js) • Advanced UsesFAQ Release Notes
RTPPM RTPPM Feed
Train Movements Train Movements Feed Train Activation Train Cancellation Train Movement Train Reinstatement Change of Origin Change of Identity Change of Location TSPEED Field Planned Cancellations Cancellation Codes
TD TD Feed C-Class Messages S-Class Messages Train Describers TD Berths
VSTP VSTP Feed
TSR TSR Feed Route Codes
SCHEDULE SCHEDULE Feed Schedule and Location Records Association Records CIF Codes How Scheduling Works Allowances
Reference Data Reference Data Feed TOC Codes CIF Codes Delay Attribution Codes Identifying Locations (STANOX, TIPLOC, NLC and 3-Alpha Codes) STANOX Geographical Areas Train Planning data