Download in Bulk
FOREX Data
Introduction
Download the FOREX dataset in bulk to get the full dataset without any selection bias. The bulk dataset packages contain data for every ticker and trading day.
Download History
To unlock local access to the Forex dataset, open the Pricing page of your organization and subscribe to at least one of the following data packages:
- OANDA Forex - Daily History
- OANDA Forex - Hour History
- OANDA Forex - Minute History
- OANDA Forex - Second History
You need billing permissions to change the organization's subscriptions.
After you subscribe to local access, follow these steps to download the data:
- Log in to the Algorithm Lab.
- On the CLI tab of the dataset listing, use the CLI Command Generator to generate your download command and then copy it.
- Open a terminal in your organization workspace and then run the command from the CLI Command Generator.
The Ticker, Start Date, and End Date fields are irrelevant for bulk downloads.
Download Daily Updates
After you bulk download the Forex dataset, new daily updates are available at 3 PM Coordinated Universal Time (UTC) after each trading day. To gain access to the data updates, open the Pricing page of your organization and subscribe to at least one of the following data packages:
- OANDA Forex - Daily Updates
- OANDA Forex - Hour Updates
- OANDA Forex - Minute Updates
- OANDA Forex - Second Updates
You need billing permissions to change the organization's subscriptions.
After you subscribe to dataset updates, to update your local copy of the Forex dataset, use the CLI Command Generator to generate your download command and then run it in a terminal in your organization workspace. Alternatively, instead of directly calling the lean data download command, you can place a Python script in the data directory of your organization workspace and run it to update your data files. The following example script updates all data resolutions:
import os
import pandas as pd
from datetime import datetime, time, timedelta
from pytz import timezone
from os.path import abspath, dirname
os.chdir(dirname(abspath(__file__)))
OVERWRITE = False
# Define a method to download the data
def __download_data(resolution, start=None, end=None):
print(f"Updating {resolution} data...")
command = f'lean data download --dataset "FOREX Data" --data-type "Bulk" --resolution "{resolution}"'
if start:
end = end if end else start
command += f" --start {start} --end {end}"
if OVERWRITE:
command += " --overwrite"
print(command)
os.system(command)
def __get_end_date() -> str:
now = datetime.now(timezone("US/Eastern"))
if now.time() > time(7,30):
return (now - timedelta(1)).strftime("%Y%m%d")
print('New data is available at 07:30 AM EST')
return (now - timedelta(2)).strftime("%Y%m%d")
def __download_high_frequency_data(latest_on_cloud):
for resolution in ["minute", "second"]:
dir_name = f"forex/oanda/{resolution}/eurusd".lower()
if not os.path.exists(dir_name):
__download_data(resolution, '19980101')
continue
latest_on_disk = sorted(os.listdir(dir_name))[-1].split('_')[0]
if latest_on_disk >= latest_on_cloud:
print(f"{resolution} data is already up to date.")
continue
__download_data(resolution, latest_on_disk, latest_on_cloud)
def __download_low_frequency_data(latest_on_cloud):
for resolution in ["daily", "hour"]:
file_name = f"forex/oanda/{resolution}/eurusd.zip".lower()
if not os.path.exists(file_name):
__download_data(resolution)
continue
latest_on_disk = str(pd.read_csv(file_name, header=None)[0].iloc[-1])[:8]
if latest_on_disk >= latest_on_cloud:
print(f"{resolution} data is already up to date.")
continue
__download_data(resolution)
if __name__ == "__main__":
latest_on_cloud = __get_end_date()
__download_low_frequency_data(latest_on_cloud)
__download_high_frequency_data(latest_on_cloud)
To update your local dataset, the preceding script checks the date of the most recent EURUSD data you have for all resolutions. If there is new data available for either of these resolutions, it downloads the new data files and overwrites your hourly and daily files. If you don't intend to download all resolutions, adjust this script to your needs.
You can also see our Videos. You can also get in touch with us via Discord.
Did you find this page helpful?
