How Citadel harnessed the weather to claim hedge fund crown examines how Citadel invested in access to weather forecasts that are more accurate than most meteorological offices as they increased their portfolio in energy trading and commodities. More accurate and faster weather forecasts can provide an edge and mitigate the risk of losses. 

While trading firms typically have multiple weather vendors for short-term forecasts up to 14 days ahead, subseasonal to seasonal (S2S) forecasts - which are weeks and months ahead - can offer an edge by extending the forecasting horizon. S2S forecasts are in a different category from short-term forecasts.

When the success of trading desks depends on having trustworthy and accurate weather information – forecasting solutions are among the most important services the energy trading industry is purchasing in 2024 (source: Commodities People) – firms should be discriminating when evaluating data sources. Our recommendation: start by asking potential vendors the right questions.

1. Are your forecasts probabilistic?

Forecasts beyond 2 weeks require probability distributions. Due to the chaos of the atmosphere, it is unrealistic to predict a single value for longer timescales. Probabilistic information provides the likelihood associated with possible future outcomes. This approach gives users a more complete understanding of future events' uncertainty, allowing for more informed decision-making.

2. What are the models that contribute to your forecast?

Increases to forecast accuracy can be obtained through calibration and debiasing of government models. However, improvements to accuracy are limited by the underlying models. Additional accuracy improvements and differentiated forecasts require proprietary models.

3. How do you measure accuracy?

Metrics like Mean Absolute Error are commonly used for deterministic forecasts that predict the single best value, but do not evaluate the full performance characteristics of probabilistic forecasts. Continuous ranked probability score (CRPS) is a more relevant accuracy metric for probabilistic forecasts. It measures accuracy (how close the prediction is to the observed historical value) and precision or sharpness (how confident the forecast is). Beware of claims of 100 percent higher accuracy using metrics like hit rate which are easily gamed.

4. Do you share your skill metrics?

Transparency around the skill of models is key to selecting a trustworthy vendor. S2S forecasts are not a crystal ball and skill varies by location, season, lead time and variable. Understanding the accuracy of a model quantitatively (as opposed to simply qualitatively) across these dimensions can help inform decision making. 

5. What is your methodology for backtesting?

It’s important to ensure that data leakage is not inflating backtesting performance. In a data-driven forecasting system, a proper cross-validation scheme is critical. Ask probing questions regarding the statistical rigor employed by the vendor when backtesting.

6. How do you ensure that your forecasts are reliable? 

Confident probabilistic decision making requires reliable forecasts. “Reliability” measures how well the forecasted probabilities match the observed frequency for particular events. In other words, when aggregating across every forecast that predicts colder than normal conditions in week 3 with a 60 percent probability, a colder-than-normal week should actually occur 60 percent of the time. Basically, “reliability” relates to whether or not the user can rely on the forecast when making decisions.

A transparent vendor will be able to provide reliability plots for their models.

7.  Do you provide hindcasts?

Hindcasts are produced in the same way as the real-time forecasts but for past dates that can be compared to the historical record. Access to hindcasts is another factor in building trust with a potential vendor and allows you to independently validate forecast accuracy.

8. Can your forecast output support traders and quant teams? 

The ability for multiple teams to use the same set of forecast data provides cost savings and a single source of information, avoiding multiple vendors and different interpretations. Do the delivery methods support your needs? Is the user interface user friendly? Does the vendor provide an API with access to data in the formats that you need?

9. What percentage of your employees are climate scientists, data scientists, machine learning and data engineers?

The proportion of employees with these types of titles tells you the level of investment a company is making in the research and development of new AI models.

10. How are you  keeping up with the rapid pace of innovation?

The use of AI and machine learning is an emerging world with rapid innovations. Organizations that have the expertise and agility to deliver new models at a rapid pace will have an edge. 

The bottom line 

When buying weather forecasts for energy and commodity trading, asking the right questions of weather vendors is crucial to making informed decisions.

PS: Join Salient’s Chief Product Officer, Janet Lee, for a sneak peek of what’s coming in late 2024 and 2025 at our upcoming webinar on September 10, 2024: Closing the Energy Trading Decision Gap with S2S Forecasts [register].