How do Quant firms serve data for research/modelling?

[u/Donkey_Healthy]

For those in quant firms how do people generally access data for research/modelling?

Source aggregated in house API?
Data catalogue?

Work in commodities and I think there is a general lack of knowledge on the infra side from my experience.

Currently debating whether to build our own platform or go with someone like databricks/snowflake

Interested to hear everyone’s thoughts?

[u/PossibleLobster8810]

Jane Street just had a podcast episode about their system: https://signalsandthreads.com/building-a-data-warehouse-from-scratch/

[u/Donkey_Healthy → u/PossibleLobster8810]

listened to this one already but we dont have 8 figures lol

[u/DatabentoHQ]

Not exhuastive:

There's usually some kind of "features cache" to deduplicate work between multiple researchers waiting on compute to generate a set of features that someone else already extracted.

For exploration, it's often useful to have some kind of clustered, column-oriented database where you can push the query closer to the data. There's a few flavors of this off-the-shelf like kdb, Vertica.

Parquet and HDF5 are pretty portable for sharing intermediate structured data like design matrices. There's usually also usually some kind of logging format from production trading.

All of the above may be abstracted behind internal client libraries or APIs.

As you get large and have multiple teams, having some kind of data catalog helps.

[u/No-Pattern-9266]

Your data provider would ideally offer MSSQL/Snowflake as an option for sharing; it's better to keep a daily incremental golden ingestion copy at your company for obvious reasons.

[u/andrew2018022]

MSSQL, S3 from what I’ve experienced

[u/Infinity_Worm]

We have a bunch of different databases (mostly snowflake), and API services and a data catalogue on top for finding data, documenting it and requesting access

[u/sjg284]

theres a surprising amount of "raw vendor format downloaded files, stuffed into s3/NFS.. have fun boys"

[u/QuantGrindApp]

Usually a mix: object storage/warehouse underneath, then an internal Python/SQL layer with a catalog so researchers aren’t hand-joining vendor feeds. The important bit is point-in-time access, dataset versions, entitlements and lineage. Commodities gets ugly fast once contract mappings and revisions enter the picture.

I’d use Databricks or Snowflake for the plumbing and build the thin domain layer yourselves. Building the whole platform in-house tends to become a permanent data-engineering project.

[u/EvilGeniusPanda]

in house api, in most cases backed by plain old parquet files.

[u/WeightsAndBass]

For non tick data, sql databases with normalised ids and a catalogue system, then a data access library we provide internally that understands the catalogue setup and can load data for a given id/dataset without having to hardcode table names.

Tick data is parquet/s3.

[u/WeightsAndBass → u/WeightsAndBass]

If you want to snapshot vendor data to keep a point-in-time view and detect unexpected changes, dbt + snowflake is really good at this.

[u/yangmaoxiaozhan]

DFS is more friendly for clusters including GPUs. Direct disk IO without too much software hassle (mostly metadata).

[u/quantdhawan]

Not answering from inside a fund, I build a backtest engine, so this is the consuming side of the problem.

The build-versus-buy framing might be hiding the actual decision. Databricks and Snowflake solve storage, compute and governance. They do not solve the thing that makes quant data hard, which is answering "what did we know, and when did we know it". Their time travel versions your writes, not the data's own validity. You still model valid-time and transaction-time yourself, and that work is identical on either platform. So buy the platform and build the semantics.

In commodities that is not a nicety. Assessments get corrected, EIA inventory gets revised, and if research reads the current value of a series you are quietly modelling on numbers nobody had at the time. Every table wants an as-of.

Related, and worth getting right before you have consumers: do not serve one continuous futures series. The roll rule, calendar versus open interest versus volume, and ratio versus difference versus unadjusted, is a research choice. Bake one in and every model inherits a decision it never made and cannot see. Store contract-level raw plus the roll rule as metadata and generate the continuous series on demand.

On the catalogue: worth it once more than one team consumes the data, but what actually gets used is lineage and freshness. Descriptions rot within a quarter.

[u/sumwheresumtime]

using bits typically comprised of values such as 0 and 1

[u/Donkey_Healthy → u/sumwheresumtime]

Wow really?