SciData Hub
One door to 40+ scientific datasets — curated Parquet on Hugging Face, federated external collections, SQL and Python access, and MCP tools for research agents.
The Problem
Scientific data is public but scattered. Galaxy rotation curves live in one archive. Dark matter detector results in another. Protein interactions in a REST API. Number theory databases behind their own query syntax. Each source has its own format, pagination, rate limits, and documentation — if any.
If you're testing a hypothesis that touches more than one domain, you spend days on plumbing before you touch science. SciData Hub is the front door: one interface to discover, load, query, and join across 40+ datasets.
One Door, Four Ways In
Hugging Face Parquet — cleaned datasets under the
scidata-hub org. Versioned, streamable, one-line load.
Python library — load() for cached data,
query() for live APIs, discover() to find what exists,
join() to cross-match catalogs by coordinates or keys.
DuckDB views — a registry file with SQL views that read Parquet directly from Hugging Face over HTTP. No install, any language, joins work out of the box.
MCP server — LLM agents call tools to discover schemas, run queries, and join datasets without knowing which API sits underneath.
Federation, Not Duplication
SciData Hub doesn't copy what others maintain. MultimodalUniverse hosts 24 astronomical surveys — JWST, DESI, TESS — over 100 TB. They stay where they are; our registry routes to them and makes them joinable with our data. science-datalake exposes ~293M papers as queryable Parquet. SciData Hub federates it: query by DOI, trace which datasets a paper references, route to the source.
Seven ontologies — GO, ChEBI, HPO, DOID, MeSH, CSO, FMA — published as Parquet alongside the research data. A gene name in one paper, a chemical ID in another, and a disease term in a third resolve to the same entity graph. Research data plus shared identifiers in one place.
What It Looks Like
from scidata import load, query, discover, join
# One line — rotation curves from Hugging Face
sparc = load("sparc")
# Live API, same interface
curves = query("lmfdb",
filters={"conductor_max": 1000, "rank": 2},
fields=["conductor", "rank", "bsd_ratio"]
)
# What exists for this question?
hits = discover("dark matter annual modulation NaI")
# → anais112, cosine100, xenon1t, ...
# Cross-catalog join — the part nobody else ships
result = join(
left=load("sparc"),
right=query("sdss", filters={"type": "galaxy"}),
on="sky_coords",
tolerance_arcsec=5.0
)
Why One Door Matters
For researchers: stop losing days to API archaeology. Load SPARC, query LMFDB, join on sky coordinates — same afternoon.
For agents: simulation pipelines import scidata or
call MCP tools instead of generating fragile one-off HTTP scripts per hypothesis.
For the commons: cleaned Parquet, documented schemas, and DuckDB views on Hugging Face — reusable by anyone, whether or not they use Impossible Papers.
Live but still growing. New domains mean new sources; each follows the same plugin
pattern — one file in sources/, one DuckDB view, discoverable from day one.
Working on a public dataset that should be behind this door?
Get in touch.