← LeaderboardThe Skill Leaderboard

data-doc

Document datasets, variables, sources, and merge keys for replication

Published by brycewang-stanford ↗brycewang-stanford/Auto-Empirical-Research-Skills

Try your first task

data-doc

Start a fresh conversation. Replace the highlighted placeholders.

Use data-doc to generate a dataset documentation template for [dataset name and path] including its overview, source data, key variables, variable construction details, sample filters, and merge keys. [primary:12-71]

Required inputs

  • [dataset name and path]
  • [primary:12-71]

User-approved starter prompt based on a pinned source version; check current skill requirements before running. Task evidence ↗

SKILL.md

name: data-docdescription: Document datasets, variables, sources, and merge keys for replication

Data Documentation Assistant

Help document datasets systematically for replication packages, co-author handoffs, and future reference.

Dataset Documentation Template

When documenting a dataset, capture:

1. Dataset Overview

  • Name: (e.g., firm_year_panel.parquet)
  • Location: (relative path from project root)
  • Created by: (script that generates it)
  • Last updated:
  • Unit of observation: (firm-year, loan-quarter, etc.)
  • N observations:
  • Time coverage:

2. Source Data

SourceAccessRaw FileNotes
WRDS CompustatSubscriptionraw/compustat_funda.csvAnnual fundamentals
Revelio LabsLicensedraw/revelio_positions.parquetVia BU server
Hand-collectedManualraw/manual_coding.xlsxSee coding protocol

3. Key Variables

VariableTypeDescriptionSourceNotes
gvkeystrCompustat firm identifierCompustatPrimary key
fyearintFiscal yearCompustat
atfloatTotal assets ($ millions)CompustatWinsorized 1/99
treatedint=1 if treated firmConstructedSee section 4

4. Variable Construction

For constructed/derived variables, document:

Variable: treated
Definition: =1 if firm received first PE investment in year t
Construction:
  1. Merge PitchBook deals to Compustat on EIN
  2. Keep first deal per firm
  3. Flag year of first investment
Script: 2a_construct_treatment.py, lines 45-78

5. Sample Filters

Document all filters applied:

FilterObservations DroppedRemaining
Raw data-150,000
Drop financials (SIC 6000-6999)25,000125,000
Require non-missing assets5,000120,000
Require 2+ years in panel10,000110,000

6. Merge Keys

Dataset ADataset BKey(s)Match RateNotes
CompustatCRSPgvkey95%Via CCM link table
CompustatPitchBookEIN72%Manual cleaning needed
RevelioCompustatcompany_name68%Fuzzy match, see script

Quick Commands

  • "document this dataset" - Generate full template for a dataset
  • "variable list" - Create variable table only
  • "merge documentation" - Focus on merge keys and match rates
  • "sample flow" - Generate sample filter table
  • "codebook" - Formal codebook format for replication package

Codebook Format (for Replication Packages)

================================================================================
CODEBOOK: firm_year_panel.dta
Generated: 2026-01-15
================================================================================

IDENTIFICATION
  gvkey         Compustat permanent firm identifier
  fyear         Fiscal year

OUTCOME VARIABLES
  roa           Return on assets = ni/at (winsorized 1/99)
  investment    Capex/lagged assets = capx/L.at

TREATMENT VARIABLES
  post          =1 for years after treatment
  treated       =1 for firms ever treated
  treat_post    Interaction: treated × post (DiD coefficient)

CONTROLS
  size          Log total assets = ln(at)
  leverage      Book leverage = (dltt+dlc)/at
  mtb           Market-to-book = (prcc_f×csho)/ceq

FIXED EFFECTS
  ff48          Fama-French 48 industry classification
  state         State of incorporation
================================================================================

Data Provenance Checklist

Before finalizing a dataset, verify:

  • All source files documented with access instructions
  • Variable definitions are unambiguous
  • Sample filters documented with observation counts
  • Merge match rates reported
  • Winsorization/trimming documented
  • Missing value treatment documented
  • Script that creates dataset is identified
  • Date of creation recorded