The Problem: Every Analysis Starts from Zero
A data science team needs to solve a specific problem: predict which enterprise customers are at risk of churn in the next 90 days. Two team members independently start researching approaches. One spends 3 days exploring gradient boosting methods and finds a technique blog post from Towards Data Science. The other spends 2 days on survival analysis approaches and finds an arXiv paper from a team at a major tech company who solved this exact problem with a different method.
They compare notes and realize: they were researching the same question from different directions, reading different papers, and had no shared starting point. The approach they end up choosing isn't informed by a systematic literature review — it's informed by what they each happened to find.
Research workflows for data scientists are the structured processes that prevent this scattered approach — ensuring that the relevant prior work is identified systematically, the right approach is selected for the specific problem and data context, and the research process produces findings that are reproducible and building blocks for future work.
What Data Science Research Actually Requires
Problem definition before method selection: The most common data science research failure is selecting a method before fully defining the problem. The question "should I use XGBoost or neural networks for this?" is unanswerable without first defining: what is the prediction target precisely? What are the constraints (inference latency, explainability, data volume)? What does success look like? Problem definition determines method appropriateness.
Literature grounding: Most applied data science problems have been solved before, in the same domain or in analogous ones. A systematic literature review — arXiv, papers from major ML conferences, relevant domain journals, engineering blogs from companies that have solved similar problems — identifies prior approaches, benchmark performance, and known pitfalls before reinventing them.
Data reality: Research hypotheses are often invalidated by the actual data. A research workflow that includes early exploratory data analysis — understanding what's actually in the data, what's missing, what the distributions look like, what the label balance is — prevents building a sophisticated approach on a data foundation that won't support it.
Reproducibility by design: A research workflow that captures dataset versions, random seeds, environment specifications, and experiment metadata as the work proceeds produces reproducible results. One that treats documentation as a post-analysis task produces results that can't be reproduced, verified, or built upon.
The Data Science Research Workflow, Stage by Stage
Stage 1: Problem Definition and Literature Review
Define the problem precisely:
Before any code is written:
- What is the exact prediction target? (Not "churn" but: "customer account closed or downgraded within 90 days of observation date")
- What are the constraints? (Real-time inference? Monthly batch? Explainability required for compliance?)
- What data is available? (Source systems, update frequency, historical depth)
- What does good look like? (AUC-ROC threshold? Business-level metric like prevented churn revenue?)
- What does failure look like? (Which errors are more costly — false positives or false negatives?)
Literature review:
With the problem defined, survey what's known:
- arXiv: Search for papers on your specific prediction problem in your industry domain
- Conference proceedings: ACM KDD, NeurIPS, ICML, RecSys — depending on the problem type
- Engineering blogs: Netflix Tech Blog, Airbnb Engineering, DoorDash Engineering, LinkedIn Engineering — applied ML teams document their solutions
- Domain literature: Healthcare prediction models, financial risk modeling, manufacturing quality prediction — domain-specific journals often have relevant prior work
Literature note:
For each relevant source, capture (in your research library):
- What problem did they solve (precisely)?
- What method did they use?
- What results did they achieve on what dataset?
- What would need to be different about your context for their approach to not apply?
Stage 2: Exploratory Data Analysis
Before committing to a modeling approach, understand the data:
Data inventory:
- What tables/datasets are available? What time periods? What's the join key?
- What's the update latency? (Daily? Real-time?)
- What's the historical depth for training?
Distribution analysis:
- What does the target variable distribution look like? (Class imbalance?)
- What do the feature distributions look like? (Outliers, skewness, seasonality)
- What's missing? (NULL rates, feature availability over time)
Quality assessment:
- Are there known data quality issues? (Talk to the data engineering team before assuming)
- Are there leakage risks? (Features computed from data that wouldn't be available at prediction time)
- Are there consistency issues across time? (Schema changes, data definition changes)
EDA findings documentation:
The EDA findings should be captured in a structured summary — not just the final approach but what the data revealed that shaped the approach:
- "Target class imbalance: 4.2% positive rate — will need to address in training and evaluation"
- "Feature X has 23% null rate in the 2022 data; the team confirmed a collection issue that was resolved in Q3 2022 — training data should start Q3 2022 to avoid this"
- "Feature Y creates leakage — it's computed after the churn event in the source system; must exclude"
These EDA findings are as important to capture as model results. They explain the approach.
Stage 3: Approach Selection and Experiment Design
With problem definition, literature grounding, and EDA complete, design the research approach:
Baseline first:
Before any sophisticated modeling — establish a naive baseline. Predict majority class for classification problems. Predict mean or prior-period value for regression. What does "predicting nothing" achieve? This is the floor.
Literature-informed candidates:
From the literature review, identify 2-3 candidate approaches:
- Method from the most directly relevant paper
- Method that best fits the specific constraints (latency, explainability)
- A simpler baseline that might be "good enough"
Experimental design:
Define before running experiments:
- Train/validation/test split strategy (time-based splitting for time-series problems; random split with stratification for cross-sectional problems)
- Primary evaluation metric (and why this metric rather than alternatives)
- Minimum meaningful improvement over baseline (what would make the new approach worth deploying?)
- Number of random seeds for robustness testing
Stage 4: Experiments and Documentation
Run experiments systematically:
Use experiment tracking from the first run (MLflow, Weights & Biases, or Neptune). Capture:
- Dataset version and split parameters
- Feature set used
- Model architecture and all hyperparameters
- Random seeds
- Evaluation results on validation and test sets
- Training time and inference time (if latency is a constraint)
Negative results count:
When an approach doesn't work as expected, document why — the finding is as valuable as a positive result. "LSTM approaches did not outperform gradient boosting on this problem (AUC-ROC: 0.821 vs. 0.847) despite 3x longer training time; likely due to limited historical data depth rather than architecture mismatch" is a useful negative result.
Experiment summaries:
After each significant experiment batch, a brief written summary: what was tested, what was found, what the interpretation is, and what the next experiment should test.
Stage 5: Findings Documentation and Communication
Technical write-up:
For any experiment or research phase that produces findings worth communicating:
- Problem definition (precise)
- Approach taken and rationale (grounded in literature and EDA)
- Results (with confidence intervals; statistical significance where applicable)
- Limitations and caveats
- Recommended next steps
Stakeholder communication:
For findings that inform business decisions:
- What is the business interpretation of the model performance? (Not AUC-ROC, but "the model correctly identifies 70% of churners while flagging 15% of non-churners")
- What are the error modes and their business cost? (Which churners do we miss? Which customers do we incorrectly flag?)
- What data or time would meaningfully improve the model?
A Recommended Tool Stack for Data Science Research
| Stage | Tool | Notes |
|---|
| Literature review | arXiv + conference proceedings + engineering blogs + WebSnips | Systematic search; capture relevant sources |
| Problem definition | Notion / Confluence | Structured problem spec document |
| EDA | Jupyter notebooks + Pandas Profiling / ydata-profiling | Interactive + documented findings |
| Experiment tracking | MLflow / Weights & Biases | Automatic experiment metadata capture |
| Dataset versioning | DVC | Track dataset versions alongside code |
| Code versioning | Git + GitHub | Including notebook versioning |
| Research library | Notion + WebSnips | Organized external research with applicability notes |
WebSnips for data science research: Engineering blog posts from Netflix, Airbnb, DoorDash, LinkedIn, and other applied ML teams document practical solutions to common problems — often more directly applicable than academic papers. These posts get updated or removed over time; a clip with date and source URL preserves the content and the context of when it was relevant. A WebSnips collection organized by technique area (survival analysis, time-series forecasting, class imbalance, feature engineering) builds a research library that's retrievable by topic rather than scattered in browser history.
A Worked Example
A data scientist, Nadia Kim, uses a structured research workflow for a purchase propensity model:
Stage 1 — Problem definition:
Target: P(customer makes a repeat purchase within 30 days of first purchase)
Constraint: Inference at purchase time; must run in <200ms
Success metric: AUC-ROC ≥ 0.80 on holdout; top-decile lift ≥ 3.5x
Data available: 14-month transaction history; customer demographic features; product category features
Literature review:
Nadia searches arXiv and engineering blogs for "purchase propensity" and "repeat purchase prediction." Finds 3 directly relevant papers and an Instacart Engineering blog post (clips saved to WebSnips, organized in "Propensity Models" collection):
- Paper 1: BG/NBD model (2005) — probabilistic, interpretable, works well with limited data
- Paper 2: Deep neural approach with transaction sequences — requires 6+ months of history; outperforms BG/NBD in large-data settings
- Engineering blog: Instacart's approach combining recency/frequency features with gradient boosting
Conclusion: With 14 months of data, gradient boosting with recency/frequency features (Instacart approach) is the best starting point; BG/NBD as an interpretable baseline.
Stage 2 — EDA:
Key findings:
- 73% of customers have 1-2 purchases only — limited sequence data; neural sequence approaches unlikely to work
- Class imbalance: 18% positive rate (repeat within 30 days) — manageable, but need to evaluate precision/recall carefully
- Recency feature shows strong bivariate correlation with target; validates the Instacart approach direction
- 8% null rate on demographic features — will test impact of imputation vs. exclusion
Stage 3-4 — Experiments:
Experiment PROP-001: BG/NBD baseline — AUC-ROC: 0.734
Experiment PROP-002: XGBoost with recency/frequency features — AUC-ROC: 0.812; top-decile lift: 4.1x
Experiment PROP-003: Add product category features — AUC-ROC: 0.831; top-decile lift: 4.4x
Experiment PROP-004: Add demographic features (imputed) — AUC-ROC: 0.829; no improvement; demographic features excluded from final model
Stage 5 — Findings:
"PROP-003 meets our success criteria (AUC-ROC 0.831, top-decile lift 4.4x). Recommended for production evaluation. Demographic features don't improve performance — simplified model (no demographic data dependency) preferred. Inference time: 47ms at p99 (well within 200ms constraint)."
Compliance and Ethics Notes
Training data fairness:
For models that may make decisions affecting people differently based on demographic characteristics (even indirectly through correlated features), fairness analysis should be part of the research workflow. Test for disparate impact across protected characteristics. Document findings.
Data lineage and governance:
In regulated industries (financial services, healthcare), training data lineage — what data was used, when it was collected, how it was transformed — is a regulatory requirement in some contexts. Research workflows should capture data lineage from the beginning, not as a post-deployment documentation task.
Reproducibility as a professional standard:
In research contexts (academic, published work, or high-stakes business decisions), failure to document sufficient information for reproduction is a professional standard issue, not just an inconvenience. Results that inform significant business decisions should be reproducible by another team member from the documented workflow.
Common Data Science Research Workflow Mistakes
Mistake 1: Method selection before problem definition.
Picking XGBoost because it worked on the last project before defining what "success" means on the current project. The best method depends on the constraints, data characteristics, and business interpretation — none of which are known without problem definition.
Mistake 2: Skipping literature review.
Starting from scratch on a problem that has established prior work in the literature. A 2-day systematic literature review often identifies the approach that months of independent experimentation would eventually converge to.
Mistake 3: EDA as an afterthought.
Discovering data quality issues, leakage, or fundamental distribution characteristics after spending weeks on modeling. EDA findings that would have changed the approach should be found before the modeling approach is selected.
Mistake 4: Experiment tracking that starts on "the important experiments."
"I'll use proper tracking when I'm running the real experiments" — meanwhile, the exploratory runs that establish the baseline and eliminate dead-end approaches aren't captured. Track everything from the first run.
Mistake 5: Research workflow that produces results but not decisions.
A technically rigorous research workflow that produces AUC-ROC scores but not business-interpretable findings doesn't inform the decisions the research was supposed to support. Every research phase should end with a stakeholder-interpretable conclusion.
Key Takeaways
- Research workflow for data scientists covers five stages: problem definition and literature review, exploratory data analysis, approach selection and experiment design, experiments with documentation, and findings communication — each generating knowledge that informs the next stage.
- Problem definition before method selection: the best modeling approach depends on constraints, data characteristics, and success criteria that must be defined before any code is written.
- Literature review identifies prior work: most applied data science problems have been solved in the same or analogous domains; 2 days of systematic literature search prevents weeks of independent rediscovery.
- EDA findings are as important as model results: data quality issues, leakage risks, and distribution characteristics that shape the modeling approach should be captured, not just the model performance numbers.
- Track everything from the first experiment run: the "exploratory" experiments that establish baselines and eliminate dead ends are part of the research record; start MLflow/W&B tracking from the beginning.
- Research findings should be business-interpretable: AUC-ROC communicates to the data science team; precision/recall at the operating threshold and business-impact framing communicates to decision-makers.
Conclusion
A research workflow for data scientists is what converts individual technical work into cumulative organizational intelligence. The data scientist who defines the problem precisely, grounds the approach in prior literature, documents EDA findings that shaped the approach, tracks every experiment, and communicates findings in business terms is building work that can be reproduced, built upon, and used to make decisions. The one who starts with a method and produces results — with no documented rationale, no reproducibility, and no business interpretation — is doing work that ends when the project ends. The workflow is how data science compounds.
Try WebSnips free — clip engineering blog posts, arXiv papers, framework documentation, and applied ML research from the web with date and source URL, building an organized research library that makes relevant prior work findable rather than repeatedly re-discovered.