Skip to content

README.md

Latest commit

 

History

History
84 lines (58 loc) · 3.77 KB

README.md

File metadata and controls

84 lines (58 loc) · 3.77 KB
Raw

AI in the Data Science Job Market

Open in Streamlit Future Proof DS Dataset on Hugging Face

What skills do Data Scientists actually need right now? This dashboard answers that by extracting and categorizing skills from thousands of real job postings, with a focus on how AI is reshaping the role.

Dashboard Preview

What You'll Find

  • Top skills employers are hiring for: ranked by frequency across all postings
  • Skills by category: Programming, Cloud, ML/DL, AI/GenAI, and more
  • AI disruption analysis: what % of DS roles now require AI skills, which seniority levels are most affected, and the salary premium for AI-skilled candidates
  • Interactive job explorer: browse individual postings with matched skills highlighted

Getting Started

Prerequisites

Installation

git clone https://github.com/andresvourakis/ai-ds-job-market.git
cd ai-ds-job-market
make install

Pages

  • AI in Data Science: how AI skills show up in Data Scientist postings.
  • ML in Production: how often postings require production skills (deploying, serving, monitoring, CI/CD) and which cloud platforms companies ask for. Keyword decisions are documented in docs/production_keyword_audit.md.

Running the Dashboard

make run

Opens at http://localhost:8501.

Data

The dataset contains thousands of Data Science job postings collected via Google's Jobs API. Each posting includes title, company, location, full description, salary (when available), posting date, and schedule type.

The dataset is refreshed weekly by an automated scraper and published to Hugging Face: futureproofds/ai-ds-job-market. The dashboard downloads jobs_merged.json.gz from there at startup (cached for 6 hours), so nothing needs to be committed to this repo when the data changes. To develop against a specific file, drop it at data/jobs_merged.json and it takes precedence.

How Skill Extraction Works

Job descriptions are matched against ~230 curated keyword groups in analyse_job_market.py. Each group maps variations of a skill to a canonical name (e.g., "PostgreSQL", "psql" → "SQL"). Both original and NLTK-lemmatized forms are matched to handle plurals and word variations.

Skills are organized into 10 categories:

Category Examples
Programming Languages Python, R, SQL, Java, Scala
Data Storage & Databases PostgreSQL, MongoDB, Snowflake, Redis
Data Processing & ETL Spark, Kafka, Airflow, Databricks
Data Visualization Tableau, Power BI, Looker, Plotly
ML/DL Frameworks PyTorch, TensorFlow, Scikit-learn, XGBoost
Cloud Platforms AWS, GCP, Azure
Development Tools Git, Jupyter Notebook, Anaconda
DevOps & MLOps Docker, Kubernetes, MLflow
Core Data Science Machine Learning, A/B Testing, Feature Engineering
Artificial Intelligence LLMs, GenAI, RAG, Prompt Engineering, LangChain

Tech Stack

  • Streamlit: Dashboard framework
  • Pandas: Data manipulation
  • Plotly: Interactive charts
  • NLTK: Lemmatization for keyword matching

License

MIT - see LICENSE.