Large-scale data science for real-world data

Summary

This hands-on course covers tools and methods used by data scientists, from researching solutions to scaling prototypes on Spark clusters. Students engage with the full data engineering and data science pipeline, from data acquisition to extracting insights, applied to real-world problems.

Content

Keywords

Data Engineering, Data Lakes, Machine Learning Operations (MLOps), Distributed Computing, Real-Time Data Stream Processing, Scalable Data Processing, Large-Scale Data Analysis, Predictive Modeling, Apache Spark, Hadoop, Kafka.

Learning Prerequisites

Important concepts to start the course

Participants are expected to have prior experience with Python programming and understanding of fundamental mathematical concepts relevant to data science. Familiarity with key data science libraries - such as NumPy, pandas, and scikit-learn - is strongly recommended. A basic understanding of using the Linux terminal, including navigating file systems and executing command-line tools, is also beneficial.

Learning Outcomes

By the end of the course, the student must be able to:

• Apply and coordinate the use of standard data science libraries and big data technologies to manage distributed and real-time data workflows.
• Design , build, and optimize data lakes using efficient storage formats to enable scalable, high-performance data engineering.
• Conduct large-scale data wrangling, transformation, and model training tasks on complex, real-world datasets.
• Design , develop, and optimize scalable data pipelines for both batch and streaming contexts.
• Integrate machine learning techniques into end-to-end data science workflows using appropriate tools and environments.
• Interpret complex datasets and evaluate outcomes to extract actionable insights that support data-driven decision-making.
• Formulate and justify technical choices in data storage, pipeline design, and machine learning model deployment, and communicate results clearly through visualizations, documentation, and oral presentations.

Transversal skills

• Continue to work through difficulties or initial failure to find optimal solutions.
• Identify the different roles that are involved in well-functioning teams and assume different roles, including leadership roles.
• Use a work methodology appropriate to the task.
• Manage priorities.
• Use both general and domain specific IT resources and tools

Teaching methods

Assessment is based on lectures and hands-on lab sessions, with all activities involving real-world datasets and the use of distributed computing and storage services to ensure practical, applied learning.

Expected student activities

• Apply: Put concepts into practice during hands-on lab sessions.
• Engage: Take part in class discussions and interactive activities.
• Collaborate: Work in teams to complete assignments and tackle real-world challenges.
• Explain: Present your ideas and results clearly and concisely.

Assessment methods

• 60% Continuous group assessments during the semester
• 40% Final group project

Supervision

Resources

Virtual desktop infrastructure (VDI)

Yes

Bibliography

• Python Data Science Handbook: Essential Tools for Working with Data by Jake VanderPlas, O'Reilly Media, 2023

Ressources en bibliothèque

• Find the references at the Library

Moodle Link

• https://go.epfl.ch/COM-490