Using cutting-edge open-source technologies to build one of the biggest industrial Data Lake of the World

Allan Sene

On this talk, we will discuss how DataSprints is using cutting-edge open-source technologies, such as Dremio and dbt, to build one of the biggest industrial Data Lakes of the World, serving more than 200 reports, processing more than 300Gb and 2000+ columns datasets in near-realtime (8s), keeping the cloud costs very low, and by very low we mean, VERY LOW. We will pass through the architecture now in production, all the challenges and lessons learned on this project.

About Allan Sene:

Allan has a background in Computer Science and Statistics, having worked with Datasince 2010, from genetic data transformation to complex industrial data lakes. He's Co-Founder & CTO at DataSprints and Co-Founder of Data Hackers.

Media

Slide deck @ SpeakerDeck

Intro to FastAPI

Sebastián Ramírez @tiangolo

Learn how to create an API ready for production in very little time using FastAPI... explained with memes. Your API will have automatic validation, documentation based on standards, high performance, and several other features. All this, having editor support including autocompletion everywhere.

In this talk you will learn what FastAPI can do, and how it could benefit you. You will see how to declare the data you want to receive in each request using standard Python type annotations. Including path parameters, query parameters, body payloads with JSON, etc.

You will also see how to use simple, standard, Python type annotations to declare complex JSON body payloads with deeply nested structures, and get automatic data validation, serialization, and documentation.

About Sebastián Ramírez:

Sebastián (tiangolo) is the creator of FastAPI, Typer, and other open source tools. Currently he's a developer at Explosion in Berlin, Germany.

Media

Slide deck @ SpeakerDeck