Meetup Online #38 - DataOps - DevOps Porto & DSPT

23/07/2020

Agenda:

  • Welcome
  • Talk: Using cutting-edge open-source technologies to build one of the biggest industrial Data Lake of the World by Allan Sene
  • Talk: Intro to FastAPI by Sebastián Ramírez
  • Closing

Video

DevOpsPorto YouTube
Image

Using cutting-edge open-source technologies to build one of the biggest industrial Data Lake of the World

Allan Sene

On this talk, we will discuss how DataSprints is using cutting-edge open-source technologies, such as Dremio and dbt, to build one of the biggest industrial Data Lakes of the World, serving more than 200 reports, processing more than 300Gb and 2000+ columns datasets in near-realtime (8s), keeping the cloud costs very low, and by very low we mean, VERY LOW. We will pass through the architecture now in production, all the challenges and lessons learned on this project.

About Allan Sene:

Allan has a background in Computer Science and Statistics, having worked with Datasince 2010, from genetic data transformation to complex industrial data lakes. He's Co-Founder & CTO at DataSprints and Co-Founder of Data Hackers.

Media

Slide deck @ SpeakerDeck

Image

Intro to FastAPI

Sebastián Ramírez @tiangolo

Learn how to create an API ready for production in very little time using FastAPI... explained with memes. Your API will have automatic validation, documentation based on standards, high performance, and several other features. All this, having editor support including autocompletion everywhere.

In this talk you will learn what FastAPI can do, and how it could benefit you. You will see how to declare the data you want to receive in each request using standard Python type annotations. Including path parameters, query parameters, body payloads with JSON, etc.

You will also see how to use simple, standard, Python type annotations to declare complex JSON body payloads with deeply nested structures, and get automatic data validation, serialization, and documentation.

About Sebastián Ramírez:

Sebastián (tiangolo) is the creator of FastAPI, Typer, and other open source tools. Currently he's a developer at Explosion in Berlin, Germany.

Media

Slide deck @ SpeakerDeck


Join the discussion

The community is regularly present in Slack, so get on there ASAP and join in on the conversation. Get your invite at https://devopsporto.herokuapp.com/

Want to present at the meetup?

Interested in presenting at a Meetup? We're interested in having you there. Checkout the how the talk submission process works. Afterwards, please fill out our Call for speakers form with your talk idea.

Code of Conduct

Please keep in mind our code-of-conduct when participating in the community. We believe this code is essential to a healthy community and clearly in line with the values of DevOps.