Democratizing Data Science

Before data can be analyzed, it is necessary to know where the data is located and what it actually means. This process is called “Schema Discovery”, and it is essential before any insights can be gained from the data. Especially in the field of Machine Learning, it has become a frequent but often underestimated activity.

This process can be improved by mapping the initially “chaotic” set of files to a database schema, which can then be iteratively refined and loaded. The goal is to automate the previously tedious parts of this process with the help of Large Language Models (LLMs).

This presentation introduces “DataLoom”, a prototype that carefully orchestrates the use of LLMs for the “soft” problems and traditional algorithms for the “hard” problems in data loading.