DuckDB

DuckDB is a popular and powerful “in-process SQL OLAP database management system

It provides APIs in multiple languages, including R, Python, and Julia.

In practical terms, it provides a very efficient way to read, query, and write big data in R, which includes datasets that are bigger than memory.

For example, it allows lazy-reading of large datasets with optional filtering, selecting, and grouping operations before reading into memory.

Resources