AWS: DuckDB will provide 'connective tissue' across the data estate

Last week, AWS bought DuckLabs, the people behind the open source OLAP system DuckDB.

Since the cloud giant already owns a data warehouse – in Redshift – and it doesn’t get the DuckDB intellectual property (it remains open source), then why would it bother? Andy Warfield, AWS VP and Distinguished Engineer, argued the acquisition is a major plank of AWS’s thinking when it considers the future of application building, data, and analytics.

“Half of why DuckDB is perceived as being valuable is as this kind of connective tissue layer across existing data processing tools,” he told The Register.

Written in C++, DuckDB is embedded within a host process.

There is no DBMS server software to install, update, or maintain.

It runs in the host application's memory space with queries executing directly within the process.

As such, developers can build analytics inside their applications, but Warfield said the design means developers and data engineers have new ways of connecting data sources and processing engines.

For example, DuckDB employs verbs in SQL such as "attach" and "connect" that allow it to work with almost any remote SQL endpoint.

Warfield said DuckDB is already capable of connecting to Redshift or RDS, issuing SQL commands, and performing cross-engine interactions.

“In a lot of senses, we can treat it as [being] like an SDK for data that customers can build with, and they can run the large proportion of their queries that are small, and work over smaller data sets locally without having to stand up an external engine.

Yet, as they graduate to work with larger datasets, or need centralized query capabilities, they can still use DuckDB as a way of passing those queries on to those other engines,” he said.

The concept applies to analytics and data systems from other vendors — such as Google’s BigQuery and Microsoft’s Fabric — as much as AWS’s own products.

“You can build with Duck, and it provides the path to graduate and to submit inquiries against something else, like a singular interface for data,” he said.

AWS acquired DuckLabs, the support and development company founded by DuckDB developers Hannes Mühleisen and Mark Raasveldt, both academics at Amsterdam's Centrum Wiskunde & Informatica mathematical and theoretical computing research center.