2024 Data Engineering Trends

Doing More with Less

The tech industry in 2024 is under pressure to optimize resources. Technology and data leaders are asked to integrate more data to support new AI-driven features while simultaneously being forced to reduce costs and headcount. Judging by the recent layoffs at a.o. Google, Amazon, Meta, Twitch even the largest tech companies are not immune to this trend toward increased efficiency.

The Impact of AI on Layoffs vs. Economic Factors

The growing capabilities of LLMs are reshaping the job market, and the data space is no exception. While it’s difficult to estimate to what extent AI progress has contributed to the growing waves of tech layoffs, many companies are cutting costs in some established lines of business and reallocating that budget toward AI development. Dropbox reduced its headcount by 16% last year and reallocated those resources toward hiring AI specialists in order to “stay competitive”.

Implications for Data Engineering

As organizations seek to do more with less, there’s a growing demand for generalists proficient in cloud-native technologies, data, AI, and platform engineering. This shift is steering the field away from highly specialized roles, such as ETL or BI engineers, in favor of a broader range of engineering skills. Data engineering teams in 2024 start resembling software engineering teams. This happens partially thanks to the growing maturity of data engineering as a discipline and partially out of necessity: data teams are expected to deliver more with less, and this requires building data products faster, often in smaller teams than before.

Job Titles Will Become Even More Confusing

Job titles in the data field will remain confusing. As mentioned before, software engineers are increasingly involved in building AI-enhanced products, and data engineers are moving closer in the direction of software and platform engineering. There’s also a growing number of product engineer positions, which are software engineers responsible for managing the entire product lifecycle, from managing backlog to development and maintenance. A similar trend can be observed in marketing roles in tech, where many companies require familiarity with Python and SQL from candidates. Such hybrid roles combining product/marketing and engineering will likely continue to grow in popularity as coding tasks become more accessible to less technical users with tools such as ChatGPt, GitHub Copilot . If you like fancy terms, you can call that trend a democratization of data, analytics, and engineering practices as a self-service.

What’s Next

In summary, the most important data engineering trends in 2024 are an accelerated integration of AI into data products, a shift toward platform engineering, and an increased focus on efficiency. As a result, we see the growing importance of diversified skills, data lakehouses, open table formats, event-driven systems, AI-augmented development, and the need to better balance open-source with commercial incentives to stay competitive.