Keith Gregory

Aggregating Files in your Data Lake – Part 2

When I ran the Lambda from my previous post against Chariot’s CloudTrail repository, it took almost four minutes to process a single day’s worth of data. That seems like a long time, and as a developer I want to optimize everything I write. In this post I look into analyzing the current runtime, and options for improving it.

Aggregating Files in your Data Lake – Part 1

As I’ve written in the past, large numbers of small files make for an inefficient data lake. But sometimes, you can’t avoid small files. Our CloudTrail repository, for example, has 4,601,675 files as-of this morning, 44% of which are under 1,000 bytes long. In this post, I develop a Lambda-based data pipeline to aggregate these files, storing them in a new S3 location partitioned by date. Along the way I call out some of the challenges that face such a pipeline.

Data Engineering is more SRE than SQL

Following my post about the Chariot Data Engineering interview, I received some comments along the lines of “wait, you don’t test their SQL skills?!?” Actually, we do: after loading up the test data into Redshift, the candidate creates three progressively difficult queries. But by then, I’m pretty sure they’ve got the skills we need, because … Read More

Developing A Coding Test for Data Engineering

Hiring good candidates is difficult. After nearly 40 years in this business, and interviewing hundreds of candidates, I’m not going to claim that I have the answer. Just some ideas.

Putting Amazon CodeGuru Reviewer To The Test

Amazon CodeGuru Reviewer promises to “detect potential defects that are difficult for developers to find,” using machine learning to identify potential problems. But how does it compare to existing rule-based tools? In this post I turn CodeGuru loose on a seven-year-old library that’s in use by 3,000 people, to see what issues it flags.

Java Virtual Threads: Worth the wait?

With version 21, Java got virtual (lightweight) threads. This feature has received a lot of press, but will it actually help you? In this post I review the theoretical benefits of virtual threads, and then show actual results from a benchmark.

Small Data: a pipeline for low-latency decision support

In my last post, I said that I didn’t think Postgres was a good choice for a decision support database, versus a task-specific DBMS such as Redshift. In this post I’m going to take the opposite stand, and say that there are cases where Postgres is appropriate: namely, low-latency systems that contain a limited amount of data.

Why Not Just Use Postgres?

My last few posts have focused on Redshift and Athena, two specialized tools for managing and querying Big Data. But there’s a meme that’s been floating around for at least a few years that you should just use Postgres for anything data-related. It may not provide all of the features and capabilities of a dedicated tool, but is one less thing to learn and manage. Should this advice also apply to your data warehouse?