My last few posts have focused on Redshift and Athena, two specialized tools for managing and querying Big Data. But there’s a meme that’s been floating around for at least a few years that you should just use Postgres for anything data-related. It may not provide all of the features and capabilities of a dedicated tool, but is one less thing to learn and manage. Should this advice also apply to your data warehouse?
Recently, a project I worked on was considering using Electron as a fallback technology for an initial Progressive Web Application. At the time, the assumption was that since Electron uses Chromium, a browser, it should allow application developers to not only use the features of a PWA but also gain native access to technologies, such as local databases, file storage, enhanced networking, and it should still be able to run the existing PWA service worker for offline support. After spending…
Earlier this year at PhillyETE, we had a great talk by Avdi Grimm and Jessica Kerr, REPLs All The Way Up: A Rubric For Virtuous Feedback Loops. In this talk, one of the key theses was to find ways to make exploring your code easier, via REPLs, scenario setups and other means. Many years ago I used to mount the BeanShell servlet and export my Spring Context so I could write little scripts in a web page in a Spring…
In this post I walk through several execution plans, explain what Redshift is doing in each, and highlight the parts of plans that indicate problems.
Some key architecture decisions to help you drive brand awareness and revenue through seamless customer experiences.
If you want to run OS X Sonoma, but can’t dedicate a computer to it, you could always install it on the UTM virtual machine engine. This allows you to test out Beta OS X features without taking over your primary machine’s OS. Pre-requisites: A Mac with Apple Silicon running OS X Ventura An Apple Developer account ($99 / yr) UTM (available on the Mac App Store) Steps: Download the IPFW restore file for OS X Sonoma for Apple Silicon…
If you’re one of those people who left Java before it got into functional programming with Lambdas and Streams, we’ve got news for you.
I’ve always been a fan of database servers: self-contained entities that manage both storage and compute, and give you knobs to turn to optimize your queries. The flip side is that I have an inherent distrust of services such as Athena, which promise to run queries efficiently on structured data split between many files in a data lake. It just doesn’t seem natural; where are the knobs?
So, since I had data generated for my post on Athena performance with different file types, I decided to use that data in a performance comparison with Redshift.
In my “Friends Don’t Let Friends Use JSON” post, I noted that I preferred the Avro file format to Parquet, because it was easier to write code to use it. I expected some pushback, and got it: Parquet is “much” more performant. So I decided to do some benchmarking.
In a perfect world, there would never be a need to connect to your resources running on AWS. In the real world, it’s sometimes necessary to get your hands dirty and look at what’s happening on the actual machine, especially during development. This post dives into a few ways to connect your workstation to resources running inside a VPC. It started out as a how-to for using bastion hosts, but quickly expanded to look beyond the bastion.
