Decision support databases have a number of quirks that are not obvious to the casual user, particularly someone coming from an OLTP background. In this post I look at how unbalanced distributions can impact your query performance, how you can identify imbalances, and what you can do to fix them.
Are you running a database with RDS? Would you like to manage it via migrations? This article explains how to use AWS CodeBuild to keep a database schema updated using Flyway, an open-source data migrations tool. Configuration is outlined via CloudFormation snippets. An AWS example repository is provided.
It’s possible to analyze your Glue jobs using just the logs they produce. Possible. But it’s not a pleasant task: your log messages are buried in messages from the framework, and in the case of a distributed PySpark job they’ll be spread amongst multiple CloudWatch log streams. In this post I look at an alternative: AWS X-Ray, which captures and aggregates “trace segments” that monitor specific sections of your code. With X-Ray, you can easily see where your jobs are spending their time, and compare different runs.
Introduction Today’s small microcontrollers offer impressive functionality and provide an opportunity to replace older, more expensive software and hardware. Consider the case where a facility wants to have control over devices or equipment, with rules that evaluate telemetry from sensors and activate, deactivate or regulate equipment and other devices. Many large facilities have systems in place that rely on a global standard named BACNet. This protocol has been an ISO standard since 2004 and allows for different types of devices…
Two of the biggest mistakes companies make in building an MVP (minimum viable product) and how you can avoid them.
In my last post I recommended using Avro for file storage in a data lake. It has the benefits of compact storage and a schema in every file that tells you what data it holds. In this post I show three ways to generate Avro files: one in Java, and two in Python.
I’ve never been a JSON hater, but I’ve recently run into enough pain with JSON as a data serialization format that my feelings are edging toward dislike. However, JSON is a fact of life in most data pipelines, especially those that receive event-stream data from a third-party supplier. This post reflects on some of the problems that I’ve seen, and solutions that I’ve used
We’ve been celebrating Chariot Solutions’ 20-year anniversary this year with a series of lookbacks across specific industries, team members and types of technologies. But any reflection on the last two decades would be incomplete without a deep dive on the tech-related changes in our hometown of Philadelphia during that same span. The pre-Y2K tech scene here was very different, as evidenced by the top startups list for 1999 published by Philadelphia Business Journal technology reporter Peter Key. It was largely…
As 2022 comes to a close, here are some of the technologies, patterns and strategies I'm watching for 2023. This list is from my perspective, I'm sure if you poll others here at Chariot they'd have their own. The Big Serverless Push Watching the re:Invent keynote last month, you'd be hard pressed to miss Werner Vogels' central message: go all in on Serverless computing, and sure, it has an allure. With a few clicks or a small (?) CloudFormation script…
The Internet of Things (IoT) permeates our lives, and has literally grown up over the course of Chariot Solutions’ 20 years in business.
