Building a Data Pipeline - Languages and Stack
Choosing a language and stack
Hello! Welcome to the exciting topic of ingesting massive amounts of pastries data! We talked about ingest a bit in the previous tutorial; in this one, we’ll examine various architectures in great detail.
Let’s get the controversial part out of the way. Language. =)
So You Want to Build a Language VM - Part 33 - Cluster Syncing
I don’t know about you, but I’m getting tired of all this clustering. But, the end is in sight! For real this time! I promise!
WWhen we ended last tutorial, we had been able to send bincoded messages. A cluster member could join another cluster, and receive a list of other nodes back.
Our next task is for the new node to take each node it receives, and establish its own, independent TCP connection to them.
So You Want to Build a Language VM - Part 32 - More Clustering?!
Hello again everyone! In this tutorial, we’re going to continue to work on clustering. When we left off in the last tutorial, we had the joiner node sending a hello message and the server node adding it to its list. The next tasks are:
Send a hello back
Send a list of all known nodes to the new joiner == Full-mesh Network Remember how I mentioned we’d be doing a full mesh network? I realized an illustration might be handy, so more beautiful text art!
Building a Data Pipeline - Part 0
Covers general elements of data pipelines, what they do, and why they do it
So You Want to Build a Language VM - Part 31 - Making Clustering Make Sense
Making the Clustering Make Sense
Sorry everyone! I’ve been busy, hence the delay. Back in tutorial 29, we had two nodes talking to each other, but we got a lot of random text on the screen. Let’s figure out why!
So You Want to Build a Language VM - Part 30 - Cleanup Time
We've got a lot of warnings and clippy things to fix!
As fun as it has been working the clustering, we’ve accumulated quite a bit of technical debt. There’s tons of warnings, and I haven’t dared to run clippy. So this post is all about going through and cleaning them up. =) It won’t be as exciting as adding features, but making sure to take time to do cleanups is just as important, if not more so. Tech debt has a way of growing faster than credit card debt.
So You Want to Build a Language VM - Part 29 - Clustering: Part 3
Explores making clustering thread-safe
If you tried to compile the code from tutorial 28, you probably saw a lot of errors about not being able to send Strings across mpsc channels. The reasons behind this is worth a paragraph or three on why.
So You Want to Build a Language VM - Part 28 - Clustering: Part 2
Finishes basic clustering
Hi, me again! In our last tutorial, we had added a separate TCP server to the Iridium VM. In this one, we’re going to finish up the client part, so that two Iridium VMs can talk to one another. Make sure you start this tutorial from this tag: https://gitlab.com/subnetzero/iridium/tags/0.0.27
Primer on Database Sharding
Covers what database sharding means and different implementations
Primer on NewSQL
Covers what NewSQL is and when to use it