Let’s talk about developing your own tools. It’s hard to come up with an introduction to this because it’s such a pervasive thing in programming that it feels like talking about breathing. “Yeah, breathing is fun and important! You should try it yourself!”
Well, whatever. Developing your own tools is fun and important! You should try it yourself!
Recently I’ve had a need to work with byte buffers in Javascript. I’ve got an Electron app that needs to communicate with a completely different machine via TCP, and this completely different machine really only likes talking in bytes. It wants a few different formats, and I’ve had to develop a few specific tests for diagnostic purposes, prototypes, and whatnot. Over that period of time, I’ve gotten very tired of the formatting that Javascript wants for a byte buffer. One delightful May morning, I finally had the bright idea to develop a tool that makes this easier.
Let’s do that! There are all sorts of ways to do this, but I want to take a crack at simply replacing that syntax entirely, somehow. This is probably not the best way. That’s fine! We don’t have to be optimal on the first try.
So the first thing I want to achieve is just reading a file in and ignoring all of the standard Javascript stuff. My ultimate goal here is to have something that is Javascript-plus-extra-thing. So we’ll have two files right away. Let’s call them “main.js” and “example.jj.” For now, I picked a new, nearly-arbitrary file extension “jj” mostly so that VS Code won’t complain at me if I write non-Javascript things in a “js” file. If you’re wondering, it stands for “Joe’s Javascript” and it amuses me (which is a great motivator for programming something).
So at the start, here’s how the world currently looks in “main.js”:
const fs = require("fs");
const program = fs.readFileSync("example.jj");
const programString = program.toString();
console.log(programString);
Here’s what we’ve got in “example.jj”:
console.log("Hey!");
const justAFunction = () => {
return "Yep";
};
As you can see, so far we’ve just got some throwaway JavaScript and something that reads it in and prints it. This isn’t very useful, though. Let’s put my new buffer syntax in there!
The contents of “example.jj”:
console.log("Hey!");
const justAFunction = () => {
return "Yep";
};
@ test = 00 01 bc 33;
That’s a pretty simple buffer syntax. I’m truly just tired of typing 0x before everything, along with commas. It made copying and pasting long, demonstrably correct series of bytes into the editor an exercise in figuring out all the formatting tricks that VS Code had to offer. Unfortunately, nothing ever felt like it shaved off enough of the headache, and any headache-inducing process is also likely to be an error-inducing process. Therein lies my impetus in fixing it: I was a little annoyed, and creating typos.
I picked “@” as the “Hey, make this into a buffer” operator because I was shooting for utmost simplicity and also because that’s not an operator of any kind in vanilla Javascript. Otherwise I’d have to go with some longer keyword and things would get incrementally more complicated from there. I’m sure they’ll get there, eventually, but this should suffice for the blog post.
The next step is to figure out how we can recognize that line as special! I’d ideally like this to blaze through everything that isn’t like that last line, and only modify what it should. Let’s add some stuff to “main.js” that allows us to identify what we’re after: something to run through the program string and something to check to see if it’s the “@” character. Those additions are shown below:
const fs = require("fs");
const isBufferSign = (chr) => {
return chr === "@";
};
const program = fs.readFileSync("example.jj");
const programString = program.toString();
for(let i = 0; i < programString.length; ++i){
if(isBufferSign(programString.charAt(i))){
console.log("Hey! Found it!");
}
}
Alright, great! We can identify that special something and leave everything else alone. Let’s see if we can’t generate regular ol’ Javascript buffer initialization from that. We know some useful things that will help us: the index of the “@”, and we know that line will terminate with a semicolon. We can use that to our advantage to create a substring that holds the entire line for some special processing. We also know that the numbers are separated by whitespace and they’re always going to be two characters. Helpfully, the last one in the series is unique in that it ends with a semicolon immediately, no whitespace to be found. That allows us to place the right amount of commas automatically, and also to generate the closing bracket, closing parenthesis, and semicolon for what will be the Javascript.
At this point, it’s worth spilling the beans and writing the whole darn thing. You can find the tool, as it stands as of this post, at https://github.com/joevonholtum/js_byte_tool/tree/byte-tool-pt-1. I know I’m playing pretty fast and loose with terminology like “parser” and “tokenizer” (and others), but it made sense to me at the time. I’m perfectly open to being argued into better names for these things.
You’ll note that in this state, it’s really only good at generating something useful from a file I included called “example.jj.” I was hoping to be able to add this as a little domain specific language, but as I worked on it with that in mind, I quickly realized that I was working significantly harder than was called for. I would miss out on all of the helpful tools that VS Code provides regarding code completion and error checking and refactoring, so I’m opting to go a new direction with this as I continue to develop it. There’s no reason we can’t include a file of byte buffer definitions with names, perhaps, and then have my tool return an object populated with those as properties. Perhaps we could also add a function that allows us to pass in a string and receive a nicely initialized byte buffer back in return!
The next step, though, is to get this thing under some form of automated testing so that changing stuff and maintenance gets easier. That might explain why I was so intensely interested in object equality in JavaScript a while ago. Of course, the plan is to build that as well, because it’s been fun!
Joe Writes Software 