A quick introduction to Talon, a hands-free input system that lets you control you computer with your voice, and Python.

When I started getting my master’s degree I had just developed a repetitive strain injury (RSI) that made it painful to impossible to use a keyboard, mouse or even a phone. Still, I was able to finish my degree without any delay, as well as a summer internship at Bekk. The reason I was able to do this was because I had discovered Talon, a hands free input system, which allowed me to use the computer proficiently using only a microphone. These days the pain is not as bad as it used to be and I’m not fully reliant on Talon any more, but when I’m primarily using a keyboard I find myself missing some of the awesome features Talon has to offer. I really believe that systems like Talon can be really effective even for people without disabilities so in this article I will demonstrate how you can use talon to improve your workflow and significantly increase your wow-factor when pair programming on Zoom 😉

Using Talon

If you’ve want to follow along with the examples you can install Talon from here.Talon does not come with any built-in commands. If you want to be able to control your computer fully using only Talon I recommend downloading a larger command set such as knausj_talon, but if you’re just playing around you can simply create a file in the user directory (%APPDATA%\Talon\user on Windows, and ~/.talon/user on macOS/Linux.).

Let’s get started with the simplest example:

Uttering the phrase say hello will now make Talon type Hello from Talon!.

You can also emulate any keystroke, for example to switch to the last opened window:



This is not really faster than pressing the key combination yourself, but voice commands are much easier to remember than keyboard shortcuts, and it’s really easy to chain them together to make macros.

Voice commands can also capture parts of a spoken phrase to do interesting things with them.



The above lets me say for example call print to produce print() with the cursor inside the parentheses. Now we have what we need to create some useful snippets!



You can of course be really productive with just the snippets in your editor, but this allows you to leverage the naturally high throughput you have with your voice, and they have the benefit of being available everywhere (notepad, slack, browsers, etc).

If this syntax reminds you of pythons f-strings, then that’s great, because they are! That means we can write python expressions and evaluate them. We could for example create a spoken calculator with the number capture:

Now saying multiply two by three will produce 6 🤯

Talon can also respond to keyboard events, so the macros you write for Talon can still be used when you’re sitting in an office and don’t want to use a microphone. For example:



Getting more advanced

You can do a lot with just the built-in scripting language, but it is when you start to add python modules things start to get really interesting. If you’re still following along, go ahead and create a python file next to the .talon file you created earlier.



I haven’t explained what modules or contexts are, but for now it’s sufficient to know that they are used to control when groups of actions and voice commands are active. This makes it so that you can’t accidentally trigger application specific commands when that application is not open, and you can define actions that work differently based on the context you’re in, such as which programming language you’re using.

The following example shows how it can use Talon’s clipboard API to grab the visually selected text and append it to a file called notes.md in your home directory.



clip.capture() allows us to work with the clipboard while making sure it gets restored once we are done with it. And just like that, we have a simple note tracking script that can be triggered in any application that allows copy pasting. The action can be invoked from a voice command like so:



Going back to a programming example, let’s revisit our command for invoking python functions, which has a couple of issues. Not all function names are easy to pronounce, so to address this we can define a Talon list which associates a spoken form with some output:



We can then reference this list in a voice command as {user.python_functions}. Another issue is that we used the word capture which only captures a single word, but we want to be able to dictate longer function names and apply snake case forming. One way to do this to define a custom capture which we can use to parse user input and process it however we like:



Rules are defined like regular expressions except that they work on whole words instead of characters, so hello* matches hello hello hello, and not hellooo. The above capture will match any phrase, but if it matches something in the python_functions list it will override the output. We can then update our voice command to be:



There’s a lot more fun to be had

The things I presented in this article only scratches the surface of what you can do with Talon. Talon also has (among other things): 

Mouse control 

Eye tracking

Noise recognition (pop, hiss, etc)

Facial gesture recognition

API for file watching

Many user scripts for working with different applications

Many cool user-defined commands in knausj_talon such as google that which will google the selected text, or launch/focus which will open / focus any application you say.

If any of this excites you, feel free to check out the official website, knausj_talon, and join the Slack channel (link on the website).

I can also recommend Emily Shea’s talk Voice Driven Development: Who needs a keyboard anyway?

Conclusion

Talon has a lot of features that I think can be really useful for anyone who likes optimising parts of their workflow. Even if you don’t want to use voice commands in your normal workflow, you can use it as a general purpose scripting environment and create macros that will work anywhere. If you need any help getting started feel free to contact me or anyone else in the slack 😊

System Level Python Macros With Voice Commands Using Talon

Hvordan spenne beina på et IT-prosjekt? (Del 3 av 3) - Abstraksjoner

With React 18, a new hook is available which can slow down our UI on purpose 🤯 Why may we need such a hook? 🤔

⏲ useDeferredValue

useDeferredValue is a new hook that is part of the concurrent mode patterns being introduced in React 18. The hook exists such that we can introduce an inconsistency in the UI, intentionally.

By default React guarantees that the UI is rendered consistent, always. This means that whatever React is told to render, React will guarantee that the components will reflect the same data that is passed to them.

In most cases this works really well and gives a great user experience, however, there may be cases where an inconsistency can give a better user experience and here useDeferredValue come in handy. 

As mentioned useDeferredValue exists such that we can create an inconsistency in the UI. The hook takes two parameters, a value to defer, and an optional config object. Currently the config object is used to tell the maximum time react can defer a value.

The block above show a simple example where useDeferredValue is used to create a deferred version of the search text.

🤔 How does it differentiate from debouncing? 

If you have been in touch with debouncing, you may think that useDeferredValue and debouncing are the same. In one way, they are doing the same - making it possible to delay a value. However, there is one major difference between the two techniques.

In React 18, useDeferredValue will only make a value "lag behind" if the rendering takes a while. On the other hand, a debounce method will always introduce a lag, it does not matter how fast the computer is.

✨An example

To give a short demonstration of the useDeferredValue hook, we have implemented a simple web application to view and search for players in Fantasy Premier League ⚽️

To see the effect of the useDeferredValue hook on your machine, throttle your CPU while looking at the example application 🐌

The web application fetches all players from the Fantasy Premier League API and displays them in one big simple table along with a search field 🤓

There is only one problem. Looking at the gif below, we can see how typing the name of a player in the search field causes stutter.

If you, like me, have created large tables with the possibility to search for a value, you would probably have thought about introducing the debounce method now, but wait!

Changing our implementation to use the new useDeferredValue hook on the value of the search field - typing a player’s name suddenly feels more responsive.

It’s worth noticing that using useDeferredValue comes with a cost - we are slowing the results with a lag. For most users the lag would be unnoticeable and most important, for users with a slower computer the experience would be better.



👩🏼‍💻 The Implementation

If you want to have a look at our example implementation using the useDeferredValue hook - it is available here🎅🏻

Thanks for reading - and happy holidays! 🎄🎁

useDeferredValue in React 18

We are visiting the digital galaxy again - today, as any other day. So grand and immense, expanding by the second. For years, we have roamed here, visiting place after place, yet not seen a percentage of what it has to offer. You would think we would become rather overwhelming at some point. Yet we seldom are. Maybe it's due to the invisible helpers we have by our side, guiding us through these various landscapes and obstacles toward our desired destination. Although many know these guides and interact with them everyday, few know their name: Recommender systems.

Alright, let’s get down to earth for a second and talk about these systems. This blog post aims to be a short introduction to the topic, including a bit about what recommender systems are, how they work, and why they are important. If you want the extremely short version, it can be summarized in one sentence: Recommender systems are systems which can recommend something to you based on information about you, other users, and the content they recommend to you. That’s it. Now, let’s dig in.





What Are Recommender Systems?

Since its beginning, the Internet has expanded immensely, containing more information to consume and options to choose from than ever before. To not be overwhelmed while traversing the digital landscape, the users rely on their appointed guides - the recommender systems - to bring them safely to their destination. The guides provide the users with personalized recommendations by navigating and filtering content to fit the users preferences and needs. Today, the systems have become such an integrated part of the users’ lives that they almost appear invisible. For instance, in 2015 Netflix revealed that the choices for about 80 % of the streaming hours were influenced by the recommendation algorithm. That is a lot!



Recommending you a good movie to watch at a late night is not the only thing these systems are good at. You can find them in all sorts of online services and you rarely go a day without using one. Picture this: On the way to work, you open Spotify and the first thing to pop up is the new Discover Weekly playlist, containing your latest obsessions. The break is spent scrolling through the Facebook feed, providing you with updates about friends and family, memes, celebrity news, and cute dogs. On the way home, you walk through a busy street filled with blissful couples and Christmas decorations. Your hand pulls up the phone, and swipes left and right until the matchmaking guru Tinder can find you a suitable date. The day is topped off by lifting your feet up on the coach and beginning the great hunt for Christmas presents from the comforts of your home. You find yourself leaning on Amazon’s trusted recommendations, as your family never seems to wish for anything. All of these applications are recommender systems, and yet there exist even more variants.





How Do Recommender Systems Work?

You may have noticed that your Netflix feed looks quite different from the first time you opened it. Somehow, it has become much better at guessing which types of movies you would like to watch. It might make you wonder how Netflix does this. How is a machine able to know what sort of movie you would like to watch or which product you would like to buy? The short answer is machine learning - most often. But we don’t really like big, scary buzzwords in this blog post, so I will try to give a few pointers without getting too technical, just so we can grasp the concepts.


Let's begin by looking at the recommendation process. There are usually three phases: Collection of user data, processing data and generating recommendations, and returning a list of recommendations to the user. First, in order to make personalized recommendations, the system needs a lot of data about the user - the more, the merrier. This data is usually collected in two ways. One way is to explicitly request the user for data - such as the user’s gender, age or nationality - when the user begins using the app. Another way is to implicitly collect the data through the user’s interactions with the system. For example, when a user clicks on a product, it might indicate the user’s interest in that sort of product. The same principle applies with the genres a user previously has listened to, or the videos they have liked.


Once the needed data has been collected, the system uses it to try to extract the user’s preferences and predict which items they will like. I make it sound as if this happens in one go, but that is not entirely true. As mentioned earlier, the system usually uses some form of machine learning. This means that the system keeps a model containing all the users and their preferences, which can be used to make recommendations. It would take a lot of time to update the model anytime a user clicks on a new item, and therefore this usually is done periodically. For instance, if you watch a new movie on Netflix you may not notice much difference in your feed until the next day. So, when a user logs into a streaming platform, the system only needs to check the model for preferences and generate some recommendations to send back.





Recommendation Strategies

Now, let’s look at some common strategies for deriving preferences and generating these recommendations. It all comes down to similarity - similarity between users and similarity between items. What do I mean by similar users? Have you ever bought a nice sweater in an online shop, and then been prompted with a pair of pants and a text saying: Other people who bought this sweater also bought these? That means there is a group of users out there which share some of your preferences based on your previous purchases. The same concept can be applied to movie recommendations. Netflix might look through your watch history, and search for other users who have watched many of the same movies. As you seem to have some common interests, they might recommend some movies from their watch history which you have not seen yet. This strategy is called Collaborative filtering. It is by far the most popular strategy and is used by many of the big applications.



Content-based filtering is another strategy that tries to generate recommendations based on the similarity between items rather than users. An item may refer to a product, a movie, or anything which can be recommended to the users. In this case, the similarity is based on the content of the items and the available information about them. For example, take a webpage which recommends articles. If a user reads a lot of articles about a certain topic, such as Computer Science, the system will assume their interest and find other related articles which they might like to read. However, this strategy is not well suited for all content types. In order to find out the characteristics of an item, there must be some way to analyze the content and extract this information, making it rather hard to apply to videos and music.



There is also a third common strategy called Hybrid filtering, which combines the two previous strategies. This is done to utilize their strengths and mellow out their weaknesses.



Why Are Recommender Systems Important?

We know that the greatest advantage of the recommender systems is their ability to help users navigate through content and find what they are looking for. And we know this is quite a valuable ability due the enormous amount of content on the Internet. In other words, the users are gaining a great deal from using these systems. But what about the businesses behind these systems, what are they gaining?



There is an overwhelming amount of businesses and services - all which are fighting daily for the users attention. Therefore, providing the users with the best recommendations and experience has a great impact on business, and is taken quite seriously by many companies. Let’s look at a quick example. In 2009, Netflix held an open competition where teams could compete to find the best collaborative filtering algorithm to predict users movie ratings. The competition lasted for three years, and the winning team received a shocking amount of US$1,000,000! The time and money Netflix spent on this competition demonstrates how important the recommendation experience is to many companies. After all, the quality of recommendations may determine whether a company gains or loses a customer, or how much time a customer spends on their platform.



It might be good to keep in mind that the recommender systems do not only cater to the users’ needs, but also serve the interests of the company. Therefore, there might come a time when their interests and the users’ interests are in conflict. For example, a social media platform which tries to keep the user entertained for as long as possible, may not be considerate of the user’s personal goal to spend less time there.



So, What Have We Learned?

Let’s summarize. We have now learned that recommender systems are systems which recommend interesting content and products to the users. This can be achieved by extracting the user’s preferences from either data about users (collaborative filtering), data about items (content-based), or both (hybrid). The systems mitigate the problem of information overload on the Internet, and are beneficial to users and businesses alike. The conclusion? I get to watch good movies. Netflix gets my money. Everyone’s happy.







Invisible Guides Through the Galaxy

TypeScript is fantastic, but without Json Decoders, your code (probably) isn't type safe.

Let me introduce you to a library I made to solve a pressing problem in the TypeScript space, and convince you it is something you need in your app.

It is called typescript-json-decoder and can be found over on github here: https://github.com/tskj/typescript-json-decoder. It's MIT licenced, tiny, dependency free, and works in any TypeScript environment where you might read external data, such as a React app, or any frontend app written i TypeScript really, or a backend app on Node or Deno.

What even is TypeScript?

To set the stage, let me first explain briefly what TypeScript is and what problems it tries to solve.

TypeScript is fantastic, working in a fully typed app is a great experience. It is in fact significantly better than traditional languages, and makes Java, C#, and friends, seem like dynamically typed languages in comparison.

Let me give a quick example of the kind of things that TypeScript does. The following code is just a simple example of some code you might see in your application, some functions passing a User object around.

Here TypeScript helps you manage the types of your values, letting you access the username property on the User object - but only if it isn't null. Compared to regular, untyped JavaScript this is a lot of help in reminding you what your functions can and cannot do. But even compared to C# or Java, TypeScript "remembers" that you have checked for null in the first function, and allows you to skip the null check in the second. This is the philosophy of having a fully typed app, doing the right thing with as little friction as possible, and only allowing legal operations to compile. This way you have no runtime errors.

Where TypeScript Falters

Or is it so simple? Let us have a look at some more code. For example, a very standard way you might load the data about the Users is the following.

Pretty standard way of fetching data from an API in a frontend app. In addition to this code, you might have a separate file with some type definitions that mirror the API. That is, you have manually typed the JSON API with some matching TypeScript types, and that might look a bit like the following.

Just to flesh out the picture of what we're talking about, the following is what a corresponding HTTP request / response might look like.

Notice how the response JSON corresponds to our type, it is a list of objects which have a "username" key with a string value, a "dateOfBirth" key with a Date value, and a "password" key with a string password in cleartext as a value. And here lies our problem. Our astute readers will have noticed that the "dateOfBirth" key does not in fact have a Date object as its value in the JSON representation, but rather a regular old string, which happens to encode a date. This is because JSON is a serialization format and does not have rich datatypes or objects such as Date, but only simple primitive types such as string, number, boolean and their aggregates, in the form of records and arrays.

This mistake will not be caught at compile time, in fact this program compiles without warnings. Yet we will see a runtime crash, which is exactly what TypeScript is designed to avoid. And even worse, the crash happens far away from all the code we have seen so far! It happens much "later" in the program execution than you might think. For example, consider a piece of code that uses the result of fetching this data, which might be something like the following.

This code tries to get the year part of the date, by calling the method .getFullYear(), which is a standard method on the Date class. However, strings don't have any such method of course, and here the program will crash with a classic getFullYear is not a function.

The Hotfix

When this happens in production you will go back to your type definitions and fix it to actually match the data. We swap the Date for the string in the type definition:

And now finally the compiler yells at us, and forces us to fix our program where it's wrong. The compiler error will be at the use site, so let's go there as well, and parse the date out of the string where we need it.

That's a slight complication, but now it's correct and works, also at runtime.

Let's Fix it with a Decoder

This isn't a full fix though, because what stops it from happening again? The API might change, we might change our type definitions to use more of the API, or our current representation of the API might be wrong right now - we just haven't hit the runtime case where it explodes.

I hope I have made a compelling case that this isn't a good way of doing this. It's full of bugs and you can never be confident that it's actually correct. So let me introduce you to a better world, a world of decoders.

What is a decoder? A decoder is some mechanism that validates that our JSON structure actually matches our idea of it, that is - our types line up with the data we receive. In our case we'll write a function that traverses the structure to check it. This function also needs some kind of mechanism to report errors, which could be returning a "Result" type or throw an exception.

Let's implement a hypothetical User decoder.

Here we simply write some straightforward code to check that the thing is an object, and if it is, we verify that it has the right keys, and that the keys have the expected types. Here I've only showed the first key, "username", but you'd have to repeat this process for all keys in the object. This is incredibly tedious. But it works! Let's revisit our example to see how it makes things better.

We simply map the userDecoder over every element in our list, and if any one of them fails, the decoder throws an exception, which causes the promise we're in to be rejected - just as if the network call itself had failed. This makes a lot of sense, because being unable to parse the data in a way that our app understands is just as bad as the endpoint or API being down.

Notice that even this code assumes that the returned data is an array (which is why we can call .map on it), however to be properly safe we would need to have an array decoder as well. And we should, but a much more pressing problem is that our decoder code is incredibly annoying to write and maintain. Which in fact is why no one does this, everyone just does the first thing and praying, because keeping up the decoder code with your types is very time consuming, it's just downright ugly, and keeping them in sync really is a chore. But it doesn't have to be that way.

Finally, automatic Json Decoders with my library

This problem annoyed me so much, especially the fact that the lack of proper tools being so bad that people don't even bother to do it safely, that I decided to make this library. The idea is as simple as it is powerful: automatic decoders would allow you to write one single definiton of the API types and have a decoder derived automatically. That would be huge, it would be no more difficult than the status quo of writing the API types out in a dedicated file and the decoders would give you actual type safety. Here is how it works:

One of the primary design goals of this library is to be as idiomatic TypeScript as possible, and as low friction, low overhead as it can. It leverages your existing knowledge, is easy to migrate to from classic type definitions (because it looks and behaves like them), and gets out of your way.

This code is supposed to look as similar (and mean the same thing!) as possible to the original type definition, all we do is swap the types with imported string and date decoders, and wrap it up in record. This defines a decoder of this type as you would expect it to work, and then we add a single line on the top, which derives the type from the decoder (that's the line type User = decodeType<typeof userDecoder>).

I know that was a mouthful, but the process of migrating your existing types is simple. The decoders mirror your existing types one to one, and all you have to do in practice is to add the single line above every decoder to have the type derived automatically (you don't actually need this either, it's just there for convenience if you want it).

Notice also that here we have an actual array(userDecoder), which I commented on in the previous (manual) example. This is even more correct, because it not only checks every object in the array, but also makes sure the JSON is in fact an array.

Notice also also that here we actually can use date! This is actually an additional benefit of using a decoder library, you are not necessarily stuck only typing your API using primitve JSON types, but you can actually "pretend" that JSON supports these kinds of rich data types. Here, the library provides a date decoder which decodes a string containing date information, encoded in a way the JavaScript Date class understands. And the result of this is an actual Date object! Which means you can safely (with complete type safety and without runtime errors) call .getFullYear() on it. And this isn't just limited to the decoders this library provides, you can compose and write your own decoders which decode any arbitrary data structure you wish.

Wrapping up, you can use this decoder anywhere you want, but I recommend putting it as close to the "edge" of your program as possible, such as where you fetch the data.

Here you don't need to cast to a Promise<LoggedInUsers>, in fact you don't need to specify any types by hand at all, it is all inferred automatically for you.

And now the use site also works, with complete type safety, as promised.

Which is exactly the code we started with!

I hope I have made a good case for why you need decoders in your app, and why this library is the best solution. You can find it on github and npm.

I look forward to hearing your feedback on how it works for you in your projects! I'm sure there are things that can be better and which need improving, and I would like to know about them. If you need any help in using the library, have any questoins or have feedback, please don't hesitate to get in touch. I'm @_tskj_ on Twitter and epost@tarjei.org on the emails. Merry Christmas!

TypeScript Decoders and Where to Find Them

Template literals (`My name is ${name}`) have existed in JavaScript for a while. There is a good chance that it is your preferred way to concatenate strings in JavaScript. A slightly lesser known sub-feature of this is the tagged template literal (myTag`My name is ${name}`). What is it and why should you care?

In fact, the tag (myTag in the example above)  is just a function with a special signature. The first parameter is an array of strings. The following arguments are the substitution values provided. A tag like this, called normal here, would just concatenate the string as expected by a normal template string:

We can have slightly more fun by adding some transformations like making this screaming function:

Fun? Yes. But perhaps not very useful. And you could easily just call .toUpperCase() on the template string or create a normal function for it.

The great advantage of tagged template strings is that you can access the individual arguments passed to the string. Let's create a debug tag that prints a table of the arguments and their types if we're not in production mode:

This is perhaps something we could use, leveraging the advantages of the tagged template string syntax. Notice that we don't need to return a string from this tag function. You can return whatever you want.

Real-life usages

Tagged template literals are used in some popular libraries out there. You might recognize it from the popular CSS-in-JS library Styled Components, where the tags are used to generate React components with certain styles:

If you have used GraphQL and the Apollo libraries, you might have encountered the gql tag. It is a helpful utility for parsing your GraphQL queries, making them easier to work with compared to using simple strings.

Another library which supports the tagged template literal syntax is timeproxy. It lets you create human-readable time values with strings. And with tagged template literals, the syntax is quite compact and readable.

Conclusion

Tagged template literals is a syntax that provides a new way of working with strings. Maybe you have some use case in your project that it is perfect for? At least now you know some of the magic behind this funky syntax when you see it!

22-Dec

Today's posts

System Level Python Macros With Voice Commands Using Talon

Hvordan spenne beina på et IT-prosjekt? (Del 3 av 3) - Abstraksjoner

useDeferredValue in React 18

Invisible Guides Through the Galaxy

TypeScript Decoders and Where to Find Them

Tagged template literals