Dark patterns are everywhere. Probably you have been the victim of a dark pattern without even knowing about it. So what are they, and what can we do about them?

Ever see one of these? On first glance, just an innocent question to ask why you don’t want those lovely newsletters anymore. But wait, what is with the wording on that button? And why is that the highlighted option, when the user was obviously in a process of unsubscribing?

This here, is a textbook dark pattern. User interfaces designed to confuse, mislead or trick you to buy more stuff, give out more personal information or to keep you hooked on a service even though you want to quit.

Think of them as the sleazy used car salesman of UX. Seemingly charming and helpful, but when you walk away, your wallet is empty, you have a bag full of faulty spark plugs that you don’t know what to do with, and a creeping feeling you just sold him your old car for a penny and a packet of gum.

Beware, tricksters ahead!

While we are usually pretty good at avoiding scumbags like that in the physical world, the internet is still so young that nobody really knows what the rules are. Nigerian princes with money problems pop up all over the place, some weirdo keeps stealing nude pictures, and we’re nowhere near getting that nasty troll infestation under control.

It feels like entering the wild west, where you have to be constantly on your guard. Even companies that claim to be trustworthy, that manage the information of millions of users, such as Facebook, LinkedIn and Snapchat, try to sneak information or money out of us with shady wording or hidden checkboxes.

Remember when Snapchat asked if you wanted to turn on visibility in their new map feature? No? That’s because they didn’t. They just turned it on for everyone, and made it ridiculously difficult to fint the off-switch. Or LinkedIn confusing you in their on-boarding, and then sending out emails to ALL the contacts in your address book. The web is crawling with malicious UX, and while these two examples are far from the worst, they are all the more frightening because of the size and power of the offenders. If you want more examples of dark patterns in action, take a look at darkpatterns.org and their Hall of Fame.

So what can we do?

One option is to swear off the internet all together, maybe ask the nice fellow in the park where he got his stylish tin hat. Or just be really really careful. As users of technology, the best advice for the time being is probably to read twice and click once.

The good news is that our laws are slowly starting to catch up to the problem. With GDPR in place, we now have at least some level of guard against rampant misuse of our information, but that has far from solved the problem.

The best thing UX-ers and developers can do, is to raise awareness by talking about dark patterns, and setting our foot down when the higher-ups push us to make morally shady interfaces. We are the ones making our digital future, and every little interface counts. The internet is still a wonderful thing, connecting billions of people and containing a staggering amount of information, including about 7 billion pictures of cats. Just as long as we remember: With great power comes great responsibility.

Dark patterns, dark times

About a year ago, Sophie Alpert and Dan Abramov presented one of the most exciting new features in React: Hooks. Release 16.8 introduced several alternatives that can be used directly (e.g. useState, useEffect and some others), but also provided the possibility to create your very own hooks – Custom Hooks.

Custom Hooks anatomy

Suppose you want to create an app that keeps track of a value even after a page refresh. This can easily be done by taking advantage of the browser's localStorage. A Custom Hook for this may look like the following:

Like all the pre-defined hooks in the react package, a Custom Hook's name should always start with the phrase "use", making them distinguishable from other concepts in our application. Custom Hook sounds tough, but luckily for us it's really just a fancy way of saying "a function containing other hooks and/or some additional logic" (and remember: Hooks don't work in class components).

If you would like to use the useLocalStorage hook from above, you could easily do so like this:

The whole idea behind a Custom Hook is to support the reuse of stateful logic and to abstract the implementation of something frequently used in your application (acts as a black box).

Popular Custom Hooks

At our client project, we use several Custom Hooks in our daily work. The most important aspect of using them is that they solve a typical problem in the application. These abstractions make sense for us in our daily work. However, there are some patterns that may be recognised even across projects, domains and countries. This is where popular Custom Hook packages come into play. Fortunately for us, many have already started to develop comprehensive abstractions and black boxes for us. Let's take a look at the most popular:



useForm – Ever struggled with state in form validations? It can really help you manage the state of your forms.



react-use – Not a single hook, but a collection of many different (contains hooks for interacting with sensors, UI, state etc.)



useHooks.com – This website presents third-party Hooks every once in a while with simple but detailed examples.

Still, it is a joyful task to create your own Custom Hooks. For example, have a look at how you can make setInterval declarative with Hooks, or check out what people already have done in other awesome react hook packages.

And remember, by creating your very own Custom Hooks, you will automatically be just as cool as the redux and react-router-dom npm packages, who recently added several hooks. Ho Ho Hoooooooks! 👊🎄

Home-baked Hooks

During standup the following day, Dag was asked to share his progress on the team Christmas card. He agreed to share it on their primary communication channel without slacking too much.

He shared his work and waited for feedback. It did not take long before he was asked if he had an abstract period. "Why do you ask?" he wrote in confusion. Selbekk, their team leader, answered with a screenshot and the comment "did you test it in Safari?". He had not.

"You know. You could use Can I use to look up browser compatibility," Selbekk suggested. Dag did as suggested and found the following table:

Can I use - clip-path

"Okay, no support for IE or Edge. That might be okay, most of the receivers of the card are using Mac or are developers, so they probably use another web browser. Hm, light-green means partially supported. So I have to see if Safari supports it." Dag hovered the relevant box and got the support note "Partial support with the prefix -webkit-."

"Hmm… Let's see if adding -webkit-. to clip-path makes it work in Safari!"

Dag updated the code and tried to rerun the card in Safari. The card worked as expected this time.

"It's been some time since I've looked into browser compatibility," Dag thought. "Either I've forgotten to test, or it seems like the build has just fixed the prefixes by itself."

Dag spent the remaining time he had left for the side project reading up on CSS builds and auto prefixing.

To be continued tomorrow.

CSS Christmas card - Part 2: But can you use this?

Fossefall. Kravspec. Use Case. Systemtest. Overleveringer. Hvordan var det å jobbe i den hverdagen?

En spøkelseshistorie fra fortiden. Var det verre før?

Dashboards and information screens are becoming increasingly common for monitoring and displaying important, relevant information at a glance. Over the following two blog posts, we will look at how I, without much programming skills, was able to build such a dashboard in the cloud using Google’s BigQuery and DataStudio tools.

Bekk’s Oslo office is located at the tip of Vippetangen. Thus our employees are mostly dependent on public transportation to get to clients, social events or back home from work. We wanted to build a simple dashboard in our reception hall that would provide employees and visitors with realtime information about nearby buses, city bikes and electric scooters on their way out of the office.

The Google Cloud Platform (GCP) combines powerful cloud tools with the simplicity and ease of use from Google services. To build the dashboard data, we made use of BigQuery. BigQuery is part of GCP and is a serverless tool ideal for performing analysis on large datasets using the SQL language, without the need for administering a local database.

1. Obtain and store your data

BigQuery has plenty of huge, open datasets you can play around with, or you can connect directly to common services such as Google Analytics.

As the dashboard had to display real-time data, this required us to continuously gather the data through APIs, and stream it into BigQuery.

Oslo City Bike and Entur (who delivers information about public transportation and e-scooters), both have open real-time APIs. GCP provides the ability to create scheduled functions that collect data through an API and insert it into rows in a BigQuery table. These simple functions can be written using Python or another language of your choice. This is the function I use for obtaining the most recent data through Oslo City Bike’s API and storing it in a BigQuery table:

Function to stream data from API to a BigQuery table:

2. Build relevant queries

When the functions are up and running, your table(s) will continuously be populated with the newest data, and you can start playing around in BigQuery to extract the information you want.

In our case, we did not just want to present the raw data points. For example, for the city bike part of the dashboard we wanted to:

Show how many bikes were available in the Vippetangen racks

Show other racks within a given radius with available bikes

Show number of available docks near public transport hubs to dock your bike

Display the station names, which were not collected in the real-time data table

These data points were not immediately available in our data table, and so we had to build different queries for manipulating the data as we wanted. Below is an example of a query to display the three racks near Stortinget metro station with available docks.

Example of showing 3 available dockings near Stortinget station:

Heads up:



In this case we only needed the real-time data, but there was no need to store the historical data. However, BigQuery is not as intuitive when it comes to updating or deleting rows. Continuously adding rows to your data table makes it grow in size quite rapidly, and both storage and querying can become costly. I wasn’t able to implement a function to replace the data in my BigQuery tables instead of just adding new rows on every API call. My quick workaround was to write a script that ran every night, removing the rows from 2 days before and prior, but I’m quite certain there are other more elegant solutions to this problem.

To be continued...

In part 2 we will look at how you can save your queries as views, and visualize the data with a DataStudio dashboard.

Building a dashboard with Google BigQuery and DataStudio (1/2)

If you missed the introduction to coroutines you can head over to the first part of the blogpost here 👈

Coroutines are made up of more than the coroutine builder launch {…}, a coroutine builder must be called in a coroutine context, and might have to be run by a specific dispatcher. What if we want to cancel coroutines? Can coroutines coexist with components with a lifecycle? Read on to get a brief explanation of the much needed details when working with coroutines.

So lets start with the most basic way we can start a coroutine.

As we have already learned, the coroutine builder launch {…} starts up a coroutine and executes everything inside the launch-block asynchronously from the main-thread. So what is GlobalScope and why do we need it? 🤔

CoroutineScope 🔭

GlobalScope is a CoroutineScope which all coroutine builders (like CoroutineScope.launch) are an extension of. The scope of a coroutine is often bound to a Job which has a lifecycle, and is cancellable.

All CoroutineScopes should be implemented in components with a lifecycle, meaning that the lifetime of the CoroutineScope is limited. We don't want coroutines to run when they don't need to.

Like you probably guessed GlobalScope is not bound to any local component with a lifecycle, but is global and is not bound to any job. This scope lives as long as your application does, meaning that if you start a coroutine from this scope it can potentially run as long as your application. You would often want to avoid starting coroutines in this scope if you want your coroutines to have a more limited lifecycle. It seems apparent that the more concise the lifetime of a coroutine is, the easier it becomes to understand it's purpose and debug it.

An example of such a lifecycle aware component is an Activity in Android. If you're unfamiliar with the basic building blocks in Android an Activity can basically just be a full screen window in your application. When we implement a CoroutineScope in an Activity we (most likely) don't want to have any coroutines running and doing work for that screen when it is not in the foreground of the application.

To avoid running work on a screen that the user has left, the CoroutineScope is attached to the Activity's lifecycle. So when the user leaves the screen and the Activity finish, all coroutines in the scope attached to the Activity are cancelled. In activities, the function onDestroy() is called when the Activity finishes.

All coroutines are started from the CoroutineScope and when we cancel the CoroutineScope, all coroutines started within this scope is cancelled. So when the user exits the screen in the application and the activity finishes, all coroutines affiliated with this component will cancel.

You may also notice that we don't need to supply the scope when using the coroutine builder launch {} in the code snippet above. That is because launch is an extension on the Activity because it implements the interface CoroutineScope. If we were to run the code above we would crash the instance the Activity is told to show UI stuff (fun showUIStuff()). Why?🤨

Since all interference with the UI needs to run on the Main thread this will not run and this is because it is on the wrong Dispatcher. To make the code above runnable there is one small, but very important change we need to make. We need to tell the coroutine we're running to run it on the MainDispatcher, like so:

Dispatchers 🔧🔨⛏

So what is a CoroutineDispatcher and why do they exist? Very simply explained, the CoroutineDispatcher tells the coroutine which type of threads to use for the execution of the coroutine block. Depending on what operation we wish to execute in our coroutine we need a specific thread dispatcher.

As we've previously learned coroutines have the ability to suspend. All coroutines are run by a dispatcher and it is the dispatcher that has the responsibility to resume them. The various Dispatchers are thread dispatchers that are specialised to do the various kinds of operations that they're created for.

In kotlin we have three main types of Dispatchers, these are the Main, the IO, and the Default dispatcher. They all have respective types of operations for which they are responsible and are specialised to execute.

Main dispatcher

The Dispathers.Main have the responsibility of handling operations that needs to run on the main thread. Such as the mentioned function of showing UI, or manipulating the UI in an application. If we do not use the Main dispatcher for operations such as this we will actually crash runtime, so these are operations that are very important to be run by the correct dispatcher.

IO dispatcher

This dispatcher is optimized for and should handle all events that deals with input/output and network operations. These include calls to an API like fetching data, reading/writing to/from disk, and operations that include communication with a database.

Default dispatcher

The Default dispatcher is the dispatcher that is selected when no dispatcher is specified when running a coroutine. It is the dispatcher we would want to assign when doing CPU intensive work such as sorting lists, pasing JSON and other such operations.

Now that we have gathered some knowledge about the three main dispatchers in the coroutine library, let's remake the above Activity to make it run. Furthermore, let's also define some functions to add some functionality for fetching data, and displaying this to the UI. To define the dispatcher for a specific function we use the keyword withContext() {…}.

Best practice when working with dispatchers is to specify a desired dispatcher on the smallest scope possible. This ensures that all coroutine calls is run by the desired and correct dispatcher. Because of this and readability, make suspend functions that uses withContext() to switch context to the desired dispatcher that you want to do your work on.

Another thing we have changed here that is different from the MyActivity above is that the MyActivity no longer extends CoroutineScope. This is because we no longer need to specify and create the CoroutineScope for all components with lifecycle in Android. Since android added the lifecycle part of the androidx library (androidx.lifecycle:lifecycle-*) they now handle how to connect the CoroutineScope with the lifecycle of the component. All we have to do is reference the current coroutine scope (lifeCycleScope) and use the coroutine builder launch {…} and the coroutine will live and die with the Activity automatically.

Disecting the very important details of coroutines

In this article, I will present to you some resources which can be useful to know about if you want to learn JavaScript, or in general, want to learn more about the language.

MDN web docs

This is a resource created for developers by developers and is loaded with articles, tutorials, references, and documentation on many subjects such as JavaScript, CSS, and WebAssembly. You can find information aimed towards all kinds of experience here, which also applies to the tutorials and the guides. When it comes to JavaScript, in particular, MDN has a lot to offer, and I think that almost everyone that reads this article can learn something here. There is one guide on MDN, which you need to check out named JavaScript Guide. This is a detailed guide that gives an excellent overview of the language and is divided into multiple chapters that cover topics such as functions, working with objects, promises, modules, and so on. This guide is directed primarily towards those who have previous experience with JavaScript or other languages. In addition to this guide, here are some other exciting tutorials on JavaScript that MDN offer:

JavaScript First Steps

Introducing JavaScript objects

Client-side web APIs

Memory Management

Egghead.io

This is a website that provides short, on point, informative video courses on JavaScript, web frameworks, and libraries. The courses are for both newcomers and experienced JavaScript developers and are made by both professionals and open source contributors. Most of the courses consist of small bite-size videos rather than long ones, and the videos go straight to the point so that you can save time. Each course is made up of 10-30 videos on exciting topics such as:

JavaScript Promises in Depth

Just Enough Functional Programming in JavaScript

JavaScript ES2019 in Practice

Egghead has a lot of great content that is free, but you have to pay to get full access to all of the content of the website.

Wes Bos

This website and its content are made by a full-stack web-developer and designer named Wes Bos from Hamilton, Canada. The website has a blog with plenty of references to courses, videos, and articles that Wes Bos has made. On this blog, you can find useful resources on learning the basics but also resources on more advanced topics in the realm of web development. Some of the resources are free and some you have to pay for. If there is one resource you have to check out from Mr.Bos, it must be his famous JavaScript30 course. This is a 30-day vanilla JavaScript coding challenge, where you can join over 300 000 others to learn vanilla JavaScript in 30 days with 30 tutorials.

YouTube

You can’t make a list like this without mentioning YouTube. There are just so many great courses on this platform, and the best about the platform is that it’s 100% free. I got into JavaScript just by watching courses on YouTube. So I will list some popular channels which I have discovered that has a lot of great video courses on JavaScript, but also web-development in general.

freeCodeCamp.org

Traversy Media

derek banas

Books

Finally, I will share two free books with you.



Eloquent javascript. Dives deep into the JavaScript to show you how to write beautiful, effective code.



You Don't Know JS Yet. A free series of books that lets you explore the ins and outs of JavaScript and its core mechanisms.

There you have it, a full list of resources about JavaScript which you can enjoy during the holidays. 🎅🤶

JavaScript resources you need to know about

This is the story of how we built the Node.js foundation.

Node.js started as any other open source project. One person had an idea and decided to share it with the rest of the world. This was back in 2009 and has since grown to become the largest software ecosystems with over 1 million open source packages in 2019 and some of the largest software companies in the world like IBM, LinkedIn, Microsoft, Netflix, PayuPal are betting on Node.js to power their business critical applications! We even have sent Node.js applications into space 🚀

As the creator of Node.js Ryan Dahl lead the development which was later sponsored by a Silicon Valley based Enterprise Infrastructure Company named Joyent. In 2012 Ryan stepped aside promoting coworker and npm creator Isaac Schlueter to manage the project. Schlueter announced that Timothy J. Fontaine, another employee of Joyent, would lead the project.

This is type of open source governance is typically referred to as Benevolent dictator for life[1], or BDFL, a title given to a small number of open-source software development leaders, typically project founders who retain the final say in disputes or arguments within the community. The title is often passed onto the next benevolent dictator by anointment as opposed to through democratic means.

At the same time that Node.js was seeing an all time growth in adoption the development had ground to a halt with an all time low in September of 2014. Contributors was becoming more and more frustrated with the stringent process imposed by Joyent. A coordinated effort through the node-forward[2] initiative in order to work with Joyent to move the project to a structure where the community could contribute effectively lead nowhere.

A private fork of Node.js had already existed for some time amongst some of the most active contributors but due to a trademark issue and legal threats they could not make it public under that name. The decision was then made to open source the fork under a new name – io.js.

The goal of io.js was to create a new and inclusive organisation for Node.js using an Open Governance model[3] or OPEN Open Source. The new organisation should be owned by the community and not one person or corporation.

Individuals making significant and valuable contributions are given commit-access to the project to contribute as they see fit. This project is more like an open wiki than a standard guarded open source project.

The io.js organisation was set up several autonomous working groups that had a specific responsibility. Each group was soley responsible for their work and only as a last resort when a consensus could not be reached within the group would the top level Technical Steering Committee (TSC) act as a final arbiter.

The Docker working group were responsible for maintaining the official Docker image for io.js, the Build working group were responsible for setting up and maintaining the test and build infrastructure servers from DigitalOcean, Rackspace, ARM and Voxer as well as from the community managed to get the infrastructure up and running just in time for the first release of io.js.

io.js also changed the even-odd version numbering that no-one understood and that Node.js had been following since it was first created by Ryan, and io.js adopted Semantic Versioning[4] instead with the first release being v1.0.0. The reason was that Node.js had already been proven stable enough in production for several years and releasing anything short of v1 would not get the same addoption and thus defeating the purpose of releasing a version at all.

In June of 2015, only a short year after io.js, it was made public that Joyent was transferring the ownership of the Node.js trademark to a new Node.js foundation which would be a part of the Linux Foundation[5] and governed by the open governance structure already established by io.js. The work that io.js had done would be the basis of the project going forward and remaining changes in Node.js was merged into io.js and version 4.0.0 of Node.js was released.

[1] https://en.wikipedia.org/wiki/Benevolent_dictator_for_life

[2] https://github.com/node-forward

[3] https://en.wikipedia.org/wiki/Open-source_governance

[4] https://semver.org

[5] https://www.linuxfoundation.org

From Node.js to io.js and back again...

Reporting API. That sounds really cool! Or really boring you say? This is one of the W3C-drafts that may not have gotten the attention it deserves so let's take a look!

Logging and monitoring our back-end have been pretty standardised and have been a standard part of any application for a long time. As long as things are well on the server then all must be good for the clients (the users of our web app). Right? Okey, that is a bit simplified and silly to say in this day and age where front-end development have matured immensely and a lot of good frameworks exists to handle complex user interfaces. But what happens if something goes wrong at the client? We can of course try to handle front-end errors and send them to the server within our own code, but that may only be efficient to catch errors in our own code. What if we use a deprecated API that will break our application soon? Or something is making our app require access to browser features that should not be necessary? Or maybe the user does not reach our site at all? Enter the Reporting API!

The Reporting API defines a standard way of handling issues that arise at the client side. It defines a generic reporting framework that use HTTP headers that can be sent to the client that describes how it should act when something goes wrong. If you have used Content Security Policy (CSP) you may already know about something similar. CSP had a Report-Uri property for sending violations of the policy to an address (for logging purposes). That property is now deprecated and replaced by the Report-To property of the Reporting API. Other standards, such as Network Error Logging (NEL) also provides a Report-To property so now you probably get the idea. Via the Reporting API you can now get reports from all these different types of standards in a uniform way.

To utilise the Reporting API you can send a header which includes a definition of the reporting setup of your application:

Note that this header defines a group (“endpoint-1”) which later can be used to specify which standards that should report to what group. I other words, you can have multiple groups that can contain multiple endpoints. You can then refer to the Reporting API groups when using CSP, NEL or other standards (note that the actual policy of the CSP has been omitted for the sake of brevity):

By doing this, the browser will send reports about violations of the Content Security Policy and problems caught by Network Error Logging to the “endpoint-1” group. The Report-To header will be cached by the browser according to the “max_age” property. Our sample Report-To definition also defines two URLs with different priority. This is used by the browser to send report to the second URL if the first one does not respond. In addition to that, endpoints can also be assigned weights to distribute load, with each endpoint receiving a specified fraction of reporting traffic. This can be useful if your application has a lot of traffic.

As the Reporting API header is cached it allows the browser to send reports even if the client has lost communication with the server. If you combine this with the NEL-standard you can actually get the browser to report things like DNS misconfigurations or other network related errors that means the clients does not reach you application at all!

Other types of reports (in addition to CSP and NEL) include these:



Deprecation: a browser API or feature has been used which is expected to stop working in a future update of the browser



Intervention: indicate that a browser has decided not to honor a request made by the application (e.g. for security, performance or user annoyance reasons)



Crash: the user was unable to continue using the page because the browser crashed. For security reasons, no details of the crash are communicated except for a unique identifier and optionally the reason for the crash (such as Out Of Memory)



Feature-Policy: code running on the client violated the rules that defines what browser features and APIs are allowed (this is currently in development)

As you can see the Reporting API can be used to report a lot of interesting data from the front-end of your application. All in a standardised way that does not require any specific front end frameworks or libraries at all. We have just covered the basics so head over to the links below if you want to know more. As of now, Chrome is the only major browser that support the Reporting-API (https://caniuse.com/#search=report-to), but hopefully others will follow soon.

Happy reporting!

Get your client side reports together!

No, not the Stream API introduced in Java 8, but the API for stream-based I/O which has been part of the JDK since Java 1.0. That is even older than the Collections Framework, which you had to wait until Java 1.2 to be available. And as with the Java Collections, even though the java.io API is quite dated, it is still heavily in use. InputStreams and OutputStreams are passed around, and consumed and written to, also in applications which are written nowadays.

Java got proper support for managing releasing I/O resources used by these classes as late as in version 7 with the try-with-resources language facility. Using it, the runtime would ensure that close() is invoked on the stream you are working on before going out of scope. This makes it easier to ensure your application is not unnecessarily holding on to I/O resources, and even eventually exhausting available resources, so called resource leaks.

But it may still not be entirely intuitive when to actually ensure to close streams, and when to leave the resource management to other parts of the code. It may be tempting to defensively just wrap any use of streams with a try(..) {, so closing is not forgotten, and while this actually works for many cases, in some circumstances leads to erroneous behaviour.

Let’s look at some simple principles to handling InputStreams, OutputStreams, and other AutoCloseable resources. We can summarize when you should take care to close streams with three prioritized rules-of-thumbs, prioritized meaning that each rule applies as long as it is not conflicting with any of the previous ones.

Do not close streams offered from someone else

Do not close streams you hand to someone asking for them

Close streams you acquire yourself

Do not close streams offered from someone else

This means that code operating on streams it gets from method arguments, should not close the streams as well. This is simply because the code does not know how the stream was created. Even though if the method is used to fully read an InputStream, it should not as a convenience close it when the stream is delivered through a method argument from some calling code.

Do not close streams you hand to someone asking for them

This may seem obvious, as why would you return an unusable (i.e. closed) stream from a method? Even so, it is included as it sets an important presedence for the next rule.

Close streams you acquire yourself

The only cases to consider whether or not to ensure proper closing of streams are when you take active means to acquire them, either through creating a new instance, or calling a method which returns a stream. There is a notable exception for streams which are created to be delivered as a return value from the method. Code which acquire streams are responsible for the resource management of the stream, and should expect methods which are handed the stream to not close it, because they follow the first principle, and therefore does not close streams that are offered from someone else.

Examples

The implications of this helps us realize that the following code is wrong:

We use a handy utility from Apache Commons IO to read a textfile completely as a String. As the IOUtils.toString method exhausts the InputStream, one may be inclined to think that it should also close it, being a nice utility library and all which lightens the burden on me having to think about such things. The javadoc does not mention anything whether it will close it or not, and if we trace the source code we will eventually come to the conclusion that, no, it does not even bother to close the stream, even though it could have, and it would help me a lot with writing less code. The thing is, it is perfectly valid for an InputStream to be exhausted, to be read until there are no more bytes available, and at a later time produce more bytes, and the IOUtils.toString method have no way to know that, because it was offered from someone else as a method argument.

The correct code would be something like this:

because the code is actively acquiring the stream using Files.newInputStream. Likewise, it would be similarily wrong to do resource management if our code got the InputStream yielding the file content from someone else, like this:

The resource management of the InputStream passed to the process-method is the responsibility of the calling code, and the calling code must apply the three principles to decide if it should do resource management.

Good ol’ I/O streams

High dimensional data can be a pain sometimes, at least when it comes to visualization and exploration. Today we will introduce one common technique that can potentially ease the pain a bit.

The main idea behind dimensionality reduction techniques is to reduce the amount of features, whilst retaining as much information in the data as possible. There are many reasons for doing this, for example lower dimensions means that less computational resources are needed and it also makes it easier to visualize and interpret the data. It is therefore a must know for every data scientist.

Principal Component Analysis (PCA) is one of the most common dimensionality reduction techniques. The magic behind it lies in the math, but today we are just going to briefly introduce it without any potentially confusing variables or formulas. As mentioned earlier, the goal is to find a lower-dimensional representation of the data that represents the original structure of the data just as well. The way PCA tries to find this representation is by first finding the directions in the data with the most variance, these directions are also known as principal components. After the principal components have been found, one can then project the original data points onto the most important principal components and discard the others to achieve a new subspace with lower dimensions.

This might sound complicated at first, but let's look at an example.

As one can see in the leftmost graph in the figure above, the data set consists of 5 points in a two-dimensional space. A green and a red line is also plotted in the graph and corresponds to the principal components of the points. The rightmost graph shows what it would look like if one were to project the points onto these principal components.

This might not look like something special at first glance. However if we view each principal component by itself, which can be seen in the figure below, we can see that the first principal component accounts for most of the variance in the data. It is barely possible to distinguish the data points in the second one and it can therefore just be discarded. Thus, we can just use the first principal components to represent our data.

This was an example of how we can use PCA to go from two dimensions to just one, while still retaining all the important information in the original data set. It can be applied to data with any number of dimensions and reduce it down to two or three components for visualization, which makes it a useful tool for every data scientist. For more in-depth information regarding PCA, please see the links down below.

Dimensionality reduction using Principal Component Analysis

This post explores how you can create your own generative art in the web browser using functional programming in the ClojureScript programming language.

Creative programming is about writing code to create an expression rather than developing some kind of functionality. It's an enjoyable process that can be used to create all kinds of art pieces, be it still images, animations, music etc. It's also a great framework for learning new programming languages.

ClojureScript, or cljs for short, is a Lisp that compiles to javascript. It's a dynamic, functional programming language with great support for an interactive development process, making it a perfect fit for the type of explorative, creative programming that generative art often resemble.

We will create an art piece based on a Perlin noise flow field. It uses Perlin noise to move some thousand particles across the screen in an organic looking fashion. As opposed to white noise, Perlin noise is a controlled type of noise where two values drawn from close to each other in the noise map will have similar, but still random values.

We will be using a library called Quil to create a drawing using the browser canvas api, along with a library called Figwheel for building and hot loading the code as it evolves.

A file deps.edn in the project root defines the dependencies for this setup.

To kick things off, we define a namespace to hold our code in src/sketches/perlin_flow.cljs, and bring in the needed Quil functions.

A developement build called dev is defined by placing a file dev.cljs.edn in the root of the project. This file instructs our build on what namespace that should run initially when the built javascript file loads.

A simple resources/public/index.html loads the resulting sketch in the browser.

Running the build is a matter of letting clojure know where to begin, and further instructing figwheel what build we want to be using, i.e. the dev one.

A couple of vars w and h used throughout our sketch is defined to hold the width and height of the browser window. This also shows a general example of how function calls are made in a lisp, by placing the name of the function to be called as the first element in the list (i.e. inside the parens). However, it also shows the somewhat cryptic approach cljs takes to javascript interop, where a .- prefix in front of the name is how properties of native javascript objects are looked up. We will generally not be needing this prefix, as most our code simply refers ClojureScript vars and functions directly.

We define a function create to create our sketch by passing it the nescessary options. This defines the essense of our sketch, namely what functions will be used for state setup, update and draw. We'll keep them minimal for now. The setup function sets up our drawing's state, i.e. an empty vector []. The update function gets the state of our sketch as and argument and simply returns it. The draw function, well, draws our sketch to the actual canvas in the browser. It also receives the state as an argument, but as of now, simply draws an ellipse to the center of the screen, on a dark background color.

By seeding Quil's random and noise functions we ensure reprodusable results across page reloads even though we will be calling a couple of functions providing random values.

A clever cljs macro, defonce, ensures that we only create one instance of the sketch, even though the namespace is reloaded by figwheel each time the sketch file is saved.

A white circle appears.

Try it yourself (Hint: click "Run" in the top left corner)

The keen-eyed would have noticed our create function also brings out Quil's fun-mode middleware to enable a functional workflow for our sketch. (It's going to be fun as well, I promise.) This enables a data flow coherent with functional thinking, such that sketch-setup returns the initial state of the sketch, while sketch-update can be free of side effects, operating only on the immutable state received as an argument, returning the state for the next iteration. sketch-draw needs side effects to be able to draw to the canvas.

A couple of iterations with saved changes to sketch-draw might result in something like the following. As the canvas remembers what's been previously drawn, multiple circles will be shown in the image.

Not a masterpiece, to say the least. Let's up the game.

Every piece of generative art needs a pretty palette. This palette is a map of predefined colors, on the form of vectors with rgb values. We'll use this palette to spice things up a bit when drawing particles to our canvas.

The state for this sketch will hold a vector of these particles, created by mapping the particle function over the range of numbers from 0 to 2000. On each change of the setup function we have to reload the page, as the defonce mentioned above keeps the sketch from re-initializing.

Each particle is a map consisting of the :id of the particle, :x and :y representing its position constrained by the screen size, :vx and :vy representing its speed in the x and y directions respectively, :size representing the particle's size in pixels, :direction representing the direction of each particle and a random :color drawn from the palette.

Our draw function starts off by clearing what was previously drawn by setting a new background color with q/background. q/no-stroke ensure there's no stroke around each particle. Subsequently, it uses doseq to process each particle p from our state of particles (returned by sketch-setup) for side effects. It uses the Quil function q/fill to set the color of what is about to be drawn to that of the :color the particle, and calls out to q/ellipse to draw a circle at the :x and :y positions of each particle with its :size.

This is what we get.

Try it yourself

Already it's starting to look good, albeit static. Let's fix that.

Our sketch-update function receives the current state of particles and its task is to update their state for the next iteration of the drawing, before sketch-draw will draw the resulting state to the canvas. The update and draw functions will be called after one another for the lifetime of the sketch, resembling our program's render-loop. The following function definition moves each particle according to its current position by adding in its velocity in each direction. By running the modulo function mod on each resulting position, we ensure that particles come back on the opposite side of the screen if they move outside the screen bounds. The cljs function assoc associates new values for the given keys in each map.

The fact that none of our defined functions have their own state, means that Figwheel is able to patch the running javascript code with redefinitions we make to the source without loosing the current state of the sketch or what has been previously drawn. This is an amazing feature when experimenting with generative art, and provides an immensely short feedback loop - a must when letting your creative juices flow!

Look! It's moving.

Try it yourself

Simply by removing the one line of code from the draw function that clears the background we see a radically different expression in our drawing. If you are coding along you would be starting to feel the effects that this kind of short feedback loop setup can have.

The circles become lines!

Rest assured, the circles are really moving, but my gifs were getting huge due to the poorly compressable content!

Changing around each particle's speed by a random amount between [-0.5, 0.5] and capping it at 2 renders a rather busy result.

Try it yourself

Let's introduce some Perlin noise to start heading for the effect we're after and attempt to calm things down a bit.

We will give each particle a direction from [0,2π] by use of Quil's Perlin noise function, based on the noise (i.e. a number between [0,1]) at the particle's location x and y in the noise map. We'll let this represent "the wind" that carries the particles across the screen with slight variations in direction. By averaging the current velocity and the Math/cos/Math/sin of the updated direction we keep it within bounds of [0,1]. To calm it down a bit we also scale the x and y values by the noise-zoom. By "zooming in" on the noise map, values close to each other will have increasingly similar random values, easing things out. This forms a great basis for creating natural looking flows of particles.

Getting there!

Try it yourself

To make things more natural looking, let's introduce some more noise on a per particle basis, to add the effect of uniqueness to each particle's movement. We'll do this by adding a touch more perlin noise to each calculated direction by using the :id of each particle as a third argument to the q/noise function. This looks up the Perlin noise value in three dimensions, making it unique per particle.

Not radically different, I hear you.

Try it yourself

Now try this: adjust the noise-zoom scale by a slight increase. In this way we are effectively "zooming out" of the Perlin noise cloud, making its randomness more varying, making the curves stand out more in the drawing.

It flows!

Try it yourself

Playing around with opacity often create stunning effects in sketches like these. Try modifying each drawn color by adding an alpha channel, of say 3, to the [r g b] color vectors making them [r g b a], e.g. so the first color of the palette becomes [32 0 40 3]. We'll do this for all our colors by appending 3 to each color vector as it is drawn. This is done using the cljs function conj.

Not immediately exciting, hang in there.

A couple of minutes later the following slowly appears!

That's it! Your own, personal, generative art masterpiece is complete!

Try it yourself

To finish off, here's a couple of more renders for inspiration, done only by tweaking some of the hard coded values in the sketch discussed.

And while your artwork might feel complete by the time you read this, what you'll find once you start experimenting with these kinds of sketches on your own is that most often that not you will never "complete" these kind of art pieces, because of the sheer joy of seeing the radical effects that minor tweaks you make will have!

Suggestions for further experiments:

extract constants to variables so you can change them while the art is drawn

adjust the size of the particles

increase the speed of the particles

increase the angle of the direction by orders of a magnitude

adjust each particle's uniqueness

change stuff around while the drawing is progressing

clear the background on each render again, so all lines become independent particles

experiment with different color palettes

try new seeds for the random functions, or simply remove the seeds altogether, creating new, unique art pieces every time you render the page

I hope you enjoyed this tutorial and that I encouraged you to experiment with generative art on your own! Don't hesitate to hit me up with questions over at twitter.com/torgeir or github.com/torgeir.

13-Dec

Today's posts