Inception, Inspirations and First Stitching

Let us introduce ourselves.

The Slashers are interested in open-source media control tools for live performances.

End to End

Maybe beginning to beginning. Performance essays is a multimodal approach to conveying ideas and concepts to an audience by taking advantage of the many avenues of perception by humans (not limited to simply listening to a written work)

One end of the performance essay is input

We're looking to make an instrument of sorts - one for the performance essayist

It should at minimum be able to begin and end (i.e. display fullscreen and minimize again) any media type

• image

• video

• audio

• and so on!

When we see media we must ask how do we believe we can interact with it. How do we connect, modify, and layer different media types? How do these interactions extend into a the physical space? How do we generalize an all-media "instrument"?

Considering the Software Stack

We are using Node framework (https://nodejs.org/)

We are developing using React (https://reactjs.org/)

We are building to a desktop app with Electron (https://electronjs.org)

Last month we began to examine the way that a live performer could - independently - interact with different forms of multimedia. Inspired by the flexibility and abstraction of MIDI controllers like the Abelton Push and Monome Grid, we sought to create an application that instead of controlling audio signals would control multimedia on a desktop. The app would effectively bring the "control booth" of a planned theatrical piece into the hands of the performer in the form of an instrument - empowering the performer to both plan the performance and make improvisational choices during.

We have a passion for both open source and accessibility which is reflected in our choices for the app's architecture. Initially, we considered using standalone OS platforms such as openFrameworks, Unity, python-Qt, and Pure Data. Most of these desktop frameworks are either outdated or overly-complex. In the case of python-Qt, image display and audio/video playback are separated into different models - a deliberate design choice that makes our desired outcome difficult. With multi-platform compatibility on the table from the very beginning, we took some inspiration from webapp-to-desktop porting to develop our project as a web app built on Node.js and React. These frameworks give us access to development paradigms like Single Page Application, components, and static serving which make React a perfect fit for us.

Further iterations have led us to develop without the need for a server - working on a local device with no need for internet. Static media assets are a major part of the functionality, but the "container" of a typical web app - the browser, is unaware of the dynamic usage of these files out of security concerns. Dynamic usage includes resolving absolute path from relative path at runtime, loading and playback on demand, etc. Normally it would require us to build a server to host these files. The app could, in principle, help a user to upload the files from local machine to the server through HTTP requests, and return certain files when user requests them from the client React app. And, this could certainly work. However, requiring the artist to have internet access, exclusively carrying out a tiny function on the server, and dependence on network latency during a performance are critical drawbacks. Electron.js can handle turning our web app framework into a desktop app. The last of the major architectural hurdles is the ability to access local files on a device - important given that we want the performer to use media that they have on hand.

By default, React sandboxes away the local filesystem out of the same security implementations in web browsers. We wanted to leverage the desktop version and we found a library, CRACO (Create React App Configuration Override), to bypass the security isolation and access the node filesystem control library "fs". With its help, users can easily drag-drop files from local folder to the app without any additional processing of the files and performance degradation. With the larger foundations now set, we look forward to refining the appearance and interaction of the application. The design itself begins to ask questions about the human-computer interaction that is afforded. How can we visually layout the media that an artist wants to organize for a performance? What types of interactions are expected with the different types of media (audio, text, video, etc)? Moreover, what are the modifications that an artist may want to make in a live setting (fade in/out, play/pause, transform)?

As we continue to develop the software and interface we are also looking towards the physical computing aspect of the project. We are looking at this from the perspective of having a physical type of instrument that can interface directly with the software (and provide all the affordances that we come up with for the software). Over the next month we will begin sketching out what an input system will look like for our software and how to make it as accessible and open as possible.

• We have a github repository!

Kelly and Bil