a creative coding toolkit for spatial interfaces

# Spatial considerations

## From screens to rooms

Traditional graphics platforms tend to put pixels in a 2D plane, where [0, 0] is the top left (or lower left) corner of the screen. 3D graphics tends to use a coordinate frame that’s completely abstract. By contrast, Greenhouse uses a single 3D coordinate frame: yours. The frame of reference for the graphics that you draw is shared by those pixels, the human user, and the room they’re in.

This coordinate system always measures space in millimeters – not pixels. Physical objects, furniture, people, input devices, pixels and software objects can all be related to each other in this same reference system. We sometimes call this concept ‘real world pixels’.

You can define (through calibration, or by fiat) where the origin is in the room – where [0, 0, 0] actually resides. You can define where the views into this space are, how big they are, and what their orientation is.

We’ve found that sharing a single coordinate system is a powerful unifying step when building applications that run across multiple displays and machines; and it also gives us leverage when dealing with gestural or other spatial input.

## Felds

A Feld is what we call a view into this single global space. A Feld is usually the same thing as a window, though it doesn’t have to be; a Feld can also be an abstract or offscreen view (think of picture-in-picture).

When we run fullscreen, the Feld is effectively the whole laptop screen, desktop screen, TV, set of VR glasses, or (if we had Greenhouse SDK for mobile, which we do not) a phone or tablet screen.

Each Feld has a camera. (If you know OpenGL, it’s just a regular OpenGL camera.) Objects are only visible to the user if they’re visible to the camera of some Feld. This is an important source of power, and also potential difficulty. So we’ll approach the subject step by step.

## Locating and Orienting Objects

Greenhouse objects, when they’re first created by a call to new, are located at [0, 0, 0]. Now, whether that object is actually visible depends on where the Felds are. But to get started, let’s assume there’s just one Feld and it’s not looking at the origin.

So to see this object, we’ll move it to the center of the Feld. This is Greenhouse’s Hello World.

``````Text *object = new Text ("Helloooo world");
object -> SlapOnFeld ();
``````

Now the object can be seen. But what did we actually do? SlapOnFeld() is a wrapper around two operations: translating (moving) the object to the center of the Feld (the center of our view), and orienting so that it’s facing perpendicular to the Feld (facing outwards, towards us). We could have accomplished the same thing this way:

``````object -> SetTranslation (Feld () -> Loc ());
object -> SetRotation (Feld () -> Norm (), Feld () -> Over ());
``````

Let’s break this down a bit.

The call to Feld(), with no arguments, grabs the main, primary, Feld. It has to be named “main”. You can add more Felds (see below), and you can call them what you like, but the main one must be called “main”.

The Loc() of any object is its center. It’s a 3D point in space, with units that are in millimeters – NOT pixels. The SetTranslation() call gives this object the same Loc that the Feld has. Locations are generally represented in a Vect object, which has x, y, and z fields. To see where the object ended up, we could print out something like:

``````INFORM ("I'm at: " + ToStr (object -> Loc ()));
``````

Now let’s look at the SetRotation() call. It’s being used to set the Norm and the Over of the object at the same time. What are those?

The Norm() of an object is simply the direction that it’s facing; it’s also known as a “normal vector”. We want our object to face the same way the Feld is, so that it’s lying “flat” with respect to the Feld, and not skewed or crooked. So we take the Feld’s Norm() and apply that as the object’s Norm().

It’s worth stopping to note, though, that unlike Loc(), which is a location – a point in 3D space, with an x, y, and z – the Norm() is a direction. It describes how far along in space something points – how much in the x direction, how much in the y direction, and how much in the z direction. If the Feld is facing straight along the Z axis, then its Norm() might be [0, 0, 1]. That’s not a point; it’s a direction.

Now, here’s something vaguely tricky for the uninitiated: by convention or convenience, directional vectors are usually represented with same the kind of object that’s used to hold points: Vect. Both points and directions have an x, y, and z, but be aware: the fields are the same, but their meaning is different.

Over() is also expressed as a directional vector. The Over() of an object is the direction running along its top edge, from left to right. Imagine a picture that’s perfectly level on a wall; if the floor runs along the Y axis, then the Over vector would be [0, 1, 0]: that is,straight over in the Y direction, nothing in the X direction, and nothing in the Z direction. A picture that hung a bit crooked would have a different Over vector, perhaps [0.11, 0.99, 0].

So we are matching the object’s Over() to the Over() of the Feld. This makes sure the object is level with respect to the window, which is what we want.

## Wrangling

Now we’ll add a layer of complexity (or power, depending on your point of view). While we have implied that objects are all located together in an absolute 3D space, that was true, but not complete. We give objects a location (and rotation, and scale) that’s relative to their parent.

In the previous section, when we moved our object around, it had no parent. (It actually had a pseudo-parent named Origin, which lives at [0, 0, 0].)

Objects can have ‘kids’ in Greenhouse, and kids inherit all the translation, rotation, and scale properties of their parent. So if we add a second object that’s a child of our existing one, and do nothing else to it, we will see it draw in the same spot as its parent.

``````Text *k1 = new Text ("^");
object -> AppendKid (k1);
``````

We haven’t explicitly told the kid where to be. But if you ran that code, you would see the “^” hat draw above one of the letter ‘o’s in Helloooo. This is because the kid’s default origin is the center of its parent.

Have a look at this line and try to figure out where the kid will go:

k1 -> SetTranslation (Vect (0, 10, 0));

It will not end up back near the origin. Instead, it will draw 10mm along the Y axis from its parent.

However, if we ask the kid what its Loc() is, it reports the absolute 3D truth:

``````INFORM ("I'm at: " + ToStr (k1 -> Loc ()));
``````

Kids are affected by ALL the geometrical and visual transformations applied to their ancestors:

• translation
• scale
• rotation
• color and transparency

We call this the wrangler chain.

For example, if we have a snippet of code like this:

``````Image *parent = new Image ("parent.png");
Image *kid = new Image ("kid.png");
kid -> SetTranslation (Vect (10, 0, 0));
parent -> SetTranslation (Vect (1, 2, 3));
parent -> AppendKid (kid);
``````

The end result would place the kid’s location at (11, 2, 3). Note this means objects have two concepts of location; it’s local translation and its global location. Here is how you would access each:

``````Vect trans = kid -> Translation (); // (1, 2, 3)
Vect loc = kid -> Loc (); // 11, 2, 3
``````

If an object is not the ‘kid’ of any other object, then these values are equal.

The wrangler chain really becomes useful when you start thinking big. When we plot data on the earth, we give the data actual lat, lon locations on an earth object that’s sized to the actual size of the earth in mm and then simply translate and scale the earth to the Feld location and size.

There is another treatment of wrangling and transforms in this article in the API reference on wrangling.

Greenhouse applications determine the room setup through two configuration files:

• /etc/oblong/screen.protein
• /etc/oblong/feld.protein

These files tell application where physical screens are located in the room and where application windows should appear within the context of those screens. The screen.protein file sets up the physical space and location of the screens in the room, while the feld.protein sets up where a traditional application window should be drawn on the screens defined in the screen.protein. Below are the default configurations installed with Greenhouse:

screen.protein - Sets up a single 15” monitor with the name “main” at a location 1000mm up and -700mm behind the origin.

``````!<tag:oblong.com,2009:slaw/protein>
descrips:
- visidrome
- screen-info
ingests:
screens:
{ main: # gives the screen a name; This value should be unique.
{ type: basic, # this value is generally unused, but can be set to
# arbitrary strings like "table" or "vertical" for
cent: [ 0.0, 1000.0, -700.0 ], # the center of the monitor's
# physical location
phys-size: [ 330.0, 210.0 ],   # physical size in mm of the monitor
norm: [ 0.0, 0.0, 1.0 ],       # the normal vector of the monitor
over: [ 1.0, 0.0, 0.0 ],       # the over vector of the monitor
px-size: [ 1440, 900 ],        # the width and height of the monitor
# in pixels
px-origin: [0, 0],             # this value is overriden by the
# "window" value in the feld.protein
eye-dist: 700.0, # the estimated distance from the monitor where the
# user will sit/stand
}
}
``````

feld.protein - Sets up a single window to be drawn on the screen defined above, 100 pixels from the left, 0 pixels from the top and with a size of 1080x675. Note: If running on Mac OS X with a Dock locked at the base of the screen, the operating system will re-center the window and throw these values off.

IMPORTANT: Greenhouse requires the first feld to be named ‘main’.

``````!<tag:oblong.com,2009:slaw/protein>
descrips:
- visidrome
- feld-info
ingests:
felds:
{ main: # gives the window (ie. 'feld') a name; This should be unique
# and it must be 'main' for the first defined feld.
{ window: [100, 0, 1080, 675], # The size and location given in pixels
# using the format:
#   [xLoc, yLoc, width, height]
screen: main  # Maps to the name of a screen defined in
# screen.protein on which this feld should be drawn
}
}
``````

For an exercise, let’s say we want to add a 30” monitor to the right of our main screen at a 45 degree angle like the diagram below:

Then our screen protein would look something like:

``````!<tag:oblong.com,2009:slaw/protein>
descrips:
- visidrome
- screen-info
ingests:
screens:
{ main:
{ type: basic,
cent: [ 0.0, 1000.0, -700.0 ],
phys-size: [ 330.0, 210.0 ],
norm: [ 0.0, 0.0, 1.0 ],
over: [ 1.0, 0.0, 0.0 ],
px-size: [ 1440, 900 ],
px-origin: [0, 0],
eye-dist: 700.0,
},
right:
{ type: basic,
cent: [ 412.5, 1160.0, -452.5 ],
phys-size: [ 650.0, 400.0 ],
norm: [ -0.5, 0.0, 0.5 ], # this value will be normalized
over: [ 0.5, 0.0, 0.5 ], # this value will be normalized
px-size: [ 2560, 1600 ],
px-origin: [0, 0],
eye-dist: 700.0,
}
}
``````

Then let’s set up 3 felds; two on the 15” monitor and one on the 30” monitor in our feld.protein:

``````!<tag:oblong.com,2009:slaw/protein>
descrips:
- visidrome
- feld-info
ingests:
felds:
{ main:
{ window: [720, 0, 720, 450], # this feld's physical location will be
screen: main                # right of main screen's center
},
left:
{ window: [0, 0, 720, 450], # this feld's physical location will be
screen: main              # left of main screen's center
},
right:
{ window: [0, 0, 2560, 1600], # this feld's physical location will
screen: right               # be the same as the right screen
}
}
``````