Windows 8.1 Panorama Files

Windows 8.1 has a level of panorama creation and viewing built into it, in the (Metro Modern Windows Store App) Camera and Photos apps. Making a panorama on a suitable device – in my case, a Surface with its frankly awful camera – produces a .pano file, which can be viewed in the Photos app, and nothing else. It also can’t be edited. (The files do at least pretend to be a normal picture, displaying a flattened partial view as a thumbnail.) Viewing it on a suitable device – again, for my purposes, the Surface suffices – also permits an ‘augmented reality’ mode where yawing and pitching the device changes the view as if it were a window. Hardly a unique feature, I’m sure, but it’s still neat.

On one hand, this made sense – presumably the file was in a new, essentially incompatible format. On the other hand… the only way to make panoramas is with the camera on the device, it’s apparently not possible to view normal photos in the panorama mode, and I wanted to be able to display panoramas I’d already created with real cameras in the Surface’s AR mode. And so, time to investigate!

For the time being, take everything from here onwards with a grain of salt, since I’m partially sleep-deprived and haven’t done all that much testing, and didn’t take all that many notes at the time. Forgive the lack of screenshots for similar reasons.

File Format

On a whim I tried opening it with 7-zip and… immediately succeeded. It turns out a .pano file is effectively a renamed zip file. Ones created by the camera app don’t have any compression – there isn’t much point when the file is only a few megabytes of jpegs anyway – but this isn’t a requirement. It does explain why image editing programs couldn’t do anything with it, as well.

In fact, though I wasn’t aware of it at the time, it’s an Open Packaging Convention package. It includes most of the usual OPC files, with the panorama image stored in the ‘formats’ folder…

Example.pano
 | _rels
    | .rels
 | formats
    | cubemap
       | back
          | 9
             | 0_0.jpg
          | 10
             | 0_0.jpg
             | 0_1.jpg
             | 1_0.jpg
             | 1_1.jpg
          | 11
             | ...
       | bottom
          | ...
       | front
       | left
       | right
       | top
       | atlas.jpg
       | cubemap.json
 | properties
    | properties.coreprop
    | thumbnail.jpg
 | [Content_Types].xml

properties.coreprop has some panorama-specific metadata – date taken, megapixels, fields of view, and location – and a namespace link to http://schemas.microsoft.com/panoramaDocument/2013/panorama-properties, but otherwise the xml files weren’t that interesting.

Panorama Image Organisation

As poorly shown by the diagram above, the panorama created by my Surface is in the form of a cubemap. (Presumably the actual specification allows for other formats.) The sides of the cubes have their own folders, each with several ‘detail level’ folders. The lowest detail folder is always 9, containing a single jpeg image of no more than 512×512 pixels that encompasses the whole side of the cube. The next level contains at most four images of the same maximum size, level 11 has 16, and so on. I don’t know how much of this is potentially variable – there’s a mention of the image size in the json file – but at least in the ‘official’ panoramas, those are the numbers used. (I assume it starts at level 9 because 2^9 = 512; levels 8 and below would still just contain one image of the entire cube face of progressively worse quality, so they’re skipped.)

Individual detail images start at the top-left of the cube face, and progress in the order x_y.jpg (i.e. one column after another). There doesn’t seem to be a requirement for the original file to be square, or even exactly divisible by four (though it might be the cause of a minor display issue I haven’t figured out yet, or that might be wrong entirely). If the last column or row is less than 512 pixels wide, the remaining image is just saved as-is. For example, the last two images in one Level 11 folder are 88×512 and 88×88 as the edge and corner respectively.

The camera on the Surface isn’t of sufficient quality to go past level 11, but I didn’t think that would be an actual limit when viewing one. (It isn’t, mostly.)

As for the other files – atlas.jpg is a compilation of the cube faces, horizontally side by side (specifically in the order: front, right, back, left, bottom, top). The faces are about half as small again as level 9 – 204 pixels high versus 406 for my example panorama. cubemap.json contains the short snippet (newlines are my addition):

{"cubemap_json_version":1,"tile_size":510,"face_size":1616,"front":{},"right":{},
"back":{},"left":{},"top":{},"bottom":{},"field_of_view_bounds":[0,360,-90,85.895],
"initial_look_direction":[-2.052500000000002,0]}

I haven’t experimented with the initial look direction values yet. Of more interest are the tile_size and face_size values – the tile_size is two pixels smaller than the actual tile sizes, and the face_size is proportionally wrong (off by two pixels for every 512). Finally, the thumbnail is at a rather unthumbnail-like size of (in this case) 1021 x 685 pixels. This one covers a horizontal FOV of about 180, not the whole panorama, and is in a equirectangular or cylindrical or similar projection (I haven’t looked much at it).

These extra files are required for the panorama to display at all; if they’re missing, the Photos app complains that the panorama is damaged. They are used, after all – the thumbnail is shown in the file browser as an icon and when initially loading in Photos, and the atlas is used for the lowest-resolution cube images (really only visible if it’s having trouble keeping up with panning and loading the higher-detail squares).

Making my own .pano

NYC

At the risk of sounding like a shill – thanks to Microsoft ICE, which is incredibly good considering it’s almost entirely automatic and unconfigurable.

At this point it seemed a lot easier than I’d been thinking to take an existing (cylindrical, single file) panorama and convert it. Exactly how easy I thought was too optimistic, but not by much!

faceThe first step was getting the full-resolution cube faces. The idea of doing this bit in code seemed more trouble than it would be worth – I hadn’t done much in C# with image editing at all – so I loaded the panorama I was testing with (a view of New York from the Empire State Building) in Hugin, lined up the preferred point as the ‘front’ of the cube, set the fields of view to 90 by 90 degrees, and manually exported each face. I’m sure I can at least automate that part a little if I have to do it more often in future.

After poring through documentation and forum posts and blogs for a while, I managed to get a horrifyingly inefficient program to cut up the full-size cube faces into the right sizes and filenames and folders for the panorama. (It is at least no longer inefficient to the point that it breaks with ‘out of memory’ exceptions, and I’ve learnt an important lesson about Dispose(), garbage collection and memory leaks today.) The original panorama image was 23040 x 2959 pixels, Hugin produced cube faces of 7334 x 7334 pixels, and all this meant that I ended up with a maximum detail level of 13 (pretty trivial to calculate – minimum integer x  where 2^x > cube length). The original panorama wasn’t a complete sphere, but exporting added a black background in the areas without an image, like the Camera app does, and meaning there were no empty/transparent cube areas.

Capture

Hey, it works. Eventually.

I’ll skip over a longish section of tedium here where I thought the zip format or compression was the problem, but in fact I’d forgotten to include some of the extra files. I ended up just copying the ‘official’ panoramas’ copies over for the time being.

Final Result

Excuse the desk - it needs repainting.

Excuse the desk – it needs repainting.

The original panorama weighed in at 12.5 MB as a jpeg; the final .pano version reaches 30 MB (with minimal compression). It displays almost correctly on my desktop computer; there are tiny (aliased) partially-visible lines along the cube edges. I haven’t yet solved this – it might be from the overall dimensions of my cube not being divisible by four (the original Camera panorama is), a math error somewhere, a tile rendering issue… more time will hopefully tell. Performance when panning is fine, although the background atlas file shows through when turning quickly – probably only noticeable because, as I copied it from the other, it’s of a completely different image. There’s also some moiré on a few buildings that shows up when panning, but that’s not a huge deal.

On the Surface, it’s a somewhat different story. The edge lines are still visible, but performance takes a noticeable dive. Panning and zooming are still fairly smooth, but the ‘atlas gaps’ at the edge when panning get a lot bigger before they fill in. It’s even possible to spin it fast enough so that only the atlas is visible, with the detail tiles filling in a second or so later. On a more positive note, the AR mode works perfectly! So I’ll consider this a success.

Further Investigation

  • What’s the minimum data required for a panorama that the Photos App will open? Does this include required metadata in the xml files, e.g. field of view, megapixels?
  • Do tiles actually have to be 512 pixels? Can I vary the value in coreprops.properties with success? (Even if they don’t, presumably this size was chosen for a reason, though.)
  • What’s causing the edgelines problem?
  • Does the length of the cube faces have to be divisible by four? (Possibly related to directly above.)
  • Are levels below 9 used? Is there a maximum? Is level 9 or above required?
  • Can portions of a cube face have higher detail, or does the whole cube face have to be tiled at each level of detail?
  • How do the Camera and Photo apps handle panoramas with a small field of view? Are the unused cube sides required at all levels, only in the atlas file, or can they be skipped entirely? (Do the field of view bounds mean they can’t even be viewed?)
  • Is cubemap the only format available?
  • What about sharing the other way – extracting the original image from the .pano file? The Share Charm doesn’t seem to produce anything useful, and while reassembling it from the cube faces would be fairly straightforward the Surface output isn’t high-quality enough for this to really be worthwhile.
  • Not a question, but I could certainly improve my little program – producing the extra files correctly, finding a better way to produce the cube faces, checking for more memory leaks and inefficiencies, etc.
Advertisements

5 thoughts on “Windows 8.1 Panorama Files

  1. Kae

    If you would be so kind, I’d very much like a link to this “horrifyingly inefficient program” you spoke of. It’s the key element missing from this article that would allow the rest of us to create .pano files. I’ve rigged these files to work with Oculus Rift, but I can’t create .pano files out of the panoramas I took outside of the surface tablet.

    Reply
    1. bdhurkett Post author

      I’m happy to share, but in hindsight it’s not just inefficient, it’s incredibly confusing (and I wrote the thing!). I’ll fix up the worst of it and see if I can get something out later today.

  2. Pingback: Making .pano files the hard way | Ben Hurkett

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s