A VR Application for Laparoscopic Surgury

An idea to enhance laparoscopic surgery using virtual reality.

Medical technology is amazing. One can use a small camera to diagnosed problems in the human body. Referring to a laparoscopic surgery, a surgeon uses a small camera navigate through a patient’s body to perform a minimally invasive operation. More information can on University of Southern California, Department of Surgery’s site.

This camera transmits video footage to small screen that the doctor monitors. In a VR application, the video stream would be replaced by a fully immersive VR display. Though, current camera technology is not ready. It would require stereoscopic 60 fps high-resolution footage from a microscopic camera. This would involve both a doctor and an assistant. The need for 360 degree footage is critical. When the doctor looks around, the camera would not move and reduce the chance of injury. The ability to look around also provides opportunity to diagnose unseen problems.

The assistant will help the doctor monitor the patient’s vital signs. The doctor will also have access to this information presented in the HUD. It would not be primary. This application will not be for the faint of heart. It will require doctors to go through many training simulations.

Finally, the application will be controlled by the same hand controller in current use, to maintain tradition. Controller using head movement would not be optimal, since the doctor would need to focus. Using head gestures would be a distraction. In summary, having a full 360 degree view is much more helpful than a small screen.

A Skydiving VR Game Idea

This is going to be a crazy virtual reality!

What if one could Skydive in virtual reality and experience falling in real life? The most realistic skydiving simulator could exist in VR. People already invented indoor skydiving. It is called iFly Indoor Skydiving. Let us turn this thrill ride into a VR game.

In this facility people, customers are lifted upwards from the force of a huge floor fan. Room scale VR exists, but this will require a volume-scale VR. Such a system would track one’s movements in all eight directions. Attach tracking beacons, but on a headset, and begin flying! Unlike regular skydiving, this is in a playable contained space. It is possible.

One might need to wear additional motion tracking devices on the their feet, chest and back for it to track smoothly. The Unreal Engine has achieved real-time cinematic motion capture provided the proper suit: See Hellblade from the GDC talk. These trackers would be attached to wing suits. The fans would be very noisy. Luckily, active noise cancelling headphones exist. These headphones cancel ambient noise. Since fans emit a predicable sound, the sound would be muffled.

Imagine playing Just Clause 3. The parachutes would be pretty crazy feeling. What about meteor smashing an opponent in Super Smash Brothers while doing off the stage spectacular acrobatics? Needless to say, it is a crazy idea that might work.

Hamster Ball VR Project Week 2

During class, we agreed that particular feature needed to be changed: When you walk outside of the hamster ball, it should not stop. That was my work for the week.

Afternoon Work

Our team worked to allow the player to walk around inside the ball. As discussed in class, the ball needs to move when the player walks in it. It would also utilize the full play area in addition to the controllers. This ended up being a very challenging task. The hamster ball uses Unity physics via the AddForce() method. We programmed a function to do this, but it conflicted with the other movement system. This took about two and a half hours of work.

My partner had to leave so I spent the other two hours fixing the bug. This bug was funny. Upon moving, the player would shoot forward. Essentially, the ball updated the player, which updated the ball and so on. I tried to use a variety of implementations. The one that worked involved the headset’s immediate delta position to influence rolling. Once finished, the rolling proceeded to be synced with the player’s steps.

Night Work

The next issue discussed in class was the ability to walk outside the ball. Previously, the ball would stop moving if the player was outside of it. I spent the night fixing this issue. Moving inside the ball alleviated most of the problems, such levitating above various ledges. One bug still persisted. If the player rolled into a wall and walked out of the ball, they could clip though that wall. I decided upon an effective solution, darken the screen when the player is not inside the ball. The black screen will be replaced by a nice texture later in development.

Continue reading

Hampter Ball VR Project Week 1b

scene view

In the last post, I logged what I worked in the first week of my VR Project. According to my class, that was “Week 0”. As such, this is week 1b, the “official week 1”

VR Game

This is what the scene currently looks like.

In the Afternoon

  1. We put together a simple ball prefab in Unity.
  2. Then we programmed it so that it can move around the screen with the player.
  3. Next, we added RigidBody’s and Colliders to the ball, so that it could collide and roll in the scene. This was pretty difficult
  4. Afterwards… comments and documentation.
  5. While testing, we learned that having the ball be off-axis and falling from gravity can cause motion sickness. (We just became dizzy)
  6. Added feature, where if the player walks outside of hamster ball, the ball stops moving. A very clever solution that works really well.

 

At Night

  • Velocity Change vs. Accelerating – Hand movements are closer to being one-to-one with the rotation of the hamster ball.
  • Made the ball feel weighted – ball stops more like it would due the weight of the hamster. Ball now rolls based on controller position. Controllers outside the ball will not influence the ball.
  • More predictable – Pressing either controller’s trigger at any point will stop the ball and its rotation.
  • Interactions – Added some fun stuff to play around with i.e. Ramps, A terrain, and more cubes.

Finally

I made some cheesy poke-stops around the game. When you roll next to them, they start spinning.

 

 

Hamster Ball VR Project Week 1

This is the first log of my contributions to our class VR project. The whole class is working together for one project, a hamster ball game. In this game, the player will be using the HTC Vive. They will be playing as a hamster, controlling a ball, around a cage. The goal of the game is to escape. The mechanics are simple and represent the puzzle platforming genre.

My role is programmer. There are no leads, since the class is very small. For week 1, I was tasked with making a controllable player that can move around the screen. Before going any further, we need to know if the controls are viable and not sick inducing. Luckily, both of these work.

I made some simple data structures to manage the position of each tracked object. Using distance calculations and a little bit of vector math, I was able to get the controls to work. Next, I wanted to make the speed and responsiveness a little better. All in all, it worked out. I have posted my code below. Future posts will not post all source code, but this is proof of the work I did.

Scripts

FollowHand.cs

A simple script to move an object to the position of the controller.

Continue reading

Playing the Oculous Rift

Today I was able to try out the Oculus Rift. It is a good mix between the Samsung Gear VR and the HTC Vive. Like the Gear VR, it is conformable to wear, compact, and breathable. It is also easy to set up; plug in a few wires, sign in, and play. Like the Vive, it has a very high resolution display, built for high end computers, had advance head tracking, and soon to be controller tracking. The Oculus is the halfway point, if you will, except for the price point, which is still very high.

I played Lucky Tails, a third person platformer similar to Banjo-Kazooie. It was a very smooth experience with responsive controls. The Oculus uses the Xbox One controller, so I felt at home as if I was playing an in immersive gaming console. It might have been me, but the pixels were pretty apparent, though the lenses might not have been adjusted correctly. They were a little on the clunky side. Like the Vive, the head tracking felt the same. Overall, it was a very fun experience where I did not have to stand and walk around, but I was not able to reach out and touch objects in the game world

Consumer VR Thoughts

My current level of interest with VR as a consumer is pretty high. I am excited to see what happens with the ongoing advancements of AR and mixed reality too. I own a Google Cardboard headset, which is quite fun to use. I use for entertainment every once in a while. I also use it to show my friends what VR is, so they may get an introduction to it. Google Cardboard was not very good, so I eventually backed a Kickstarter campaign offering a more rugged headset with better quality lenses. It looked similar to the Oculus Dev Kit, but for mobile.

If these blockers disappeared, I could have the perfect VR setup. I would play exploration games,or game set in an interactive landscape. I enjoy my VR experiences to be more freeform rather than action or story driven. The amount of content for VR is in an excellent growth phase, with many great titles being developed. This is ideal, since my theoretical perfect setup would allow me to play these games on a weekly basis. It would not be every day, but I would find time to try them out.

What about an ideal price point? Google Cardboard costs around $20 plus the phone. The Samsung Gear VR and other higher end VR devices cost around $80. The Oculus rift and the Vive cost over $500 plus the computer. To me, the price point would need to be around $200; I have a very long time to go before I will purchase one.

To make it more appealing, I would want gesture to be available controls all platforms.I feel that gesture controls are important, because they allow players to use their individual fingers in a natural way, rather than wearing “mittens”, with open close hand controls. The leap motion and the HoloLens are two technologies that offer this, but are not mainstream yet.

Experience With Google Cardboard

In my Virtual Reality in Games class, I was also able to use the Google Cardboard. I want to go into a little bit more detail about how it felt.

I played the game Lands End, which is a neat VR puzzle game. The important aspect which made it work was fluid handling of objects. To pick up objects, you look at them until the indication is present. You can swing rocks around into the correct position. Never once did the rocks glitch out; it felt natural. At the same time, the world was simple and beautiful. The sound used 3D positioning, so it chanced depending on which way you looked. When I was near the waterfall, this was especially present. This is one example which makes mobile VR nice: Simple Graphics to combat phone performance capabilities and an immersive world to simulate a new reality.

No wires attached VR gaming is pretty nice.

Experience with the HTC Vive

I was able to use the HTV Vive this week as part of my Virtual Reality in Games course. I also used other VR devices, such as the Samsung VR and the Google Cardboard. They are all great experiences, but the first was the most immersive.

Google Cardboard is great for its portability and cost, but the Samsung Gear VR was much more conformable to wear. It also has more controls enabling me to play more games. Both have basic head tracking.

The HTV Vive is a different story. It was conformable and immersive. The ability to walk around in rather than using “teleportation” feels much more natural. It also spiked my inner curiosity. Also, the controllers are motion tracked too, so I was able to pick up objects rather than using look-and-see approach. The later emphasizes what some may consider loading times. The development of new VR hardware and supporting games is great, but the ease and natural use of VR is just as important. If you ever get a chance, you should try it out.

 

C# references

I spent most of this summer programming in c#. I worked on a game in Unity and my old XNA game. In both cases, I worked closely with references. Although I am not an expert, I did find them to be very useful. This is a simple, basic, programming example or references in C#.

 

In my old XNA game, I converted all of my structs to classes besides for a select few; the performance improvements were amazing. Essentially, in C#, classes are passed by references. Structs are passed by value.