Gaming “Sword” Controller

Team Project: 2022 Spring

Chang Chen, Ivan Shao, William Huang

Shikai Zheng, Jingtao Tan

1. Background and Motivation

In recent years, Virtual Reality (VR) and Augmented Reality (AR) have taken a big leap in their development. Gamers were introduced to new kinds of gaming experiences. The VR games can track the player’s motion using a controller and display corresponding reactions in the game. However, the VR equipment requires a large upfront cost. The weight and shape of the controller do not resemble the in-game items. In this case, our team was seeking a way to build a wireless sword-shaped controller to play video games on any platform.

Triangle part

Handle

2. Implementations

a. Real-time

The real-time feature of our project was implemented by the ESP32 timer interrupt and its pin interrupt. There are two different timers for the program as shown in figure (2.b). One for the Neopixel LED strips control, the interrupt frequency was set as 50Hz (20ms). Changing the brightness and color of the light strip at this frequency allows the sword to present the breathing and pulsing animation in a different game stage. Another timer was implemented for the data communication between Esp32 to unreal-Engine. The interrupt frequency is 10Hz (100ms). Based on the motions of the player, IMU data was analyzed to determine the “attack mode” of the character in the game. At the same time, the joystick reading value and the “attack mode” are packed into a JavaScript Object Notation (JSON) and sent to unreal-Engine by User Datagram Protocol (UDP). At last, the Pin interrupt was built for the joystick switch.

b. Multitasking

The multitasking feature was implemented by the ESP32 interrupt and the unreal engine. On the ESP32 side, data acquisition from IMU and joystick, data transfer between ESP32 and unreal engine via WIFI, as well as LED control are running at the same time with different timer setup in micropython. On the unreal engine side, blueprints for the main character, enemies, and game run at the same time when the game starts. The blueprint for the character keeps receiving commands from the sword and displaying attack animation of the character. It also updates the health and the score of the character, where the health is calculated by subtracting total health to total attacks from enemies, and the score is obtained by counting the number of enemies’ death. There are two blueprints for the enemy. One of them keeps spawning enemies in three different locations at different spawn rates once the game has started. Once the enemy is spawned, the other blueprint will control the enemy to move toward the player and displays the attack animation of the enemy when it approaches the player. The blueprint for the game times how long the game has started and decides whether the player wins the game or not.

c. GUI

LabVIEW was our first choice in creating the GUI. During our project meetings, we discussed the feasibility of making animation in the VIs. Using 2D images would introduce excessive work to create and import images into LabVIEW. Soon after, we found that it is out of our knowledge range to overlay images on top of a 3D simulator, which is going to plot the motion detected by the IMU. Then we looked into using a 2D color box array to make an 8-bit pixel-style game. However, due to the lack of a zooming feature in the Front Panel of LabVIEW, we found that it was not possible to select a fine pixel element in the color box to adjust its properties. After the progress meeting, we considered the advice and explored the game development software such as Unity and Unreal Engine. Although Unreal Engine is C++ programming, it has a Blueprint coding feature that closely resembles LabVIEW.

The default GUI in Unreal Engine when the game started is the Main Menu Level, it is a Starter Level that is separated from the main Gaming Level. The START Button on the Main Menu Level is connected to a click button. After clicking on the button, the game will turn to the main gaming level. SETTINGS Button are for users to change the display resolution of the game, while QUIT Button is binded with the Unreal Engine built-in quit game function in the Gaming Blueprints.

For the gaming scene part, we made full use of the free resources in the UE5 to build it. Our game scene is set in a small valley, the sky is gray without sunlight, surrounded by old crumbling buildings. We created a silent and scary atmosphere by adjusting the lighting settings, such as SkyAtmosphere, skylight,etc. We also added a red spot light at the top of character and a scary BGM in this game level making a more tense game atmosphere. Moreover, we added a cool sound effect for each of the character’s special attacks. The main gaming level has a health bar widget along with score and timer. The score counts the number of zombies the character kills while the timer is set as 60 seconds. The score count is binded with the variable score in the blueprints, each time the player hits the zombies, the score will increase by 1. The victory condition for this game is not being killed by the zombies in 60 seconds. The character has a health bar with 50 health points, the player needs to control the sword to kill as many zombies as they can and try to survive in 60 seconds. Each time a zombie hits the character, the character will lose one point of health bar.

d. Unreal Engine Blueprints

3. Testing

Closing Remark

In this project, I was responsible for sensor data analysis, algorithm design, communication between ESP32 and desktop, and parts of gaming setup. in the future update and development, we will try the method and implement a full version of the motion tracking system. Unlike typical VR systems which employ additional vision sensors to enhance motion tracking ability, our system would use only IMUs to perform the tracking tasks. By using two IMUs we can map the motion of the player’s upper arm and the sword into UE.

Page updated

Google Sites

Report abuse