Dominik Hackl is a musician and computer programmer who focuses on 3D gaming and Virtual/Augmented Reality. A piano player for over twenty years specializing in classical music, jazz, and improvisation he created Magic Keys, a mixed reality piano learning app. With a very accessible 3D notation system, similar to Guitar Hero, users can play simple pieces within minutes without having to read music.
Dominik plans on releasing the beta version at the end summer 2022 with a full version available on the Oculus store next year. Dominik knows and works within the following computer programming languages: C#, Java, JavaScript, C++, and PHP.
I have been playing piano since I was about seven years old and I am now 26. I have been playing for almost twenty years. I come from a musically inclined family and my parents were always very supportive and also played instruments. There were always a lot of instruments around.
We didn’t have a proper piano at our home but we had a very old small synthesizer. It was enough to get me started. After messing around with it for a little while, my parents signed me up with a private teacher. I took two years of lessons with a teacher learning the keyboard. We then bought a digital piano which was pretty much a proper piano. I played a lot of classical music from then on.
Magic Keys is an augmented reality piano learning app with the idea that the player/user is sitting at a piano and wearing a VR (Virtual Reality) headset or an AR (Augmented Reality) headset. Through the headset the user looks at the piano keyboard and right on top of the piano keyboard the notes the user plays are projected on top of the keyboard.
With the technology you can see through the display. There are two ways of doing this from a technical point of view. There are see through displays where you have a transparent display and there are video based displays which are traditional VR devices. This means you are looking through a screen but also through the cameras within the device so you can see what is right in front of you. If you are familiar with rhythm games like Guitar Hero the same principle is applied but it is now on top of your piano keyboard.
In general, I have been working on the idea/project on and off since 2017. I started it as a student project while I was still at my university. The first prototype I did was very promising but the hardware wasn’t really ready yet. I put it aside and steadily improved it as a side project. As of last year, I decided to start working on the project full-time.
My short-term goal is to launch the first public beta version of the app this summer. For the first full version, the plan is to have it be available on the Oculus store and have it go live by early next year. The short-term goal is to get the app out and get a lot of public feedback. I am currently giving it out to selected testers and I am really excited about the public launch even if it is just a beta version. That will dictate how I continue. If people like the state of the app, then I will I try and put all of my resources into making the first version as good as possible. After the first minimal version is out, there are a lot of potential directions I can take the app.
For instance, I could take it in a learning direction. I have plans on partnering up with a teacher and maybe devising a learning schedule that is tailor made for this technology. I can think of a lot of cool things that can be done with a learning program. Possibly, virtual hands could show the player how to hand posture and how to align certain fingers. Maybe there could even be a complete character who is a virtual teacher. Also, this could aid remote piano teachers who are teaching. With Covid this has been a huge topic. I have spoken to a lot of teachers who really aren’t satisfied with remote teaching as no one was really prepared and there aren’t any good solutions yet for it.
Teachers are doing standard Zoom calls which is not really intended for music teaching. I imagine syncing the notes that the student is playing for the teacher or vice-versa. Another thing would be for live performance or improvisation. Obviously, it is not currently viable as you would have to wear the headset during the live performance. In the future as devices get smaller and smaller a lot of cool things can be done. Also, with students practicing for live performance there could be virtual crowds to help them prepare. As you can see, I have a lot of thoughts and many ideas to develop it.
I studied software engineering so I have a software engineering background and I’ve done a lot of programming in my life. I’ve focused in 3D gaming and interactive applications so I have always done a lot of 3D programming. In general, if you are doing interactive 3D work the core of your app is based on a game engine. In my case it is the Unity Game engine which is currently one of the most popular and commonly used. Unity uses the C# programming language.
The engine does a lot of the heavy lifting for the underlying program involving rendering, physics, networking, and input handling. It does a lot of the common functionality so I didn’t have to do everything on my own from scratch. There is also an editor to create 3D scenes. I started by creating the 3D virtual keyboard and then developed the system to detect notes that the user is playing. Then I developed the system to take note input in the form of MIDI data and render it out in the way that my app is currently visualizing it. These are some of the earliest things that I did.
I have been working alone since I started this while I was going to a university and developing my Bachelors and Master’s Thesis. I now have an additional VR developer who is helping me. Roughly eighty to ninety percent has been done on my own. At some point, I would love to find a co-founder. As of last year, it was a very spontaneous decision to start working on this full-time.
That is a really good question and one that I have asked myself a lot as well. These are my greatest passions in life, music and programming. Both areas require creativity and both train you in creativity. Other than that, I would rather say that they complement each other very well.
After a long hard work day of doing a lot of programming, I really enjoy going to my piano and improvising and playing to free my head. I enjoy both.
Again, as of now everybody is using the common game engines, Unity and Unreal. These are the two main game engines. Even the bigger game companies are starting to use them as they are getting so good and they have really attractive pricing models. For starters, everything is free and with the Unreal Engine you have to pay them only if you are making more than a million dollars in revenue. It is a very attractive model for starters.
For both engines, there are endless tutorials on the internet so I would really suggest downloading Unity and trying to watch tutorials on YouTube. If you want to learn programming, I recommend learning C# as it is the language that is used in Unity but if you have learned any other language that is completely fine because it is really easy to transition to new languages.
As far as other skills, a little bit of math does not hurt especially linear Algebra, vector math, and a little bit of physics. It is not required but it depends on what kind of game you are making. That would be my best advice to download it as it is very user friendly and entry level friendly.
I am pretty confident that it will be pretty widespread. If you want to compare it to the iPhone, I would say we are many years before that. There are a lot of differences between VR and AR. With AR we are further away because the display technology is much more difficult. With VR I would say the headsets that are available, The Oculus/Meta Quest is already very affordable and they cost around $300.00 U.S. dollars.
It’s really good already and there are a lot of cool apps. I think a lot of people do not know how good it is already. It will be very widespread. I don’t like the theory that everyone expects that everything in life will transition into virtual reality and all of your social life will be in VR. I am not a fan of that. I think that wouldn’t improve people’s lives. I think there will be a lot of jobs where it will be used at the workplace. If you are an architect, it makes perfect sense to use a lot of VR apps. It will be very widespread but it is going to take time.
I would say there is a lot of potential in music and with learning instruments. People are always asking me if I am planning on doing this with other instruments. It would require a different approach. If you want to do it on the guitar, you would have a completely different angle to deal with as to where you are looking at the tabs. I could imagine looking at a T.V. and augmenting your tabs there.
I would say there is a lot of potential with instrument learning. There are a lot of other uses for A.R. that currently come up. It is a very new technology and we are only getting started. I am really excited and fascinated with what people are going to come up with in this space.
© Brant Buckley Web Design 2024