Flutter AR — solutions overview

If you want to implement some Augmented Reality features inside your Flutter app — get ready for hard times. The first big challenge of yours with Flutter AR app will be to pick your technology stack and you should do it with caution. It will heavily determine your future possibilities, but also potential problems.

The story behind this article is our struggle with finding a way around this topic in our first attempt to create a Flutter application with AR functionalities. It was tough to find a way around all available tools and approaches. It was difficult to even start understanding what we need and what this is all about. Now, we want to help you skip this hard part and receive a general understanding of the topic.

In this article, you will find a complete overview of the most common AR development approaches in Flutter. Also, you’ll learn about some known catches, what kind of files with 3D models you need. Eventually, you will receive some more articles to learn more about topics that may interest you.

Before you start, answer these questions:

• Is your AR feature complex and crucial to your application or is it just a little bonus for your users?
• Do you want to display static models in mid-air, place them on surfaces, augment faces or make complex animations?
• Do you want to have fixed 3D models inside your application or to fetch them dynamically from a remote source?

Coding approaches

The high-level overview is not very complex. You need to have your 3D models created by a graphic designer. Then, compile them into an application or fetch them from a remote source in the runtime. Finally, you will use one of the following methods to display this model to your users.

We’ll cover three coding AR approaches in Flutter that we’ve considered during our projects:

Unity

If you wish to implement breathtaking AR features, you should consider embedding a Unity project.

Unity is one of the top-shelf engines to work with Augmented Reality. It provides the widest range of features and IDE proficient in working with 3D graphics. What is more, it allows you to keep your AR features in a single codebase, without implementing those features separately on Android and iOS. You can also choose if you wish to implement a user interface inside the Unity project or use Stack and some Flutter Widgets on top of the Unity screen.

There are two main cons of using Unity — it requires a paid licence and integrating Unity projects in Flutter may be cumbersome. You can use one of the available packages like flutter_unity or implement embedding yourself in the native code. While native code will be much more time consuming, you may not want to depend on a Flutter package, which is not maintained by any resourceful third party. This is a tradeoff you have to make — risky package dependency or additional work to be done.

Pros

• The widest range of AR features,
• Single and mature AR engine,
• IDE made for graphical development,
• Single codebase for both platforms.

Cons

• Paid licence (unless you don’t mind Unity logo on AR startup),
• Risky Flutter package dependency or additional work manually embedding Unity project on two platforms,
• AR development using programming language and platform you may not be familiar with.

Flutter Packages

The quickest, yet the most risky and the least flexible solution. My recommendation is to use this approach only when developing an MVP or a fairly simple AR feature.

You need to know one thing up-front: so far there is no Flutter AR package that works both on Android and iOS. This means that you have to implement separate solutions for both platforms anyway. The good thing here is that you write those pretty fast and use only one programming language. Coding these two solutions in the native code takes more time and may be difficult for a programmer who is not skilled with a given platform.

This approach is very risky because it requires you to depend on two packages, which are not maintained by any resourceful third party. For example, you could use the arkit_plugin for iOS and the arcore_flutter_plugin for Android. While these libraries provide convenient API that makes the initial development fast, they lack access to low-level controls. This makes it difficult or impossible to work with the performance or with more subtle behaviour.

Pros

• The fastest way to implement simple AR features,
• Everything is written in Flutter,
• No Android/iOS knowledge required.

Cons

• Two separate packages required to implement AR for Android and iOS,
• Dependency on two risky packages,
• Little or no low level and performance control.

Native code

Verbose, but safe and flexible solution.

At first glance, this is not much different than using Flutter packages, since we have to write two implementations anyway. What is different is that we have complete control over what’s going on. On Android we can for example use SceneForm (a high-level framework) or ARCore (low-level choice). Also, in my opinion, this is the safest approach. We depend only on high-quality libraries and we don’t have to integrate an external Unity engine into our application.

The price to pay is more time coding these solutions. Also, since AR is pretty complex stuff, it may be difficult to implement it without platform-specific knowledge.

WARNING: The most common way of implementing AR for Android used to be SceneForm from Google, which is archived and no longer maintained. Currently, it is unknown if Google will introduce any new framework for high-level AR development.

Pros

• Complete control over AR solution,
• No dependencies on risky, third-party libraries,
• You can write user interface in Flutter or native code, as you prefer,
• The easiest way for a developer with experience in a specific platform.

Cons

• Platform-specific knowledge highly recommended,
• Unknown future of Android AR libraries,
• May take more time than other approaches.

More info

To learn more visit:

• Flutter Europe conference talk about AR in Flutter, by the author of arkit_plugin package.
• Unity documentation for AR on mobile devices.

Types of AR models

Bad news, once again. You will most probably need two different models for different platforms. Hopefully, it won’t be a serious issue if your graphic designer can export them from one project.

There are so many types for different use-cases, but it is highly probable that you will end up with:

• Collada's Digital Asset Exchange (.DAE) or USDZ for iOS,
• GL Transmission Format (.GLTF) for Android.

WARNING: If you choose .dae files for iOS and you wish to fetch them from a remote data source, you will have to make an extra step before doing so. When compiling .dae files into an application they are converted to .scn format by Xcode. If you want to fetch them from an outside location, you need to perform this conversion manually. You can do so, using the Xcode script:

/Applications/Xcode.app/Contents/Developer/usr/bin/scntool --convert [inputFilePath] --format c3d --output [outputFilePath] --force-y-up --force-interleaved --look-for-pvrtc-image

For example:

/Applications/Xcode.app/Contents/Developer/usr/bin/scntool --convert flower.dae --format c3d --output newFlower.dae --force-y-up --force-interleaved --look-for-pvrtc-image

It requires you to have a Mac.

More info

To learn more about available 3D models types visit:

• Available 3D file types and their pros and cons
• USDZ type for Apple devices
• GLTF type and additional info about other options

Some other obstacles

There are at least a few things that will make your AR development slower and you should know it.

Forget about using hot reload if you don’t use the Flutter packages approach. Even if you do, you will have to rebuild an entire app when you switch model files. This alone causes the whole AR development process to be significantly longer than coding a regular feature.

Your graphic designers WILL send you incorrect 3D models files. GLTF files with incorrect paths to textures for example. Fortunately, many file types are human-readable so you can find their mistakes, fix them and instruct them what they should avoid in the future.

Conclusion

You have a lot of work and research yet to be done, but hopefully, you understand what you are dealing with and you know what your options are. Maybe, tell your graphic designers what kind of models they ought to prepare for you and you’re ready to start playing around.

You have to understand your unique business requirements and choose an implementation approach that is suited best for your needs. Once again, you should make this decision with caution.

If you prepare yourself upfront for what is ahead, everything should go just fine. At the end of the day, AR is just a little fancy idea for implementing a user interface. It is complex, yes, but the heavy lifting is already done by brilliant people who developed tools like ARCore or ARKit. You only need to integrate them with your unique ideas.

Good luck and have a great time implementing something new!

And if you've never worked with Flutter and want to learn more about it, check out this article.