Stylus Low Latency. Creating natural Stylus experiences… | by Cedric Ferry | Android Builders | Dec, 2022

Creating natural Stylus experiences with Low Latency Graphics and Movement Prediction APIs

With the rising reputation of tablets and foldables, customers are adopting new behaviors with their units. The bigger display screen space permits new experiences and empowers productiveness duties that have been beforehand reserved to laptops and desktops.

Handwriting, drawing, and sketching with a stylus unlock exact and natural workflows, like we may carry out within the analog world. But the delay between gestures and on-screen rendering can stop the pure really feel like writing on a bit of paper, that is known as latency.

I’m glad to share that the Android group has launched two new libraries: low latency graphics and movement prediction, at the moment alpha02. The libraries handle stylus enter latency on Android and ChromeOS.

The aim of the low latency graphics library is to scale back the processing time between stylus enter and display screen rendering. The movement prediction library leverages a strong algorithm to foretell the place the following stylus transfer will probably be. Implement each libraries collectively to supply the bottom perceived latency potential.

On this article, we’re going to discover how these libraries work and how one can implement a best-in-class stylus expertise in your app. Whereas we give attention to the stylus expertise, these two libraries work with any pointer, finger, stylus, capacitive stylus, and mouse.

However, earlier than getting into into the main points, let’s check out how good this expertise may be:

Slowmotion showcasing the low latency features — Gadget: Samsung Galaxy Tab S7 + S-Pen

With none latency optimizations, the system has to undergo many steps to render pixels on the display screen. The low latency graphics library minimizes latency by leveraging front-buffered rendering. The result’s sooner rendering to the display screen.

Conventional rendering makes use of a number of buffers. This ensures smoothness in consumer expertise with no visible tearing, however the trade-off is further latency between the consumer interplay with the display screen and when the content material is seen on the show. To be taught extra about rendering and buffers, watch the Challenge Butter presentation.

With a multi-buffered rendering strategy, the display-buffer accommodates knowledge the consumer sees at the moment on the display screen, and the rendering buffers are utilized by the graphics engine to renders the information (into reminiscence).

As soon as a render is completed, the display-buffer is swapped with the rendering-buffer so the consumer can see the consequence on the display screen. The rendering-buffer turns into the display-buffer and the display-buffer is freed for future use, the place the rendering now occurs.

That is potential because of the SurfaceFlinger and WindowManager. The animation under showcases the method.

Animation exhibiting how multi-buffer rendering work

To attain the entrance buffer method, entrance and double buffered layers are launched. The entrance layer is designed for fast rendering of small parts of the display screen. The layer is clear and ephemeral, as we’ll see in a second.

The entrance layer inputs are rendered to the double buffered layer to persist the work. After which the entrance buffer is hidden.

However chances are you’ll marvel how the entrance buffer manages to render sooner than the common doubled buffered strategy and what are the caveats.

The low latency graphics library writes on to the entrance buffer — in different phrases, on to the display screen. This works as a result of solely a small space of the display screen is modified (solely a small portion of the stylus stroke is modified every body).

As soon as the stylus is lifted, the common rendering with the double buffered method is resumed, and the stroke is persevered.

Animation exhibiting how entrance buffer rendering copy the strokes to doubled buffer as soon as accomplished

The entrance buffer method works finest for particular use circumstances — particularly, handwriting, drawing, and sketching. These actions modify solely a small portion of the display screen, which is the place the library shines. If we have been to switch larger areas, this might create tearing and artifacts.

The low latency graphics library isn’t meant for rendering a complete display screen, for instance a recreation, or when massive areas are modified like panning or zooming. Android has different instruments, resembling Android Sport Improvement, and the common OpenGL rendering, that assist builders in these circumstances.

The low latency graphics library is out there for Android 10 (API stage 29) and better and ChromeOS units operating Android 11 and better.

Earlier than we see how the code seems to be, chances are you’ll need to take into consideration your utility and consider what work ought to be rendered to the entrance buffer (quick work that impacts a small a part of the display screen, like a stroke) and what ought to be processed within the double buffered layer (bigger areas, like panning and zooming).

Do and Don’t — use the low latency for small display screen space updates, don’t use for full display screen rendering, resembling zooming, panning, gaming

Let’s begin coding! The GLFrontBufferedRenderer works with your individual knowledge sort, which might sometimes include x and y coordinates but additionally shade, brush sort, stress, and so forth.

The tactic requires two callbacks the place OpenGL code will probably be executed, one for rendering on the entrance buffer (delta of the stroke) with onDrawFrontBufferedLayer, and the opposite for double buffer (persistence) with onDrawDoubleBufferedLayer.

I like to recommend the Stylus to your App presentation, from my colleague Nader Jawad, in addition to the Android Graphics presentation launched on the 2022 Android Developer Summit.

You may as well seek the advice of the low latency graphics library documentation.

Now that you understand how to scale back latency on the graphics layer, let’s see how we are able to additional enhance responsiveness by lowering the perceived latency with the movement prediction library.

The aim of the movement prediction library is to remain a step forward of rendering and cut back the perceived latency even additional by creating synthetic factors primarily based on the mathematical-prediction-powered Kalman Filter algorithm. Movement prediction improves the consumer expertise by saving further milliseconds and contributing to the natural feeling that the ink flows from the stylus.

With out diving too deep into how the Kalman Filter algorithm works, it’s essential to know that the algorithm relies on course, velocity, stress and time to foretell the place the following level will probably be.

The algorithm predicts in a short time by leveraging a set of equations with little knowledge and by understanding the variation and uncertainty of the enter.

This technique of prediction is quick and reminiscence environment friendly. The algorithm is used to trace planes, satellites, and now stylus movement on Android. Different prediction methods could also be used on completely different units, relying on OEMs.

The movement prediction library will get actual consumer inputs as MotionEvent objects. MotionEvent accommodates details about x and y coordinates, stress, time… all leveraged to feed the movement predictor and predict the following MotionEvent.

As now we have seen earlier, the contact pattern price can fluctuate from one gadget to a different. The upper the contact pattern price, the sooner knowledge factors may be supplied to the movement predictor, the sooner the algorithm can predict precisely. The movement prediction library is ready to detect the sampling price and use it to assemble data sooner and enhance the prediction accuracy.

The movement prediction library relies on MotionEvent, so so long as you’re working with MotionEvent objects, you’ll be able to leverage the library.

Predicted movement occasions are synthetic and inaccurate by definition. Use the expected occasion to scale back the perceived latency, however guarantee the expected knowledge is changed with the true knowledge as soon as it’s obtained.

The movement prediction library is out there from Android 4.4 (API stage 19) and better and ChromeOS units operating Android 9 and better.

The movement occasion predictor presents two strategies, file that takes MotionEvent objects and predict that returns a MotionEvent .

See the movement prediction library documentation for extra particulars.

For contact screens to register pointer inputs with accuracy, the system should incessantly examine the display screen for contact. The contact or stylus sampling price is how typically the system checks the display screen. Most screens have a contact sampling price that’s twice the display screen refresh price. For instance, on an ordinary 60Hz show, the sampling price can be 120Hz. In different phrases, the display screen will file contact enter each 8ms.

The stylus sampling price may be even increased, on ChromeOS some units have a stylus sampling price as much as 400Hz. On these screens, the consumer expertise is even higher because the display screen is extra responsive.

On this article, we explored two methods to enhance the stylus expertise and cut back latency in terms of drawing, sketching, or handwriting. The low latency graphics Jetpack library permits functions to render sooner. As well as, the movement prediction Jetpack library predicts the following MotionEvent to all the time keep one step forward of the rendering engine.

Mixed, these revolutionary libraries make it easier to obtain best-in-class stylus experiences.

The low latency graphics library and movement prediction library can be found as an alpha launch in the present day, so you can begin integrating and testing them in your apps. We stay up for incorporating your priceless early suggestions as we proceed to enhance the libraries in future releases.

Please share your suggestions for the low latency graphics library at suggestions, and for the movement prediction library at androidx.enter suggestions. Thanks a lot for studying. Completely satisfied coding!

Related Articles


Please enter your comment!
Please enter your name here

Latest Articles