Graphics and UI

The application composes the graphic user interface by creating a window with a toolkit. The display server composites an application’s windows and shows the result on the screen. For this procedure, the graphics and UI middleware offers the following 3 modules for both client and server:

• Tizen Buffer Manager (TBM)
• Tizen Display Manager (TDM)
• TPL-EGL

Figure: Graphics UI diagram

The modules are hardware abstraction layers for graphics and UI. They allow the client and server to render with the GPU, share buffers with other processes, and organize hardware output devices for various chipsets. Their backend module needs to be implemented for the new hardware device.

• TBM provides an abstraction interface for the Tizen graphic buffer manager.
• TDM provides an abstraction interface for a display server, such as X or Wayland, to allow direct access to graphics hardware in a safe and efficient manner as a display HAL.
• TPL-EGL is an abstraction layer for surface and buffer management on the Tizen platform aimed to implement the EGL™ porting layer of the OpenGL® ES driver over various display protocols.

For an application to handle input device events, the Input Manager is provided, and is mainly comprised of libinput and a thin wrapper around it. It handles input events in Wayland compositors and communicates with Wayland clients.

Buffer Management

TBM has a frontend library and a backend module. The TBM frontend library is hardware-independent and provides a generic buffer interface. On the other hand, the TBM backend module is hardware-dependent and provides a buffer interface dependent on the target system. Chipset vendors have to provide their own backend modules in order for TBM to work well on the Tizen platform. This is because the way each vendor manages the graphic buffers can be different between various chipset devices. TBM already has several reference backends, such as libtbm-dumb, and libtbm-shm.

Figure: TBM backend

With TBM, the client and server can allocate buffers and share buffers between them. For example, a client allocates a graphic buffer, draws something on it with GL and sends it to the display server for displaying it on the screen without buffer copying. The TBM backend module is implemented as a shared library and the TBM frontend finds the libtbm-default.so file and loads it from the /usr/lib/bufmgr directory at runtime.

sh-3.2# ls -al
lrwxrwxrwx  1 root root    14 Jul 28  2016 libtbm_default.so -> libtbm_sprd.so
lrwxrwxrwx  1 root root    20 Jul 28  2016 libtbm_sprd.so -> libtbm_sprd.so.0.0.0
lrwxrwxrwx  1 root root    20 Jul 28  2016 libtbm_sprd.so.0 -> libtbm_sprd.so.0.0.0
-rwxr-xr-x  1 root root 26728 Jun 29  2016 libtbm_sprd.so.0.0.0

Initialing the TBM Backend Module

The TBMModuleData is for the entry point symbol to initialize the TBM backend module. The TBM backend module must define the global data symbol with the name of tbmModuleData. The TBM frontend loads the tbmModuleData global data symbol and calls the init() function at the initial time.

Note

Do not change the name of the symbol in the TBM backend module.

/*
   @brief tbm module data
   Data type for the entry point of the backend module
*/
typedef struct {
    TBMModuleVersionInfo *vers;	/* TBM module information */
    ModuleInitProc init; /* init function of a backend module */
} TBMModuleData;

typedef int (*ModuleInitProc) (tbm_bufmgr, int);

The TBM backend module initialization consists of allocating the tbm_bufmgr_backend instance (tbm_backend_alloc), entering the necessary information, and the initialization itself (tbm_backend_init).

tbm_bufmgr_backend tbm_backend_alloc(void);
void tbm_backend_free(tbm_bufmgr_backend backend);
int tbm_backend_init(tbm_bufmgr bufmgr, tbm_bufmgr_backend backend);

MODULEINITPPROTO (init_tbm_bufmgr_priv);

static TBMModuleVersionInfo DumbVersRec = {
    "shm",
    "Samsung",
    TBM_ABI_VERSION,
};

TBMModuleData tbmModuleData = {&DumbVersRec, init_tbm_bufmgr_priv};

int
init_tbm_bufmgr_priv(tbm_bufmgr bufmgr, int fd) {
    tbm_bufmgr_backend bufmgr_backend;

    bufmgr_shm = calloc(1, sizeof(struct _tbm_bufmgr_shm));

    bufmgr_backend = tbm_backend_alloc();

    bufmgr_backend->priv = (void *)bufmgr_shm;
    bufmgr_backend->bufmgr_deinit = tbm_shm_bufmgr_deinit,
    bufmgr_backend->bo_size = tbm_shm_bo_size,
    bufmgr_backend->bo_alloc = tbm_shm_bo_alloc,
    bufmgr_backend->bo_free = tbm_shm_bo_free,
    bufmgr_backend->bo_import = tbm_shm_bo_import,
    bufmgr_backend->bo_import_fd = NULL,
    bufmgr_backend->bo_export = tbm_shm_bo_export,
    bufmgr_backend->bo_export_fd = NULL,
    bufmgr_backend->bo_get_handle = tbm_shm_bo_get_handle,
    bufmgr_backend->bo_map = tbm_shm_bo_map,
    bufmgr_backend->bo_unmap = tbm_shm_bo_unmap,
    bufmgr_backend->bo_lock = NULL;
    bufmgr_backend->bo_unlock = NULL;
    bufmgr_backend->surface_get_plane_data = tbm_shm_surface_get_plane_data;
    bufmgr_backend->surface_supported_format = tbm_shm_surface_supported_format;

    if (!tbm_backend_init(bufmgr, bufmgr_backend)) {
        tbm_backend_free(bufmgr_backend);
        free(bufmgr_shm);

        return 0;
    }

    return 1;
}

Porting the OAL Interface

TBM provides the header files to implement the TBM backend module.

Table: TBM backend module header files

TBM Backend Interface

The following table lists the TBM backend interface functions for initializing and deinitializing.

Table: Initializing and deinitializing functions

The following table lists the TBM backend interface functions for tbm_bo.

Table: tbm_bo functions

The following table lists the TBM backend interface functions for tbm_surface.

Table: tbm_surface functions

The following table lists the TBM buffer memory types.

Table: TBM buffer memory types

The following table lists the TBM buffer device types.

Table: TBM buffer device types

The following table lists the TBM buffer access options.

Table: TBM buffer access options

TBM DRM Helper Functions

If the target uses the drm interface, the client needs to get the authenticated fd from the display server and the display server must share the drm master fd with the TDM backend module. The TBM frontend provides the helper functions for drm authentication with the Wayland protocol and shares the master fd with the TDM backend module.

Table: DRM helper functions

TBM Backends

The following table lists the TBM backends.

Table: TBM backends

Reference

For more information about TBM and the TBM backend, see Tizen Buffer Manager (TBM).

Display Management

The display server composites and shows the client’s buffers on screen. The display server sometimes needs to convert or scale an image to a different size or format. To make it possible for various chipset devices, the display server needs the display hardware resource information and control over the resources. Tizen Display Manager (TDM) offers these functionalities for the display server with the unified interface for various chipset devices.

Figure: TDM backend

With TDM, the display server can perform mode setting, DPMS control, and showing a buffer (framebuffer or video buffer) on the screen in the most efficient way. If the hardware supports the m2m converting and capture device, the display server can also convert an image and dump a screen including all hardware overlays with no compositing.

The vendor has to implement the TDM backend module. The TDM backend module has the responsibility to let the TDM frontend know the display hardware resource information. The display server gets this information and controls hardware devices through the TDM frontend APIs. TDM already has several backends for reference, such as libtdm-drm and libtdm-fbdev.

The TDM backend is implemented as a shared library. The TDM frontend finds the libtdm-default.so file and loads it in the /usr/lib/tdm directory at runtime.

sh-3.2# ls -l /usr/lib/tdm
total 40
lrwxrwxrwx 1 root root    14 Jul 28  2016 libtdm-default.so -> libtdm-drm.so
-rwxr-xr-x 1 root root 37152 Jul 12  2016 libtdm-drm.so

The TDM backend module must define the global data symbol with the name tdm_backend_module_data. The TDM frontend reads this symbol at the initialization time. TDM calls the init() function of the tdm_backend_module_data. For more information, see tdm_backend.h.

typedef struct _tdm_backend_module {
    const char *name; /* The module name of the backend module */
    const char *vendor; /* The vendor name of the backend module */
    unsigned long abi_version; /* The ABI version of the backend module */
    tdm_backend_data *(*init)(tdm_display *dpy, tdm_error *error);
    void (*deinit)(tdm_backend_data *bdata);
} tdm_backend_module;

#include <tdm_backend.h>

static tdm_drm_data *drm_data;

tdm_backend_data*
tdm_drm_init(tdm_display *dpy, tdm_error *error) {
    drm_data = calloc(1, sizeof(tdm_drm_data));

    return (tdm_backend_data*)drm_data;
}

void
tdm_drm_deinit(tdm_backend_data *bdata) {
    free(bdata);
}

tdm_backend_module tdm_backend_module_data = {
    "drm",
    "Samsung",
    TDM_BACKEND_SET_ABI_VERSION(1,1),
    tdm_drm_init,
    tdm_drm_deinit
};

The TDM backend must register the tdm_func_display(), tdm_func_output(), and tdm_func_layer() functions with the tdm_backend_register_func_display(), tdm_backend_register_func_output(), and tdm_backend_register_func_layer() functions in the tdm_backend_module_data init() function.

#include <tdm_backend.h>

tdm_backend_data*
tdm_drm_init(tdm_display *dpy, tdm_error *error) {
    memset(&drm_func_display, 0, sizeof(drm_func_display));
    drm_func_display.display_get_capability = drm_display_get_capability;
    drm_func_display.display_get_pp_capability = drm_display_get_pp_capability;
    drm_func_display.display_get_outputs = drm_display_get_outputs;
    drm_func_display.display_get_fd = drm_display_get_fd;
    drm_func_display.display_handle_events = drm_display_handle_events;
    drm_func_display.display_create_pp = drm_display_create_pp;
    ret = tdm_backend_register_func_display(dpy, &drm_func_display);
    if (ret != TDM_ERROR_NONE)
        goto failed;

    memset(&drm_func_output, 0, sizeof(drm_func_output));
    drm_func_output.output_get_capability = drm_output_get_capability;

    ret = tdm_backend_register_func_output(dpy, &drm_func_output);
    if (ret != TDM_ERROR_NONE)
        goto failed;

    memset(&drm_func_layer, 0, sizeof(drm_func_layer));
    drm_func_layer.layer_get_capability = drm_layer_get_capability;

    ret = tdm_backend_register_func_layer(dpy, &drm_func_layer);
    if (ret != TDM_ERROR_NONE)
        goto failed;

    return (tdm_backend_data*)drm_data;
}

After loading the TDM backend module, the TDM frontend calls the display_get_capability(), display_get_outputs(), output_get_capability(), output_get_layers(), and layer_get_capability() functions to get the hardware-specific information. That means that the TDM backend module must implement these 5 functions.

In addition, if a target has a memory-to-memory converting hardware device and the capture hardware device, the TDM backend module can register the tdm_func_pp() and tdm_func_capture() functions with the tdm_backend_register_func_pp() and tdm_backend_register_func_capture() functions.

Porting the OAL Interface

TDM provides the header files to implement the TDM backend module.

Table: TDM backend module header files

The display backend interface is mandatory. For more information, see tdm_backend.h.

Table: Display backend interface functions

The output backend interface is mandatory. For more information, see tdm_backend.h.

Table: Output backend interface functions

The layer backend interface is mandatory. For more information, see tdm_backend.h.

Table: Layer backend interface functions

The pp backend interface is optional. For more information, see tdm_backend.h.

Table: pp backend interface functions

The capture backend interface is optional. For more information, see tdm_backend.h.

Table: Capture backend interface functions

TDM backends

There are several backends which can be used as reference when implementing the TDM backend.

Table: TDM backends

Testing the Porting Result

TDM offers the tdm-test-server tool to allow you to easily test the porting result. The tdm-test-server tool is included in the libtdm-tools package, which can be downloaded from the platform binary’s snapshot repository. Make sure that TBM porting is done before using the following commands, because TDM works on top of TBM.

$ systemctl stop display-manager  (stop the display server)
$ export XDG_RUNTIME_DIR=/run
$ export TBM_DISPLAY_SERVER=1
$ tdm-test-server                 (show all options)
$ tdm-test-server -a              (test all layers)
$ tdm-test-server -a -v           (test all layers with vblank events)

The following image shows the result of a test performed using the tdm-test-server -a command. The fullscreen buffer is set to the PRIMARY layer, and the small buffer is set to the OVERLAY layer.

Figure: Tdm-test-server results

Checking TDM Log Messages

TDM uses dlog to print debug messages. To show TDM runtime log messages:

$ dlogutil -v threadtime TDM

References

For detailed information about TDM and the TDM backend, see Tizen Display Manager (TDM).

Input Management

The input manager supports a libinput-based input device backend. libinput is a common input library for the Wayland compositor. With libinput, the input stack is simpler without the Xorg input drivers. Since Tizen 3.0, the input manager is not a HAL component.

Figure: Tizen 3.0 input management

libinput

The libinput library handles input devices for display servers and other applications that need to directly deal with input devices:

• Device detection
• Device handling
• Input device event processing
• Scaling touch coordinates
• Generating pointer events from touchpads
• Pointer acceleration

For more information, see the libinput wiki.

libevdev

The libevdev library handles evdev kernel devices. It abstracts the evdev ioctls through type-safe interfaces and provides functions to change the appearance of the device. For more information, see https://en.wikipedia.org/wiki/Evdev.

mtdev

The mtdev standalone library transforms all variants of kernel MT events to the slotted type B protocol. For more information, see http://www.linuxfromscratch.org/blfs/view/svn/general/mtdev.html.

libinput Backends

libinput: platform/upstream/libinput

OpenGL®

This section describes the essential elements of the Tizen platform-level graphics architecture related to OpenGL® ES and EGL™, and how it is used by the application framework and the display server. The focus is on how graphical data buffers move through the system.

The Tizen platform requires the OpenGL® ES driver for the acceleration of the Wayland display server and the wayland-egl client. This platform demands an OpenGL® ES and EGL™ driver which is implemented by the Tizen EGL Porting Layer.

Tizen OpenGL® ES and EGL™ Architecture

The following figure illustrates the Tizen OpenGL® ES and EGL™ architecture.

Figure: Tizen OpenGL® ES architecture

• CoreGL

  An injection layer of OpenGL® ES that provides the following capabilities:

  • Support for driver-independent optimization (FastPath)
  • EGL™/OpenGL® ES debugging
  • Performance logging
  • Supported versions
    • EGL™ 1.4
    • OpenGL® ES 1.1, 2.0, 3.0, 3.1

  CoreGL loads the manufacturer’s OpenGL® ES driver from the /usr/lib/driver directory. CoreGL provides the libEGL.so, libGLESv1_CM.so, and libGLESvs.so driver files in the /usr/lib directory.

• GPU vendor GL/EGL™ driver

  The Tizen platform demands that the GPU vendor implements the GL and EGL™ driver using libtpl-egl. The GPU vendor GL/EGL™ driver (libEGL.so, libGLESv1_CM.so, libGLESv2.so) must be installed in the /usr/lib/driver path.

Tizen Porting Layer (TPL) for EGL™

TPL-EGL is an abstraction layer for surface and buffer management on the Tizen platform. It is used for implementation of the EGL™ platform functions.

Figure: TPL architecture

The background for the Tizen EGL Porting Layer for EGL™ is in various window system protocols in Tizen. There was a need for separating common layer and backend.

Tizen uses the Tizen Porting Layer for EGL™, as the TPL-EGL API prevents burdens of the EGL™ porting on various window system protocols. The GPU GL Driver’s Window System Porting Layer can be implemented by TPL-EGL APIs which are the corresponding window system APIs. The TBM, Wayland, and GBM backends are supported.

Tizen Porting Layer for the EGL™ Object Model

TPL-EGL provides interfaces based on an object-driven model. Every TPL-EGL object can be represented as a generic tpl_object_t, which is reference-counted and provides common functions. Currently, display and surface types of TPL-EGL objects are provided. A display, like a normal display, represents a display system which is usually used for connecting to the server. A surface corresponds to a native surface, such as wl_surface. Surfaces can be configured to use N-buffers, but are usually double-buffered or triple-buffered. A buffer is what you render on, usually a set of pixels or a block of memory. For these 2 objects, the Wayland, GBM, TBM backend are defined, and they correspond to their own window systems. This means that you do not need to care about the window systems.

The TPL-EGL has the following core objects:

• TPL-EGL Object

  Base class for all TPL-EGL objects.

• TPL-EGL Display

  Encapsulates the native display object (Display *, wl_display). Like a normal display, this represents a display system which is usually used for connecting to the server, scope for other objects.

• TPL-EGL Surface

  Encapsulates the native drawable object (Window, Pixmap, wl_surface). The surface corresponds to a native surface, such as tbm_surface_queue or wl_surface. A surface can be configured to use N-buffers, but they are usually double-buffered or triple-buffered.

TPL-EGL Objects and Corresponding EGL™ Objects

Both TPL-EGL and vendor OpenGL® ES/EGL™ driver handles a tbm_surface as the corresponding TPL surface buffer. It is represented by the TBM_Surface part in the following figure.

Figure: TPL-EGL architecture

The following figure illustrates the OpenGL® ES drawing API flow.

Figure: OpenGL® ES drawing API flow

TPL-EGL Frontend API

TPL-EGL Object is a base class for all TPL-EGL objects. It provides common functionalities to all TPL-EGL objects.

Table: TPL-EGL Object functions

TPL-EGL Display encapsulates the native display object (Display *, wl_display). Any other objects created from TPL-EGL Display inherit its backend type.

Table: TPL-EGL Display functions

TPL-EGL Surface encapsulates the native drawable object (Window, Pixmap, wl_surface). The main features of the class are retrieving the buffer for a frame and posting the surface to a screen.

Table: TPL-EGL Surface functions

The following code snippet shows a simple example of the Tizen Porting Layer:

dpy = tpl_display_create(...);
sfc = tpl_surface_create(dpy, ...);

while (1) {
    buf = tpl_surface_dequeue_buffer(sfc);

    /* Draw something */

    tpl_surface_enqueue_buffer(sfc, buf);
}

In the GPU vendor driver, the GPU frame builder handles the drawing. TPL-EGL exposes the native platform buffer identifiers and managers so that the buffer can be used in other modules. Currently, dma_buf/DRM is supported for these purposes. The EGL porting layer calls TPL-EGL functions to execute commands requested of it, and returns the results to the GPU vendor driver. TPL-EGL performs all protocol-dependent actions. Such protocol-dependent parts can be separated into TPL-EGL backends. TPL-EGL backend can also be configured at runtime, and you can specify which type of backend to use when initializing a display object.

TPL-EGL and Wayland Server and Client

Tizen uses the wl_tbm protocol instead of wl_drm. The wl_tbm protocol is designed for sharing the buffer (tbm_surface) between the wayland_client and wayland_server. Although the wayland_tbm_server_init and wayland_tbm_client_init pair is a role for the eglBindWaylandDisplayWL, the EGL™ driver is required to implement the entry points for the eglBindWaylandDisplayWL and eglUnbindWaylandDisplayWL as dummy. For more information, see https://cgit.freedesktop.org/mesa/mesa/tree/docs/specs/WL_bind_wayland_display.spec.

Figure: TPL-EGL and Wayland

Buffer Flow Between the Wayland Server and OpenGL® ES/EGL™ Driver

The following figure shows the buffer flow between the Wayland server and the OpenGL® ES/EGL™ driver. The passed buffer is of the tbm_surface type.

Figure: Buffer flow between Wayland server and OpenGL® ES/EGL™ driver

Project Git Repository

The following table lists the available project Git repositories.

Table: Git repositories

libtpl-egl Reference Driver

The Emulator YAGL (OpenGL® ES / EGL™ driver for the emulator) is implemented by libtpl-egl.

The following commit explains how to port the driver with libtpl-egl from the traditional drm-based driver:

• Porting YAGL to the Tizen platform https://review.tizen.org/gerrit/#/c/67921/

Testing and Verifying the OpenGL® ES Driver

The Khronos OpenGL® ES CTS supports wayland-egl. libtpl-egl has a test case for the libtpl-egl. The ws-testcase’s tpl-novice has sample code for the libtpl-egl.