Introduction to drone video image stitching using Carmenta Engine 5.17

In dynamic environments or areas without orthophotos, it’s often helpful to use a drone to capture up-to-date imagery. A drone can record video footage that a Carmenta Engine based application can project onto a map in real time or review later.

This article introduces the new MosaicWriter functionality in Carmenta Engine 5.17. This allows developers to stitch together georeferenced video frames into a mosaic, represented as a GeoPackage image tile pyramid, for real-time or post-mission mapping.

By using a CameraProjectOperator as input, the MosaicWriter receives georeferenced video frames and stitches them into a mosaic GeoPackage that can then be read using a MapPackageDataset. Creating such GeoPackage allows for easy distribution and reusability of the data.

Technical Overview

The writing process is implemented entirely in code. The reading part can be handled either in code or in a Carmenta Engine configuration file. However, implementing both parts in code is often simpler when working with the MosaicWriter.

The example code shown here is written in C++ using Qt 6.7.

Note: MosaicWriter is not an operator and cannot be used directly in a Carmenta Engine configuration file using Carmenta Studio.

Creating the Writer

The MosaicWriter constructor takes three arguments:

• The path to the GeoPackage file to write.
• The name of the GeoPackage table that will store the mosaic.
• The operator that provides the projected video frames, typically a CameraProjectOperator.

In this example, the input is an existing CameraProjectOperator that receives frames from a CustomDataSet through a ReadOperator:

MosaicWriterPtr writer = new MosaicWriter("package.gpkg", "mosaic", cameraProjectOperator);

This line creates a new MosaicWriter instance. You can then call its writeFeatures method to write projected frame features into the GeoPackage.

Before writing, you can configure several MosaicWriter properties to control the behavior of the mosaic generation process. Here, we will focus on the strategy property.

Caution: You can modify properties only if the specified table does not already exist. This is indicated by the initialized property being False.

Setting the Strategy

The strategy property defines the MosaicStrategy used when building the mosaic.

You can choose between:

• MosaicStrategyLatestView: Always draws the frame, regardless of existing content. It is the default.
• MosaicStrategyBestView: Only writes pixels that provide a more vertical view than what’s already stored.

writer->strategy(MosaicStrategyLatestView);

Caution: When using MosaicStrategyBestView, and if the input frames come from a CameraProjectOperator, set CameraProjectOperator::keepAttributes to True.

Writing the mosaic

After setting the strategy, you can call writeFeatures. This method accepts an AttributeSetPtr called updateAttributes.

This AttributeSet is passed through the operator chain and can affect how a CustomDataSet behaves. For instance, a CustomDataSet may check for an attribute like provideAll: true and return all video frames in one query.

In this example, we pass an empty AttributeSet:

writer->writeFeatures(new AttributeSet());

At this point, package.gpkg is created but initially empty. If the operator chain returns only the current frame while the video plays, you need to call writeFeatures continuously to update the mosaic.

In this example, using Qt, we run the write loop with QtConcurrent:

void MosaicPainter::startPainting()
{
    m_future = QtConcurrent::run([this]() {
        if (!m_writer)
            return;

        while (!m_future.isCanceled())
        {
            m_writer->writeFeatures(new AttributeSet());
            QThread::sleep(std::chrono::milliseconds{ 33 });
        }
    });

    m_watcher.setFuture(m_future);
}

Creating the Reader

To read the GeoPackage and display it on a 2D View, we will add a new operator chain to the current view. This chain uses a ReadOperator with a MapPackageDataset.

To ensure the application picks up newly written tiles, set the monitorInterval property on the MapPackageDataSet. This property defines how often (in seconds) the dataset checks for updates in the GeoPackage. You’ll also need to connect a mosaicChangedHandler method to the dataset so that when changes are detected, we can refresh the view.

MapPackageDataSetPtr mosaicDataSet = new MapPackageDataSet(gpkgPath, "package.gpkg", "mosaic");
mosaicDataSet->monitorInterval(1);
mosaicDataSet->addDataSetChangedHandler(mosaicChangedHandler,this);

In the 2D View configuration file, make sure there is a TileLayer or LayerSet where you can attach the new operator chain that reads and displays the mosaic.

Next, assign the mosaicDataSet to a ReadOperator, then add it to an OrdinaryLayer. Finally, dock this layer to the LayerSet defined in your configuration.

Important: Since we are modifying the configuration when in use, we need a Guard to lock it during the update.

ReadOperatorPtr reader = new ReadOperator(mosaicDataSet);
OrdinaryLayerPtr layer = new OrdinaryLayer(reader);

{
    Guard g;
    m_mosaicLayer->layers()->add(layer);
}

Conclusion

The introduction of the MosaicWriter in Carmenta Engine 5.17 provides an efficient and flexible approach to creating dynamic mosaics from georeferenced video frames.

Properties such as overlapLimit, tileSize and strategy, offer precise customization options.

More information can be found in the MosaicWriter documentation.