Shape the window when there is no compositor

On Xfce with compositing off, X11 has no alpha to blend against, so the
transparent background came out as a black box around the cat.

When a window comes up without an alpha channel, TNCat now cuts the window down
to the pixels the drawing covers, using the X11 shape extension, and refreshes
that 15 times a second so it follows the animation. A sitting cat in a 120 x 106
window comes out as 96 rectangles over 29 percent of the window, so most of the
box is gone. Edges are harder than with a compositor, one bit per pixel, and the
log says so and names the Xfce setting that gives smooth ones instead.

The image-to-rectangles step is platform independent and folds every device row
into its window row, which both halves the rectangle count on a scaled display
and keeps a pixel that is opaque in only one of them.

xcb and xcb-shape are picked up through pkg-config on Unix and the build still
works without them. Opaque mode now paints a neutral card rather than the coat
colour.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CMakeLists.txt

@@ -17,6 +17,7 @@ qt_add_executable(tncat
     src/catcontroller.h src/catcontroller.cpp
     src/appsettings.h src/appsettings.cpp
     src/platforminfo.h src/platforminfo.cpp
+    src/windowshape.h src/windowshape.cpp
 )
 
 target_include_directories(tncat PRIVATE src)
@@ -38,6 +39,24 @@ qt_add_qml_module(tncat
 target_link_libraries(tncat PRIVATE
     Qt6::Core Qt6::Gui Qt6::Quick Qt6::QuickShapes)
 
+# Without a compositor, X11 renders a transparent background as black. The
+# shape extension is the way out: it cuts the window down to the cat itself.
+# Both libraries ship with X11 and Qt's own xcb plugin already uses them.
+if(UNIX AND NOT APPLE)
+    find_package(PkgConfig QUIET)
+    if(PkgConfig_FOUND)
+        pkg_check_modules(TNCAT_XCB IMPORTED_TARGET xcb xcb-shape)
+    endif()
+    if(TARGET PkgConfig::TNCAT_XCB)
+        target_link_libraries(tncat PRIVATE PkgConfig::TNCAT_XCB)
+        target_compile_definitions(tncat PRIVATE TNCAT_HAVE_XCB_SHAPE)
+    else()
+        message(STATUS "TNCat: xcb-shape not found. Without a compositor the cat "
+                       "will sit in a black box; install libxcb-shape0-dev or "
+                       "xcb-util-dev and reconfigure.")
+    endif()
+endif()
+
 set_target_properties(tncat PROPERTIES
     MACOSX_BUNDLE TRUE
     MACOSX_BUNDLE_INFO_PLIST ${CMAKE_SOURCE_DIR}/Info.plist.in

README.md

@@ -102,9 +102,21 @@ This is the main target. Two things to know.
 **Transparency needs a compositor.** Without one, X11 has no alpha channel to
 composite against and the transparent window background renders as a black box
 around the cat. In Xfce it is one checkbox: Settings, Window Manager Tweaks,
-Compositor, Enable display compositing. TNCat prints a warning naming that
+Compositor, Enable display compositing. That is the best answer, because alpha
-setting when its window comes up without an alpha channel. `TNCAT_OPAQUE=1`
+gives smooth edges.
-paints a solid background instead, which is usable but not pretty.
+
+When TNCat sees a window come up without an alpha channel it says so and falls
+back to shaping the window with the X11 shape extension: the window is cut down
+to the pixels the cat covers, so there is no box left to be black. The shape is
+one bit per pixel, so edges are harder than with a compositor, and it is
+refreshed 15 times a second to follow the animation. Measured on a sitting cat
+in a 120 x 106 window, the shape is 96 rectangles covering 29 percent of the
+window.
+
+Shaping needs `xcb` and `xcb-shape` at build time (`libxcb-shape0-dev` on
+Debian and Ubuntu, `xcb-util-devel` on SUSE). CMake says so if they are
+missing, and the build still works without them. `TNCAT_OPAQUE=1` skips the
+whole question and paints a solid background.
 
 **Keeping the cat on top.** The default is a frameless `Qt::Tool` window that
 stays on top. If the window manager puts it behind other windows or shows it in

qml/CatWindow.qml

@@ -17,7 +17,9 @@ Window {
     height: Math.round(width * sprite.implicitHeight / sprite.implicitWidth)
     x: cat.x
     y: Screen.desktopAvailableHeight - height - settings.bottomMargin
-    color: Platform.opaqueBackground ? settings.coat : "transparent"
+    // A neutral card reads better than the coat colour when the background
+    // cannot be transparent at all.
+    color: Platform.opaqueBackground ? "#f2f0ec" : "transparent"
     flags: Qt.FramelessWindowHint | Qt.WindowStaysOnTopHint | Qt.Tool
            | Qt.NoDropShadowWindowHint | Qt.WindowDoesNotAcceptFocus
            | (Platform.bypassWindowManager ? Qt.X11BypassWindowManagerHint : 0)
@@ -48,6 +50,17 @@ Window {
     }
     Screen.onDesktopAvailableWidthChanged: cat.stageWidth = Screen.desktopAvailableWidth
 
+    // Without a compositor the window is cut to the cat's own outline. The
+    // drawing animates, so the shape has to be refreshed; 15 Hz is enough for
+    // a swinging leg and cheap enough for a window this small.
+    Timer {
+        interval: 66
+        repeat: true
+        running: Platform.shapeWindows && win.visible
+        triggeredOnStart: true
+        onTriggered: Platform.updateWindowShape(win)
+    }
+
     // The eyes need to know where the face sits on the desktop.
     Binding {
         target: win.cat

qml/ChipWindow.qml

@@ -27,6 +27,14 @@ Window {
         NumberAnimation { duration: 320; easing.type: Easing.OutQuad }
     }
 
+    Timer {
+        interval: 66
+        repeat: true
+        running: Platform.shapeWindows && win.visible
+        triggeredOnStart: true
+        onTriggered: Platform.updateWindowShape(win)
+    }
+
     Chip {
         id: curl
         anchors.fill: parent

qml/CupWindow.qml

@@ -22,6 +22,14 @@ Window {
            | (Platform.bypassWindowManager ? Qt.X11BypassWindowManagerHint : 0)
     visible: cat.cupVisible
 
+    Timer {
+        interval: 66
+        repeat: true
+        running: Platform.shapeWindows && win.visible
+        triggeredOnStart: true
+        onTriggered: Platform.updateWindowShape(win)
+    }
+
     Cup {
         id: mug
         anchors.fill: parent

src/platforminfo.cpp

@@ -1,5 +1,7 @@
 #include "platforminfo.h"
 
+#include "windowshape.h"
+
 #include <QGuiApplication>
 #include <QRegion>
 
@@ -18,14 +20,34 @@ void PlatformInfo::checkTranslucency(QQuickWindow *window)
     if (!window || m_opaque || m_warnedTranslucency)
         return;
     if (window->format().alphaBufferSize() > 0)
-        return;
+        return; // a compositor is doing the blending, nothing to fix
 
     m_warnedTranslucency = true;
+
+    if (WindowShape::supported()) {
+        m_shapeWindows = true;
+        emit shapeWindowsChanged();
+        qInfo("TNCat: no alpha channel on this window, which means no compositor "
+              "is running. Falling back to a shaped window. Edges will be a "
+              "little hard, because a shape is one bit per pixel. Turning the "
+              "compositor on gives smooth edges: in Xfce, Settings, Window "
+              "Manager Tweaks, Compositor, Enable display compositing.");
+        return;
+    }
+
     qWarning("TNCat: this window has no alpha channel, so the transparent "
-             "background will render as a black box. On X11 that means no "
+             "background renders as a black box, and window shaping is not "
-             "compositor is running. In Xfce turn on Settings, Window Manager "
+             "available either. On X11 that means no compositor is running. In "
-             "Tweaks, Compositor, Enable display compositing. Or start TNCat "
+             "Xfce turn on Settings, Window Manager Tweaks, Compositor, Enable "
-             "with TNCAT_OPAQUE=1 to paint a solid background instead.");
+             "display compositing. Or start TNCat with TNCAT_OPAQUE=1 to paint "
+             "a solid background instead.");
+}
+
+void PlatformInfo::updateWindowShape(QQuickWindow *window)
+{
+    if (!m_shapeWindows || !window || !window->isVisible())
+        return;
+    WindowShape::applyFromImage(window, window->grabWindow());
 }
 
 void PlatformInfo::setInputRegion(QQuickWindow *window, const QRectF &region)

src/platforminfo.h

@@ -21,6 +21,7 @@ class PlatformInfo : public QObject
     Q_PROPERTY(bool bypassWindowManager READ bypassWindowManager CONSTANT)
     Q_PROPERTY(bool opaqueBackground READ opaqueBackground CONSTANT)
     Q_PROPERTY(bool debug READ debug CONSTANT)
+    Q_PROPERTY(bool shapeWindows READ shapeWindows NOTIFY shapeWindowsChanged)
 
 public:
     explicit PlatformInfo(QObject *parent = nullptr);
@@ -46,10 +47,23 @@ public:
     // window that a window manager or a bad binding collapsed.
     bool debug() const { return m_debug; }
 
-    // Warns once if the window came up without an alpha channel, which is what
+    // True once a window has been seen without an alpha channel and the X11
-    // a missing compositor looks like: transparency renders as black.
+    // shape extension is there to make up for it. QML then keeps the window
+    // shaped to the cat instead of leaving a black box around it.
+    bool shapeWindows() const { return m_shapeWindows; }
+
+    // Looks at the window's format. Turns shaping on, or warns if there is no
+    // way to avoid the black box.
     Q_INVOKABLE void checkTranslucency(QQuickWindow *window);
 
+    // Grabs the window and cuts it down to the pixels the drawing covers.
+    Q_INVOKABLE void updateWindowShape(QQuickWindow *window);
+
+signals:
+    void shapeWindowsChanged();
+
+public:
+
     // Restricts the window's input region to one rectangle so clicks land on
     // whatever is behind the rest of the overlay. Implemented by the X11 and
     // Wayland plugins; a no-op on platforms that ignore the mask.
@@ -62,4 +76,5 @@ private:
     bool m_opaque = false;
     bool m_debug = false;
     bool m_warnedTranslucency = false;
+    bool m_shapeWindows = false;
 };

src/windowshape.cpp

@@ -0,0 +1,143 @@
+#include "windowshape.h"
+
+#include <QQuickWindow>
+#include <QtMath>
+
+#ifdef TNCAT_HAVE_XCB_SHAPE
+#include <QGuiApplication>
+#include <QtGui/qguiapplication_platform.h>
+#include <xcb/shape.h>
+#include <xcb/xcb.h>
+#endif
+
+namespace {
+
+// Anything this opaque counts as cat. Low on purpose: it keeps the soft edge
+// of an antialiased outline inside the shape instead of biting into it.
+constexpr int kAlphaThreshold = 24;
+
+} // namespace
+
+namespace WindowShape {
+
+// One rectangle per run of opaque pixels in a row. Neighbouring rows are not
+// merged: an X server copes fine with a few hundred rectangles at this size,
+// and merging them would cost more than it saves.
+QList<QRect> opaqueRuns(const QImage &image, qreal scale)
+{
+    QList<QRect> rects;
+    if (image.isNull())
+        return rects;
+    if (scale <= 0)
+        scale = 1.0;
+
+    const QImage rgba = image.format() == QImage::Format_ARGB32
+            ? image : image.convertToFormat(QImage::Format_ARGB32);
+
+    // The grab is in device pixels, the shape has to be in window coordinates.
+    // Every device row that falls into a window row is folded into it, so a
+    // pixel that is opaque in either of them keeps its place in the shape.
+    const int cols = qMax(1, qCeil(rgba.width() / scale));
+    const int rows = qMax(1, qCeil(rgba.height() / scale));
+    QList<bool> row(cols);
+
+    for (int ly = 0; ly < rows; ++ly) {
+        row.fill(false);
+        const int first = qFloor(ly * scale);
+        const int last = qMin(rgba.height(), qCeil((ly + 1) * scale));
+        for (int y = first; y < last; ++y) {
+            const QRgb *line = reinterpret_cast<const QRgb *>(rgba.constScanLine(y));
+            for (int x = 0; x < rgba.width(); ++x) {
+                if (qAlpha(line[x]) >= kAlphaThreshold)
+                    row[qMin(cols - 1, int(x / scale))] = true;
+            }
+        }
+
+        int runStart = -1;
+        for (int lx = 0; lx <= cols; ++lx) {
+            const bool opaque = lx < cols && row[lx];
+            if (opaque && runStart < 0)
+                runStart = lx;
+            else if (!opaque && runStart >= 0) {
+                rects.append(QRect(runStart, ly, lx - runStart, 1));
+                runStart = -1;
+            }
+        }
+    }
+    return rects;
+}
+
+#ifdef TNCAT_HAVE_XCB_SHAPE
+
+namespace {
+
+xcb_connection_t *connection()
+{
+    const auto *x11 = qGuiApp->nativeInterface<QNativeInterface::QX11Application>();
+    return x11 ? x11->connection() : nullptr;
+}
+
+bool haveShapeExtension(xcb_connection_t *conn)
+{
+    static int cached = -1;
+    if (cached >= 0)
+        return cached == 1;
+
+    const xcb_query_extension_reply_t *ext = xcb_get_extension_data(conn, &xcb_shape_id);
+    cached = (ext && ext->present) ? 1 : 0;
+    return cached == 1;
+}
+
+} // namespace
+
+bool supported()
+{
+    xcb_connection_t *conn = connection();
+    return conn && haveShapeExtension(conn);
+}
+
+bool applyFromImage(QQuickWindow *window, const QImage &image)
+{
+    xcb_connection_t *conn = connection();
+    if (!window || !conn || !haveShapeExtension(conn))
+        return false;
+
+    const QList<QRect> runs = opaqueRuns(image, image.devicePixelRatio());
+    if (runs.isEmpty())
+        return false;
+
+    QList<xcb_rectangle_t> rects;
+    rects.reserve(runs.size());
+    for (const QRect &r : runs) {
+        rects.append({ static_cast<int16_t>(r.x()), static_cast<int16_t>(r.y()),
+                       static_cast<uint16_t>(r.width()), static_cast<uint16_t>(r.height()) });
+    }
+
+    xcb_shape_rectangles(conn, XCB_SHAPE_SO_SET, XCB_SHAPE_SK_BOUNDING,
+                         XCB_CLIP_ORDERING_UNSORTED,
+                         static_cast<xcb_window_t>(window->winId()), 0, 0,
+                         rects.size(), rects.constData());
+    xcb_flush(conn);
+    return true;
+}
+
+void clear(QQuickWindow *window)
+{
+    xcb_connection_t *conn = connection();
+    if (!window || !conn || !haveShapeExtension(conn))
+        return;
+
+    xcb_shape_mask(conn, XCB_SHAPE_SO_SET, XCB_SHAPE_SK_BOUNDING,
+                   static_cast<xcb_window_t>(window->winId()), 0, 0, XCB_PIXMAP_NONE);
+    xcb_flush(conn);
+}
+
+#else // no xcb-shape at build time
+
+bool supported() { return false; }
+bool applyFromImage(QQuickWindow *, const QImage &) { return false; }
+void clear(QQuickWindow *) {}
+
+#endif
+
+} // namespace WindowShape

src/windowshape.h

@@ -0,0 +1,33 @@
+#pragma once
+
+#include <QImage>
+#include <QList>
+#include <QRect>
+
+class QQuickWindow;
+
+// Cuts the window down to the pixels the cat actually covers, using the X11
+// shape extension. Without a compositor X11 has no alpha to blend against and
+// a transparent background comes out black, so the only way to lose the box
+// around the cat is to make the window itself that shape.
+//
+// Everywhere else this is a no-op: with a compositor, plain alpha is both
+// cheaper and smoother, since a shape is one bit per pixel and cannot
+// antialias an edge.
+namespace WindowShape {
+
+// Runs of opaque pixels, one rectangle per run, in window coordinates.
+// Platform independent so it can be tested without an X server.
+QList<QRect> opaqueRuns(const QImage &image, qreal scale);
+
+// True when the platform can shape windows and the build has the pieces.
+bool supported();
+
+// Shapes the window to the opaque pixels of the image, which is expected to be
+// a grab of that same window. Returns false if nothing was applied.
+bool applyFromImage(QQuickWindow *window, const QImage &image);
+
+// Drops the shape again, leaving a plain rectangular window.
+void clear(QQuickWindow *window);
+
+} // namespace WindowShape