Credit Andrea - correct HDR references

[goodguy/cin-manual-latex.git] / parts / Developer.tex

\chapter{Developer's Section}%
\label{cha:developer's_section}

\section{How Builds Really Work and Even More Options}
\label{sec:builds_really_work_more_options}
\index{build!more options}

This section describes how builds really work if you want to know more about making changes or understanding the process; probably only for a developer or system administrator.

Builds occur in 4 basic steps:

\begin{enumerate}[nosep]
        \item unpack/patch source code
        \item configure build
        \item make build targets
        \item installation
\end{enumerate}

So, an example of what happens in 4 steps for a single-user build would be as follows:

\begin{enumerate}[nosep]
        \item unpack/patch source code: \\
        \texttt{git clone -{}-depth 1 ``git:/{\dots}/target'' cinelerra5} \\
        \texttt{./autogen.sh}
        \item configure build:\\
        \texttt{./configure --with-single-user}
        \item make build targets:\\
        \texttt{make 2 > \&1 | tee log}
        \item installation:\\
        \texttt{make install}
\end{enumerate}

A lot of things can be tweaked to change the results. Mostly these changes are parameters to the configure step, which can change important build related items, like the application name, or where and what the target system directories should be. This makes it possible to have several versions at the same time on the same computer if needed. To see what it is that the makefiles use to build \CGG{}, look at the resulting top-level global\_config file which is created by the ./configure step.

Building \CGG{} requires many thirdparty libraries, and it is recommended that you use the static build version included in the git repo. Some of them are patched, and fix significant bugs. It is important to note that because system installation historically has been with as many shared objects as possible, the defaults are that any system library detected during configuration setup will be used, when the package is built \textit{-{}-without-single-user}, which is the default build. To build with static thirdparty libraries for system install to the system /usr area, use:

\hspace{2em}\texttt{.configure -{}-enable-static-build --prefix=/usr}

Sometimes, additional package parameters and variables are needed during thirdparty builds. These optional values occur before and after the \textit{configure} and \textit{make} commands during a build. A presentation of the format of the package qualified variable names and how they appear in the build procedure are:

\hspace{2em}
\begin{tabular}{@{}ll}
        pkg.cfg\_vars & prepended to configure\\
        pkg.cfg\_params & appended to configure\\
        pkg.mak\_vars & prepended to make\\
        pkg.mak\_params & appended to make\\
    pkg.cflags & added as CFLAGS+=\$(cflags) to pkg.vars\\
        pkg.cppflags & added as CPPFLAGS+=\$(cppflags) to pkg.vars\\
\end{tabular}

These steps are done for EACH of the packages in the thirdparty build:

\hspace{2em}\texttt{<pkg.cfg\_vars> ./configure <pkg.cfg\_params>}

\hspace{2em}\texttt{<pkg.mak\_vars> make <pkg.mak\_params>}

The thirdparty Makefile has a set of default vars and params used to build each of the needed thirdparty packages, but you can specify new or overriding values for these Makefile substitutions. These thirdparty build config changes are specified in the top-level file: \textit{cin\_config}. By using this file, you can save the configuration changes made for the current build to use the next time you do a new build. For example, to add an include file path to the giflib build, add this line to \textit{cin\_config}:

\hspace{2em}\texttt{giflib.cflags := -I/usr/local/include/giflib5}

To have a param/var change apply to all thirdparty builds, use:

\hspace{2em}\texttt{CFG\_VARS, CFG\_PARAMS, MAK\_VARS, MAK\_PARAMS}

CFLAGS, CXXFLAGS and LDFLAGS are forwarded to the thirdparty build environment via:

\hspace{2em}\texttt{CFLAGS=-ggdb ./configure -{}-with-single-user}

However, there is no guarantee that the thirdparty build will honor the environmental flags.

Finally, there are build controls, which enable/disable and set build features.

A few of the more useful ./configure -{}-parameters are:

\begin{tabular}{ll}
        -{}-with-jobs=n & where n=number of build jobs; defaults to 1.5*cpus+2\\
        -{}-enable-static-build & build all 3rd party libs; defaults to yes if single-user, else no\\
        -{}-with-single-user& build installs to <build\_path>/bin; no system installation\\
\end{tabular}


The ./configure command builds \textit{global\_config}. The global\_config is read by the thirdparty/Makefile to create the recipes and definitions used by the thirdparty build.

There are a lot of different options.  Thirdparty library build control is available in the configure step of the build.  Thirdparty libraries are built on a demand basis.  So if you use:

\begin{tabular}{l p{11cm}}
  -{}-enable-libname=auto & the static-build enable, or the lack of a system library causes a build\\
  -{}-enable-libname=yes  &  this forces the thirdparty build\\
  -{}-enable-libname=no   &  this forces no thirdparty build\\
\end{tabular}

FFmpeg is a \textit{strongly connected} component in the build linkage and widely influences the \CGG{} library demands.  It is possible to make small additions to the ffmpeg configuration step using the environment variable \texttt{FFMPEG\_EXTRA\_CFG}.  For example, to eliminate the use of libvdpau (an nvidia support library) in the ffmpeg configuration step after you have determined that it is causing an error, use:

\begin{itemize}[label={},nosep]
        \item \texttt{make clean}
        \item \texttt{autogen.sh}
        \item \texttt{export FFMPEG\_EXTRA\_CFG=" -{}-disable-vdpau"} 
        \item \texttt{./configure} {\dots}
\end{itemize}

\section{Experimental Builds}
\label{sec:experimental_builds}
\index{build!experimental}

The main compilation we have seen leads to building \CGG{} with its own internal ffmpeg, which includes its stability and feature patches. Normally ffmpeg is updated to version x.1 of each release. The reasons why it is best to use thirdparty with their static versions of the libraries and ffmpeg are explained in detail in \nameref{sec:latest_libraries} and in \nameref{sub:unbundled_builds}.

You can still compile \CGG{} with ffmpeg-git to enjoy the latest version. This build may lead to feature variation and less stability, but in most cases will work perfectly fine.
You have to supply the actual URL location of the ffmpeg git as you can see in this example \texttt{bld.sh} script:

\begin{lstlisting}[numbers=none]
        #!/bin/bash
        ./autogen.sh
        ./configure --with-single-user --with-booby --with-git-ffmpeg=https://git.ffmpeg.org/ffmpeg.git
        make && make install ) 2>1 | tee log
        mv Makefile Makefile.cfg
        cp Makefile.devel Makefile
\end{lstlisting}

Since the procedure for obtaining the latest ffmpeg version is not always kept up-to-date and the line numbers will always change, you may have to create some patch first. Generally those line numbers are only updated by a developer when a new stable version with worthwhile features is actually included in the \CGG{} build. FFmpeg is constantly changing and many times the git version is not as stable as desired.

Finally, it is possible to compile \CGG{} so that it uses ffmpeg which is already installed on the system. This build takes less time to compile and may increase performance in both rendering and timeline manipulation. Again, there may be variations in functionality and less stability.
Getting a build to work in a system environment is not easy. If you have already installed libraries which are normally in the thirdparty build, getting them to be recognized means you have to install the devel version
so the header files which match the library interfaces exist. If you want to build using only the thirdparty libraries installed in your system, just include \texttt{-–without-thirdparty} to your configure script. For example:

\begin{lstlisting}[numbers=none]
./confgure --with-single-user --disable-static-build --without-thirdparty --without-libdpx
\end{lstlisting}

The library, libdpx, is just such an example of lost functionality: this build of \CGG{} will not be able to use the DPX format.

\section{Configuration Features}
\label{sec:configuration_features}
\index{build!configuration}

A listing of the current configuration features as of January 11, 2020:

\begingroup
    \fontsize{10pt}{12pt}\selectfont
\begin{verbatim}

`configure' configures \CGG{} to adapt to many kinds of systems.

Usage: ./configure [OPTION]... [VAR=VALUE]...

To assign environment variables (e.g., CC, CFLAGS...), specify them as
VAR=VALUE.  See below for descriptions of some of the useful variables.

Defaults for the options are specified in brackets.

Configuration:
  -h, --help              display this help and exit
      --help=short        display options specific to this package
      --help=recursive    display the short help of all the included packages
  -V, --version           display version information and exit
  -q, --quiet, --silent   do not print `checking ...' messages
      --cache-file=FILE   cache test results in FILE [disabled]
  -C, --config-cache      alias for `--cache-file=config.cache'
  -n, --no-create         do not create output files
      --srcdir=DIR        find the sources in DIR [configure dir or `..']

Installation directories:
  --prefix=PREFIX         install architecture-independent files in PREFIX
                          [/usr/local]
  --exec-prefix=EPREFIX   install architecture-dependent files in EPREFIX
                          [PREFIX]

By default, `make install' will install all the files in
`/usr/local/bin', `/usr/local/lib' etc.  You can specify
an installation prefix other than `/usr/local' using `--prefix',
for instance `--prefix=$HOME'.

For better control, use the options below.

Fine tuning of the installation directories:
  --bindir=DIR            user executables [EPREFIX/bin]
  --sbindir=DIR           system admin executables [EPREFIX/sbin]
  --libexecdir=DIR        program executables [EPREFIX/libexec]
  --sysconfdir=DIR        read-only single-machine data [PREFIX/etc]
  --sharedstatedir=DIR    modifiable architecture-independent data [PREFIX/com]
  --localstatedir=DIR     modifiable single-machine data [PREFIX/var]
  --libdir=DIR            object code libraries [EPREFIX/lib]
  --includedir=DIR        C header files [PREFIX/include]
  --oldincludedir=DIR     C header files for non-gcc [/usr/include]
  --datarootdir=DIR       read-only arch.-independent data root [PREFIX/share]
  --datadir=DIR           read-only architecture-independent data [DATAROOTDIR]
  --infodir=DIR           info documentation [DATAROOTDIR/info]
  --localedir=DIR         locale-dependent data [DATAROOTDIR/locale]
  --mandir=DIR            man documentation [DATAROOTDIR/man]
  --docdir=DIR            documentation root [DATAROOTDIR/doc/cinelerra]
  --htmldir=DIR           html documentation [DOCDIR]
  --dvidir=DIR            dvi documentation [DOCDIR]
  --pdfdir=DIR            pdf documentation [DOCDIR]
  --psdir=DIR             ps documentation [DOCDIR]

Program names:
  --program-prefix=PREFIX            prepend PREFIX to installed program names
  --program-suffix=SUFFIX            append SUFFIX to installed program names
  --program-transform-name=PROGRAM   run sed PROGRAM on installed program names

Optional Features:
  --disable-option-checking  ignore unrecognized --enable/--with options
  --disable-FEATURE       do not include FEATURE (same as --enable-FEATURE=no)
  --enable-FEATURE[=ARG]  include FEATURE [ARG=yes]
  --enable-silent-rules   less verbose build output (undo: "make V=1")
  --disable-silent-rules  verbose build output (undo: "make V=0")
  --enable-dependency-tracking
                          do not reject slow dependency extractors
  --disable-dependency-tracking
                          speeds up one-time build
  --enable-a52dec         build a52dec (yes)
  --enable-djbfft         build djbfft (yes)
  --enable-audiofile      build audiofile (no)
  --enable-encore         build encore (no)
  --enable-esound         build esound (no)
  --enable-ffmpeg         build ffmpeg (yes)
  --enable-fftw           build fftw (auto)
  --enable-flac           build flac (auto)
  --enable-giflib         build giflib (yes)
  --enable-ilmbase        build ilmbase (auto)
  --enable-lame           build lame (auto)
  --enable-libavc1394     build libavc1394 (auto)
  --enable-libraw1394     build libraw1394 (auto)
  --enable-libiec61883    build libiec61883 (auto)
  --enable-libdv          build libdv (auto)
  --enable-libjpeg        build libjpeg (auto)
  --enable-opus           build opus (auto)
  --enable-openjpeg       build openjpeg (auto)
  --enable-libogg         build libogg (auto)
  --enable-libsndfile     build libsndfile (auto)
  --enable-libsvtav1      build libsvtav1 (no)
  --enable-libtheora      build libtheora (auto)
  --enable-libuuid        build libuuid (yes)
  --enable-libvorbis      build libvorbis (auto)
  --enable-mjpegtools     build mjpegtools (yes)
  --enable-openexr        build openexr (auto)
  --enable-tiff           build tiff (auto)
  --enable-twolame        build twolame (auto)
  --enable-x264           build x264 (auto)
  --enable-x265           build x265 (auto)
  --enable-libvpx         build libvpx (auto)
  --enable-lv2            build lv2 (auto)
  --enable-sratom         build sratom (auto)
  --enable-serd           build serd (auto)
  --enable-sord           build sord (auto)
  --enable-lilv           build lilv (auto)
  --enable-suil           build suil (auto)
  --enable-libaom         build libaom (auto)
  --enable-dav1d          build dav1d (auto)
  --enable-libwebp        build libwebp (auto)
  --enable-ffnvcodec      build ffnvcodec (auto)
  --enable-static-build   build static ([auto])
  --enable-x264_hidepth   build x264 10bit ([no])
  --enable-x265_hidepth   build x265 10bit ([no])

Optional Packages:
  --with-PACKAGE[=ARG]    use PACKAGE [ARG=yes]
  --without-PACKAGE       do not use PACKAGE (same as --with-PACKAGE=no)
  --with-jobs             parallel build jobs (auto)
  --with-exec-name        binary executable name (cin)
  --with-single-user      to install cin in bin (no)
  --with-ladspa-build     build ladspa library (yes)
  --with-lv2              lv2 library support (yes)
  --with-cinlib           cinelerra library path (auto)
  --with-cindat           cinelerra share path (auto)
  --with-plugin-dir       plugin install dir (auto)
  --with-ladspa-dir       ladspa install dir (auto)
  --with-config-dir       .bcast config dir ($$HOME/.bcast5)
  --with-nested-dir       nested proxy dir ($$HOME/Videos)
  --with-browser          cin_browser path (firefox)
  --with-git-ffmpeg       git ffmpeg using url (no)
  --with-noelision        use noelision/libpthread (auto)
  --with-booby            window lock trace booby trap (no)
  --with-libzmpeg         build libzmpeg (yes)
  --with-commercial       enable commercial capture (yes)
  --with-thirdparty       use thirdparty build (yes)
  --with-shuttle          shuttle device (yes)
  --with-wintv            usb 2040:826d wintv device (yes)
  --with-x10tv            usb 0bc7:0004 X10 remote device (yes)
  --with-vaapi            video acceleration api (yes)
  --with-vdpau            video decode+presentation api for unix (yes)
  --with-nv               nvenc/nvdec ffnvcodec api (yes)
  --with-cuda             nv cuda plugins (auto)
  --with-clang            use clang instead of gcc/g++ (no)
  --with-gl               use opengl (auto)
  --with-oss              use OSS audio (auto)
  --with-xft              use libXft (auto)
  --with-alsa             use libasound/alsa (auto)
  --with-firewire         use firewire (auto)
  --with-dv               use dv (auto)
  --with-dvb              use dvb (auto)
  --with-video4linux2     use v4l2 (auto)
  --with-xxf86vm          use xf86vmode (auto)
  --with-esound           use esd (auto)
  --with-shuttle          shuttle dev support (auto)
  --with-shuttle_usb      use libusb-1.0 (auto)
  --with-lv2              use lv2 (auto)
  --with-cuda             build cuda plugins (auto)
  --with-dl               system has libdl (auto)
  --with-opencv           opencv=sys/sta/dyn,git/tar=url (auto)
  --with-numa             system has libnuma (auto)
  --with-openexr          use openexr (auto)
  --with-onevpl           use Intel hardware oneAPI Video Processing Library (no)

Some influential environment variables:
  CC          C compiler command
  CFLAGS      C compiler flags
  LDFLAGS     linker flags, e.g. -L<lib dir> if you have libraries in a
              nonstandard directory <lib dir>
  LIBS        libraries to pass to the linker, e.g. -l<library>
  CPPFLAGS    (Objective) C/C++ preprocessor flags, e.g. -I<include dir> if
              you have headers in a nonstandard directory <include dir>
  CCAS        assembler compiler command (defaults to CC)
  CCASFLAGS   assembler compiler flags (defaults to CFLAGS)
  CXX         C++ compiler command
  CXXFLAGS    C++ compiler flags

Use these variables to override the choices made by `configure' or to help
it to find libraries and programs with nonstandard names/locations.

Report bugs to <mail@lists.cinelerra-gg.org>.

\end{verbatim}
\endgroup

\section{Thirdparty Parallel Build}
\label{sec:thirdparty_parallel_build}
\index{build!thirdparty}

The Makefile in the thirdparty build directory employs a set of macros used to create a build rule set of thirdparty library build dependencies.  The standard build sequence of [source, config, build] is used to prepare thirdparty products as static libraries.  Build package dependency can be specified in the Makefile std-build macro call.  These Makefile macro calls define the rules used for each thirdparty build.  The expanded rule definitions may be viewed by using:

\hspace{2em}\texttt{make -C thirdparty rules}

Individual package libraries can be rebuilt, via:

\hspace{2em}\texttt{make -C thirdparty <pkg>-clean;  make -C thirdparty <pkg>}

The rule targets create the set of thirdparty packages which are built from local source archive copies of thirdparty source code and patches, if needed.  The build rule set of dependencies allows for compiling multiple thirdparty programs simultaneously using maximum computer resources.  This parallel build speeds up the process considerably.  For example, these are full static build timings on the production build machine (full build includes building all thirdparty programs as well as all of \CGG{}):

\begin{center}
        \begin{tabular}{@{}lcr}
                1 cpu & = & 61 mins\\
                12 cpus & = & 7.5 mins\\
                24 cpus & = & 2 mins\\
        \end{tabular}
\end{center}

\section{Using the very latest Libraries}
\label{sec:latest_libraries}
\index{build!use latest library}

Using the most current libraries can be a challenge for some of the Operating System distros that use
stable compilers, assemblers, and their own libraries.  Because they are stable, they frequently do
not keep up with the very latest of thirdparty libraries.  Consequently, some program constructs may
not yet be implemented.  \CGG{} tries to maintain stability since it is better to have less features
and no crashes.  The goal is to make \CGG{} widely available on many platforms rather than
dependent on advanced tools that are not supported on some distros.

\CGG{} attempts to upgrade to the latest releases of many thirdparty libraries about every 3-4
months. But it is often difficult to keep some of these thirdparty libraries up to date as their
developers switch from tried and true standard tools to newer, less standard tools. As a result 
in order to build \CGG{} on 2-3 versions of any distro rather than only the most current version,
some thirdparty libraries can not be kept up to date and may even be to the point of no further
updates.  In a lot of cases, the updated releases provide little new capabilities but rather
are bug fixes that may not even be relevant to \CGG{}'s use.

So as a computer savvy user or a developer, if you would like to build \CGG{} with the latest
thirdparty releases for some of the packages here are a few suggestions based on other 
developer's feedback and experimentation.

\textbf{dav1d} 
\begin{description}[noitemsep]
     \item Status - currently \CGG{} is staying at 0.5.1.  This is disappointing because there
may be speed gains in later versions that would be beneficial. However, it is usable for decoding
whereas libaom is a lot slower.  Unfortunately, it has no effective encoder.
     \item Problem - 0.6 dav1d requires NASM 2.14 (and later versions of dav1d use even later versions of NASM) and uses instructions like vgf2p8affineqb,
not exactly an "add" instruction. It also uses meson which is not widely available on all
distros.  The more recent NASM requirement apparently provides for using AVX-512 
instructions (like vgf2p8affineqb, which is more like a whole routine than a simple instruction).
     \item Workaround already in use by \CGG{} - a Makefile was generated to replace Meson usage
but has to be continuously updated for new releases. Dav1d 0.5.1 requires NASM 2.13 so at this level
the newer distros will work.  The availability of meson and nasm are a significant problem on
many systems which are still in wide use.
     \item Your workaround - Because a request to dav1d developers to consider changes to
ensure their library is more widely usable does not appear to be in their future, since it works
for them, you can upgrade NASM to 2.14 to stay up to date.  Even then, you will have to build using meson and incorporate it into \CGG{}.
\end{description}

\textbf{OpenExr} 
\begin{description}[noitemsep]
     \item Status - stable at 2.4.1 from February 2020.
     \item Problem - the OpenExr tarball is not a single package but is 2 packages instead.
     \item Workaround already in use by \CGG{} - reworked the packages so that it looks like
one package with 2 stubs.
     \item Your workaround - perhaps use the same workaround.
\end{description}

\textbf{OpenCV}
\begin{description}[noitemsep]
     \item Status - 2 different versions specific for O/S but none for Ubuntu 14, 32 or 64 bit.
     \item Problem - There are really 2 problems here.  The first is OpenCV is not really
"Open" in that Surf is patented/non-free and there is no actual source available for certain
capabilities. The second is that cmake 3.5.1 is required for OpenCV 4.2.
     \item Workaround already in use by \CGG{} - using 3.4.1 for older distros and 4.2 for newer.
     \item Your workaround - upgrade cmake to 3.5.1 for upgrade to 4.2; add non-free to the
compile; and use binaries that you do not know what they contain since no source code to compile.
Look into opencv4/opencv2/core/types.hpp:711;27.
\end{description}

\textbf{libaom}
\begin{description}[noitemsep]
     \item Status - currently at version 3.6.0 for older O/S and 3.8.0 for newer O/S.
     \item Problem - requires cmake 3.5 at v3.6.0 and 3.7.2 for v3.8.0.
     \item Workaround already in use by \CGG{} - modify configure.ac to switch from 3.8.0 to 3.6.0 for Ubuntu 16 and delete thirdparty/src/libaom-v3.8.0*.*.
     \item Your workaround - upgrade on some systems to cmake 3.7.2, switch to using 3.6.0 as in last sentence, or add --libaom-enable=no to configure line when building.
\end{description}

\textbf{x10tv}
\begin{description}[noitemsep]
     \item Status - this is the x10 TV remote control.
     \item Problem - INPUT\_PROP\_POINTING\_STICK not defined error on older distros.
     \item Workaround already in use by \CGG{} - leaving out of Ubuntu14, Ubuntu, Centos7.
     \item Your workaround - look into /usr/include/linux/input-event-codes.h.
\end{description}

\textbf{lv2 plugins, consisting of 6 routines}
\begin{description}[noitemsep]
     \item Status - currently at version 1.18.0 for lv2 and different for other 5.
     \item Problem - the current versions use cmake but the updated versions now all use meson and \CGG{} is not set up to handle that.
     \item Workaround already in use by \CGG{} - not upgrading at this time.
     \item Your workaround - if you are familiar with meson, you can independently upgrade the 6 routines.
\end{description}

\section{Find Lock Problems with Booby Trap}
\label{sec:find_lock_problems_booby_trap}
\index{build!booby trap}

A Booby Trap is used in \CGG{} for setting a trap to catch lock problems that might have been missed. It will trap boobies only if compile by adding \textit{-{}-with-booby} on the configuration command line. This is the default if you compile using \texttt{./bld.sh} from the GIT repository. It should not interfere with normal execution.

If you have the time and inclination, enable \textit{-{}-with-booby} and send any trap output that you find. Maybe you will catch some boobies and if you do, send a snapshot of any boobies you find.

There are 2 potential traps:

\begin{itemize}[nosep]
        \item If you try to unlock a lock when it is not locked
        \item If you execute a drawing operation without holding the window lock
\end{itemize}

The trap prints the following in the controlling terminal window:

\hspace{2em} \textit{BOOBY! \qquad <backtrace>}

An example backtrace is below along with a hint below on how to analyze:

\begin{lstlisting}[numbers=none]
/home/cin5/bin/./cin(_Z5boobyv+0x3f) [0x557069fa9b2f] /home/cin5/bin/./cin(_ZN13BC_WindowBase9draw_lineEiiiiP9BC_Pixmap+0x3b)0x557069fb9a9b]
/home/cin5/bin/./cin(\_ZN10BC_ListBox11draw_borderEi+0x73)[0x557069f7dc73]
/home/cin5/bin/./cin(+0x9707fb) [0x557069f7e7fb]
/home/cin5/bin/./cin(\ZN10BC\ListBox16center\selectionEv+0x4e)[0x557069f7f2ae]
/home/cin5/bin/plugins/video/sketcher.plugin(_ZN17SketcherCurveList6updateEi+0x1a0)[0x7f1b8002a4c0]
/home/cin5/bin/plugins/video/sketcher.plugin(_ZN18SketcherCurveColor17handle_done_eventEi+0x76)[0x7f1b8002a5f6]
/home/cin5/bin/./cin(_ZN15BC_DialogThread3runEv+0xd8)[0x557069f6fb78]
/home/cin5/bin/./cin(_ZN6Thread10entrypointEPv+0x45)[0x557069fc5995]
/usr/lib/libpthread.so.0(+0x7a9d) [0x7f1b91b4ea9d]
/usr/lib/libc.so.6(clone+0x43) [0x7f1b90accb23]
\end{lstlisting}

To see which routine is reporting the booby key in:

\hspace{2em} \texttt{c++filt}

And then the $2^{nd}$ line in the backtrace above:

\hspace{2em}\texttt{\_ZN13BC\_WindowBase9draw\_lineEiiiiP9BC\_Pixmap}

It comes back with the routine as:

\hspace{2em}\texttt{BC\_WindowBase::draw\_line(int, int, int, int, BC\_Pixmap*)}

\section{Valgrind Support Level}
\label{sec:valgrind_support_level}
\index{build!valgrind}

Valgrind is a memory mis-management detector.  It shows you memory leaks, deallocation errors, mismanaged threads, rogue reads/writes, etc.  \CGG{} memory management is designed to work with Valgrind detection methods.  This assists in developing reliable code.  Use of Valgrind points out problems so that they can be fixed.  For example, when this version of \CGG{} shuts down, it deallocates memory instead of just stopping, thus making memory leak detection possible.  An alternative to Valgrind is
\textit{heaptrack} which has good documentation and for large programs can run
faster.  You can find the source and information about it at {\small\url{https://github.com/KDE/heaptrack}}.
\index{valgrind!heaptrack}

The best way to compile and run valgrind is to run the developer static build. This takes 2 steps and you must already have gdb and valgrind installed:

\begin{enumerate}[nosep]
        \item The standard static build:\\
                \texttt{cd /path/cinelerra-5.1}\\
                \texttt{make clean}\\
                \texttt{./bld.sh}
        \item run the incremental rebuild for debug objs:\\
                \texttt{CFLAGS=-ggdb make -j8 rebuild\_all}
\end{enumerate}

If you frequently make mods and changes in \CGG{} or the thirdparty libraries, do not depend on those
compiled versions to have the headers updated; so be sure to fully rebuild as shown in the previous 2
steps before running valgrind or you will get false errors.

Now your \CGG{} obj has all of the debug stuff. Next run valgrind as root for the most useful results:

\hspace{2em}\texttt{cd /path/cinelerra-5.1/cinelerra}

\hspace{2em}\texttt{valgrind -{}-log-file=/tmp/log -{}-leak-check=full\\
        -{}-num-callers=32 ./ci}

This runs \CGG{} under the control of valgrind, and produces a log file in /tmp which will list information about any leaks, usually clearly identifiable. Be sure to Quit out of \CGG{} normally instead of Ctrl-C or a SEGV otherwise the program does not have a chance to cleanup and there will be some false alarms. But it runs very slowly, and is basically single threaded, which means that race conditions may be impossible to catch$\dots$ like one thread deletes memory that another thread is currently using. But overall it is a big help and if you test any new features, please email the log output. A lot of effort when writing the code was put into trying to be sure that all of the object constructors have matching destructors so that the leaks can be identified. There are already several libraries that create predictable memory leaks and valgrind does a good job for most of these.

It is impossible to test everything with valgrind because some things are just too big and slow for a practical test. Occasionally you can find a leak or an illegal memory access. There are several false alarms that are difficult to avoid \textit{Conditional jump} messages, and \textit{unhandled DW\_OP\_}, but anything with the word \textit{illegal} in the message is important. Memory leaks that originate in \CGG{} are good to find and fix, but are usually not
deadly. The listing of the memory leaks can be quite voluminous so locating the \textit{LEAK SUMMARY} section
towards the end of the report is most useful.

Another very useful valgrind run to locate unitialized variables while executing is:

\hspace{2em}\texttt{valgrind -{}-log-file=/tmp/log -{}-tool=memcheck\\
        -{}-num-callers=32 ./ci}

\section{CFLAGS has -Wall}
\label{sec:cflags_has_-wall}
When compiling \CGG{} Infinity a CFLAGS option used is \textit{Wall} where the "W" represents warnings and "all" means all.  This causes the compile to check for simple mistakes that can be detected automatically and issue warnings when the code is questionable.  It can also detect situations where the compiler will generate incorrect code, like type-punned pointer.  By turning on this flag, when new code is vetted for predictable mistakes, the code can be corrected before becoming manifested in the application.

\section{Prof2 -- A Profiler}
\label{sec:prof2_profiler}
\index{build!prof2}

Frequently there is a problem with a program running slow and you do not know why. You need a thumbnail analysis of where the program is spending most of its time without all of the overwhelming details. This is when a Profiler comes in handy.

There are many different profilers available -- this particular one does not do anything special and is fairly ordinary in that it just characterizes frequency of execution of various regions in the program. However, it has some pretty good strengths as listed next.

\begin{enumerate}[nosep]
        \item It takes very little or no preconditioning, i.e. setup
        \item The effect it has on the running program is minimal
        \item It runs almost at full speed, which is really nice
        \item It is not particularly thread aware
        \item Reports where it is executing at intervals of 100 times a second
\end{enumerate}

\subsection{Setup}
\label{sub:setup}

This profiler works on x86\_64 systems and you must be root to compile and run it. Also, you must install your operating system's \textit{iberty} -- for example, which would be binutils-devel for Fedora or libiberty-dev for Ubuntu 18.

Go to the top level \CGG{} directory.

Key in: \qquad \texttt{prof2}

Key in: \qquad \texttt{make clean all install}

Because \textit{smap} may have to be found in the system if \textit{How to use} below does not work, you will have to do the following:
\newline
\newline
Key in: \qquad \texttt{cp -a smap /usr/local/bin}


Later, if you want to remove this from the system,

Key in: \qquad \texttt{make uninstall}

\subsection{How to use}
\label{sub:how_to_use}

The program you are profiling should be compiled debuggable with stack frame and symbols enabled.

To see help, key in: \qquad \texttt{prof -h}

usage: -h [-o path] [-d] [-e] [-p libpath] [-012] [-u \#] cmd args...

\hspace{2em}
\begin{tabular}{@{}ll}
        -o & profile output pathname, -=stdout, -{}-=stderr\\
        -d & debug otitleutput enabled\\
        -e & child debug output enabled\\
        -p & specify path for libprofile.so\\
        -0 & usr+sys cpu timer intervals (sigprof)\\
        -1 & usr only cpu timer intervals (sigvtalrm)\\
        -2 & real time timer intervals (sigalrm)\\
        -u & profile timer interval in usecs\\
\end{tabular}

To execute the profiler, key in:

\hspace{2em}\texttt{prof -o /tmp/prof\_list.txt ./cin}

where \texttt{/tmp/prof\_list.txt} is the output file and in this case \texttt{cin} is the \CGG{} binary file. The pid of this command will be displayed on the startup window. This comes in handy in the use case where there is a lot of initial load and possible configuration setup inside of \CGG{} and you want to profile plugins and not necessarily all of the setup steps. Then you can use the following command in another window to continue running \CGG{} and obtain the more useful information:

\hspace{2em}\texttt{kill -USR1 pid}

Running this command refreshes the memory maps used for profiling. When you are profiling a plugin, you want to run this AFTER the plugin loads.

\subsection{Results}
\label{sub:results}

There are 3 sections in the resulting output file of this stochastic quick analysis.

\begin{enumerate}[nosep]
        \item The first section is a histogram of the timer intervals of that sample set. Each function occupies a region of addresses. One hundred times a second, the profiler samples the program address in the region of execution.
        \item In the second section, there is another histogram of the cumulative frequency of all things in the call stack and this should point out who is the culprit. For example, a codec calls a bunch of subroutines, the cost of the subroutines is accumulated to the codec parent caller. This makes the actual cpu user much more apparent.
        \item The last section is for the library spaces. Each library occupies a region and the profiler adds up the time spent in each of the libraries by frequency.
\end{enumerate}

On the very bottom is a 1 line summary which shows you if there is a bad guy and who it is.

\subsection{Sample output}
\label{sub:sample_output}

\textit{--- profile start ---}

\textbf{1020 ticks 43 modules 81412 syms}\\
\begin{tabular}{@{}rrp{\linewidth-6em}}
 0.010s & 0.1\% & Autos::copy(long, long, FileXML*, int, int) /mnt0/build5/cinelerra-5.1/bin/cin\\
 0.010s & 0.1\% & BinFolders::copy\_from(BinFolders*) /mnt0/build5/cinelerra-5.1/bin/cin\\
 0.010s & 0.1\% & cstrdup(char const*)     /mnt0/build5/cinelerra-5.1/bin/cin\\
 0.010s & 0.1\% & XMLBuffer::encode\_data(char*, char const*, int) /mnt0/build5/cinelerra-5.1/bin/cin\\
 0.010s & 0.1\% & XMLBuffer::encoded\_length(char const*, int) /mnt0/build5/cinelerra-5.1/bin/cin\\
 0.010s & 0.1\% & PluginClient::send\_configure\_change() /mnt0/build5/cinelerra-5.1/bin/cin\\
 0.010s & 0.1\% & UndoVersion::scan\_lines(UndoHashTable*, char*, char*) /mnt0/.../cin\\
 0.010s & 0.1\% & UndoStackItem::set\_data(char*) /mnt0/build5/cinelerra-5.1/bin/cin\\
 0.010s & 0.1\% & UndoStack::load(\_IO\_FILE*) /mnt0/build5/cinelerra-5.1/bin/cin\\
 0.010s & 0.1\% & BC\_Bitmap::cur\_bfr()     /mnt0/build5/cinelerra-5.1/bin/cin\\
 0.010s & 0.1\% & YUV::init\_tables(int, int*, int*, int*, int*, int*, int*, int*, int*, int*, int*, int*, int*, int*, int*) /mnt0/build5/cinelerra-5.1/bin/cin\\
\end{tabular}\\
$\dots$\\
$\dots$\\
\textit{--- profile calls ---}

\begin{tabular}{@{}rrl}
 0.010s & 0.1\% & AutoConf::save\_xml(FileXML*)   1.0 /mnt0/build5/cinelerra-5.1/bin/cin\\
 0.010s & 0.1\% & Automation::copy(long, long, FileXML*, int, int)   1.0 /mnt0/.../cin\\
 0.010s & 0.1\% & AWindow::run()             1.0 /mnt0/build5/cinelerra-5.1/bin/cin\\
 0.010s & 0.1\% & Canvas::stop\_single()      1.0 /mnt0/build5/cinelerra-5.1/bin/cin\\
 0.010s & 0.1\% & ColorPicker::new\_gui()     1.0 /mnt0/build5/cinelerra-5.1/bin/cin\\
 0.010s & 0.1\% & ColorWindow::create\_objects()   1.0 /mnt0/build5/cinelerra-5.1/bin/cin\\
 0.010s & 0.1\% & PaletteWheel::draw(float, float)   1.0 /mnt0/build5/cinelerra-5.1/bin/cin\\
 0.010s & 0.1\% & PaletteHex::update()       1.0 /mnt0/build5/cinelerra-5.1/bin/cin\\
 0.010s & 0.1\% & CWindowGUI::draw\_status(int)   1.0 /mnt0/build5/cinelerra-5.1/bin/cin\\
 0.010s & \textbf{0.1\%} & CWindowCanvas::status\_event()   1.0 /mnt0/build5/cinelerra-5.1/bin/cin\\
\end{tabular}\\
$\dots$\\
$\dots$\\
\begin{tabular}{@{}rrl}
 0.990s & \textbf{9.7\%} & BC\_Xfer::xfer\_slices(int)   1.0 /mnt0/build5/cinelerra-5.1/bin/cin\\
 1.880s & \textbf{18.4\%} & DirectUnit::process\_package(LoadPackage*)   1.0 /mnt0/build5/cinelerra-5.1/bin/cin\\
 1.880s & \textbf{18.4\%} & DirectUnit::rgba8888()     1.0 /mnt0/build5/cinelerra-5.1/bin/cin\\
 3.910s & \textbf{38.3\%} & \_\_init\_array\_end           1.1 /mnt0/build5/cinelerra-5.1/bin/cin\\
 5.450s & \textbf{53.4\%} & LoadClient::run()          1.0 /mnt0/build5/cinelerra-5.1/bin/cin\\
 7.890s & \textbf{77.4\%} & Thread::entrypoint(void*)   1.0 /mnt0/build5/cinelerra-5.1/bin/cin\\
 7.890s & \textbf{77.4\%} & start\_thread               1.0 /lib64/libpthread-2.28.so\\
\end{tabular}\\
---\\
---\\
\begin{tabular}{@{}rrl}
 0.010s & 0.1\%/  0.0\% & /lib64/libm-2.28.so\\
 0.010s & 0.1\%/  0.0\% & /lib64/libexpat.so.1.6.8\\
 0.020s & 0.2\%/  0.1\% & /lib64/libXext.so.6.4.0\\
 0.020s & 0.2\%/  0.1\% & /lib64/libXft.so.2.3.2\\
 0.020s & 0.2\%/  0.1\% & /lib64/libxcb.so.1.1.0\\
 0.040s & 0.4\%/  0.2\% & /lib64/ld-2.28.so\\
 0.050s & 0.5\%/  0.2\% & /lib64/libpng16.so.16.34.0\\
 0.130s & 1.3\%/  0.6\% & /lib64/libX11.so.6.3.0\\
 0.180s & 1.8\%/  0.8\% & /lib64/libz.so.1.2.11\\
 0.200s & 2.0\%/  0.9\% & /lib64/libfontconfig.so.1.12.0\\
 0.380s & 3.7\%/  1.8\% & /lib64/libpthread-2.28.so\\
 1.660s & 16.3\%/ 7.7\% & /lib64/libc-2.28.so\\
 7.480s & 73.3\%/34.7\% & /mnt0/build5/cinelerra-5.1/bin/cin\\
\end{tabular}

\textbf{10.200t 0.001u+0.000s 21.566r  47.3\%}\\
\textit{--- profile end ---}

The summary line above in Bold represents the User time, System time, Real time and the percentage is how much Timer time elapsed over Real time so in this case the measurement covers 47.3\% of time.

So why use a Profiler? Because it is the ``ls'' for executable functions!!

\section{Working on AppImage}
\label{sec:working_on_appimage}

You can work on the appimage file to make changes and fix errors, or you can create a new appimage from scratch containing customizations. For example, you can add new rendering presets, update the Context-Help, change libraries that are no longer supported by the current distro, or make other modifications.

\subsection{Managing AppImage}
\label{sub:managing_appimage}
\index{appimage!management}

A limitation of using AppImage instead of installing the binary or compiling from git, is that there is only a single file without the ability to browse the directory structure or to look for files to edit or check. So if using \CGG{} leads to some errors, it is not possible to investigate and fix the problem. Which means if you want to add the most up-to-date Context-Help or want to introduce some custom presets, that can not be done.

Because the appimage file is nothing more than a compressed file containing the same structure as the installed program plus other libraries that allow the program to run independently from the system, the content can be extracted so that you can work on it as you would have on the normally installed program.  To do this you will need the appimage management program.
Many Linux distros come with this managment program by default, but others may not. For instance in the case of Arch Linux the \texttt{appimagetool-bin} package from AUR needs to be installed. 

To work on the appimage, first unpack it using the command\protect\footnote{Example provided by Glitterball3} (note that you do not have to be root to do any of the following):

\begin{lstlisting}[numbers=none]
        /{path to appimage}/CinGG-yyyymmdd.AppImage --appimage-extract
\end{lstlisting}

You will now have a \texttt{squashfs-root} folder in your current directory containing \texttt{/usr/bin/} as well as other files and directories such as \texttt{/usr/lib} and \texttt{/usr/share}. \texttt{Bin} is the folder similar to the one installed with \CGG{} and contains the files that you can work on. Now it is possible to make changes like adding a custom preset in \texttt{/ffmpeg/video} or replacing a library that no longer works with a more recent version by working in \texttt{/squashfs-root/usr/lib}.

To start the unpacked program from the bin folder use the command:

\begin{lstlisting}[numbers=none]
        /{path to appimage}/squashfs-root/usr/bin/./cin
\end{lstlisting}

After making your changes, to get back to having an appimage file instead of having to run the program from the bin folder, you can take the extra step of recompressing the squashfs-root folder.
To do so use linuxdeploy's appimage, a program that can also be useful for creating appimages from scratch (\url{https://github.com/linuxdeploy/linuxdeploy/releases/continuous}).

The steps to recreate the appimage are:

\begin{enumerate}
        \item Copy the linuxdeploy appimage to \texttt{/\{path to appimage\}} and make sure it is executable.
        \item Then use the command:
        \begin{lstlisting}[numbers=none]
                ./linuxdeploy-x86_64.AppImage --appdir squashfs-root --output appimage
        \end{lstlisting}
\end{enumerate}

A new appimage will be created like the original but containing the changes.

Alternatively, download the \texttt{appimagetool} version from \url{https://github.com/AppImage/AppImageKit/releases} if available for your distro and use the command:

\begin{lstlisting}[numbers=none]
        ./appimagetool --comp /{path to appimage}/squashfs-root /tmpCinGG-yyyymmdd.AppImage
\end{lstlisting}

Now there will be an appimage called \textit{CinGG-yyyymmdd.AppImage} with the changes that were made.

NOTE: for the \textbf{bdwrite} program which is used to create the udfs image to burn Blu-ray
media (or any other standalone program), you will find it in the squashfs-root/usr/bin
subdirectory. You only need to do the initial unpacking extract step and then just copy
bdwrite and the entire squashfs-root/usr/lib directory to a convenient permanent place for
usage.  Then when running bdwrite, you will have to first export the libary path by typing
this command in the window first:
\begin{lstlisting}[numbers=none]
export LD_LIBRARY_PATH=/{put your copied usr/lib path here}:$LD_LIBRARY_PATH
\end{lstlisting}

\subsection{Build the CinGG.AppImage from scratch}
\label{sub:built_appimage_scratch}
\index{appimage!creating}

If a developer wants to create an appimage from git, follow these next few steps. An existing automated script is available, \texttt{bld\_appimage.sh}, in the directory \texttt{/\{path to cinelerra-5.1\}}.

This follows four steps:

\begin{itemize}
        \item Build static version of \CGG{}.
        \item If desired and needed, build the HTML manual for context-sensitive help.
        \item Organize code and if present, the manual, in a AppImage specific format \textit{AppDir}. Do this with tool makeappimage, which is built if needed.
        \item Call an external tool to make the AppDir into an AppImage
\end{itemize}

Start by downloading the \CGG{} source from Cinelerra's git. The last parameter is a directory name of your choice, the directory must not exist. As example, the name \textit{cinelerra5} is used.

\begin{lstlisting}[numbers=none]
        git clone --depth 1 git://git.cinelerra-gg.org/goodguy/cinelerra.git cinelerra5
\end{lstlisting}

The source will be in a subdirectory \texttt{cinelerra-5.1} of the directory created by the \textit{git clone} operation.

If context-sensitive help is needed, download the manual sources too, with a different destination directory.

\begin{lstlisting}[numbers=none]
        git clone --depth 1 git://git.cinelerra-gg.org/goodguy/cin-manual-latex.git cin-manual-latex
\end{lstlisting}

Then move to the \texttt{/\{path to cinelerra-5.1}/\} directory.


There are two preliminaries to do before running the script:

1- If context sensitive help in the appimage version is required, the source of the manual and the tools (packages) to build it must be on the system. In the bld\_appimage.sh script, set the variable \texttt{MANUAL\_DIRECTORY=\$(pwd)/../../ cin-manual-latex} to the path of the source of the manual. If the variable is empty, or the specified directory does not exist, \CGG{} will be built without built-in help. The path to the manual source can be an absolute or relative one. An easier method to include the help from the manual rather than having to install a bunch of latex building software, is to simply download the latest tgz version from {\small\url{https://cinelerra-gg.org/download/images/HTML_Manual-202xxxxx.tgz}}.
Then extract the files using "tar xvf" into the cinelerra AppDir/usr/bin/doc directory.
This alternative method may not contain the most recent changes to the Manual but rather will contain what had been checked into Git by the date of the tgz file.

2- The script bld\_appimage.sh uses a platform specific version of appimagetool so that it can create appimages for \textit{x86\_64}, \textit{i686}, \textit{aarch64}, or \textit{armv7l} architecture. We need to add appimagetool-(platform).AppImage to the \texttt{/\{path to cinelerra- 5.1\}/tools} directory, or somewhere in your path. You can download the tool for your system (e.g. appimagetool-x86\_64.AppImage) from git: {\small\url{https://github.com/AppImage/AppImageKit/releases}}. Be aware of the possibility that an older appimagetool from 2020 might work better on some systems compared to the latest release.

Always remember to make it executable. The four supported platforms are:

\begin{itemize}
        \item appimagetool-aarch64.AppImage
        \item appimagetool-armhf.AppImage
        \item appimagetool-i686.AppImage
        \item appimagetool-x86\_64.AppImage
\end{itemize}

With the preparations done, let's go back to the bld\_appimage.sh script.

Make the script executable if it isn't already:

\begin{lstlisting}[numbers=none]
chmod x+u bld_appimage.sh
\end{lstlisting}

Start the script with:

\begin{lstlisting}[numbers=none]
./bld_appimage.sh
\end{lstlisting}

The first part of the script customizes \texttt{configure}, depending on the architecture of our system. The next step compiles and installs \CGG{} in the \texttt{bin} subdirectory. The third part creates the manual in HTML; It will be the basis for the context help (see \nameref{sec:help_context_help}). The fourth part creates the directory \texttt{/AppDir/usr}, which is the basic structure of an appimage, and then populates it with the contents of \texttt{/\{path to cinelerra-5.1\}/bin}. Finally, the fifth step starts a script, called \texttt{makeappimage}, which points to \textit{bld.sh} contained in \texttt{/tools/makeappimagetool/}.

The script specifies all executables that are part of \CGG{}, so makeappimage can pick up dependencies. Any executable code in other places is not picked up. makeappimage will populate the AppDir directory, and then call the platform dependent appimagetool found in \texttt{/\{path to cinelerra-5.1\}/tools}.

At the end of the compilation there will be an appimage in the \texttt{cinelerra-5.1} directory, which we can move and use.

\section{How to Create a new Theme}
\label{sec:how_create_theme}
\index{theme!create new theme}

A \textit{Theme} is a base class object that is created and customized as \textit{ThemeName}.
It is constructed during program initialization in a theme plugin
\texttt{PluginTClient},
defined in \texttt{plugins/theme\_name} source directory.

\texttt{theme\_name.C} and \texttt{theme\_name.h} are derived \textit{Theme} class object constructors.

A \textit{Theme} is constructed during initialization in \texttt{init\_theme} (\texttt{mwindow.C}). The theme plugin is accessed using the \textit{name} from preferences and then the theme plugin is loaded which contains the code to construct the theme.  A \textit{Theme} object has functions and data that \CGG{} uses to do a variety of customizations, such as \texttt{default\_window\_positions}, and it can modify GUI defaults like \\
\texttt{default\_text\_color} when it is initialized.

The theme plugin contains a \textit{new\_theme} function that allocates and constructs a
\textit{ThemeName} object with base classes of \textit{BC\_Theme} (gui setup), \textit{Theme} (\CGG{} defaults), and \textit{ThemeName}, with definitions and overrides that create the custom theme. To create a new theme, a new plugin is needed:

\begin{lstlisting}[numbers=none]
        #include "header files.h"
        PluginClient* new_plugin(PluginServer *server)
        {
                return new NameMain(server);
        }
        
        NameMain::NameMain(PluginServer *server)
        : PluginTClient(server)
        {
        }
        
        NameMain::~NameMain()
        {
        }
        const char* NameMain::plugin_title() { return N_("Name"); }
        
        Theme* NameMain::new_theme()
        {
                theme = new ThemeName;
                extern unsigned char _binary_theme_name_data_start[];
                theme->set_data(_binary_theme_name_data_start);
                return theme;
        }
        
        Name::Name()
        : Theme()
        {
        }
        
        Name::~Name()
        {
                delete stuff;
        }
\end{lstlisting}

When a theme is constructed by \texttt{NameMain::new\_theme()}, it sets a pointer to a
block of data created in the plugin build that contains all of the png data
files in the \texttt{plugins/theme\_name/data} directory.  These images may define or override the appearance of gui images, such as \textit{ok.png} (the ok button).  There are usually a large number of images that need to be defined.  The theme plugin adds them to the theme image data in the \texttt{theme $\rightarrow$ initialize()} function.  The best list of theme image setup is probably in SUV (\texttt{plugins/theme\_suv/suv}).

The easy way to create a new theme is to copy an existing theme and change
its name to \textit{ThemeName}, change \texttt{plugin\_title()} to the new name, and then tweak the definitions until you are happy with the results.   The file
names and Makefile also need to be updated to the new theme name.  The source
can by manually rebuilt by invoking \textit{make} in the \texttt{plugins/theme\_name}
directory.

Once the new theme is built into the plugin library, it will automatically be discovered by the plugin probe
and it will become an available theme in \textit{Preferences}.

If you are ready to add it to the main build, then \textit{theme\_name} should be
included in the DIRS targets of the \texttt{plugins/Makefile}, and \texttt{plugin\_defs} needs \textit{theme\_name} in the themes list.

Themes usually require considerable time to create from scratch.  For
example, the SUV theme has over 800 lines in the initialize function, and has over
500 png images in the data directory.  Most of these images and data values are
required to be initialized by the custom theme constructor; very tedious and
time consuming work.  Creating a new theme is usually a lot of work.

\section{How Context Help works in the Program Code}
\label{sec:context_help_coding}
\index{context help}

All class methods related to context help start with the common pattern \texttt{context \_help} in their names. It is easy to get all occurences in the code with the following command (example):

\begin{lstlisting}[style=sh]
        grep -F context_help `find . -name '*.[Ch]' -print`
\end{lstlisting}

The base functionality is defined in several \textit{BC\_WindowBase} class methods in \\ \texttt{guicast/bcwindowbase.C} (search on \texttt{context\_help}). All BC\_Window's and BC\_SubWindow's inherit these methods.

For the simplest case of context help definition, it is sufficient to add the call to \texttt{context\_help\_set\_keyword()} in the most inclusive widget constructor. If \texttt{Alt/h} is pressed with the mouse over this widget's window, its \texttt{keypress\_event()} method (inherited from BC\_WindowBase) will catch this hotkey, fetch the keyphrase defined by \texttt{context\_help\_set\_keyword()} and call the \texttt{doc/ContextManual.pl} script with this keyphrase. Then ContextManual.pl script does the whole processing of the keyphrase given and calls web browser to display the found HTML manual page. The browser is called in the background to prevent from blocking the calling \CGG{} thread.

An example from \textit{cinelerra/zoombar.C}:

\begin{lstlisting}[style=sh]
        ZoomBar::ZoomBar(MWindow *mwindow, MWindowGUI *gui)
        : BC_SubWindow(...)
        {
                this->gui = gui;
                this->mwindow = mwindow;
                context_help_set_keyword("Zoom Panel");
        }                                                        
\end{lstlisting}

If \texttt{Alt/h} is pressed with the mouse over some subwindow (arbitrary depth) of the \texttt{context\_help\_set\_keyword()} caller, the \texttt{keypress\_event()} method of that subwindow catches the hotkey. Its own context help keyword, probably, will be empty. In this case the whole widget hierarchy is traced up to \textit{top\_level} widget, their context help keywords are checked, and the first nonempty keyword is used for context help.

This approach allows us to define the help keyword common to the whole dialog window with a bunch of diverse buttons with a single call to \texttt{context\_help\_set \_keyword()}, without placing such a call into each button constructor. And at the same time, this approach allows to assign different help keywords to GUI elements belonging to the same window but documented in different manual pages.

\subsubsection{An example with several different help keywords from cinelerra/mwindowgui.C:}%
\label{ssub:exemple_different_help_keywords}

\begin{lstlisting}[style=sh]
        MWindowGUI::MWindowGUI(MWindow *mwindow)
        : BC_Window(_(PROGRAM_NAME ": Program"), ...)
        {
                this->mwindow = mwindow;
                ...
                context_help_set_keyword("Program Window");
        }
        ...
        FFMpegToggle::FFMpegToggle(MWindow *mwindow, MButtons *mbuttons, int x, int y)
        : BC_Toggle(...)
        {
                this->mwindow = mwindow;
                this->mbuttons = mbuttons;
                set_tooltip(get_value() ? FFMPEG_EARLY_TIP : FFMPEG_LATE_TIP);
                context_help_set_keyword("FFmpeg Early Probe Explanation");
        }
        ...
        StackButton::StackButton(MWindow *mwindow, int x, int y)
        : BC_GenericButton(x, y, mwindow->theme->stack_button_w, "0")
        {
                this->mwindow = mwindow;
                set_tooltip(_("Close EDL"));
                context_help_set_keyword("OpenEDL");
        }
        ...
        ProxyToggle::ProxyToggle(MWindow *mwindow, MButtons *mbuttons, int x, int y)
        : BC_Toggle(...)
        {
                this->mwindow = mwindow;
                this->mbuttons = mbuttons;
                ...
                context_help_set_keyword("Proxy");
        }
        \end{lstlisting}
If the widget we wish to add context help to, overloads keypress\_event() of the base class with its own method, then it is necessary to introduce a small addition to its keypress\_event() handler: the call to \texttt{context\_help\_check\_and\_show()} has to be added in a suitable place in the handler.

An example with \texttt{context\_help\_check\_and\_show()} from \textit{cinelerra/mbuttons.C}:
\begin{lstlisting}[style=sh]
        int MButtons::keypress_event()
        {
            int result = 0;
            if(!result) {
                result = transport->keypress_event();
            }
            if(!result) {
                result = context_help_check_and_show();
            }
            return result;
        }
\end{lstlisting}

All the keypress handlers are not equal, therefore we provide different variants of the methods context\_help\_check\_and\_show() and context\_help\_show() (the latter for explicit checking on hotkey).

An example with context\_help\_show() from cinelerra/editpanel.C:
\begin{lstlisting}[style=sh]
        int EditPanelTcInt::keypress_event()
        {
                if( get_keypress() == 'h' && alt_down() ) {
                        context_help_show("Align Timecodes");
                        return 1;
                }
                ...further processing...
                return 1;
        }
\end{lstlisting}

A single example of much more sophisticated keypress\_event() handler can be found in \texttt{cinelerra/trackcanvas.C} (search on context\_help). The problem here was to track the particular object drawn on the canvas to figure out whose context help is to be requested. Another rare example of difficulty may appear when context help is desired for some GUI object which is not subclassed from BC\_WindowBase.

\textit{ContextManual.pl} looks for occurence of keyphrases in the following order:

\begin{enumerate}
        \item In \textit{Contents.html}
        \item In \textit{Index.html}
        \item In all the \textit{CinelerraGG\_Manual/*.html} files via grep
\end{enumerate}

Keyphrase matching is tried first exact and case sensitive, then partially and case insensitive. The special keyword \texttt{TOC} shows Contents, \texttt{IDX} shows Index. If the keyword starts with \texttt{FILE:}, the filename after \textit{FILE:} is extracted and that file is shown. You can look in the ContextManual.pl script text.

For debugging purposes the script can be called from command line. For this to work, the environment variable \texttt{\$CIN\_DAT} has to be set to the parent directory of doc.


\subsubsection{Providing context help for plugins}%
\label{ssub:providing_context_help_plugins}

In the simplest case, nothing has to be done at all. The plugin context help functionality introduced in \texttt{cinelerra/pluginclient.C} automatically assigns context help keyword from the plugin's title for all plugins subclassed from \textit{PluginClient}. For this to work, the manual page documenting the plugin must be entitled identically to the programmatic title of the plugin. A special treatment can be necessary in the following cases:

\begin{enumerate}
        \item \textit{Rendered effects} are not subclasses of PluginClient and require an explicit context help definition via \texttt{context\_help\_set\_keyword().}
        \item Only the main plugin dialog inherits context help functionality from the parent class. If a plugin opens additional dialog windows, they probably require explicit context help definitions.
        \item If the plugin title differs from its subsection title in the documentation, either its help keyword or the corresponding title in the manual should be renamed. Another possibility can be an addition to the \textit{\%rewrite table} in \texttt{doc/ContextManual.pl}.
        \item If the plugin title contains some characters special for HTML and/or Perl regular expression syntax, the special characters have to be escaped. For example, the keyphrase corresponding to the title \textit{Crop \& Position (X/Y)} should be converted to:

        \begin{lstlisting}[style=sh]
                Crop & Position \\(X\\/Y\\)
        \end{lstlisting}

        Such a rewriting can be done either in the C++ code or in ContextManual.pl.
\end{enumerate}

One additional explanation about previous user information when plugin tooltips are off. Help for the \textit {selected plugin} is shown because in this case it would be difficult to figure out, without a visual feedback, which particular item is currently under the mouse.

\subsection{Manual Maintenance is now Tightly Coupled with the Program Code}
\label{sub:manmaintain}

If some section of the \CGG{} manual gets renamed and is used for Context Help, the corresponding help keywords should be adjusted too. All the keywords used can be listed via the following example command:

\begin{lstlisting}[style=sh]
        grep -F context_help `find . -name '*.C' -print` | grep -F '"'
\end{lstlisting}
----------------

Note that the keyword does not have to exactly match the section title; it can also be a case insensitive substring of the section title.

If some new \CGG{} window or dialog is created and documented, the inclusion of context\_help\_set\_keyword() calls and/or adaptation of keypress\_event() handler (if any) should be included in the program code.

If some new \CGG{} plugin is created, it is best to document it in the section entitled exactly equal to that plugin's title. Then probably its context help will work out of the box. Otherwise, some adaptation to its keypress\_event() (if any) or to the %rewrite table in ContextManual.pl may be necessary.

Why the local copy of \CGG{} manual should be used?

\begin{enumerate}
        \item For context help keyphrases matching, the local copy of \textit{Contents.html} and \textit{Index.html} is necessary anyway.
        \item Grepping \textit{CinelerraGG\_Manual/*.html} files of the remote manual from the website cannot work per definition.
        \item The local copy usage ensures exact matching of the version of the manual to the version of \CGG{}. Otherwise, if one uses for some reason an older version of \CGG{} with the help fetched from the newer version of the website manual, various incompatibilities can be expected.
        \item Packing the manual into AppImage, the current method of \CGG{} packaging, should be much easier than merging two different git branches when building from source packages, as was done earlier.
\end{enumerate}

What about Localization?

For now, \CGG{} context help is not localized at all. There is no complete \CGG{} manual in any language except English. But even should the manual be translated to other languages, the context help keywords must remain unlocalized literal string constants. First, because the set of languages known to \CGG{} itself is not the same as the set of languages the manual will be translated to. If some "translated keyword" is fed to the help system, while the manual in this language does not exist, keyword matching cannot succeed. Second, such a help localization with the translation of all keywords can be done and then maintained much easier and much more reliably inside the ContextManual.pl script rather than in the \CGG{} binary.

More about ContextManual.pl file?

The bin/doc/ContextManual.pl script can be configured further. Look in the script text. You can define your preferable web browser, or redefine it to 'echo' for debug purposes. There are also some predefined HTML pages for Contents and Index, and several explicitly rewritten keyphrases. 

After the first invocation of context help the system-wide script, bin/doc/ContextManual.pl, copies itself into
the user's config directory, \$HOME/.bcast5. If the user needs to modify ContextManual.pl, it should be done
on that copy from .bcast5 rather than the system-wide one. On all subsequent context help requests this
(possibly modified) .bcast5 version of the script will be called.  If later the \CGG{} package gets
upgraded containing a newer, not 100\% compatible version of ContextManual.pl script, and/or not compatible
structure of the HTML manual itself, the new system-wide version of the script will be copied into .bcast5
again to ensure context help functionality. The older version of the script will be backed up (with the .bak
suffix) as a reference of the modifications done by the user.

\section{Unique Blend plugins workflow}
\label{sec:ba_bp_workflow}

The uniqueness of the Blend Algebra/Blend Program plugins, different then most of the other plugins, provides a universal tool for creating new effects and may be handy in unexpected cases.
These 2 plugins provide the advanced feature of combining and modifying color pixels and transparencies of several tracks according to a mathematic algorithm written entirely by the user in the form of compact and simple piece of code. Such user defined algorithms (blend functions) are compiled, dynamically linked and immediately executed inside \CGG{} on the fly without the need to restart the application, reload the project, reattach plugins, and so on.
Because of this uniqueness, the Blend Algebra and Blend Program plugins workflow is documented here to explain how they work. Additional helpful information for developers can be found in the section on the Blend Algebra/Blend Program plugins themselves at \nameref{sub:blend_algebra}. You will most likely need to read parts to get a better understanding.

\subsection{Preparation phase}%
\label{preparatio_phase}

As in any realtime plugin, the \texttt{process\_buffer()} method in Blend Algebra/Blend Program gets a set of frames from the tracks the plugin is attached to. Then the following events take place. It is checked if the configuration (the parameters) got changed. There is one parameter which requires special treatment, the \textit{user function name}.

In order to prevent resource consuming recompilation of the functions on each new frame, the plugin maintains the successfully compiled and attached functions in cache. If at some keyframe the function name gets changed, the plugin searches if this function is already known and cached. In addition to important function related objects such as entry points, there is a timestamp representing  when this function was last checked to be up to date.

If the current function name is empty, it means a function is not used. Nothing else has to be done; all tracks are fetched and passed along in the processing pipeline unchanged. If the function is not empty and seen for the first time, or its timestamp is older than the global timestamp, it is checked as follows.

\begin{enumerate}
\item File lock is placed on the function source file to prevent concurrent modification of object files in case of several simultaneous compilations.
\item Compilation script \texttt{BlendAlgebraCompile.pl}/\texttt{BlendProgramCompile.pl} is started. The script checks if the resulting shared object file exists and is newer than the source and recompiles it if not.
\item The plugin checks if the shared object timestamp became newer than the timestamp of this function in cache (if any). If the cached version of the function in memory is up to date, it stays there. If not, the outdated function is detached from the plugin, the updated one is reattached, and its entry points are fetched and put into cache. The function's timestamp in cache is set to the current time since the function has just been checked.
\end{enumerate}

While recompiling or dynamic linking, various things may go wrong.
\begin{enumerate}
\item If in the unlikely scenario where the given function file does not exist, the program does nothing, same as for an empty function. No error message is shown in this case in order to prevent a possible deadlock. 
\item If recompilation was unsuccessful because of a syntax error, the error message is shown. More detailed diagnostics from the compiler can be seen in the terminal window in which \CGG{} was started. 
\item If compilation succeeded, but dynamic linking did not, the error message is shown. In case of any error, the failed function is marked with the current timestamp in cache so that the error messages appear only once before the global timestamp gets updated.
\end{enumerate}

Updating global timestamp forces all cached functions to be checked for recompilation when first accessed. The global timestamp is updated when the following events occur: changing function name, pressing \texttt{Edit...} or \texttt{Refresh} button, or exiting the \texttt{Attach...} dialog with the \texttt{OK} button.

If the current active function is up to date and attached, the plugin fetches video frames from all of the affected tracks with the important parameters like frame width and height. Then the \texttt{INIT} phase of the function is executed (once for each frame). Several important parameters are requested as defined by the function. They are:

\begin{itemize}
        \item Working color space needed inside the function. If it is not the same as the color space of the project, then color space conversions have to be done.
        \item The required number of tracks the function works on. If less than the required number of tracks is available, an error message is shown and the function is not executed.
        \item Whether the function supports parallelizing or not. If the function does not claim parallelizing support, it will be executed sequentially even if the \textit{PARALLE\_REQUEST} checkbox is ON in the plugin GUI.
\end{itemize}

\subsection{Processing phase}%
\label{processing_phase}

After the preparation phase, the processing itself takes place. If running sequentially instead of parallel, the following is done for each frame pixel individually.

For each input track, the corresponding pixel is split into color components according to the actual color space of the project. All color components are converted to float (type of the C language) in the ranges $[0.0 .. 1.0]$ for R, G, B, A, Y or $[-0.5 .. +0.5]$ for U, V. If the project color space has no alpha channel, the alpha component is set to A=1.0.

If the function uses a different color space than the project uses, the required conversions are performed. The key color components (selected in the plugin GUI) are also converted to the function color space in the same manner.

For Blend Algebra, the values for output are preinitialized from the track which the output is to go to. All the other tracks are cleared if the corresponding checkbox in the plugin GUI is checked. For Blend Program, this step is not needed.

\subsection{User function phase}%
\label{user_function_phase}

The user function is called with the parameters: actual \textit{number of tracks}, \textit{4 arrays for the 4 color components} (dimensions are equal to the number of tracks), \textit{4 key color components}, \textit{current pixel coordinates} (x, y: upper left frame corner is (0,0), lower right is (width-1,height-1)), and \textit{width and height}. In addition for Blend Algebra, there ae \textit{placeholders} for the result.

The user function returns. First of all, the color components of the relevant pixels are clipped to range if the corresponding checkbox in the plugin GUI is on. Relevant for Blend Program are pixels of all the tracks (all tracks can be modified); for Blend Algebra the result only.

After optional clipping, the color components are checked for not being NaNs.
If so, the pixel is replaced with the substitution color, then backconversion to the project color space is performed.

If the project has no alpha channel, but the function returned something not equal to 1.0, alpha channel is simulated as if an opaque black background were used.

If the project color space is not FLOAT, unconditional clipping followed by 8-bit transformation takes place.

For Blend Algebra, the result is placed in the right output track. For Blend Program, this step is unnecessary since all tracks are modified in place.

Then the loop is repeated for the next pixel in a row, next row in a frame, next frame in the whole sequence, and so on.

If the function is to be run parallelized, the necessary number of threads is created (as defined in \texttt{Settings $\rightarrow$ Preferences $\rightarrow$ Performance; Project SMP cpus}). Running parallel on 1 cpu is not exactly the same as running sequential because an extra processing thread is created, while sequential execution takes place inside the main plugin thread. This could induce some subtle differences if the function uses static variables inside or something else which is thread unsafe.