\label{sec:batch_rendering}
\index{batch rendering}
-Batch Rendering automates the rendering of audio/video files in that
+Batch Rendering as implemented in \CGG{} is considered to be more of
+an advanced feature and careful usage is advised. It automates the
+rendering of audio/video files in that
you can establish a set of job parameters, save them, and use them
-repeatedly. It also allows for \CGG{} to be run by external
-programs, with no need for the user to manually interact with the
-user interface (figure~\ref{fig:batch01}).
+repeatedly (figure~\ref{fig:batch01}). It also allows for \CGG{} to
+be run by external programs, with no need for the user to manually
+interact with the user interface.
\begin{figure}[htpb] \centering
\includegraphics[width=1.0\linewidth]{batch01.png}
\item[Save Jobs] when you have set up the batch jobs the way you
want and you think you may have to run them more than once, it is
beneficial to save the jobs for later use so you easily run them
- again.
+ again. It is recommended to use a filename with .rc as the extension
+ so that it is obvious that it is a list of batch jobs to be run.
\item[Load Jobs] reload a previous set of saved jobs. This can come
in handy if you did not have the time to render them when you
originally set them up, if you need to rerun, or if you got
You can automate \CGG{} batch renders from other programs. In the
batch render dialog, once you have created your list of batch render
jobs, you can click the button \textit{Save Jobs} and choose a file
-to save your batch render list to. Once you have created this file,
+to save your batch render list to. It is recommended that you use
+a filename with the extension of .rc in order to make it obvious that
+this is a list of batch jobs to render. Once you have created this file,
you can start up a batch render without needing to interact with the
\CGG{} user interface. From a shell prompt, from a script, or other
program, execute:
\begin{lstlisting}[style=sh]
-{path_to_cinelerra}/cin -r batchjob.xml
+{path_to_cinelerra}/cin -r batchjob.rc
\end{lstlisting} substituting your actual filename for
-\texttt{batchjob.xml}. When invoked with these parameters, \CGG{}
-will start up and perform the rendering jobs in that list, without
-creating its usual windows.
+\texttt{batchjob.rc}. \textbf{Warning} this file will be modified
+so if you use any filename that is not a legitimate list of batch jobs to
+render, that file will be overwritten and its previous contents destroyed.
+When invoked with these parameters, \CGG{} will start up and run the
+rendering jobs in the list contained in that file starting at the defined
+\textit{active region}, without creating its usual windows. If you do not
+specify a filename, the default will be \$HOME/.bcast5/batchrender.rc.
+Possible messages you might see where you started up the job are as follows.
+\begin{description}
+\item[The following files exist: - filename - Won't overwrite existing files] that batch job will not run in order to prevent writing over previous run.
+\item["filename" No such file or directory] the specified batch job file does not exist.
+\item["filename": Permission denied] the specified batch job file does not have write permission so can not be updated.
+\item[Render::run: filename] the batch job with the name of filename will be processed.
+\item[** rendered 0 frames in 0.000 secs, 0.000 fps] either you used a file that is not a list of batch jobs or the batch jobs within the file were not enabled.
+\end{description}
\subsection{Advanced features}%
\label{sub:advanced_features}
\index{batch rendering!more options}
-\textbf{Warning}: one of these advanced functions overwrites the original EDL, which will then be lost!
+\textbf{Warning}: \textit{Save to EDL path} overwrites the current EDL thus destroying the original contents.
Although the operation of Batch Rendering in \CGG{} is similar to that of other NLEs, there is one big difference that we need to take into account. The render setup is not done on a project-by-project basis, which are then brought into the Batch window to be rendered automatically. The setup must be done in the Batch rendering window, where various projects are loaded and set up. In the case of similar projects, derived from a single EDL with some variation, this mode offers the possibility of altering the projects without having to open each individual project, make the changes, set up the rendering, save and import into the Batch window. The procedure is to select the batch we want to modify in the Batches to render window; operate on the currently open timeline (even if it does not correspond to the one we want to modify) making the desired changes and then press the \textit{Save to EDL path} button. Thus the chosen batch, while retaining its original name, will now contain the modified project. Since this possibility destroys the original EDL overwriting it with the modified one, you must be very careful. This procedure is convenient in case the batches are similar, i.e. they are variations of the same EDL, where we want to experiment with other effects, other output formats or when trying out various cuts of a DVD/BD before the final production. It might also be useful to use an \textit{active region} of the timeline, so as to speed up rendering times but still have an indicative result for comparison. Instead operating on different projects, we can do a \textit{save as...} of the project on the timeline to have a new EDL with a new name and then replace it with the batch selected in the joblist using the \textit{Use Current EDL} button. The new project (with its name) overwrites the original project.
if any output files already exist.
To perform rendering from the command line, first run \CGG{} in
-graphical mode. Go to \texttt{File $\rightarrow$ Batch
- Render}. Create the batches you intend to render in the batch window
-and close the window. This saves the batches in a file. Set up the
+graphical mode. Go to \texttt{File $\rightarrow$ Batch Render}.
+Create the batches you intend to render in the batch window and
+close the window. This automatically saves the batches in a file
+with the name of \$HOME/.bcast5/batchrender.rc. Set up the
desired render farm attributes in \texttt{Settings $\rightarrow$
Preferences} and quit out of \CGG{} if you want to use the Render
Farm capability. These settings are used the next time command line
rendering is used to process the current set of batch jobs without a
-GUI\@.
+GUI\@. It is important to remember that the rendering will begin at
+the defined \textit{active region} saved when the project was saved.
On the command line run:
path. Since hundreds of thousands of image files are usually
created, ls commands will not work in the background rendering
directory. The browse button for this option normally will not work
- either, but the configuration button for this option works.
+ either, but the configuration button for this option works. The
+ default value will be /tmp/brender .
\item[File format] The file format for background rendering has to
be a sequence of images. The format of the image sequences
determines the quality and speed of playback. JPEG generally works
- well.
+ well and is the default.
\end{description}
+Tip: If you have rendered your whole project with \textit{File format}
+set to JPEG and there are no missing numbers in the sequence, you can
+create a video from that sequence outside of \CGG{}.
+For example, if using the default output so that your files are named
+/tmp/brender000000, /tmp/brender000001, ... in a window, you would type:
+
+\begin{lstlisting}[style=sh]
+ffmpeg -f image2 -i /tmp/brender0%5d -c:v copy brender.mov
+\end{lstlisting}
+which would create the video file brender.mov - be sure to delete
+existing brender files before creating a new sequence to ensure there
+are no missing numerical values in the sequence.
\section{Render Farm Usage}%
\label{sec:render_farm_usage}
video is rendered in the background, to speed up rendering
significantly. Because rendering is memory and cpu intensive, using
multiple computers on a network via a render farm is a significant
-gain. With \CGG{} installed on all nodes, the master node and the
-clients communicate via a network port that you specify.
-
+gain. With \CGG{} installed on all nodes, the master node and the clients communicate via a network port that you specify.
+The Master node is the one of the instance of \CGG{} that we normally start with its gui; the other nodes are the instances of \CGG{} that we decide to start in parallel from the terminal to create the Render farm (clients).
\CGG{} can distribute the rendering tasks over the network to the
-other computers of the Render Farm. The render farm software tries
+other computers of the Render Farm; or among all threads of a multicore CPU. The render farm software tries
to process all of the rendering in parallel so that several
computers can be used to render the results. The \textit{Total jobs
- to create} in the setup or labels on the timeline are used to divide
+to create} in the setup or labels on the timeline are used to divide
a render job into that specified number of tasks. Each background
job is assigned a timeline segment to process and the jobs are sent
to the various computer nodes depending upon the load balance. The
The following steps are just a guideline to start your render farm.
It is assumed that you already have the master and client nodes
communication, shared filesystem, permissions and usernames synched.
+Let's take the example of a network with 2 PCs: the master and the client. On the client we set 5 tasks; on the master we set 2 tasks.
\begin{enumerate}
\item On the master computer, use \texttt{Settings} $\rightarrow$
\item in the \textit{Hostname} box, keyin your hostname or ip
address such as 192.168.1.12 or \textit{localhost};
\item enter in a port number such as 401--405 (only a root user
- can use privileged ports) or $1025$ and click on \textit{Add Nodes};
+ can use privileged ports) or $10650...$ for non-root and click on \textit{Add Nodes}. To find a range of free ports to use you can look at the file \texttt{/etc/services};
\item you will see something like the following in the Nodes
listbox to the right:\newline
\begin{tabular}{lllc} On & Hostname & Port & Framerate
\\\midrule
- X & 192.168.1.12 & 401 & 0.0 \\
- X & 192.168.1.12 & 402 & 0.0 \\
- X & 192.168.1.12 & 403 & 0.0 \\
- X & 192.168.1.12 & 404 & 0.0 \\
- X & 192.168.1.12 & 405 & 0.0 \\
- X & localhost & 406 & 0.0 \\
- X & localhost & 407 & 0.0 \\
+ X & 192.168.1.12 & 10650 & 0.0 \\
+ X & 192.168.1.12 & 10651 & 0.0 \\
+ X & 192.168.1.12 & 10652 & 0.0 \\
+ X & 192.168.1.12 & 10653 & 0.0 \\
+ X & 192.168.1.12 & 10654 & 0.0 \\
+ X & localhost & 10655 & 0.0 \\
+ X & localhost & 10656 & 0.0 \\
\end{tabular}
- \item set the Total number of jobs to create;
+ \item set the Total number of jobs to create. This number only pertains to client nodes, so we do not need to consider the master node;
\item click OK on the bottom of the Preferences window.
\end{itemize}
-\item Now we must join the nodes created to instances of \CGG{}. On the client computers ($192.168.1.12$), on the terminal, start 5 background \CGG{} tasks via:
+\item For external network nodes, now we must join the nodes created to instances of \CGG{}. On the client computers ($192.168.1.12$), on the terminal, start 5 background \CGG{} tasks via:
\begin{lstlisting}[style=sh]
cd {path_to_cinelerra}
-cin -d 401 cin -d 402
+cin -d 10650 cin -d 10651
...
-cin -d 405
+cin -d 10654
\end{lstlisting}
-\item Similarly, on the terminal, we must join the local nodes created to instances of \CGG{}. On the master node (localhost), start the 2 background \CGG{} tasks via:
+In practice, at each instance that we start, the cursor will be positioned in a new line ready to enter the next command, without exiting the task. If we have several PCs on the network, repeat these steps for each new client (with its own ip address).
+\item Similarly, on the terminal, we must join the local nodes (internal to the Master node) created to instances of \CGG{}. On the Master node (localhost), start the 2 background \CGG{} tasks via:
\begin{lstlisting}[style=sh]
cd {path_to_cinelerra}
-cin -d 406
-cin -d 407
+cin -d 10655
+cin -d 10656
\end{lstlisting}
+Similar to the previous point, the cursor positions itself in a new line ready to enter the next command, without exiting the task.
\item When your video is ready, setup a render job via \texttt{File
$\rightarrow$ Render} or \texttt{File $\rightarrow$ Batch Render}
and check OK.
Scripts icon.
\item If you plan on doing more rendering, you can just leave the
master/client jobs running to use again and avoid having to restart
- them. Or you can kill them when you no longer are using them.
+ them. You can also close the terminal, but the jobs will remain active until you turn off the PC. Or you can kill them when you no longer are using them.
\end{enumerate}
\subsection{Render Farm Menu and Parameter Description}%
existing node or enter a new node.
\item[Port] keyin the port number of an existing or new node here.
You can also type in a range of port numbers using a hyphen, for
- example $1501-1505$ when you need to add many.
+ example $10650-10799$ when you need to add many.
\item[Apply Changes] this will allow you to edit an existing node
and to then commit the changes to hostname and port. The changes
will not be committed if you do not click the OK button.
\end{lstlisting} As it completes its jobs, you will should see:
\begin{lstlisting}[style=sh]
RenderFarmClientThread::run: Session finished
-\end{lstlisting} A quick way to start a sequence of clients is to
- use:
-\begin{lstlisting}[style=sh,mathescape]
-for n in `seq 1501 1505`; do
- cin -d $\$$n
-done
\end{lstlisting}
\item[Render Using Render Farm] After you have followed the
preceding steps, you are ready to use the render farm. Click on
\item[Assemble the Output Files] Once all of the computer jobs are
complete, you can put the output files together by using the shell
script, \textit{RenderMux} (from the menubar \textit{scripts} button
- just above FF), if the files were rendered using ffmpeg, or you can
- load these by creating a new track and specifying concatenate to
+ just above FF), if the files were rendered using ffmpeg (see \nameref{sec:menu_bar_shell_commands}).
+
+ If you want to remain within the open project in \CGG{}, you can load these files by creating a new track and specifying concatenate to
existing tracks in the load dialog in the correct numerical order.
File types which support direct copy can be concatenated into a
single file by rendering to the same file format with render farm
disabled as long as the track dimensions, output dimensions, and
asset dimensions are equal.
+
+ Finally if you want to use ffmpeg from terminal externally to \CGG{} you can go to the directory where the rendered files are, it creates a text file \texttt{list.txt} containing the list of all our files:
+ \begin{lstlisting}[style=sh]
+ file 'name.webm001'
+ file 'name.webm002'
+ ...
+ file 'name.webm00n'
+ \end{lstlisting}
+ and finally give the command
+ \begin{lstlisting}[style=sh]
+ ffmpeg -f concat -i list.txt -c copy output.webm
+ \end{lstlisting}
\end{description}
\subsection{Quick and Easy Render Farm Setup – The Buddy System
add the plugins, such as a Title, etc.
\item Check for a set of unused ports in \texttt{/etc/services}
file, if username is root usually $401-425$ are available; if
- non-root, then $1024-1079$.
+ non-root, then $10650-10799$.
\item On the master computer, in \texttt{Settings $\rightarrow$
Preferences, Performance} tab:
\begin{itemize}
/{cinelerra_pathname}/cin -d port#
\end{lstlisting}
\item On the master, bring up the render menu and name the output
- files, for example \texttt{/tmp/myoutput.mp4}.
+ files, for example \texttt{/tmp/myoutput.webm}.
\item The client nodes output results will be on their local
\texttt{/tmp} filesystems so you will have to again use
\textit{rsh/ftp} or \textit{usb sneaker net} to move them over to
the main computer. File names will be the render job output file
name with port number tacked on
- (e.g. \texttt{/tmp/hb.mp4001...mp4005}).
+ (e.g. \texttt{/tmp/hb.webm001...webm005}).
\item Load the files by concatenate to existing track on the master
node or use RenderMux shell script.
\end{enumerate}
If you are lucky enough to have a computer with a large cpu core
count, setting up a render farm can really take advantage of using
-all of the cpus. This is much faster than the default automatic
+all of the cores at 100\%. This is much faster than the default automatic
threading capability. Since you don’t need to communicate with other
computers, you will not have to be concerned about TCP communication
-or shared disks/files; only localhost nodes. On the terminal, we will open many instances of \CGG{} by connecting them to the jobs created. The number of such jobs can be the total number of CPU threads or not. When you are going to be doing other work
+or shared disks/files; only localhost nodes. On the terminal we will open many instances of \CGG{} by connecting them to the jobs created. The number of such jobs can be the total number of CPU threads (-1) or not. When you are going to be doing other work
simultaneously while rendering a large job, you will want to leave
some of the cpus available for that. Be sure to set \textit{Project SMP
cpus} in the \texttt{Settings $\rightarrow$ Preferences, Performance} tab to your CPU
section of the video so keep that in mind when designating port
numbers.
\item If not running as root, a port number in the higher range of
- $1024$ and above must be used instead of the $400+$ range.
+ $10650$ and above must be used instead of the $401+$ range.
\item The master and client jobs on the ports do not go away so if
you want to stop them, you will have to kill them via: \texttt{kill
PID\#}.
time. You can adjust the timer in \texttt{Settings $\rightarrow$
Preferences, Performance} tab.
\item When you get the message \texttt{RenderFarmClient::main\_loop:
- bind port 400: Address already in use}, use a different port.
+ bind port 400: Address already in use}, use a different port. See \texttt{/etc/services} for free ports.
\item A message of \texttt{RenderFarmServerThread::open\_client:
unknown host abcompany} means that the hostname of abcompany is not
in \texttt{/etc/hosts} so you will have to add it or use the ip
projects / goodguy / cin-manual-latex.git / blobdiff