Much faster Avid ingest from any format

The venerable FFmpeg audio/video tool can now package its output in Avid Op-Atom format directly, without always needing to have its output wrapped by the raw2bmx tool. This method is very fast and, crucially, can be used on any computer; not just the machine with your Avid licence. However, certain features of the Avid Op-Atom MXF wrapper are either not yet tested, or not available. For these features, I still advise using the bmxlib suite.

The advantage of this method is that video and audio data is very quickly imported into your Avid, at the full rate that the FFmpeg encoder can manage. Furthermore, your Avid will be using its native formats (e.g. DNxHD), rather than converting, say, XAVC on-the-fly with the AMA functions. The disadvantage is that metadata is quite messy, and lacking certain elements altogether, until I’ve figured out the full MXF Op-Atom metadata tags. Particularly, audio and video tracks are not linked into a single multi-track clip in an Avid bin: you must synchronise them yourself.

So, in this article, I will show you how to take video and/or audio from any format that FFmpeg will read, and output immediately MXF files in Avid-friendly codecs that Avid Media Composer’s “Media Tool” will pick up for you.

This post shows my earliest tests. I have not yet fully explored how to link files, or add other attributes, that raw2bmx adds or, indeed, Avid’s own import and capture tools add. However, the advantage of not using raw2bmx is a much faster import process.

Here, we start with a file of any format with a mono soundtrack, and output Avid DNxHD files at a resolution of 1280×720 and a bit-rate of 90Mbit/s, straight into the Avid MediaFiles directory, and ready for editing. This workflow assumes the frame-rate is for UK television, at 25fps, progressive encoding. I have not attempted to detect the number of audio channels in use, and therefore they are not encoded separately. This is, however, easy to achieve by using FFmpeg’s “asplit” audio filter and is detailed on the FFmpeg website.

ffmpeg -i "MY_CLIP.f4v" -vf scale=1280:720:lanczos -an -metadata project="MY PROJECT" -metadata material_package_name="MY CLIP" -b:v 90M -f mxf_opatom "M:\Avid MediaFiles\MXF\1\MY_CLIP_v1.mxf" -vn -metadata project="MY PROJECT" -ac 1 -ar 48000 -metadata material_package_name="MY CLIP" -f mxf_opatom "M:\Avid MediaFiles\MXF\1\MY_CLIP_a1.mxf"

Here is a break-down of that command line:

-i "MY_CLIP.f4v"

Here is the incoming clip.

-vf scale=1280:720:lanczos

We start with producing an output video MXF. Here, we ensure that the video is resized to the particular flavour of HD we’re editing in. In this case, we’re using 720p, and resizing using the algorithm I consider to be the best.

-an

The video output can contain only one track: video, in this case. This command instructs that the output must contain no audio. (Literally: “audio, none”)

-metadata project="MY PROJECT"

Here, we embed into the MXF metadata a value for “project”. This corresponds to your Avid project name. It is true that, upon analysing Avid’s own MXF files, the project name is contained within the metadata tag “project_name”, but my shorter tag appears also to work.

-metadata material_package_name="MY CLIP"

This is the name of your clip, as it will appear in your Avid bins and in the Media Tool.

-b:v 90M

Here, set the bit-rate. Using FFmpeg’s built-in DNxHD encoder, you can choose from several bit-rates, which FFmpeg’s error messages will be happy to tell you about if you set this wrongly. We don’t explicitly set the encoder itself, because FFmpeg does that for you: its default for this muxer is DNxHD.

-f mxf_opatom

This explicitly instructs FFmpeg to wrap your data in an Op-Atom MXF wrapper, ready for your Avid Media Composer to use.

"M:\Avid MediaFiles\MXF\1\MY_CLIP_v1.mxf"

Finally, for the video file, here is the output file. In this case, I’m putting it straight into Avid’s media file storage area on my ‘M’ drive, for media. There is a "_v1" suffix so that the file is marked as a video file, for my own ease of comprehension.

-vn

After the video output file is named, we start listing the options for encoding the audio file. This first option instructs FFmpeg to produce an audio-only MXF file, without a video track. (Literally: “video, none”)

-metadata project="MY PROJECT"

As with the video file, we embed into the metadata the name of the Avid project that needs this file.

-metadata material_package_name="MY CLIP"

As above, this is the clip name as it will appear in your Avid bins or Media Tool.

-ac 1

This is a quick-and-dirty kludge to mix down all the incoming audio tracks to a single track. In real life, you’ll want to use FFmpeg’s filter “asplit” to handle each incoming audio track separately. At the moment, this command line produces only a single audio file, mixing together all incoming tracks.

-ar 48000

With this command, we convert the sample-rate of the incoming audio to the standard sampling rate for television: 48,000 samples per second. Avid can, of course, convert sample rates on-the-fly while editing, but it is better to perform this work at the import stage to give Avid less to do when editing. Again, the codec itself (pcm_s16le, meaning linear PCM, 16-bit, little-endian) is not explicitly specified because FFmpeg sets this as the default for Avid import.

-f mxf_opatom

As before, this explicitly instructs FFmpeg to wrap your data in an Op-Atom MXF wrapper, ready for your Avid Media Composer to use.

"M:\Avid MediaFiles\MXF\1\MY_CLIP_a1.mxf"

Here is the filename for the audio output. In this example, I have placed it in the same directory as the video output created earlier in this command line. It is suffixed with "_a1" for my own ease of comprehension.

There are additional options associated with this FFmpeg muxer, which are listed here. I have not experimented with these, but can see that the -mxf_audio_edit_rate might need to be adjusted for non-European television or film work e.g. 30000/1001 for American television work, or 24 or 24000/1001 for 24fps film work. Also, you would probably want to set the -signal_standard bt601 or -signal_standard 1 for standard definition television work.

Muxer mxf_opatom [MXF (Material eXchange Format) Operational Pattern Atom]:

Common extensions:
mxf
Mime type:
application/mxf.
Default video codec:
dnxhd.
Default audio codec:
pcm_s16le.

MXF-OPAtom muxer AVOptions:

-mxf_audio_edit_rate
Audio edit rate for timecode (from 0 to INT_MAX) (default 25/1)
-signal_standard
Force/set Sigal Standard (from -1 to 7) (default -1)
bt601
ITU-R BT.601 and BT.656, also SMPTE 125M (525 and 625 line interlaced)
bt1358
ITU-R BT.1358 and ITU-R BT.799-3, also SMPTE 293M (525 and 625 line progressive)
smpte347m
SMPTE 347M (540 Mbps mappings)
smpte274m
SMPTE 274M (1125 line)
smpte296m
SMPTE 296M (750 line progressive)
smpte349m
SMPTE 349M (1485 Mbps mappings)
smpte428
SMPTE 428-1 DCDM

6 thoughts on “Much faster Avid ingest from any format”

  1. Hi, I am playing around with your code, thanks.
    When I get the mxf into avid it applys a frameflex effect to it. Do you have this experience as well?

    1. Hello Laurance. Thanks for trying my method. I don’t see the frameflex effect going onto the clip here. Can you tell me, please: (1) which version of Avid MC you’re running; (2) what the frame rate of your original clip is before you import it; (3) the frame rate and format of your project? I’m using 25fps progressive projects at both 720p and 1080p and running MC 8.2, not upgraded yet because a project that’s taken over a year to complete is currently being finished and I’m wary of changing Avid versions mid-task. If you have even a short extract of your original clip that you can send me (possibly prepared using ffmpeg -i YOURCLIP.ORIGINALEXT -acodec copy -vcodec copy -tt 00:00:03.000 SHORTCLIP.ORIGINALEXT) it might help. (That command should produce a three-second clip consisting of the front of your clip)
      J

  2. I’ve been interesting in transcoding with this method.
    Is this method still current?
    What would be the command in anothergui? So I can just drag and drop and select my destination. Thank You for the great info.

  3. Wonderful post. I’ve just started testing with the mxf muxer and although the mxf is working in MC, I have some issues with the metadata. The ‘project_name’ or ‘project’ metadata do not get set in the output file. I have tried both to no avail. Has there been a change to this process which still allows for this tag to be set? Using the latest ffmpeg build as of 5th Nov 2018. Thanks.

  4. I’ve been experimenting with this and overall it works, but my files come in as two separate VIDEO and AUDIO master clips.

    Am I doing something incorrectly? How do I do this so avid knows they’re meant to be the same master clip?

    1. I am basicly stuck at the exact same point as Russell. I made quite a lot of research on importing different types of EDL/ AAF/ XML/ OMF and so on….
      but none of these work without a ridiculous ammount of work (aafs are binary, and need ID Values that need to be calculated, and all of the other solutions dont work for different reasons. )
      btw, u should take a loo at this: http://www.ffastrans.com/
      You can build great automated workflows with it, and it has a tool to include your own code quite effectively.
      I dont really know how to go on anymore, and would appreciate any kind of help. I work at a media production company and we have a LOT of Footage and Dailies to Transcode, so my goal is to automate as much of this stupid task as possible.
      For those u have a use for it, i made a small (and EXTREMELY dirty) powershell script, which i have no real use for (and therefore no intention to clean up).
      #Import Settings
      Get-Content “$PSScriptRoot\Transcoding Settings.txt” | foreach-object -begin {$h=@{}} -process { $k = [regex]::split($_,’=’); if(($k[0].CompareTo(“”) -ne 0) -and ($k[0].StartsWith(“[“) -ne $True)) { $h.Add($k[0], $k[1]) } }

      if (!(Test-Path $h.TargetLocation)) {
      New-Item -Path $h.TargetLocation -ItemType Directory
      }
      Set-Location -Path $PSScriptRoot;
      $ffmpegPath = “$PSScriptRoot\ffmpeg.exe”
      $oldVideos=Get-ChildItem -Include @(“*.mp4”, “*.avi”, “*.divx”, “*.mov”, “*.mpg”, “*.wmv”, “*.mkv”, “*.mxf”) -Path $h.SourceLocation -Recurse;
      foreach ($oldVideo in $oldVideos) {
      $oldVideoFile = (Get-Item $oldVideo).BaseName
      $newVideo = [io.path]::ChangeExtension($oldVideo.FullName, ‘newv.mxf’)
      $newVideoFile = Split-Path -Path $newVideo -Leaf
      $newVideoFilePath = “{0}\$newVideoFile” -f $h.TargetLocation;
      $newAudio1 = [io.path]::ChangeExtension($oldVideo.FullName, ‘newa1.mp3’)
      $newAudioFile1 = Split-Path -Path $newAudio1 -Leaf
      $newAudioFilePath1 = “{0}\$newAudioFile1” -f $h.TargetLocation;
      $newAudio2 = [io.path]::ChangeExtension($oldVideo.FullName, ‘newa2.mp3’)
      $newAudioFile2 = Split-Path -Path $newAudio2 -Leaf
      $newAudioFilePath2 = “{0}\$newAudioFile2” -f $h.TargetLocation;
      $newAudio3 = [io.path]::ChangeExtension($oldVideo.FullName, ‘newa3.mp3’)
      $newAudioFile3 = Split-Path -Path $newAudio3 -Leaf
      $newAudioFilePath3 = “{0}\$newAudioFile3” -f $h.TargetLocation;
      $newAudio4 = [io.path]::ChangeExtension($oldVideo.FullName, ‘newa4.mp3’)
      $newAudioFile4 = Split-Path -Path $newAudio4 -Leaf
      $newAudioFilePath4 = “{0}\$newAudioFile4” -f $h.TargetLocation;
      $newAudio5 = [io.path]::ChangeExtension($oldVideo.FullName, ‘newa5.mp3’)
      $newAudioFile5 = Split-Path -Path $newAudio5 -Leaf
      $newAudioFilePath5 = “{0}\$newAudioFile5” -f $h.TargetLocation;
      $newAudio6 = [io.path]::ChangeExtension($oldVideo.FullName, ‘newa6.mp3’)
      $newAudioFile6 = Split-Path -Path $newAudio6 -Leaf
      $newAudioFilePath6 = “{0}\$newAudioFile6” -f $h.TargetLocation;
      $newAudio7 = [io.path]::ChangeExtension($oldVideo.FullName, ‘newa7.mp3’)
      $newAudioFile7 = Split-Path -Path $newAudio7 -Leaf
      $newAudioFilePath7 = “{0}\$newAudioFile7” -f $h.TargetLocation;
      $newAudio8 = [io.path]::ChangeExtension($oldVideo.FullName, ‘newa8.mp3’)
      $newAudioFile8 = Split-Path -Path $newAudio8 -Leaf
      $newAudioFilePath8 = “{0}\$newAudioFile8” -f $h.TargetLocation;

      $StreamArgumentsa = ‘-map 0:a:0? -map_metadata 0:g? -copyts “{0}” -map 0:a:1? -map_metadata 0:g? -copyts “{1}” -map 0:a:2? -map_metadata 0:g? -copyts “{2}” -map 0:a:3? -map_metadata 0:g? -copyts “{3}” -map 0:a:4? -map_metadata 0:g? -copyts “{4}” -map 0:a:5? -map_metadata 0:g? -copyts “{5}” -map 0:a:6? -map_metadata 0:g? -copyts “{6}” -map 0:a:7? -map_metadata 0:g? -copyts “{7}”‘ -f $newAudioFilePath1, $newAudioFilePath2, $newAudioFilePath3, $newAudioFilePath4, $newAudioFilePath5, $newAudioFilePath6, $newAudioFilePath7, $newAudioFilePath8;
      $MetadataStatement1 = ‘-metadata material_package_name=”‘
      $MetadataStatement2 = ‘”‘
      $ArgumentListv = ‘-i “{0}” {1} {2}{3}{4} “{5}”‘ -f $oldVideo, $h.FormatArgumentsv, $MetadataStatement1, $oldVideoFile, $MetadataStatement2, $newVideoFilePath;
      $ArgumentLista = ‘-i “{0}” {1} {2}’ -f $oldVideo, $h.FormatArgumentsa, $StreamArgumentsa;
      Write-Host -ForegroundColor Green -Object $ArgumentListv;
      Write-Host -ForegroundColor Green -Object $ArgumentLista;

      # Pause the script until user hits enter
      # $null = Read-Host -Prompt ‘Press enter to continue, after verifying command line arguments.’;

      # Kick off ffmpeg
      Start-Process -FilePath “$ffmpegPath” -ArgumentList $ArgumentListv -Wait -NoNewWindow;
      Start-Process -FilePath “$ffmpegPath” -ArgumentList $ArgumentLista -Wait -NoNewWindow;
      }

      $oldAudios = Get-ChildItem -Include @(“*.mp3”) -Path $h.TargetLocation -Recurse;
      foreach ($oldAudio in $oldAudios) {
      $newAudio = [io.path]::ChangeExtension($oldAudio.FullName, ‘.mxf’)
      $newAudioFile = Split-Path -Path “$newAudio” -Leaf
      $newAudioFilePath = “{0}\$newAudioFile” -f $h.TargetLocation;
      $ArgumentListre = ‘-i “{0}” {1} {2}{3}{4} “{5}”‘ -f $oldAudio, $h.FormatArgumentsre, $MetadataStatement1, $oldVideoFile, $MetadataStatement2, $newAudioFilePath;
      Write-Host -ForegroundColor Green -Object $ArgumentListre;

      # Kick off ffmpeg
      Start-Process -FilePath “$ffmpegPath” -ArgumentList $ArgumentListre -Wait -NoNewWindow;
      Remove-Item “$oldAudio”
      }

      It requires a settings file in its location:

      [Path]
      TargetLocation=X:\04 ENCODING\Test\out
      SourceLocation=Z:\04 ENCODING\test\in
      [Arguments]
      FormatArgumentsv=-an -f mxf_opatom -vcodec dnxhd -vb 120M -pix_fmt yuv422p -flags ildct -threads 12
      FormatArgumentsa=-vn -ar 48000 -acodec libmp3lame -threads 12
      FormatArgumentsre=-f mxf_opatom -threads 12

Leave a Reply to carlos d Cancel reply

Your email address will not be published. Required fields are marked *