iOS: 处理来自 AVPlayer 视频轨道的音频

Question

我计划在我的 iOS 应用程序中重构我的录音系统。语境：到目前为止，我分别录制视频和音频，大约同时开始录制。录制完成后，同一个系统，我分别播放视频和音频，在音频上即时应用 AudioUnits。最后，我合并视频和修改后的音频。碰巧两条记录不同时开始（出于任何原因），产生了不同步的结果。

是否可以像这样重构我的系统：

1) Record normal video with audio into mov file --> I would be sure that audio+video would be synchronized.

2) During viewing the result with AVPlayer, process the audio part on the fly. (I will use AudioKit) --> that's the part I m not confident. 
   Would I be able to send the audio buffer to Audiokit (which would process it) and give back the processed audio to AVPlayer like if it was the original AVPlayer audio part?

3) Save a final file with video and audio modified --> easy part with AVFundation

请询问任何信息;)

Answer 1

我能想到一种相当简单的方法来做到这一点。

基本上你只需要在 AKPlayer 实例中打开你的视频文件。然后，您将视频音频静音。现在，您在 AudioKit 中有了视频音频。使用公共时钟将视频和音频锁定在一起非常简单。流程的伪代码：

// This will represent a common clock using the host time
var audioClock = CMClockGetHostTimeClock()

// your video player
let videoPlayer = AVPlayer( url: videoURL )
videoPlayer.masterClock = audioClock
videoPlayer.automaticallyWaitsToMinimizeStalling = false

....

var audioPlayer: AKPlayer?

// your video-audio player
if let player = try? AKPlayer(url: videoURL) {
    audioPlayer = player
}

func schedulePlayback(videoTime: TimeInterval, audioTime: TimeInterval, hostTime: UInt64 ) {
    audioPlay( audioTime, hostTime: hostTime )
    videoPlay(at: 0, hostTime: hostTime)
}


func audioPlay(at time: TimeInterval = 0, hostTime: UInt64 = 0) {
    audioPlayer.play(when: time, hostTime: hostTime)    
}

func videoPlay(at time: TimeInterval = 0, hostTime: UInt64 = 0 ) {
    let cmHostTime = CMClockMakeHostTimeFromSystemUnits(hostTime)
    let cmVTime = CMTimeMakeWithSeconds(time, 1000000)
    let futureTime = CMTimeAdd(cmHostTime, cmVTime)
    videoPlayer.setRate(1, time: kCMTimeInvalid, atHostTime: futureTime)
}

您可以按正常方式将播放器连接到任何 AudioKit 处理链。

当您想导出音频时，运行最终输出处理链上的 AKNodeRecorder。将其记录到文件中，然后将您的音频合并到视频中。我不确定正在处理的 AudioKit 离线处理是否已经准备就绪，因此您可能需要实时播放音频以捕获处理输出。

iOS: 处理来自 AVPlayer 视频轨道的音频

iOS: Process audio from AVPlayer video track

video

audiounit

ios

avplayer

audiokit