Translate any video or audio content with natural sounding translations and voices.

OPENAI_API_KEY

ELEVEN_LABS_API_KEY

source_file

A comma-separated list of languages to which the audio will be translated. Default is "spanish". See supported languages in README.

target_language

The translation engine to use. Default is "gpt4". Supported backends are "seamless" and "gpt4". See the README for more information.

translation_engine

The TTS engine to use. Default is "elevenlabs". Supported backends are "elevenlabs" and "openai". See the README for more information.

voice_engine

Whether to return the transcript of the video along that includes the translated text. Default is False.

return_transcript

Whether to preserve the background audio. If true, the background audio will be added to the dubbed video. Default is True.

preserve_background_audio

A comma-separated list of words that should not be translated. Default is "".

safewords

a JSON object that allows more control over how words are translated. See README for more information on using this parameter.

translation_dictionary

The start time of the media to dub from. Default is 0.

start_time

The end time of the media to dub to. If -1, the end time is the length of the media.

end_time

(ALPHA FEATURE) Whether to enable lip-syncing on the original video to the dubbed audio. Defaults to False. Only applicable for video inputs.

enable_lipsyncing

dubbing

Dubbing

Smart, automatic cropping of a video to a given aspect ratio based on subject detection and speaker tracking.

file

whether or not to determine crops based on who's actively speaking in the video. Defaults to False.

active_speaker_detection

the aspect ratio of the video to create. Must be "1:1", "4:3", "3:4", "4:5", "5:4", or "9:16". You may also specify multiple aspect ratios separated by commas to get metadata for each one. Defaults to "9:16".

aspect_ratio

whether or not to return the fully edited video file or just the metadata so you can render on your own. Defaults to False. If multiple aspect ratios are specified, while return_video is True, only the first aspect ratio will be used.

return_video

the seconds into the video to start processing from. Defaults to 0.

the seconds into the video to stop processing at. Defaults to -1, which means the end of the video.

whether or not to use a faster, but less accurate, model for the object detection that powers the cropping. Defaults to False.

speed_boost

whether or not to let the function auto-pick the main subject to focus on. Note: this setting is still experimental but it could yield better results on a broader range of videos. Defaults to False.

smart_edit

whether or not to return a visualization of the crops and bounding boxes overlaid onto the original video (for debugging purposes). Defaults to False.

visualize

whether or not to include the information about the subjects (people, faces, etc) of the crops in the metadata. Defaults to False.

include_subjects

whether or not to include the versions of the layouts if speaker detection was off in the output metadata. This is useful if you want the option to switch between active speaker and non-active speaker layouts without having to reprocess the video. Defaults to False and is only used if active_speaker_detection is True and return_video is False.

include_non_active_layouts

a comma-separated list of natural language class prompts to detect in the video. Defaults to "person". You can specify any class in natural language.

prompt

a comma-separated list of natural language class prompts to exclude from the detection in the video. Defaults to "".

negative_prompt

whether or not to only allow single crop layout versus multi-crop layouts. Defaults to False.

single_crop_only

the speed at which the crops moves from one frame to another. Defaults to 0.1. Must be between 0 and 1.

crop_movement_speed

the number of seconds between each sample taken for an updated crop position. Should be lower for faster moving subjects, and higher for still subjects. Defaults to 3.

crop_sampling_interval

whether or not to return the scene data in the output metadata. Defaults to False.

return_scene_data

the minimum length of a scene in seconds. Short scenes will be merged with succeeding scenes until the minimum length is reached. Defaults to 0.0 (no merging or minimum scene length).

min_scene_length

autocrop

Auto Crop

A comprehensive solution for video lipsyncing with a suite of different model and enhancements options.

Audio file to sync with the driver image/video file.

audio

Whether to enhance the output using the default enhancers like CodeFormer. Enabled by default. More info in README.

enhance

Backend to use for lipsync. Default is "SieveSync". SieveSync is a custom backend, recommended for the best performance. More info in README.

backend

Whether to downsample the video to 720p. Default is true. More info in README.

downsample

Method of cutting the audio and video when their durations mismatch. Default is "audio" which cuts the video to the audio's duration. "video" cuts the audio to the video's duration. "shortest" cuts the longer file to the shorter file's duration. More info in README.

cut_by

lipsync

Lipsync

State-of-the-art audio-visual active speaker detection based on new, efficent face and speaker detection models.

Whether to use the faster but less accurate object detection model when processing the video.

The maximum number of faces to return per frame. If there are more than this number of faces, only the largest x faces will be returned.

max_num_faces

Whether to only return the frame data at the start of each scene cut or to return the frame data for every frame in the video.

return_scene_cuts_only

Whether to return the scene data along with the frame data. If True, the scene data will be returned in the "related_scene" field of the output.

The seconds into the video to start processing from. Defaults to 0.

The seconds into the video to stop processing at. Defaults to -1, which means the end of the video.

The framerate to run object detection at for speaker detection. Defaults to 2.

processing_fps

A threshold that determines the minimum size of a face to run speaker detection on. Defaults to 0.5. Lower values allow for smaller faces to be used.

face_size_threshold

Active Speaker Detection

Fast, high quality speech transcription with many available backends, word-level timestamps, speaker diarization, and translation capabilities.

The transcription model to use, see README for more information.

Whether to return word-level timestamps. Defaults to True.

word_level_timestamps

Language of the audio. Defaults to auto-detect if not specified. See README for supported language codes.

source_language

The backend to use for speaker diarization. Defaults to "None".

diarization_backend

Minimum number of speakers to detect for diarization. Defaults to auto-detect when set to -1.

min_speakers

Maximum number of speakers to detect for diarization. Defaults to auto-detect when set to -1.

max_speakers

A dictionary of words to add to the vocabulary, can be used to find and replace or spell words.

custom_vocabulary

The backend to use for translation. Defaults to "None".

translation_backend

Language code of the language to translate to (doesn't translate if left blank). See README for supported language codes.

The backend to use for splitting audio into segments. Defaults to "ffmpeg-silence".

segmentation_backend

Minimum length of audio segment in seconds to use for splitting audio for parallel processing. If set to -1, we pick a value based on your settings.

min_segment_length

Minimum length of silence in seconds to use for splitting audio for parallel processing. Defaults to 0.4. Not available if pyannote segmentation_backend is selected.

min_silence_length

The threshold for VAD. Defaults to 0.2, only used if segmentation_backend is "vad".

vad_threshold

The threshold for Pyannote segmentation when merging segments. This value determines the number of seconds between words to merge segments. Defaults to 0.8, only used if segmentation_backend is "pyannote-3.1.1".

pyannote_segmentation_threshold

A list of strings specifying custom time ranges for audio segmentation. Each string should contain a start and end time in seconds, separated by a comma. For example: ["0,10", "10,20", "20,30"]. If provided, this overrides the segmentation_backend setting and uses these exact chunks for processing. Leave as an empty list to use automatic segmentation.

chunks

The backend to use for denoising the audio. Defaults to "None".

denoise_backend

A prompt to correct misspellings and style. Defaults to "".

initial_prompt

transcribe

Speech Transcription

Filters for removing background noise, enhancing speech, and more in audio files.

'all' to enhance and denoise, 'enhance' to enhance speech, 'denoise' to only remove background noise. Learn more in the README.

task

(only applicable to `sieve-resemble-enhance`). The number of steps (1-100) to take in enhancing the audio. Higher values yield better results but can make the process slow.

enhancement_steps

audio-enhance

Audio Enhance

Generate and render video or audio highlights for long-form content based on search phrases.

If True, the function will render the clips and return the clips as files. If False, the function will just return the metadata of the highlights.

render_clips

The search phrases to use to generate highlights.

highlight_search_phrases

highlights

Highlight Generation

Given a video or audio, generate a title, chapters, summary, tags, and highlights.

Backend to use for transcription. Defaults to "groq-whisper".

transcription_backend

LLM to use for processing the transcript. Defaults to "gpt-4o-2024-08-06".

llm_backend

Custom prompt to guide the LLM's analysis. This influences the analysis, focus areas, and overall output of the generated content. Use this to tailor the analysis to specific requirements or to emphasize particular aspects of the transcript.

Whether to generate a summary or not. Defaults to True.

generate_summary

Whether to generate a title or not. Defaults to True.

generate_title

Whether to generate tags or not. Defaults to True.

generate_tags

Whether to generate chapters or not. Defaults to True.

generate_chapters

Whether to generate highlights or not. Defaults to False.

generate_highlights

Whether to perform sentiment analysis on each segment. If True, analyzes the sentiment of each segment and classifies it as "positive", "negative", or "neutral". Also provides a confidence score for each sentiment classification. Defaults to False.

generate_sentiments

List of custom chapters, Pass in a list of strings, and chapters will be generated based on these, if applicable.

custom_chapters

Mode for custom chapters. If mode is "strict" only the chapters in the custom chapters list will be generated. If mode is "extended" chapters will be generated based on the custom chapters if any is applicable, in addition to the default chapters. Defaults to "extended".

custom_chapter_mode

Maximum number of sentences in summary. Defaults to 5.

max_summary_length

Maximum number of words in title. Defaults to 10.

max_title_length

Number of tags to generate. Defaults to 5.

num_tags

The target language of the output. If left empty, the analysis and transcript will be in the language of the input file.

Whether to use speaker diarization to split audio into segments. Results in less repetition on borders of transcription segments. Defaults to False.

speaker_diarization

Whether to use voice activity detection to split audio into segments. Defaults to False.

use_vad

Whether to denoise audio before analysis. Results in better transcription but slower processing. Defaults to False.

denoise_audio

Whether to return each output as a JSON file (useful for efficient fetching of large payloads). Defaults to False.

return_as_json_file

transcript-analysis

Analyze Transcripts

A set of text-to-speech models and tooling that helps generate natural-sounding speech, clone voices, control emotions, access word timestamps, and more.

CARTESIA_AI_API_KEY

voice

text

The reference audio to be used for voice cloning. Only required when a voice-cloning backend is selected.

reference_audio

The emotion you want the voice to have. Currently available for cartesia voice engine only.

emotion

The speaking rate of the voice. Controls wether the voice should speak fast, slow or normally.

pace

The stability determines how stable the voice is and the randomness between each generation. Currently available for elevenlabs. range ∈ [0-1].

stability

The style determines how much the voice is influenced by the reference audio. Currently available for elevenlabs. range ∈ [0-1].

style

Whether to return the timestamps of the spoken audio.

word_timestamps

Text to Speech

A comprehensive visual question answering app that integrates image and video analysis with text-based queries to provide accurate, structured, and context-aware responses.

The image, video, or audio file to generate content from.

The model to use for processing, for simple tasks use "gemini-1.5-flash" and for more complex tasks use "gemini-1.5-pro".

The framerate to use when processing a video file.

Whether to use audio context when processing a video file.

audio_context

Specify a OPENAPI schema to get a structured response. If not specified, the response will be in plain text. Refer to README for more details.