Using the Prefect Task
Start triggering Airbyte jobs with Prefect in minutes
Airbyte is an official integration Task in the Prefect project. The Airbyte Task allows you to trigger synchronization jobs in Prefect, and this tutorial will walk through configuring your Prefect Flow to do so.
The Airbyte Task documentation on Prefect project can be found here.

First, make sure you have Docker installed. We'll be using the docker-compose command, so your install should contain docker-compose.

If this is your first time using Airbyte, we suggest going through our Basic Tutorial. This tutorial will use the Connection set up in the basic tutorial.
For the purposes of this tutorial, set your Connection's sync frequency to manual. Prefect will be responsible for manually triggering the Airbyte job.

If you don't have a Prefect instance, we recommend following this guide to set one up.

prefect create project "airbyte"

We'll need the Airbyte Connection ID so our Prefect Flow knows which Airbyte Connection to trigger.
This ID can be seen in the URL on the connection page in the Airbyte UI. The Airbyte UI can be accessed at localhost:8000.

Create a new folder called airbyte_prefect and create a file
from prefect import Flow
from prefect.tasks.airbyte.airbyte import AirbyteConnectionTask
airbyte_conn = AirbyteConnectionTask(
with Flow("first-airbyte-task") as flow:
# Register the flow under the "airbyte" project
The Airbyte Prefect Task accepts the following parameters:
  • airbyte_server_host: The host URL to your Airbyte instance.
  • airbyte_server_post: The port value you have selected for your Airbyte instance.
  • airbyte_api_version: default value is v1.
  • connection_id: The ID of the Airbyte Connection to be triggered by Prefect.
After running the file, python3 this will register the Flow in Prefect Server.
Access the link from the output from the previous command to see the Flow in Prefect Server, or you can navigate in Prefect UI to find the new Flow -> Access the link from the output from the previous command to see the Flow in the Prefect Server. Alternatively, you can go to the Prefect UI to find the new Flow.
Click on the button Run and configure your first run.
After a few moments you should see the finished run.
After that you have the option to configure a more complex Schedule to your Flow. See the Prefect Schedule docs.

Don't be fooled by our simple example of only one Prefect Flow. Airbyte is a powerful data integration platform supporting many sources and destinations. The Airbyte Prefect Task means Airbyte can now be easily used with the Prefect ecosystem - give it a shot!
We love to hear any questions or feedback on our Slack. We're still in alpha, so if you see any rough edges or want to request a connector, feel free to create an issue on our Github or thumbs up an existing issue.
Copy link
On this page
1. Set up the tools
Start Airbyte
Start Prefect
2. Create a Flow in Prefect to trigger your Airbyte job
Create a new Prefect Project
Retrieving the Airbyte Connection ID
Creating a simple Prefect DAG to run an Airbyte Sync Job
That's it!