To build a new connector in Java or Python, we provide templates so you don't need to start everything from scratch.
Note: you are not required to maintain the connectors you create. The goal is that the Airbyte core team and the community help maintain the connector.
To add a new connector you need to:
- 1.Implement & Package your connector in an Airbyte Protocol compliant Docker image
- 3.Document how to build & test your connector
- 4.Publish the Docker image containing the connector
Each requirement has a subsection below.
If you are building a connector in any of the following languages/frameworks, then you're in luck! We provide autogenerated templates to get you started quickly:
- Python Source Connector
- Singer-based Python Source Connector. Singer.io is an open source framework with a large community and many available connectors (known as taps & targets). To build an Airbyte connector from a Singer tap, wrap the tap in a thin Python package to make it Airbyte Protocol-compatible. See the Github Connector for an example of an Airbyte Connector implemented on top of a Singer tap.
- Generic Connector: This template provides a basic starting point for any language.
- Java Destination Connector
- Python Destination Connector
Run the interactive generator:
and choose the relevant template by using the arrow keys. This will generate a new connector in the
Search the generated directory for "TODO"s and follow them to implement your connector. For more detailed walkthroughs and instructions, follow the relevant tutorial:
As you implement your connector, make sure to review the Best Practices for Connector Development guide. Following best practices is not a requirement for merging your contribution to Airbyte, but it certainly doesn't hurt ;)
If you're writing in Python or Java, skip this section -- it is provided automatically.
If you're writing in another language, please document the commands needed to:
- 1.Build your connector docker image (usually this is just
docker build .but let us know if there are necessary flags, gotchas, etc..)
- 2.Run any unit or integration tests in a Docker image.
Your integration and unit tests must be runnable entirely within a Docker image. This is important to guarantee consistent build environments.
When you submit a PR to Airbyte with your connector, the reviewer will use the commands you provide to integrate your connector into Airbyte's build system as follows:
:airbyte-integrations:connectors:source-<name>:buildshould run unit tests and build the integration's Docker image
:airbyte-integrations:connectors:source-<name>:integrationTestshould run integration tests including Airbyte's Standard test suite.
Typically this will be handled as part of code review by an Airbyter. There is a section below on what steps are needed for publishing a connector and will mostly be used by Airbyte employees publishing the connector.
The steps for updating an existing connector are the same as for building a new connector minus the need to use the autogenerator to create a new connector. Therefore the steps are:
- 1.Iterate on the connector to make the needed changes
- 2.Run tests
- 3.Add any needed docs updates
- 4.Create a PR to get the connector published
Once you've finished iterating on the changes to a connector as specified in its
README.md, follow these instructions to ship the new version of the connector with Airbyte out of the box.
- 1.Bump the version in the
Dockerfileof the connector (
- 2.Submit a PR containing the changes you made.
- 3.One of Airbyte maintainers will review the change and publish the new version of the connector to Docker hub. Triggering tests and publishing connectors can be done by leaving a comment on the PR with the following format (the PR must be from the Airbyte repo, not a fork):# to run integration tests for the connector# Example: /test connector=connectors/source-hubspot/test connector=(connectors|bases)/<connector_name># to run integration tests and publish the connector# Example: /publish connector=connectors/source-hubspot/publish connector=(connectors|bases)/<connector_name>
- 4.Update the connector definition in the Airbyte connector index to use the new version:
Then run the commend
airbyte-config/init/src/main/resources/seed/source_definitions.yamlif it is a source
airbyte-config/init/src/main/resources/seed/destination_definitions.yamlif it is a destination.
./gradlew :airbyte-config:init:processResourcesto generate the seed spec yaml files, and commit the changes to the PR. See this readme for more information.
- 5.The new version of the connector is now available for everyone who uses it. Thank you!
In order to run integration tests in CI, you'll often need to inject credentials into CI. There are a few steps for doing this:
- 1.Place the credentials into Google Secret Manager(GSM): Airbyte uses a project 'Google Secret Manager' service as the source of truth for all CI secrets. Place the credentials exactly as they should be used by the connector into a GSM secret here i.e.: it should basically be a copy paste of the
config.jsonpassed into a connector via the
--configflag. We use the following naming pattern:
SECRET_<capital source OR destination name>_CREDSe.g:
- 2.Add the GSM secret's labels:
connector(required) -- unique connector's name or set of connectors' names with '_' as delimiter i.e.:
filename(optional) -- custom target secret file. Unfortunately Google doesn't use '.' into labels' values and so Airbyte CI scripts will add '.json' to the end automatically. By default secrets will be saved to
- 4.That should be it.
Access to GSM storage is limited to Airbyte employees. To give an employee permissions to the project:
- 2.Add a new principal to
- input their login email
- select the role