How to use the Hub
The InfinyOn Hub serves as a centralized repository for InfinyOn Connectors, SmartModules, and Dataflows. It facilitates the discovery and deployment of extensions to enhance Fluvio's data streaming capabilities. The hub provides developers and data engineers a platform to access pre-built solutions or share their extensions easily.
Connectors
List the Connectors available for download. The official connectors (prepended with infinyon/) can run on InfinyOn Cloud and in your local environment.
List Connectors
Use fluvio hub connector list or cdk hub list to list the available connectors.
$ fluvio hub connector list
CONNECTOR Visibility
infinyon-labs/graphite-sink@0.2.0 public
infinyon/duckdb-sink@0.1.0 public
infinyon/http-sink@0.2.11 public
infinyon/http-source@0.4.3 public
infinyon/ic-webhook-source@0.1.4 public
infinyon/inf-webhook-source@0.1.9 public
infinyon/kafka-sink@0.2.9 public
infinyon/kafka-source@0.2.7 public
infinyon/mqtt-source@0.2.9 public
infinyon/sql-sink@0.4.3 public
qdrant/qdrant-sink@0.1.0 public
Check out Connectors for Developers to learn how to build and publish your own connectors for the Hub.
Use Connectors
If you use a connector in InfinyOn Cloud, the Cloud infrastructure will automatically download and run the connector package. If you use connectors in your local environment, you must use the cdk command to download and run the connector package.
Download and Run Locally
Connectors are independent binaries that run outside of Fluvio. Use the cdk command to download and run connectors locally:
$ cdk hub download infinyon/http-sink@0.2.11
The same connector configuration file can run the connector locally or in InfinyOn Cloud.
$ cdk deploy start --ipkg infinyon-http-sink-0.2.11.ipkg --config <CONFIG>
To stop a connector, use the cdk deploy shutdown command.
Deploy on InfinyOn Cloud
Connectors are automatically downloaded to InfinyOn Cloud. The management of InfinyOn Cloud is performed using the fluvio cloud commands. To run a connector in InfinyOn Cloud, use the command fluvio cloud connector create.
$ fluvio cloud connector create --config <CONFIG>
Note List your profiles (fluvio profile list) to ensure your CLI points to the InfinyOn Cloud cluster.
SmartModules
SmartModules are custom-defined programs that connectors use to transform data records during processing. Similar to connectors, SmartModules can be published to the Hub and made available for download. However, unlike connectors, SmartModules are downloaded to the cluster and are not executed outside of Fluvio.
List SmartModules
List the SmartModules available for download. The official SmartModules (prepended with infinyon/).
$ fluvio hub smartmodule list
SMARTMODULE Visibility
infinyon-labs/array-map-json@0.1.1 public
infinyon-labs/csv-json@0.2.0 public
infinyon-labs/dedup-filter@0.0.2 public
infinyon-labs/json-formatter@0.1.3 public
infinyon-labs/key-gen-json@0.1.1 public
infinyon-labs/regex-map-json@0.1.2 public
infinyon-labs/regex-map@0.1.1 public
infinyon-labs/rss-json@0.1.1 public
infinyon-labs/stars-forks-changes@0.1.4 public
infinyon/csv-json@0.2.0 public
infinyon/jolt@0.4.1 public
infinyon/json-sql@0.2.1 public
infinyon/regex-filter@0.2.0 public
Download and Run SmartModules
Use fluvio hub smartmodule download to download specific SmartModules to a local or InfinyOn Cloud cluster. Use fluvio profile list to identify the target cluster.
$ fluvio hub smartmodule download infinyon/jolt@0.4.1
... cluster smartmodule install complete
SmartModules downloaded to a cluster can be used by the producer CLI or consumer CLI to transform records. They can also be accessed within the Connectors via the configuration file.
To see the SmartModules currently installed on the cluster, use the following command:
$ fluvio smartmodule list