Login Sign Up

Frequently Asked Questions

What data connectors are supported?

Data Pipelines currently supports the following:
  • AWS S3 (CSV, Parquet, JSON lines, AVRO, Orc)
  • MySQL
  • PostgreSQL
  • Google Sheets
  • DynamoDB
  • BigQuery

Why isn't the pipeline view updated instantly when I click Preview?

Data Pipelines is primarily aimed at large workloads. Our platform uses Apache Spark to generate previews from your data. Each preview launches a Spark job which has some overhead and usually takes a few seconds to complete. This means that the time it takes to generate previews remains the same regardless of the size of your data. It also ensures that your pipeline is valid and will run successfully.

Can I mix different types of data connections in the same pipeline?

You can combine data from different types of connectors in a single pipeline. You can also write to different connections on the same run.

Is there a self-hosting option?

Self-hosting is available on our enterprise plan. Get in touch for more information.