Things to think about when working with Azure Data Factory

Azure Data Factory brings ETL to a whole new level. With the great responsive GUI, you can build great ETL pipelines that are easily tested and debugged. While Azure Data Factory is a great service, there are some downfalls to think about.

The lookup activity with a limit of 5000 records

Luckily for every problem, we have a solution. We added a second lookup activity and used a “like” statement in the where clause.

Convert SQL data to CSV or Parquet

We did find a workaround by sending the data first to a SQL database inside Azure and after that, we send the data to a data lake. In the last step, we convert the data to parquet. This speeded up the process by almost 40%.

Sending an email from an ADF pipeline

I also implement it this way. You can find the link to the tutorial at the end of this article.

Links