Data Factory Error - "Found More Columns Than Expected"

One popular ETL tool in Azure is Data Factory, which is like a cloud version of SSIS for those who are familiar with SSIS. While we could get this error in a variety of situations, the key to pay attention to is the number in the error - such as the row number and have an idea of the expected columns that should be in the result. We see more details about this in the video, Error found when processing 'csv' source 'file' with row number: found more columns than expected.

Some questions that are answered in the video:

In the example, what was the mistake that caused this error?
How would we solve this error in the example? Considering our environment and where we may see this, how might we solve this error?
What is at least one way that we could prevent this error from happening?

The latter question is one to consider if you've seen this error several times. Data factory is not a tool that makes it easy to dynamically update content and you want to be careful about building convoluted pipelines to handle rare issues (unless these rare issues cause serious disruptions). The reason that this could be a drawback is that you'll have to support these pipelines and it's not always clear what the issue is.

For mastering data transformation from one form to another form, check out the highest-rated Automating ETL course on Udemy. For a coupon to the course, check out the trailer video on the channel SQL In Six Minutes.

My general preference in ETL is to avoid tools like SSIS or data factory because they don't consider what could go wrong and how to easily solve the issue. I would be careful about using either of these tools, as they feel easy to develop, but come with significant costs in troubleshooting.

Are you looking for tech consultants that can assist with design and development? From building custom applications to working with existing infrastructure that's causing you trouble, we can get you connected to consultants who can assist. You can contact for assistance.

SQL In Six Minutes | Automating ETL | T-SQL In 2 Hours | Consumer Guide To Digital Security

Sort:

Trending

[-]

hivebuzz (74) 4 years ago

Congratulations @sqlinsix! You have completed the following achievement on the Hive blockchain and have been rewarded with new badge(s) :

You distributed more than 5000 upvotes.
Your next target is to reach 6000 upvotes.

_{You can view your badges on your board and compare yourself to others in the Ranking}
_{If you no longer want to receive notifications, reply to this comment with the word STOP}

Check out the last post from @hivebuzz:

	Feedback from the June 1st Hive Power Up Day
	Hive Power Up Day - June 1st 2021 - Hive Power Delegation

Support the HiveBuzz project. Vote for our proposal!

$0.00

Congratulations @sqlinsix! You received a personal badge!

Happy Hive Birthday! You are on the Hive blockchain for 5 years!

_{You can view your badges on your board and compare yourself to others in the Ranking}

Feedback from the June 1st Hive Power Up Day