Some datasets can manage with a certain percentage of duplicate data. Understand your business requirements and tolerance of duplicate data. Solutions for handling duplicate data Solution #1: Don't remove duplicate data | extend duplicate_percentage = (duplicateRecords / _sample) / _totalRecords Let _totalRecords = toscalar(_data | count) Sample query to identify the percentage of duplicate records: let _sample = 0.01 // 1% sampling Once the percentage of duplicate data is discovered, you can analyze the scope of the issue and business impact and choose the appropriate solution. Monitor the percentage of duplicate data. However, in situations where the source system can't be modified, there are various ways to deal with this scenario. If possible, fix the issue earlier in the data pipeline, which saves costs associated with data movement along the data pipeline and avoids spending resources on coping with duplicate data ingested into the system. The best solution for data duplication is preventing the duplication. This topic outlines best practices for handling duplicate data for these types of scenarios. You want to safeguard your analytical databases from malfunctioning devices that resend the cached data and cause data duplication in the analytical database. Depending on the data size, the local cache could be storing data for days or even months. Devices sending data to the Cloud maintain a local cache of the data.
0 kommentar(er)
