• State Suggested Answer
  • Replies 5 replies
  • Answers 2 answers
  • Subscribers 2 subscribers
  • Views 10 views
  • Users 0 members are here
Options
Related

Import customer list without creating duplicates

Anita Sharma

I am preparing to import approximately 12,000 customer records from our legacy CRM into FieldPulse. I have reviewed the bulk import documentation and have a concern regarding duplicate prevention.

Our legacy system contains multiple entries for the same customer due to historical data quality issues. I have confirmed that:

  1. Approximately 2,300 records share phone numbers with at least one other record
  2. Approximately 1,800 records share email addresses with at least one other record
  3. Approximately 890 records have matching first name, last name, and street address but different contact methods

My questions are as follows:

  1. What match key(s) does the FieldPulse import process use to identify potential duplicates?
  2. Is there a configurable merge rule set, or is deduplication handled automatically according to a fixed hierarchy?
  3. Can I specify which record should be considered "master" when duplicates are detected (e.g., most recent updated date, most complete data)?
  4. Does the import provide a preview or dry-run option showing which records would be merged versus created as new?
  5. Are there any constraints on which fields can be used as custom match keys?

I have noted that the import template includes columns for "Customer ID (External)" — it is not clear whether this field participates in deduplication logic or is strictly for reference.

Prior to executing this import, I require a reproducible process that does not create duplicate customer records, as remediation would be operationally costly.

Environment details: I will be using the web interface bulk import function, not the API, for this operation. Our account is on the Enterprise tier.

Parents
  • 0

    Hey Anita — good questions, and yeah, 12K records with that much duplication potential definitely warrants a careful approach.

    Short answer: the web import uses a fixed match key hierarchy, not configurable rules. Here's the actual order it checks:

    1. Email address (exact match, case-insensitive)
    2. Phone number (normalized, so "(555) 123-4567" and "5551234567" match)
    3. First name + Last name + Street address (all three must match)

    The "Customer ID (External)" field is reference only — it does not participate in matching. Useful for your own tracking, not for dedup logic.

    There's no native dry-run in the web UI, but here's what I do:

    // Quick script to preview what would merge
// Run this on your CSV before import

const seen = new Map();
const duplicates = [];

records.forEach(r => {
  const key = r.email?.toLowerCase() || 
              r.phone?.replace(/\D/g, '') ||
              `${r.first_name}|${r.last_name}|${r.street}`.toLowerCase();
  
  if (seen.has(key)) {
    duplicates.push({
      row: r.row_number,
      conflicts_with: seen.get(key),
      matched_on: r.email ? 'email' : r.phone ? 'phone' : 'name+address'
    });
  } else {
    seen.set(key, r.row_number);
  }
});

    When duplicates are found, the first encountered record in the CSV wins and becomes the master. So sort your export by "last updated" descending if you want newer data to take precedence.

    The actual merge behavior preserves the most complete individual fields (non-empty values from both records), not a strict winner-take-all. Phone numbers and emails accumulate rather than replace.

    If you need deterministic master selection (e.g., always prefer the record with more populated fields), you'd need to pre-process externally or use the API with explicit update logic. The web import doesn't expose that level of control.

    Want me to share a Python script for that pre-merge analysis? Happy to adapt it for your specific priority rules.

Reply
  • 0

    Hey Anita — good questions, and yeah, 12K records with that much duplication potential definitely warrants a careful approach.

    Short answer: the web import uses a fixed match key hierarchy, not configurable rules. Here's the actual order it checks:

    1. Email address (exact match, case-insensitive)
    2. Phone number (normalized, so "(555) 123-4567" and "5551234567" match)
    3. First name + Last name + Street address (all three must match)

    The "Customer ID (External)" field is reference only — it does not participate in matching. Useful for your own tracking, not for dedup logic.

    There's no native dry-run in the web UI, but here's what I do:

    // Quick script to preview what would merge
// Run this on your CSV before import

const seen = new Map();
const duplicates = [];

records.forEach(r => {
  const key = r.email?.toLowerCase() || 
              r.phone?.replace(/\D/g, '') ||
              `${r.first_name}|${r.last_name}|${r.street}`.toLowerCase();
  
  if (seen.has(key)) {
    duplicates.push({
      row: r.row_number,
      conflicts_with: seen.get(key),
      matched_on: r.email ? 'email' : r.phone ? 'phone' : 'name+address'
    });
  } else {
    seen.set(key, r.row_number);
  }
});

    When duplicates are found, the first encountered record in the CSV wins and becomes the master. So sort your export by "last updated" descending if you want newer data to take precedence.

    The actual merge behavior preserves the most complete individual fields (non-empty values from both records), not a strict winner-take-all. Phone numbers and emails accumulate rather than replace.

    If you need deterministic master selection (e.g., always prefer the record with more populated fields), you'd need to pre-process externally or use the API with explicit update logic. The web import doesn't expose that level of control.

    Want me to share a Python script for that pre-merge analysis? Happy to adapt it for your specific priority rules.

Children
No Data