Data Import Tools

The Data Import Tools enable bulk importing of customer, ride, and vehicle data from CSV files. This guide covers all import capabilities, file format requirements, and best practices for successful data migration.

Accessing Data Import Tools

Navigate to Admin > Database Tools to access the import interface.

Requirements:

• Admin access to the system
• Properly formatted CSV files
• Data that matches the expected schema

Overview

Available Import Types

Import Workflow

---

Customer Import

Accessing Customer Import

In Database Tools, locate the "Customers Import" card and click to access.

CSV Format Requirements

Delimiter: Semicolon (;)

Required Columns:

• ID - Unique identifier (becomes original_id)

Supported Columns:

Import Process

Small Files (< 4 MB):

1. Click "Select File" or drag-and-drop
2. Click "Import"
3. Processing happens synchronously
4. Results display immediately

Large Files (4+ MB):

1. Upload initiates batch processing
2. File is split into 500-row batches
3. Progress tracked via import jobs
4. Notification when complete

Import Results

After import completes, you'll see:

• Processed: Total rows read from file
• Imported: New records created
• Updated: Existing records updated
• Errors: Rows that failed to import

Error Details: First 10 errors shown with:

• Row number
• Customer ID (if available)
• Error message

---

Rides Import

Accessing Rides Import

In Database Tools, locate the "Rides Import" card and click to access.

CSV Format Requirements

Delimiter: Semicolon (;)

Key Columns:

Additional Columns (50+ total):

• Zone IDs (parking, paid, no-parking, bonus, speed-limit, no-go)
• Location coordinates (start/end positions)
• Subscription details
• Pricing package info
• Dynamic pricing data

Import Behavior

The rides import automatically:

1. Creates missing customers (if User ID provided)
2. Creates missing vehicles (if Vehicle ID provided)
3. Links rides to existing customers and vehicles
4. Handles duplicate rides via upsert on ride ID

Import Results

After import:

• Check total counts in Database Statistics
• Verify customer and vehicle creation
• Review any error messages

---

Vehicle Import

Format

Vehicle import uses JSON format (not CSV):

[
  {
    "vehicle_number": "VH-001",
    "iot_imei": "123456789012345",
    "vin": "1HGBH41JXMN109186",
    "vehicle_model": "Scooter Pro"
  },
  {
    "vehicle_number": "VH-002",
    "iot_imei": "123456789012346"
  }
]

Required Fields

• vehicle_number - Required - Unique identifier for the vehicle

Optional Fields

• iot_imei - IoT device IMEI number
• vin - Vehicle Identification Number
• vehicle_model - Model name or ID
• Additional fields as needed

Import Result

Returns per-vehicle results:

• Success or failure status
• Error message for failed rows
• Automatic linking to IoT devices by IMEI

---

IoT Device Import

Format

IoT device import uses JSON format:

[
  {
    "imei": "123456789012345",
    "device_type": "Tracker",
    "firmware_version": "2.1.0"
  }
]

Required Fields

• imei - Required - Device IMEI number

Linking to Vehicles

After import, devices can be linked to vehicles:

• Manually via vehicle edit
• Automatically if vehicle has matching IMEI
• Via PUT request to the API

---

Import Job Tracking

How Jobs Work

For large files and background imports:

1. Job Created: System creates an import_jobs record
2. Status: Processing: File is being parsed and imported
3. Progress Updates: Rows processed count increases
4. Completion: Status changes to completed or completed_with_errors

Job Statuses

Viewing Active Jobs

On the Stripe Import page, active jobs display with:

• Progress bar (processed/total rows)
• Status indicator
• Imported/Updated/Skipped/Error counts
• Start time and file name

Job Polling

The system polls every 2 seconds for job updates while you're on the import page.

---

Error Handling

Common Errors

Error Limits

• Processing stops after 100 consecutive errors
• First 10 error details shown in UI
• Full error log available in import job record

Handling Partial Imports

If import stops partway:

1. Note the last successful row
2. Fix errors in remaining rows
3. Re-import only the remaining data
4. Or clear and re-import everything

---

File Size Guidelines

Large File Tips

1. Split very large files: Break into chunks of 50,000 rows
2. Test first: Import a small sample before full import
3. Off-peak hours: Run large imports during low-traffic periods
4. Monitor progress: Watch the job status for issues

---

Staging Tables

How Staging Works

1. CSV data loads into a staging table (e.g., users_import)
2. RPC function transforms and validates data
3. Valid records upserted into main table
4. Staging table cleared (or retained for audit)

Staging Tables

---

Best Practices

Before Import

1. Backup existing data: Export current data first
2. Validate CSV format: Check delimiter, encoding, headers
3. Test with sample: Import 10-20 rows first
4. Note current counts: Record statistics for comparison

During Import

1. Don't navigate away: Stay on page for sync imports
2. Monitor progress: Watch job status for large imports
3. Note any errors: Screenshot or record error messages

After Import

1. Verify counts: Check Database Statistics
2. Spot check records: Review a few imported records
3. Test functionality: Ensure imported data works correctly
4. Document the import: Record what was imported and when

---

Data Transformation

Automatic Transformations

During import, data is automatically:

• Trimmed of whitespace
• Empty strings converted to NULL
• Dates parsed to timestamps
• Numbers parsed from strings
• Boolean strings (TRUE/FALSE/Yes/No) converted

Manual Preparation

Before import, ensure:

• No extra header rows
• Consistent date formats
• Valid email formats
• Phone numbers formatted consistently
• No special characters in IDs

---

Troubleshooting

File Upload Fails

Possible Causes:

• File too large
• Wrong file type
• Network timeout

Solutions:

• Reduce file size
• Ensure .csv extension
• Check network connection

Import Completes but No New Records

Possible Causes:

• All records already exist
• ID column maps to existing records
• Upsert updated instead of inserted

Solutions:

• Check for updates (not just inserts)
• Verify ID column values
• Review import statistics carefully

Many Records Failing

Possible Causes:

• Wrong delimiter (comma vs semicolon)
• Column name mismatch
• Data format issues

Solutions:

• Verify semicolon delimiter
• Check column headers exactly match
• Validate data formats

---

What's Next?

• Super Admin Features - Learn about platform administration
• Stripe Data Sync - Import payment data from Stripe
• Quick Start Guide - Get started with fleet management