Note: The job is a remote job and is open to candidates in USA. Lumenalta is a technology solutions company partnering with organizations to enhance business growth. They are seeking an experienced Data Engineer / Data Analyst to build a real-time Databricks Lakehouse platform, focusing on data profiling, quality assurance, and collaboration with stakeholders to ensure data integrity.
Responsibilities
- Profile high-frequency, multi-sensor data as it lands in the Bronze layer—characterizing distributions, null and completeness rates, value ranges, cardinality, uniqueness, referential integrity, schema drift, duplicate and late/out-of-order records, and sensor dropouts across telemetry, optical tracking, and timing/scoring feeds
- Partner with business and domain stakeholders—race engineers, analysts, and competition officials—to understand how each feed is used and translate their domain knowledge into concrete, testable expectations: what "valid," "complete," and "in range" actually mean for telemetry, timing, and optical tracking data
- Define, document, and codify the data quality rules and expectations—validity, completeness, uniqueness, timeliness, range and threshold checks—that govern the promotion of data from Bronze to Silver
- Build and maintain the Bronze-to-Silver transformation logic: schema enforcement, type casting, deduplication, standardization and conforming, and quarantine of records that fail quality checks—producing clean, trusted Silver tables
- Implement data quality enforcement using Databricks-native tooling (Delta Live Tables / Lakeflow Declarative Pipelines expectations, or frameworks such as Great Expectations or Databricks DQX), including quarantine tables and alerting for records that don't meet the rules
- Establish data contracts and clear documentation of expected schemas and semantics for each source, working with upstream data producers and downstream consumers to keep expectations aligned as feeds evolve
- Monitor data quality over time—tracking rule pass/fail rates, surfacing data drift and anomalies within the pipeline, and alerting the team when source data or ingestion quality degrades
- Work within the Unity Catalog governance framework to ensure Silver datasets are trusted, lineage-tracked, and analysis- and model-ready for downstream engineering and data science functions
- Collaborate with data engineers to define the models, conformed schemas, and serving-layer outputs that clean Silver data must support—so downstream analytics and predictive race analysis are built on reliable inputs
Skills
- 4+ years as a Data Engineer, Analytics Engineer, or data-focused Analyst, with a track record of delivering production-grade data pipelines and data quality workflows in data-rich environments
- Strong data profiling skills—able to characterize unfamiliar, high-volume datasets using descriptive statistics, distribution analysis, and null, cardinality, uniqueness, and integrity checks
- Hands-on experience defining and implementing data quality rules and expectations, with familiarity in frameworks such as Delta Live Tables / Lakeflow expectations, Great Expectations, or Databricks DQX
- Solid understanding of the medallion architecture (Bronze / Silver / Gold) and Bronze-to-Silver promotion patterns: schema enforcement, deduplication, conforming, and quarantine of bad records
- Advanced SQL and strong PySpark/Spark skills for profiling and transforming large datasets on Databricks; experience with Delta Lake and Databricks SQL
- Working knowledge of Databricks—including Delta Lake, Unity Catalog, SQL Warehouses, and streaming ingestion (Auto Loader / Structured Streaming)
- Comfort working with high-frequency, real-time data feeds—understanding latency trade-offs, streaming vs. batch freshness, watermarking, idempotency, and handling late or out-of-order records
- Experience validating data against governed sources; familiarity with data lineage, data contracts, and working within a cataloged data environment (Unity Catalog or equivalent)
- Proven ability to work directly with business and domain stakeholders—eliciting requirements, translating domain knowledge and business rules into concrete data quality expectations, and validating those rules with the people who rely on the data
- Strong written and verbal communication skills in English, with the ability to document data quality rules and data expectations clearly for both engineering and non-technical stakeholders
- Databricks certifications are required (Databricks Certified Data Engineer Associate or Professional preferred)
- Experience with high-frequency sensor, telemetry, or IoT data is a strong plus; motorsport, aerospace, or industrial real-time contexts are especially welcome
Benefits
- Employment_type: FULL_TIME
- Job_location_type: TELECOMMUTE
- Be 100% dedicated to one project at a time so that you can innovate and grow.
- Be part of a team of talented, friendly senior-level developers.
- Work on projects that allow you to use leading tech.
- This is a fully remote position; however, candidates must be based in regions that align with the Pacific, Central, or Eastern U.S. time zones to ensure effective collaboration with client and team schedules.
- Occasional travel to Georgia may be required.
Company Overview