Data Engineering Services for Mid-Market Teams

What are data engineering services, and why does the weekly Excel rebuild persist?

Data engineering services build and run the pipelines that move data from your source systems into one tested, reconciled place that a dashboard can read. The weekly Excel rebuild persists because each source system answers its own question well and none answers the joined-up one. CRM, billing, support, finance and operations each define "customer" differently, so one person stitches them together by hand every week.

The spreadsheet survives because it is the only place the join logic lives. One analyst knows which export has the right date column, which customer IDs need manual mapping, and which refunds to exclude. That knowledge is undocumented, unversioned and leaves when the analyst does.

Replacing the spreadsheet therefore means making the logic explicit, tested and scheduled. That work is the dashboard's foundation, and the dashboard is the smallest visible part of it. A unified reporting dashboard built on unreconciled data simply makes the disagreement prettier.

What the manual process is really costing

Cost shows up as delay, not invoices. Reports arrive on Tuesday for a Monday question, and nobody trusts a number enough to act on it without asking the analyst how it was made. Because the logic sits in one person's head and one file, the risk is structural, not a matter of someone being careless. Every hand-edited cell is a place where a wrong number can enter unnoticed, and nothing records who changed it or why.

A concrete failure mode: the analyst goes on leave in the week the billing system changes its export format. The date column shifts from day-first to month-first, the spreadsheet formulas still run without error, and the weekly report quietly assigns March invoices to a different month. Nobody sees an error message, so the wrong figures reach the leadership meeting. A tested pipeline would fail loudly at the load step instead.

Do you need a warehouse, or is a direct connection enough?

As a working rule of thumb, you need a warehouse once any one of these is true:

• You have three or more sources to combine.
• You need history that the sources do not keep.
• You have already seen two reports disagree.

Below that, connecting a BI tool directly is usually cheaper and good enough. If you are unsure which side you fall on, the broken workflow assessment is a quick way to check whether your reporting process is the right place to start.

Direct connection fails in predictable ways. Source APIs rate-limit, exports change shape without notice, and a dashboard that queries five live systems is only as fast as the slowest one.

When a direct connection is enough

Two small sources, one owner, and no need for history. A scheduled export into a BI tool's own model works, and building a warehouse would be over-engineering. The signs that you are still safe are simple:

• Both sources share a customer key, so no mapping table is needed.
• Nobody asks for last year's figures as they stood at the time.
• One person can explain every number on the report.

When those stop being true, the direct connection becomes the thing you maintain instead of the thing that saves you effort.

When a warehouse for mid-market companies pays off

A data warehouse for mid-market firms does not need to be large. Managed options such as Amazon Redshift Serverless or BigQuery remove cluster administration, and BigQuery's published pricing includes a free monthly tier for queries and storage. Data volumes for a five-source reporting stack are usually modest, so the engineering effort goes into modelling, not capacity.

The trade-off is that a warehouse adds a layer someone must understand. You gain history, because each load can be snapshotted, and you gain a single place to enforce definitions. You pay with an extra moving part, a modelling discipline and a small recurring bill. For a team that already feels the pain of disagreeing reports, that exchange is usually worth making.

How do data engineering services design pipelines for five source systems?

Land each source untouched in a raw layer, transform it into cleaned staging tables, then build a small set of business-facing models. Keeping those three layers separate is what lets you fix logic without re-pulling data.

Extract and load

Prefer the source's native API or export over screen scraping, and load incrementally using an updated-at field or change feed where one exists. Sources without reliable change tracking get a full reload on a schedule, which is simple but costs more as data grows.

Transform and test

Put transformations in version-controlled SQL and attach automated tests. dbt's documentation on data tests describes the standard checks: uniqueness, not-null, accepted values and referential integrity. Those four often catch silent breakages, such as a source renaming a status value.

In our integration work the pattern tends to hold: the fragile part is usually the oldest source, not the newest one.

How do you make the numbers agree?

Reconciliation means agreeing one definition per metric, then proving the warehouse total matches the system of record. Do the definition work first, because no pipeline can resolve a disagreement between sales and finance about what counts as revenue.

Agree definitions before writing code

Write each metric as a sentence an executive signs off: what is included, what is excluded, which date applies and which system wins on conflict. Customer identity needs particular care, because the same company often appears under three IDs across CRM, billing and support. A mapping table, owned by a named person, resolves it.

Prove the totals tie out

Pick the finance ledger or billing system as the anchor and compare monthly totals to the warehouse. Differences should be explainable line by line. The usual causes are:

• Timing, where a record lands in different months in different systems.
• Refunds and credit notes handled differently.
• Currency rounding.
• Test or internal accounts left in one system.

Keep that check running as an automated test so drift is caught the day it starts, not at quarter end.

How often should the dashboard refresh?

Refresh cadence is a cost dial: each step from monthly to daily to hourly to near real time adds compute, pipeline complexity and monitoring effort. Choose the slowest cadence that still supports the decision the report informs.

What the tools themselves limit

Check your BI tool's limits before promising anything. Microsoft's documentation states that Power BI scheduled refresh is capped at eight times a day on shared capacity and 48 on Premium, so an hourly dashboard may force a licensing change, not just a pipeline change.

Cut refresh cost without cutting freshness

Incremental refresh reloads only recent partitions instead of the whole dataset. Most reporting tables are append-heavy, so this alone often makes hourly feasible where full reloads were not. Real-time streaming is rarely justified for management reporting; a daily board pack and an hourly operations view cover most mid-market needs.

What do data engineering services cost to build and run?

Build cost scales with source count, source quality, reconciliation depth and refresh frequency. Run-rate is dominated by warehouse compute, connector or licence fees, and the people time to keep pipelines healthy. We scope this per project after a discovery call.

What drives the build cost

We have no published figure to quote here, and we would rather say so than invent one. The list below shows where cost comes from and what moves it, which is more useful for comparing quotes than a headline number. Treat any ETL consulting cost estimate that skips these inputs as a guess.

• Source extraction (one-off, plus upkeep): driven by API quality, authentication and rate limits. Prefer sources with stable APIs.
• Modelling and metric definitions (one-off): driven by the number of disputed metrics. Settle definitions before the build starts.
• Reconciliation tests (one-off, plus upkeep): driven by ledger access, currency and identity. Anchor to one system of record.

What drives the monthly run-rate

These items recur every month, so they matter more over a year than the build does:

• Warehouse compute and storage: driven by data volume, query load and refresh rate. Use incremental loads and a sensible cadence.
• Connectors and BI licences: driven by vendor pricing tiers. Check published tiers early.
• Monitoring and fixes: driven by how often sources change. Automated tests and alerts keep the effort down.

If you want a working slice first, our Moonship MVP is one way to scope a fixed-price first phase, and a single-source-to-dashboard slice is a sensible candidate. We agree the scope and timeline with you during discovery.

Who maintains it after we leave?

Someone on your side must own metric definitions, and someone must own pipeline health. These are different jobs, and a stack without both quietly decays back into a spreadsheet.

Pick an owner for each layer

Business definitions belong to a finance or operations lead who can approve a change. Pipeline health belongs to an engineer, in-house or on a retained basis. Without the first, definitions drift. Without the second, a source API change breaks the dashboard on a Friday and nobody notices until Monday.

Plan for the handover

Ask any provider for documented models, a runbook, alerting that reaches your team, and a recorded walkthrough. If you have no data engineer, a fractional CTO can own the architecture and vendor relationship on a fractional basis, which is cheaper than a hire you cannot yet keep busy.

Older systems often need work before they can be extracted at all, which is where system modernisation overlaps with reporting. If the work above sounds like your Monday morning, see how data engineering services from Viithiisys are scoped.