Summary
We identified and resolved a data quality issue affecting how AI-assisted coding sessions (Claude Code) were captured and displayed in Cogniscape dashboards. Instead of showing meaningful details about what developers worked on — files touched, technologies used, objectives and outcomes — the system was producing generic entries such as “Developer completed a session.” This significantly reduced the value of the five core visibility dimensions Cogniscape provides: what, who, when, how, and why. The issue was fully resolved within 24 hours. Session data quality improved from 4/10 to approximately 8/10 across all dimensions.What Was Affected
Engineering managers relying on Cogniscape for AI coding session insights were seeing:- Vague descriptions — no detail about what was actually built or investigated
- Missing file data — no visibility into which files or areas of the codebase were touched
- Incorrect branch information — always showed the default branch, even for feature work
- Incomplete session capture — only the first few minutes of long sessions were reflected
Root Cause
Multiple issues in the data processing pipeline compounded to degrade session quality:- Data loss during processing — session details (title, result, files) were being dropped when certain metadata was present, replaced by a generic description
- Only the first snapshot was kept — long-running sessions send periodic updates, but only the earliest update was retained, missing hours of subsequent work
- Unstructured summaries — raw conversation fragments were used instead of structured descriptions, making it difficult to extract meaningful insights
- Missing metadata — rich session data (files modified, commands run, tools used) was available but not being captured
- Noisy file references — file paths included developer-specific system paths instead of clean, project-relative references
Resolution
We deployed a series of targeted fixes over four releases:
Additional improvements:
- File paths are now clean and project-relative
- Branch detection works correctly for all Git workflows
- Metadata noise was reduced by 90%
Results
Quality Score Comparison
Key Improvements
Privacy
During the investigation, we evaluated capturing developer prompts (the messages typed to the AI assistant) to improve “why” context. This was immediately rejected as a privacy violation — no such data was ever persisted or made available. Cogniscape’s policy remains unchanged: we never capture what developers type to their AI coding assistants. All insights are derived from structured activity metadata (files, commands, tool usage) and AI-generated summaries of the assistant’s responses only.Lessons Learned
- Test with realistic data — the primary bug went undetected because test scenarios didn’t match real-world session patterns
- Progressive data matters — coding sessions evolve over hours; capturing only the initial state misses the majority of the work
- AI summarization is essential — raw conversation text is not a summary; AI-powered structuring dramatically improves downstream analysis
- Privacy by design — even when a data point would improve analytics, it must be evaluated against trust and compliance requirements first
Timeline
Total time to resolution: ~24 hours
Next Steps
- Minor client patch (v1.6.1) to clean up a residual configuration item
- Continued improvements to entity classification and deduplication
- Additional data sanitization for sensitive command-line content